Spark:scala集合转化为DS/DF

scala集合转化为DS/DF

 case class TestPerson(name: String, age: Long, salary: Double)
 val tom = TestPerson("Tom Hanks",,35.5)
 val sam = TestPerson("Sam Smith",,40.5)
 val PersonList = mutable.MutableList[TestPerson]()
 PersonList += tom
 PersonList += sam

 val personDS = PersonList.toDS()
 println(personDS.getClass)
 personDS.show()

 val personDF = PersonList.toDF()
 println(personDF.getClass)
 personDF.show()
 personDF.select("name", "age").show()

输出结果：

class org.apache.spark.sql.Dataset

+---------+---+------+
|     name|age|salary|
+---------+---+------+
|Tom Hanks| |  35.5|
|Sam Smith| |  40.5|
+---------+---+------+

class org.apache.spark.sql.DataFrame

+---------+---+------+
|     name|age|salary|
+---------+---+------+
|Tom Hanks| |  35.5|
|Sam Smith| |  40.5|
+---------+---+------+

+---------+---+
|     name|age|
+---------+---+
|Tom Hanks| |
|Sam Smith| |
+---------+---+

参考资料：

https://stackoverflow.com/questions/39397652/convert-scala-list-to-dataframe-or-dataset