大数据实战手册-开发篇之IO

• 2.4 sparkContext IO:读
• 2.4.1 textFile

 # Load a text file and convert each line to a Row.
 lines = sc.textFile("examples/src/main/resources/people.txt")

• 2.4.2 hadoopFile
• 2.4.3 newAPIHadoopFile

parquet_rdd = sc.newAPIHadoopFile(
        path,
        'org.apache.parquet.avro.AvroParquetInputFormat',
        'java.lang.Void',
        'org.apache.avro.generic.IndexedRecord',
        valueConverter='org.apache.spark.examples.pythonconverters.IndexedRecordToJavaConverter')

• 2.4.4 pickleFile

  备注：Load an RDD previously saved using RDD.saveAsPickleFile method.

• 2.4.5 parallelize

• 2.4.6 broadcast

• 2.5 sparkSql IO

• 2.5.1 DataFrameReader

**parquet**
df = spark.read.parquet("examples/src/main/resources/users.parquet")

**JSON**
peopleDF = spark.read.json("examples/src/main/resources/people.json")

**ORC**
df = spark.read.orc("examples/src/main/resources/users.orc")

**JDBC支持的db**
jdbcDF = spark.read \
        .format("jdbc") \
        .option("url", "jdbc:postgresql:dbserver") \
        .option("dbtable", "schema.tablename") \
        .option("user", "username") \
        .option("password", "password") \
        .load()

• 2.5.2 DataFrameWriter

**parquet**
df.select("name", "favorite_color").write.save("namesAndFavColors.parquet")

**JSON**
 (df.write
        .partitionBy("favorite_color")
        .bucketBy(42, "name")
        .saveAsTable("people_partitioned_bucketed"))

**ORC**
(df.write.format("orc")
        .option("orc.bloom.filter.columns", "favorite_color")
        .option("orc.dictionary.key.threshold", "1.0")
        .save("users_with_options.orc"))

**JDBC支持的db**
  jdbcDF.write \
        .format("jdbc") \
        .option("url", "jdbc:postgresql:dbserver") \
        .option("dbtable", "schema.tablename") \
        .option("user", "username") \
        .option("password", "password") \
        .save()