[Spark][Python]DataFrame中取出有限个记录的例子：

sqlContext = HiveContext(sc)

peopleDF = sqlContext.read.json("people.json")

peopleDF.limit(3).show()

===

[training@localhost ~]$ hdfs dfs -cat people.json
{"name":"Alice","pcode":"94304"}
{"name":"Brayden","age":30,"pcode":"94304"}
{"name":"Carla","age":19,"pcoe":"10036"}
{"name":"Diana","age":46}
{"name":"Etienne","pcode":"94104"}
[training@localhost ~]$

In [1]: sqlContext = HiveContext(sc)

In [2]: peopleDF = sqlContext.read.json("people.json")
17/10/05 05:03:11 INFO hive.HiveContext: Initializing execution hive, version 1.1.0
17/10/05 05:03:11 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0-cdh5.7.0
17/10/05 05:03:11 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.7.0
17/10/05 05:03:14 INFO hive.metastore: Trying to connect to metastore with URI thrift://localhost.localdomain:9083
17/10/05 05:03:14 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/10/05 05:03:15 INFO hive.metastore: Connected to metastore.
17/10/05 05:03:16 INFO session.SessionState: Created HDFS directory: file:/tmp/spark-99a33db4-b69a-46a9-8032-f87d63299040/scratch/training
17/10/05 05:03:16 INFO session.SessionState: Created local directory: /tmp/4e1c5259-7ae8-482c-ae77-94d3a0c51f91_resources
17/10/05 05:03:16 INFO session.SessionState: Created HDFS directory: file:/tmp/spark-99a33db4-b69a-46a9-8032-f87d63299040/scratch/training/4e1c5259-7ae8-482c-ae77-94d3a0c51f91
17/10/05 05:03:16 INFO session.SessionState: Created local directory: /tmp/training/4e1c5259-7ae8-482c-ae77-94d3a0c51f91
17/10/05 05:03:16 INFO session.SessionState: Created HDFS directory: file:/tmp/spark-99a33db4-b69a-46a9-8032-f87d63299040/scratch/training/4e1c5259-7ae8-482c-ae77-94d3a0c51f91/_tmp_space.db
17/10/05 05:03:16 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
17/10/05 05:03:16 INFO json.JSONRelation: Listing hdfs://localhost:8020/user/training/people.json on driver
17/10/05 05:03:19 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 251.1 KB, free 251.1 KB)
17/10/05 05:03:20 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 21.6 KB, free 272.7 KB)
17/10/05 05:03:20 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:55073 (size: 21.6 KB, free: 208.8 MB)
17/10/05 05:03:20 INFO spark.SparkContext: Created broadcast 0 from json at NativeMethodAccessorImpl.java:-2
17/10/05 05:03:20 INFO mapred.FileInputFormat: Total input paths to process : 1
17/10/05 05:03:21 INFO spark.SparkContext: Starting job: json at NativeMethodAccessorImpl.java:-2
17/10/05 05:03:21 INFO scheduler.DAGScheduler: Got job 0 (json at NativeMethodAccessorImpl.java:-2) with 1 output partitions
17/10/05 05:03:21 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (json at NativeMethodAccessorImpl.java:-2)
17/10/05 05:03:21 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/10/05 05:03:21 INFO scheduler.DAGScheduler: Missing parents: List()
17/10/05 05:03:21 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at json at NativeMethodAccessorImpl.java:-2), which has no missing parents
17/10/05 05:03:21 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.3 KB, free 277.1 KB)
17/10/05 05:03:21 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.4 KB, free 279.5 KB)
17/10/05 05:03:21 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:55073 (size: 2.4 KB, free: 208.8 MB)
17/10/05 05:03:21 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
17/10/05 05:03:21 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at json at NativeMethodAccessorImpl.java:-2)
17/10/05 05:03:21 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/10/05 05:03:21 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2149 bytes)
17/10/05 05:03:21 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
17/10/05 05:03:21 INFO rdd.HadoopRDD: Input split: hdfs://localhost:8020/user/training/people.json:0+179
17/10/05 05:03:21 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/10/05 05:03:21 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/10/05 05:03:21 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/10/05 05:03:21 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/10/05 05:03:21 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/10/05 05:03:22 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 2354 bytes result sent to driver
17/10/05 05:03:22 INFO scheduler.DAGScheduler: ResultStage 0 (json at NativeMethodAccessorImpl.java:-2) finished in 0.931 s
17/10/05 05:03:22 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 850 ms on localhost (1/1)
17/10/05 05:03:22 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/10/05 05:03:22 INFO scheduler.DAGScheduler: Job 0 finished: json at NativeMethodAccessorImpl.java:-2, took 1.388410 s
17/10/05 05:03:23 INFO hive.HiveContext: default warehouse location is /user/hive/warehouse
17/10/05 05:03:23 INFO hive.HiveContext: Initializing metastore client version 1.1.0 using Spark classes.
17/10/05 05:03:23 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0-cdh5.7.0
17/10/05 05:03:23 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.7.0
17/10/05 05:03:23 INFO spark.ContextCleaner: Cleaned accumulator 2
17/10/05 05:03:23 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on localhost:55073 in memory (size: 2.4 KB, free: 208.8 MB)
17/10/05 05:03:25 INFO hive.metastore: Trying to connect to metastore with URI thrift://localhost.localdomain:9083
17/10/05 05:03:25 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/10/05 05:03:25 INFO hive.metastore: Connected to metastore.
17/10/05 05:03:25 INFO session.SessionState: Created local directory: /tmp/684b38e5-72f0-4712-81d4-4c439e093f5c_resources
17/10/05 05:03:25 INFO session.SessionState: Created HDFS directory: /tmp/hive/training/684b38e5-72f0-4712-81d4-4c439e093f5c
17/10/05 05:03:25 INFO session.SessionState: Created local directory: /tmp/training/684b38e5-72f0-4712-81d4-4c439e093f5c
17/10/05 05:03:25 INFO session.SessionState: Created HDFS directory: /tmp/hive/training/684b38e5-72f0-4712-81d4-4c439e093f5c/_tmp_space.db
17/10/05 05:03:25 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.

In [3]: peopleDF.limit(3).show()
17/10/05 05:04:09 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 65.5 KB, free 338.2 KB)
17/10/05 05:04:10 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 21.4 KB, free 359.6 KB)
17/10/05 05:04:10 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:55073 (size: 21.4 KB, free: 208.8 MB)
17/10/05 05:04:10 INFO spark.SparkContext: Created broadcast 2 from showString at NativeMethodAccessorImpl.java:-2
17/10/05 05:04:10 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 251.1 KB, free 610.7 KB)
17/10/05 05:04:11 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 21.6 KB, free 632.4 KB)
17/10/05 05:04:11 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:55073 (size: 21.6 KB, free: 208.7 MB)
17/10/05 05:04:11 INFO spark.SparkContext: Created broadcast 3 from showString at NativeMethodAccessorImpl.java:-2
17/10/05 05:04:12 INFO mapred.FileInputFormat: Total input paths to process : 1
17/10/05 05:04:12 INFO spark.SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:-2
17/10/05 05:04:12 INFO scheduler.DAGScheduler: Got job 1 (showString at NativeMethodAccessorImpl.java:-2) with 1 output partitions
17/10/05 05:04:12 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (showString at NativeMethodAccessorImpl.java:-2)
17/10/05 05:04:12 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/10/05 05:04:12 INFO scheduler.DAGScheduler: Missing parents: List()
17/10/05 05:04:12 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[9] at showString at NativeMethodAccessorImpl.java:-2), which has no missing parents
17/10/05 05:04:12 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 5.9 KB, free 638.2 KB)
17/10/05 05:04:12 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 3.3 KB, free 641.5 KB)
17/10/05 05:04:12 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:55073 (size: 3.3 KB, free: 208.7 MB)
17/10/05 05:04:12 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006
17/10/05 05:04:12 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[9] at showString at NativeMethodAccessorImpl.java:-2)
17/10/05 05:04:12 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
17/10/05 05:04:12 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, partition 0,PROCESS_LOCAL, 2149 bytes)
17/10/05 05:04:12 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 1)
17/10/05 05:04:12 INFO rdd.HadoopRDD: Input split: hdfs://localhost:8020/user/training/people.json:0+179
17/10/05 05:04:14 INFO codegen.GenerateUnsafeProjection: Code generated in 1563.240244 ms
17/10/05 05:04:14 INFO codegen.GenerateSafeProjection: Code generated in 182.529448 ms
17/10/05 05:04:15 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 1). 2328 bytes result sent to driver
17/10/05 05:04:15 INFO scheduler.DAGScheduler: ResultStage 1 (showString at NativeMethodAccessorImpl.java:-2) finished in 2.549 s
17/10/05 05:04:15 INFO scheduler.DAGScheduler: Job 1 finished: showString at NativeMethodAccessorImpl.java:-2, took 2.852393 s
17/10/05 05:04:15 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 2547 ms on localhost (1/1)
17/10/05 05:04:15 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
+----+-------+-----+-----+
| age| name|pcode| pcoe|
+----+-------+-----+-----+
|null| Alice|94304| null|
| 30|Brayden|94304| null|
| 19| Carla| null|10036|
+----+-------+-----+-----+

In [4]:

[Spark][Python]DataFrame中取出有限个记录的例子的更多相关文章

  1. [Spark][Python]DataFrame where 操作例子

    [Spark][Python]DataFrame中取出有限个记录的例子 的 继续 [15]: myDF=peopleDF.where("age>21") In [16]: m ...

  2. [Spark][Python]DataFrame select 操作例子

    [Spark][Python]DataFrame中取出有限个记录的例子 的 继续 In [4]: peopleDF.select("age")Out[4]: DataFrame[a ...

  3. [Spark][Python]DataFrame select 操作例子II

    [Spark][Python]DataFrame中取出有限个记录的   继续 In [4]: peopleDF.select("age","name") In ...

  4. [Spark][Python][DataFrame][RDD]DataFrame中抽取RDD例子

    [Spark][Python][DataFrame][RDD]DataFrame中抽取RDD例子 sqlContext = HiveContext(sc) peopleDF = sqlContext. ...

  5. [Spark][Python][DataFrame][RDD]从DataFrame得到RDD的例子

    [Spark][Python][DataFrame][RDD]从DataFrame得到RDD的例子 $ hdfs dfs -cat people.json {"name":&quo ...

  6. [Spark][Python][DataFrame][Write]DataFrame写入的例子

    [Spark][Python][DataFrame][Write]DataFrame写入的例子 $ hdfs dfs -cat people.json {"name":" ...

  7. [Spark][Python][DataFrame][SQL]Spark对DataFrame直接执行SQL处理的例子

    [Spark][Python][DataFrame][SQL]Spark对DataFrame直接执行SQL处理的例子 $cat people.json {"name":" ...

  8. [Spark][Python]DataFrame的左右连接例子

    [Spark][Python]DataFrame的左右连接例子 $ hdfs dfs -cat people.json {"name":"Alice",&quo ...

  9. Python dataframe中如何使y列按x列进行统计？

    如图:busy=0 or 1,求出busy=1时los的平均,同样对busy=0时也求出los的平均 Python dataframe中如何使y列按x列进行统计? >> python这个答 ...

随机推荐

  1. 三. Redis 主从复制

    特点 1. Master可以拥有多个Slave 2. 多个Slave除可以连接一个Master外,还可以连接多个Salve(避免Master挂掉不能同步,当Master挂掉,其中一个Slave会立即变 ...

  2. Java 数据驱动测试

    适用场景 测试搜索功能, 基于数据类型需要测多种不同的数据, 比如最大值, 小数, 负数, 字符串, 特异符号等等. 如果直接写代码, 每一种数据类型都需要写一遍代码, 冗长且不方便调试. 如果采用数 ...

  3. Python进阶点

    1. 模块化设计,分而治之 2. 组合数据类型 2.1 集合类型:list.set(无序/不重复),用于数据去重 2.2 序列类型:字符串.元组.列表(有序) 2.3 字典类型:根据字典中 k/v 来 ...

  4. 洗礼灵魂，修炼python（28）--异常处理（2）—>运用异常

    你可能会想,卧槽这标题取的,前面不是说异常就是报错吗?异常还能运用? 是的,异常确实可以运用,可以刻意制造异常,在出现异常时捕获异常并对异常处理,所以进入本篇博文的话题—异常处理 异常处理: 异常处理 ...

  5. entity framework异常 The specified cast from a materialized 'System.Int32' type to the 'System.String' type is not valid

    ROW_NUMBER() OVER (ORDER BY (select Null)) AS Id entity framework 查询中有这句会有异常

  6. SVN服务端VisualSVN数据转移说明

    两台服务器,进行SVN的迁移: 系统平台:windows server 2008 and windows server 2012 版本库:meishu 源服务器:192.168.0.245 目标服务器 ...

  7. D-Link DIR-600 - Authentication Bypass

    #Exploit Title: D-Link DIR-600 - Authentication Bypass (Absolute Path Traversal Attack) # CVE - http ...

  8. Alpha版本 - 测试报告

    Alpha版本 - 测试报告 总体测试计划 前端 模块 子模块 测试项 预期结果 测试工具 执行人 登录/注册模块 无网络 提示无网异常 robolectric 陈龙江 登录 输入用户名/密码为空,点 ...

  9. 从零上手Python关键代码

    来源 https://www.kesci.com/home/project/59e4331c4663f7655c499bc3

  10. RelativeLayout 总结

    1)参考元素获取:id: 2)位置关系设置: 3)对齐关系设置: