[Spark][Python]DataFrame中取出有限个记录的例子

的 继续

In [4]: peopleDF.select("age")
Out[4]: DataFrame[age: bigint]

In [5]: myDF=people.select("age")
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-5-b5b723b62a49> in <module>()
----> 1 myDF=people.select("age")

NameError: name 'people' is not defined

In [6]: myDF=peopleDF.select("age")

In [7]: myDF.take(3)
17/10/05 05:13:02 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 230.1 KB, free 871.7 KB)
17/10/05 05:13:02 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 21.4 KB, free 893.1 KB)
17/10/05 05:13:02 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:55073 (size: 21.4 KB, free: 208.7 MB)
17/10/05 05:13:02 INFO spark.SparkContext: Created broadcast 5 from take at <ipython-input-7-745486715568>:1
17/10/05 05:13:02 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 251.1 KB, free 1144.2 KB)
17/10/05 05:13:02 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 21.6 KB, free 1165.8 KB)
17/10/05 05:13:02 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:55073 (size: 21.6 KB, free: 208.7 MB)
17/10/05 05:13:02 INFO spark.SparkContext: Created broadcast 6 from take at <ipython-input-7-745486715568>:1
17/10/05 05:13:03 INFO mapred.FileInputFormat: Total input paths to process : 1
17/10/05 05:13:03 INFO spark.SparkContext: Starting job: take at <ipython-input-7-745486715568>:1
17/10/05 05:13:03 INFO scheduler.DAGScheduler: Got job 2 (take at <ipython-input-7-745486715568>:1) with 1 output partitions
17/10/05 05:13:03 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (take at <ipython-input-7-745486715568>:1)
17/10/05 05:13:03 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/10/05 05:13:03 INFO scheduler.DAGScheduler: Missing parents: List()
17/10/05 05:13:03 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[14] at take at <ipython-input-7-745486715568>:1), which has no missing parents
17/10/05 05:13:03 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 4.3 KB, free 1170.2 KB)
17/10/05 05:13:03 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.5 KB, free 1172.6 KB)
17/10/05 05:13:03 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:55073 (size: 2.5 KB, free: 208.7 MB)
17/10/05 05:13:03 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006
17/10/05 05:13:03 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[14] at take at <ipython-input-7-745486715568>:1)
17/10/05 05:13:03 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
17/10/05 05:13:03 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, partition 0,PROCESS_LOCAL, 2149 bytes)
17/10/05 05:13:03 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 2)
17/10/05 05:13:03 INFO rdd.HadoopRDD: Input split: hdfs://localhost:8020/user/training/people.json:0+179
17/10/05 05:13:03 INFO codegen.GenerateUnsafeProjection: Code generated in 113.719806 ms
17/10/05 05:13:03 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 2). 2235 bytes result sent to driver
17/10/05 05:13:03 INFO scheduler.DAGScheduler: ResultStage 2 (take at <ipython-input-7-745486715568>:1) finished in 0.493 s
17/10/05 05:13:03 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 487 ms on localhost (1/1)
17/10/05 05:13:03 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
17/10/05 05:13:03 INFO scheduler.DAGScheduler: Job 2 finished: take at <ipython-input-7-745486715568>:1, took 0.737231 s
Out[7]: [Row(age=None), Row(age=30), Row(age=19)]

In [8]:

[Spark][Python]DataFrame select 操作例子的更多相关文章

  1. [Spark][Python]DataFrame select 操作例子II

    [Spark][Python]DataFrame中取出有限个记录的   继续 In [4]: peopleDF.select("age","name") In ...

  2. [Spark][Python]DataFrame where 操作例子

    [Spark][Python]DataFrame中取出有限个记录的例子 的 继续 [15]: myDF=peopleDF.where("age>21") In [16]: m ...

  3. [Spark][Python]RDD flatMap 操作例子

    RDD flatMap 操作例子: flatMap,对原RDD的每个元素(行)执行函数操作,然后把每行都“拍扁” [training@localhost ~]$ hdfs dfs -put cats. ...

  4. [Spark][Python][DataFrame][SQL]Spark对DataFrame直接执行SQL处理的例子

    [Spark][Python][DataFrame][SQL]Spark对DataFrame直接执行SQL处理的例子 $cat people.json {"name":" ...

  5. [Spark][Python][DataFrame][RDD]DataFrame中抽取RDD例子

    [Spark][Python][DataFrame][RDD]DataFrame中抽取RDD例子 sqlContext = HiveContext(sc) peopleDF = sqlContext. ...

  6. [Spark][Python][DataFrame][RDD]从DataFrame得到RDD的例子

    [Spark][Python][DataFrame][RDD]从DataFrame得到RDD的例子 $ hdfs dfs -cat people.json {"name":&quo ...

  7. [Spark][Python][DataFrame][Write]DataFrame写入的例子

    [Spark][Python][DataFrame][Write]DataFrame写入的例子 $ hdfs dfs -cat people.json {"name":" ...

  8. [Spark][Python]DataFrame的左右连接例子

    [Spark][Python]DataFrame的左右连接例子 $ hdfs dfs -cat people.json {"name":"Alice",&quo ...

  9. [Spark][Python]DataFrame中取出有限个记录的例子

    [Spark][Python]DataFrame中取出有限个记录的例子: sqlContext = HiveContext(sc) peopleDF = sqlContext.read.json(&q ...

随机推荐

  1. 修改eclipse的背景色(转载)

    eclipse操作界面默认颜色为白色.对于我们长期使用电脑编程的人来说,白色很刺激我们的眼睛,所以我经常会改变workspace的背景色,使眼睛舒服一些. 设置方法如下: 1.打开window-> ...

  2. JavaScript 字符串转json格式

    第一种:浏览器支持的转换方式(Firefox,chrome,opera,safari,ie)等浏览器: JSON.parse(jsonstr); //可以将json字符串转换成json对象 JSON. ...

  3. Python笔记（十四）：操作excel openpyxl模块

    (一)  常遇到的情况 就我自己来说,常遇到的情况可能就下面几种: 读取excel整个sheet页的数据. 读取指定行.列的数据 往一个空白的excel文档写数据 往一个已经有数据的excel文档追加 ...

  4. python第九十六天 ---Django(1)

    django 模块 一  安装: pip3 install django 或 python -m pip install django 二  添加环境变量 相关命令: #cmd 下 django-ad ...

  5. 第三章————用SQL语句操作数据

    第三章————用SQL语句操作数据 *********************新增**************************** 1. insert into 表名(列名1,列名2...) ...

  6. Python面试题（一）【转】

    注:本面试题来源于网络,转载自http://www.cnblogs.com/goodhacker/p/3366618.html. 1. (1)python下多线程的限制以及多进程中传递参数的方式 py ...

  7. .net 调用R语言的函数（计算统计值pvalue 对应excel :ttest）

    Pvalue 计算 项目设计pvalue计算,但是由于.net 没有类似的公式或者函数,最终决定使用.net 调用R语言 采用.net 调用r语言的公用函数 需要安装 r语言环境 https://mi ...

  8. java web中java和python混合使用

    利用java web技术展示python算法处理后的数据 工具/原料   myeclipse10 pycharm+Anaconda2 方法/步骤     首先安装配置好pycharm+Anaconda ...

  9. jQuery搜索框自动补全功能插件实现-autocomplete.js

    最近用nodeclub实现股票的输入关键字自动补全股票信息进行搜索功能,原先用jQuery-ui,结果jQuery-ui库太大,所以考虑用其他插件,最终选择使用autocomplete.js,控件简单 ...

  10. break和continue语句（初学者）

    1.break语句可以从循环体内跳出循环体,即提前结束循环,接着执行循环下面的语句. 一般形式:break: break不能用于循环语句和switch语句之外的任何其他语句中. 注意:(1)break ...