配好了Spark集群后,先用pyspark写了两个小例子，但是发现Tab键没有提示，于是打算转到scala上试试，在spark-shell下有提示了，但是发现不能退格，而且提示也不是复写，而是追加，这样根本就没法写程序.

      解决办法：

1.打开会话选项

2.终端-仿真    在终端中选择Linux

3.映射键   勾选两个选项

4.至此已经成功了,但是如果远程长时间未操作 就会中断连接，下次再操作时需要等待，其实也很影响使用，在这里也附上解决办法(可选)

val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")

lines.count()

-----
scala> val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(77922) called with curMem=179665, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 76.1 KB, free 267.0 MB)
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(31262) called with curMem=257587, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 30.5 KB, free 267.0 MB)
17/10/13 23:09:24 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41619 (size: 30.5 KB, free: 267.2 MB)
17/10/13 23:09:24 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
17/10/13 23:09:24 INFO SparkContext: Created broadcast 1 from textFile at <console>:12
lines: org.apache.spark.rdd.RDD[String] = hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12

scala> lines.count()
17/10/13 23:09:45 INFO FileInputFormat: Total input paths to process : 1
17/10/13 23:09:48 INFO SparkContext: Starting job: count at <console>:15
17/10/13 23:09:48 INFO DAGScheduler: Got job 0 (count at <console>:15) with 1 output partitions (allowLocal=false)
17/10/13 23:09:48 INFO DAGScheduler: Final stage: Stage 0(count at <console>:15)
17/10/13 23:09:48 INFO DAGScheduler: Parents of final stage: List()
17/10/13 23:09:48 INFO DAGScheduler: Missing parents: List()
17/10/13 23:09:48 INFO DAGScheduler: Submitting Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12), which has no missing parents
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(2544) called with curMem=288849, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.5 KB, free 267.0 MB)
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(1898) called with curMem=291393, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1898.0 B, free 267.0 MB)
17/10/13 23:09:48 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41619 (size: 1898.0 B, free: 267.2 MB)
17/10/13 23:09:48 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
17/10/13 23:09:48 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:838
17/10/13 23:09:48 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12)
17/10/13 23:09:48 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/10/13 23:09:48 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 1307 bytes)
17/10/13 23:09:48 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
17/10/13 23:09:49 INFO HadoopRDD: Input split: hdfs://alamps:9000/wordcount/input/test.txt:0+88
17/10/13 23:09:49 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/10/13 23:09:49 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/10/13 23:09:49 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/10/13 23:09:49 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/10/13 23:09:49 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/10/13 23:09:53 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1920 bytes result sent to driver
17/10/13 23:09:53 INFO DAGScheduler: Stage 0 (count at <console>:15) finished in 4.875 s
17/10/13 23:09:53 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4812 ms on localhost (1/1)
17/10/13 23:09:53 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/10/13 23:09:53 INFO DAGScheduler: Job 0 finished: count at <console>:15, took 5.480197 s
res2: Long = 8

[hadoop@alamps sbin]$ jps
3596 Master
3733 Worker
2558 DataNode
2748 SecondaryNameNode
3814 Jps
2884 ResourceManager
2986 NodeManager
2467 NameNode
[hadoop@alamps sbin]$ hadoop fs -ls /
Found 11 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:29 /aaa
drwxr-xr-x   - hadoop supergroup          0 2017-10-06 04:04 /external
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 09:14 /flowsum
-rw-r--r--   1 hadoop supergroup         43 2017-10-02 02:52 /hello.txt
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 21:10 /index
-rw-r--r--   1 hadoop supergroup  143588167 2017-10-01 08:38 /jdk-7u65-linux-i586.tar.gz
drwx------   - hadoop supergroup          0 2017-10-05 22:43 /tmp
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:18 /upload
drwxr-xr-x   - hadoop supergroup          0 2017-10-05 22:44 /user
drwxr-xr-x   - hadoop supergroup          0 2017-10-03 06:20 /wc
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount
cat: `/wordcount': Is a directory
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:00 /wordcount/input
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount/out
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount/input
Found 1 items
-rw-r--r--   1 hadoop supergroup         88 2017-10-01 09:00 /wordcount/input/test.txt
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount/input/test.txt
hello tom
hello java
hello c
hello python
hello scala
hello spark
hello baby
hello java
[hadoop@alamps sbin]$

val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")

lines.count()

-----
scala> val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(77922) called with curMem=179665, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 76.1 KB, free 267.0 MB)
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(31262) called with curMem=257587, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 30.5 KB, free 267.0 MB)
17/10/13 23:09:24 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41619 (size: 30.5 KB, free: 267.2 MB)
17/10/13 23:09:24 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
17/10/13 23:09:24 INFO SparkContext: Created broadcast 1 from textFile at <console>:12
lines: org.apache.spark.rdd.RDD[String] = hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12

scala> lines.count()
17/10/13 23:09:45 INFO FileInputFormat: Total input paths to process : 1
17/10/13 23:09:48 INFO SparkContext: Starting job: count at <console>:15
17/10/13 23:09:48 INFO DAGScheduler: Got job 0 (count at <console>:15) with 1 output partitions (allowLocal=false)
17/10/13 23:09:48 INFO DAGScheduler: Final stage: Stage 0(count at <console>:15)
17/10/13 23:09:48 INFO DAGScheduler: Parents of final stage: List()
17/10/13 23:09:48 INFO DAGScheduler: Missing parents: List()
17/10/13 23:09:48 INFO DAGScheduler: Submitting Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12), which has no missing parents
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(2544) called with curMem=288849, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.5 KB, free 267.0 MB)
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(1898) called with curMem=291393, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1898.0 B, free 267.0 MB)
17/10/13 23:09:48 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41619 (size: 1898.0 B, free: 267.2 MB)
17/10/13 23:09:48 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
17/10/13 23:09:48 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:838
17/10/13 23:09:48 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12)
17/10/13 23:09:48 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/10/13 23:09:48 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 1307 bytes)
17/10/13 23:09:48 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
17/10/13 23:09:49 INFO HadoopRDD: Input split: hdfs://alamps:9000/wordcount/input/test.txt:0+88
17/10/13 23:09:49 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/10/13 23:09:49 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/10/13 23:09:49 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/10/13 23:09:49 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/10/13 23:09:49 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/10/13 23:09:53 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1920 bytes result sent to driver
17/10/13 23:09:53 INFO DAGScheduler: Stage 0 (count at <console>:15) finished in 4.875 s
17/10/13 23:09:53 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4812 ms on localhost (1/1)
17/10/13 23:09:53 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/10/13 23:09:53 INFO DAGScheduler: Job 0 finished: count at <console>:15, took 5.480197 s
res2: Long = 8

[hadoop@alamps sbin]$ jps
3596 Master
3733 Worker
2558 DataNode
2748 SecondaryNameNode
3814 Jps
2884 ResourceManager
2986 NodeManager
2467 NameNode
[hadoop@alamps sbin]$ hadoop fs -ls /
Found 11 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:29 /aaa
drwxr-xr-x   - hadoop supergroup          0 2017-10-06 04:04 /external
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 09:14 /flowsum
-rw-r--r--   1 hadoop supergroup         43 2017-10-02 02:52 /hello.txt
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 21:10 /index
-rw-r--r--   1 hadoop supergroup  143588167 2017-10-01 08:38 /jdk-7u65-linux-i586.tar.gz
drwx------   - hadoop supergroup          0 2017-10-05 22:43 /tmp
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:18 /upload
drwxr-xr-x   - hadoop supergroup          0 2017-10-05 22:44 /user
drwxr-xr-x   - hadoop supergroup          0 2017-10-03 06:20 /wc
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount
cat: `/wordcount': Is a directory
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:00 /wordcount/input
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount/out
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount/input
Found 1 items
-rw-r--r--   1 hadoop supergroup         88 2017-10-01 09:00 /wordcount/input/test.txt
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount/input/test.txt
hello tom
hello java
hello c
hello python
hello scala
hello spark
hello baby
hello java
[hadoop@alamps sbin]$

spark-shell下有提示了，但是发现不能退格的更多相关文章

  1. 02、体验Spark shell下RDD编程

    02.体验Spark shell下RDD编程 1.Spark RDD介绍 RDD是Resilient Distributed Dataset,中文翻译是弹性分布式数据集.该类是Spark是核心类成员之 ...

  2. ubuntu下使用vi是方向键变乱码 退格键不能使用的解决方法

    ubuntu下使用vi是方向键变乱码 退格键不能使用的解决方法 转载:http://blog.csdn.net/yao_qinwei/article/details/8761777 在插入模式下,按删 ...

  3. ubuntu 下使用vi时方向键乱码，退格键不能使用

    ubuntu 下使用vi时方向键乱码,退格键不能使用的解决方法 问题表现:Ubuntu下,初始使用vi,编辑模式下使用方向键的时候,并不会使光标移动,而是在命令行中出现A B C D 之类的字母,并且 ...

  4. 在Scala IDEA for Eclipse或IDEA里程序编译实现与在Spark Shell下的对比（其实就是那么一回事）

    不多说,直接上干货! 比如,我这里拿主成分分析(PCA). 1.主成分分析(PCA)的概念介绍 主成分分析(PCA) 是一种对数据进行旋转变换的统计学方法,其本质是在线性空间中进行一个基变换,使得变换 ...

  5. Cygwin下安装vim后，vim中退格键无法正常使用

    问题描述: 在Cygwin中安装完vim后 进入vim,发现上下左右键和退格键都无法正常使用 问题分析: 首先考虑到的就是缺少vim的配置文件,首先查看/etc路径下是否有vim的配置文件 admin ...

  6. Ubuntu下使用Vi时方向键变乱码 退格键不能使用的解决方法

    要在Ubuntu下编辑一些文件,这就涉及到了vi这个编辑器了.在Ubuntu下,初始使用vi的时候有点问题,就是在编辑模式下使用方向键的时候,并不会使光标移动,而是在命令行中出现[A [B [C [D ...

  7. 解决Ubuntu 下 vi编辑器不能使用方向键和退格键问题

    转自:http://blog.csdn.net/sky101010ws/article/details/51012103 使用vi命令时,不能正常编辑文件,使用方向键时老是出现很多字母 这个问题主要是 ...

  8. 解决Ubuntu14.04下vi编辑器不能使用方向键和退格键问题

    参考:http://blog.sina.com.cn/s/blog_7d0c2fed01010zbi.html 系统:Ubuntu14.04 使用vi命令时,不能正常编辑文件,使用方向键时老是出现很多 ...

  9. ubuntu 下解决sublime v3 中文输入法时 退格键删除不了拼音的问题

    ubuntu下,sulime想要支持中文需要这样设置: 1.安装中文输入解决的github git clone https://github.com/lyfeyaj/sublime-text-imfi ...

随机推荐

  1. MVC 实用架构设计（〇）——总体设计

    〇.目录 一.前言 二.结构图 三.结构说明 一.前言 一直以来都想写这个系列,但基于各种理由(主要是懒惰),迟迟没有动手.今天,趁着周末的空档,终于把系列的目录公布出来了,算是开个头,也给自己一个坚 ...

  2. 优化网站设计（十）：最小化JAVASCRIPT和CSS

    前言 网站设计的优化是一个很大的话题,有一些通用的原则,也有针对不同开发平台的一些建议.这方面的研究一直没有停止过,我在不同的场合也分享过这样的话题. 作为通用的原则,雅虎的工程师团队曾经给出过35个 ...

  3. MyCAT 在 Cobar 的基础上，完成了彻底的 NIO 通讯，并且合并了两个线程池

    研读: 1.http://www.mycat.io <Mycat权威指南> 第 2 章 Mycat 前世今生: 浏览: 深度认识 Sharding-JDBC:做最轻量级的数据库中间层 - ...

  4. 使用c#反射实现接口可视化调试页面

    直接上代码,引用CommTools.dll.包括aspx显示页面和aspx.cs获取反射数据源代码 using System; using System.Collections.Generic; us ...

  5. 树形dp的深入讨论

    越发向dp深入越发现dp越有意思! 这道题做的时候感觉十分的难,然后看完学长的题解恍然大悟.设状态不好导致想了一中午,一直感觉不可做,其实是自己的状态设的不对,这道题呢,首先是一个求在树上建多个厂,而 ...

  6. 《HTTP - 状态码》

    推荐一首歌 - 那吾克热<儿子娃娃> 非常喜欢前奏 1:状态码的作用? - 职责是当客户端发起请求时候,描述请求返回结果.根据状态码,客户端可以知道服务端是正确处理了请求,还是返回了错误. ...

  7. 使用脚本与orm模型交互对数据库操作

    场景:如不想启动服务在框架中查看数据库数据,同时使用ORM框架对数据库操作带来的好处 import os import sys #将脚本所在的工程添加到环境变量 sys.path.append('.. ...

  8. sharepoint webapp 部署注意点

    只有在配置文件或 Page 指令中将 enableSessionState 设置为 true 时,才能使用会话状态.还请确保在应用程序配置的 // 节中包括 System.Web.SessionSta ...

  9. myeclipse项目导入到eclipse， HttpServletRequest报红现象

    eclipse项目中关于导入的项目里提示HttpServletRequest 不能引用的解决办法 当使用eclipse导入外部的web工程时,有时会提示HttpServletRequest, Serv ...

  10. ASCLL码中的一些小知识

    其次要记住asill值中   65是A      97是a A与a之间相隔32,用int转换后再用char转换回来. char b = s.charAt(i);为字符串转换成一个一个的.