配好了Spark集群后,先用pyspark写了两个小例子，但是发现Tab键没有提示，于是打算转到scala上试试，在spark-shell下有提示了，但是发现不能退格，而且提示也不是复写，而是追加，这样根本就没法写程序.

      解决办法：

1.打开会话选项

2.终端-仿真    在终端中选择Linux

3.映射键   勾选两个选项

4.至此已经成功了,但是如果远程长时间未操作 就会中断连接，下次再操作时需要等待，其实也很影响使用，在这里也附上解决办法(可选)

val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")

lines.count()

-----
scala> val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(77922) called with curMem=179665, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 76.1 KB, free 267.0 MB)
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(31262) called with curMem=257587, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 30.5 KB, free 267.0 MB)
17/10/13 23:09:24 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41619 (size: 30.5 KB, free: 267.2 MB)
17/10/13 23:09:24 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
17/10/13 23:09:24 INFO SparkContext: Created broadcast 1 from textFile at <console>:12
lines: org.apache.spark.rdd.RDD[String] = hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12

scala> lines.count()
17/10/13 23:09:45 INFO FileInputFormat: Total input paths to process : 1
17/10/13 23:09:48 INFO SparkContext: Starting job: count at <console>:15
17/10/13 23:09:48 INFO DAGScheduler: Got job 0 (count at <console>:15) with 1 output partitions (allowLocal=false)
17/10/13 23:09:48 INFO DAGScheduler: Final stage: Stage 0(count at <console>:15)
17/10/13 23:09:48 INFO DAGScheduler: Parents of final stage: List()
17/10/13 23:09:48 INFO DAGScheduler: Missing parents: List()
17/10/13 23:09:48 INFO DAGScheduler: Submitting Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12), which has no missing parents
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(2544) called with curMem=288849, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.5 KB, free 267.0 MB)
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(1898) called with curMem=291393, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1898.0 B, free 267.0 MB)
17/10/13 23:09:48 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41619 (size: 1898.0 B, free: 267.2 MB)
17/10/13 23:09:48 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
17/10/13 23:09:48 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:838
17/10/13 23:09:48 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12)
17/10/13 23:09:48 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/10/13 23:09:48 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 1307 bytes)
17/10/13 23:09:48 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
17/10/13 23:09:49 INFO HadoopRDD: Input split: hdfs://alamps:9000/wordcount/input/test.txt:0+88
17/10/13 23:09:49 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/10/13 23:09:49 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/10/13 23:09:49 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/10/13 23:09:49 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/10/13 23:09:49 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/10/13 23:09:53 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1920 bytes result sent to driver
17/10/13 23:09:53 INFO DAGScheduler: Stage 0 (count at <console>:15) finished in 4.875 s
17/10/13 23:09:53 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4812 ms on localhost (1/1)
17/10/13 23:09:53 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/10/13 23:09:53 INFO DAGScheduler: Job 0 finished: count at <console>:15, took 5.480197 s
res2: Long = 8

[hadoop@alamps sbin]$ jps
3596 Master
3733 Worker
2558 DataNode
2748 SecondaryNameNode
3814 Jps
2884 ResourceManager
2986 NodeManager
2467 NameNode
[hadoop@alamps sbin]$ hadoop fs -ls /
Found 11 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:29 /aaa
drwxr-xr-x   - hadoop supergroup          0 2017-10-06 04:04 /external
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 09:14 /flowsum
-rw-r--r--   1 hadoop supergroup         43 2017-10-02 02:52 /hello.txt
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 21:10 /index
-rw-r--r--   1 hadoop supergroup  143588167 2017-10-01 08:38 /jdk-7u65-linux-i586.tar.gz
drwx------   - hadoop supergroup          0 2017-10-05 22:43 /tmp
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:18 /upload
drwxr-xr-x   - hadoop supergroup          0 2017-10-05 22:44 /user
drwxr-xr-x   - hadoop supergroup          0 2017-10-03 06:20 /wc
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount
cat: `/wordcount': Is a directory
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:00 /wordcount/input
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount/out
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount/input
Found 1 items
-rw-r--r--   1 hadoop supergroup         88 2017-10-01 09:00 /wordcount/input/test.txt
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount/input/test.txt
hello tom
hello java
hello c
hello python
hello scala
hello spark
hello baby
hello java
[hadoop@alamps sbin]$

val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")

lines.count()

-----
scala> val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(77922) called with curMem=179665, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 76.1 KB, free 267.0 MB)
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(31262) called with curMem=257587, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 30.5 KB, free 267.0 MB)
17/10/13 23:09:24 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41619 (size: 30.5 KB, free: 267.2 MB)
17/10/13 23:09:24 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
17/10/13 23:09:24 INFO SparkContext: Created broadcast 1 from textFile at <console>:12
lines: org.apache.spark.rdd.RDD[String] = hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12

scala> lines.count()
17/10/13 23:09:45 INFO FileInputFormat: Total input paths to process : 1
17/10/13 23:09:48 INFO SparkContext: Starting job: count at <console>:15
17/10/13 23:09:48 INFO DAGScheduler: Got job 0 (count at <console>:15) with 1 output partitions (allowLocal=false)
17/10/13 23:09:48 INFO DAGScheduler: Final stage: Stage 0(count at <console>:15)
17/10/13 23:09:48 INFO DAGScheduler: Parents of final stage: List()
17/10/13 23:09:48 INFO DAGScheduler: Missing parents: List()
17/10/13 23:09:48 INFO DAGScheduler: Submitting Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12), which has no missing parents
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(2544) called with curMem=288849, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.5 KB, free 267.0 MB)
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(1898) called with curMem=291393, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1898.0 B, free 267.0 MB)
17/10/13 23:09:48 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41619 (size: 1898.0 B, free: 267.2 MB)
17/10/13 23:09:48 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
17/10/13 23:09:48 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:838
17/10/13 23:09:48 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12)
17/10/13 23:09:48 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/10/13 23:09:48 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 1307 bytes)
17/10/13 23:09:48 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
17/10/13 23:09:49 INFO HadoopRDD: Input split: hdfs://alamps:9000/wordcount/input/test.txt:0+88
17/10/13 23:09:49 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/10/13 23:09:49 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/10/13 23:09:49 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/10/13 23:09:49 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/10/13 23:09:49 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/10/13 23:09:53 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1920 bytes result sent to driver
17/10/13 23:09:53 INFO DAGScheduler: Stage 0 (count at <console>:15) finished in 4.875 s
17/10/13 23:09:53 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4812 ms on localhost (1/1)
17/10/13 23:09:53 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/10/13 23:09:53 INFO DAGScheduler: Job 0 finished: count at <console>:15, took 5.480197 s
res2: Long = 8

[hadoop@alamps sbin]$ jps
3596 Master
3733 Worker
2558 DataNode
2748 SecondaryNameNode
3814 Jps
2884 ResourceManager
2986 NodeManager
2467 NameNode
[hadoop@alamps sbin]$ hadoop fs -ls /
Found 11 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:29 /aaa
drwxr-xr-x   - hadoop supergroup          0 2017-10-06 04:04 /external
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 09:14 /flowsum
-rw-r--r--   1 hadoop supergroup         43 2017-10-02 02:52 /hello.txt
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 21:10 /index
-rw-r--r--   1 hadoop supergroup  143588167 2017-10-01 08:38 /jdk-7u65-linux-i586.tar.gz
drwx------   - hadoop supergroup          0 2017-10-05 22:43 /tmp
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:18 /upload
drwxr-xr-x   - hadoop supergroup          0 2017-10-05 22:44 /user
drwxr-xr-x   - hadoop supergroup          0 2017-10-03 06:20 /wc
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount
cat: `/wordcount': Is a directory
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:00 /wordcount/input
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount/out
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount/input
Found 1 items
-rw-r--r--   1 hadoop supergroup         88 2017-10-01 09:00 /wordcount/input/test.txt
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount/input/test.txt
hello tom
hello java
hello c
hello python
hello scala
hello spark
hello baby
hello java
[hadoop@alamps sbin]$

spark-shell下有提示了，但是发现不能退格的更多相关文章

  1. 02、体验Spark shell下RDD编程

    02.体验Spark shell下RDD编程 1.Spark RDD介绍 RDD是Resilient Distributed Dataset,中文翻译是弹性分布式数据集.该类是Spark是核心类成员之 ...

  2. ubuntu下使用vi是方向键变乱码 退格键不能使用的解决方法

    ubuntu下使用vi是方向键变乱码 退格键不能使用的解决方法 转载:http://blog.csdn.net/yao_qinwei/article/details/8761777 在插入模式下,按删 ...

  3. ubuntu 下使用vi时方向键乱码，退格键不能使用

    ubuntu 下使用vi时方向键乱码,退格键不能使用的解决方法 问题表现:Ubuntu下,初始使用vi,编辑模式下使用方向键的时候,并不会使光标移动,而是在命令行中出现A B C D 之类的字母,并且 ...

  4. 在Scala IDEA for Eclipse或IDEA里程序编译实现与在Spark Shell下的对比（其实就是那么一回事）

    不多说,直接上干货! 比如,我这里拿主成分分析(PCA). 1.主成分分析(PCA)的概念介绍 主成分分析(PCA) 是一种对数据进行旋转变换的统计学方法,其本质是在线性空间中进行一个基变换,使得变换 ...

  5. Cygwin下安装vim后，vim中退格键无法正常使用

    问题描述: 在Cygwin中安装完vim后 进入vim,发现上下左右键和退格键都无法正常使用 问题分析: 首先考虑到的就是缺少vim的配置文件,首先查看/etc路径下是否有vim的配置文件 admin ...

  6. Ubuntu下使用Vi时方向键变乱码 退格键不能使用的解决方法

    要在Ubuntu下编辑一些文件,这就涉及到了vi这个编辑器了.在Ubuntu下,初始使用vi的时候有点问题,就是在编辑模式下使用方向键的时候,并不会使光标移动,而是在命令行中出现[A [B [C [D ...

  7. 解决Ubuntu 下 vi编辑器不能使用方向键和退格键问题

    转自:http://blog.csdn.net/sky101010ws/article/details/51012103 使用vi命令时,不能正常编辑文件,使用方向键时老是出现很多字母 这个问题主要是 ...

  8. 解决Ubuntu14.04下vi编辑器不能使用方向键和退格键问题

    参考:http://blog.sina.com.cn/s/blog_7d0c2fed01010zbi.html 系统:Ubuntu14.04 使用vi命令时,不能正常编辑文件,使用方向键时老是出现很多 ...

  9. ubuntu 下解决sublime v3 中文输入法时 退格键删除不了拼音的问题

    ubuntu下,sulime想要支持中文需要这样设置: 1.安装中文输入解决的github git clone https://github.com/lyfeyaj/sublime-text-imfi ...

随机推荐

  1. Installing Precise (12.04.2) using netboot onto a Marvell ArmadaXP Development Board

      https://wiki.ubuntu.com/ARM/Server/Install https://wiki.ubuntu.com/ARM/Server/Install/ArmadaXP Arm ...

  2. XA Transactions Restrictions on XA Transactions

    小结: 1.innodb支持XA事务: 2.XA协议作为资源管理器(数据库)与事务管理器的接口标准: 3.提交或者回滚的点:必须所有的组件被提交或者被回滚: 4.2阶段 PC-1,TM告知所有RM要准 ...

  3. HZAU2018年十大阅读之星演讲稿

    写在前面:本博客为本人原创,严禁任何形式的转载!本博客只允许放在博客园(.cnblogs.com),如果您在其他网站看到这篇博文,请通过下面这个唯一的合法链接转到原文! 本博客全网唯一合法URL:ht ...

  4. Appium入门（4）__ Appium Client安装

    打算使用 Python 语言编写 appium 自动化测试脚本 一.前提: 1.安装Python语言 到Python官网下载最新版本:https://www.python.org/ 2.安装Pytho ...

  5. Amazon SQS(Simple Queue Service) 简单介绍

    版权声明:本文为博主原创文章,未经博主同意不得转载. https://blog.csdn.net/hongchangfirst/article/details/25877059 SQS即Simple ...

  6. 使用shell脚本监控用户登陆服务器并发送提示信息给微信

    1.需要在/etc/ssh/目录下面创建一个名为sshrc的文件,执行权限可给可不给,那么在有人通过ssh远程登录这台服务器的时候,这段脚本就会被执行 #!/bin/bash ###V1---### ...

  7. mysql-utilities1.6

    mysql-utilities1.6 mysql-utilities是一个用python编写的mysql工具集 mysql-utilities是Oracle专门开发的 一共有28个工具 /usr/bi ...

  8. Unable to cast COM object of type 'Shell32.ShellClass' to interface type 'Shell32.IShellDispatch6'.

    VS 里面的Interop.Shell32.dll(1.0) 这个版本太低了,需要高版本的(我使用的是1.2.107.0) 具体下载地址  自己找吧

  9. Linux ssh将命令放入后台

    如何在关闭ssh连接的情况下,让程序继续运行? 对Unix,Linux类服务器维护经常是通过ssh完成的,而有些操作比较费时,如更新程序等.此时如果断开ssh连接的话,更新程序就会随之被中断.如何保证 ...

  10. PHP策略模式2

    <?php /** PHP 策略模式 * 策略模式是对象的行为模式,用意是对一组算法的封装.动态的选择需要的算法并使用. * 策略模式指的是程序中涉及决策控制的一种模式.策略模式功能非常强大,因 ...