配好了Spark集群后,先用pyspark写了两个小例子，但是发现Tab键没有提示，于是打算转到scala上试试，在spark-shell下有提示了，但是发现不能退格，而且提示也不是复写，而是追加，这样根本就没法写程序.

      解决办法：

1.打开会话选项

2.终端-仿真    在终端中选择Linux

3.映射键   勾选两个选项

4.至此已经成功了,但是如果远程长时间未操作 就会中断连接，下次再操作时需要等待，其实也很影响使用，在这里也附上解决办法(可选)

val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")

lines.count()

-----
scala> val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(77922) called with curMem=179665, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 76.1 KB, free 267.0 MB)
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(31262) called with curMem=257587, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 30.5 KB, free 267.0 MB)
17/10/13 23:09:24 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41619 (size: 30.5 KB, free: 267.2 MB)
17/10/13 23:09:24 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
17/10/13 23:09:24 INFO SparkContext: Created broadcast 1 from textFile at <console>:12
lines: org.apache.spark.rdd.RDD[String] = hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12

scala> lines.count()
17/10/13 23:09:45 INFO FileInputFormat: Total input paths to process : 1
17/10/13 23:09:48 INFO SparkContext: Starting job: count at <console>:15
17/10/13 23:09:48 INFO DAGScheduler: Got job 0 (count at <console>:15) with 1 output partitions (allowLocal=false)
17/10/13 23:09:48 INFO DAGScheduler: Final stage: Stage 0(count at <console>:15)
17/10/13 23:09:48 INFO DAGScheduler: Parents of final stage: List()
17/10/13 23:09:48 INFO DAGScheduler: Missing parents: List()
17/10/13 23:09:48 INFO DAGScheduler: Submitting Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12), which has no missing parents
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(2544) called with curMem=288849, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.5 KB, free 267.0 MB)
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(1898) called with curMem=291393, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1898.0 B, free 267.0 MB)
17/10/13 23:09:48 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41619 (size: 1898.0 B, free: 267.2 MB)
17/10/13 23:09:48 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
17/10/13 23:09:48 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:838
17/10/13 23:09:48 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12)
17/10/13 23:09:48 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/10/13 23:09:48 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 1307 bytes)
17/10/13 23:09:48 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
17/10/13 23:09:49 INFO HadoopRDD: Input split: hdfs://alamps:9000/wordcount/input/test.txt:0+88
17/10/13 23:09:49 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/10/13 23:09:49 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/10/13 23:09:49 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/10/13 23:09:49 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/10/13 23:09:49 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/10/13 23:09:53 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1920 bytes result sent to driver
17/10/13 23:09:53 INFO DAGScheduler: Stage 0 (count at <console>:15) finished in 4.875 s
17/10/13 23:09:53 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4812 ms on localhost (1/1)
17/10/13 23:09:53 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/10/13 23:09:53 INFO DAGScheduler: Job 0 finished: count at <console>:15, took 5.480197 s
res2: Long = 8

[hadoop@alamps sbin]$ jps
3596 Master
3733 Worker
2558 DataNode
2748 SecondaryNameNode
3814 Jps
2884 ResourceManager
2986 NodeManager
2467 NameNode
[hadoop@alamps sbin]$ hadoop fs -ls /
Found 11 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:29 /aaa
drwxr-xr-x   - hadoop supergroup          0 2017-10-06 04:04 /external
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 09:14 /flowsum
-rw-r--r--   1 hadoop supergroup         43 2017-10-02 02:52 /hello.txt
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 21:10 /index
-rw-r--r--   1 hadoop supergroup  143588167 2017-10-01 08:38 /jdk-7u65-linux-i586.tar.gz
drwx------   - hadoop supergroup          0 2017-10-05 22:43 /tmp
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:18 /upload
drwxr-xr-x   - hadoop supergroup          0 2017-10-05 22:44 /user
drwxr-xr-x   - hadoop supergroup          0 2017-10-03 06:20 /wc
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount
cat: `/wordcount': Is a directory
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:00 /wordcount/input
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount/out
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount/input
Found 1 items
-rw-r--r--   1 hadoop supergroup         88 2017-10-01 09:00 /wordcount/input/test.txt
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount/input/test.txt
hello tom
hello java
hello c
hello python
hello scala
hello spark
hello baby
hello java
[hadoop@alamps sbin]$

val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")

lines.count()

-----
scala> val lines =sc.textFile("hdfs://alamps:9000/wordcount/input/test.txt")
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(77922) called with curMem=179665, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 76.1 KB, free 267.0 MB)
17/10/13 23:09:24 INFO MemoryStore: ensureFreeSpace(31262) called with curMem=257587, maxMem=280248975
17/10/13 23:09:24 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 30.5 KB, free 267.0 MB)
17/10/13 23:09:24 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41619 (size: 30.5 KB, free: 267.2 MB)
17/10/13 23:09:24 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
17/10/13 23:09:24 INFO SparkContext: Created broadcast 1 from textFile at <console>:12
lines: org.apache.spark.rdd.RDD[String] = hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12

scala> lines.count()
17/10/13 23:09:45 INFO FileInputFormat: Total input paths to process : 1
17/10/13 23:09:48 INFO SparkContext: Starting job: count at <console>:15
17/10/13 23:09:48 INFO DAGScheduler: Got job 0 (count at <console>:15) with 1 output partitions (allowLocal=false)
17/10/13 23:09:48 INFO DAGScheduler: Final stage: Stage 0(count at <console>:15)
17/10/13 23:09:48 INFO DAGScheduler: Parents of final stage: List()
17/10/13 23:09:48 INFO DAGScheduler: Missing parents: List()
17/10/13 23:09:48 INFO DAGScheduler: Submitting Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12), which has no missing parents
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(2544) called with curMem=288849, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.5 KB, free 267.0 MB)
17/10/13 23:09:48 INFO MemoryStore: ensureFreeSpace(1898) called with curMem=291393, maxMem=280248975
17/10/13 23:09:48 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1898.0 B, free 267.0 MB)
17/10/13 23:09:48 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41619 (size: 1898.0 B, free: 267.2 MB)
17/10/13 23:09:48 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
17/10/13 23:09:48 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:838
17/10/13 23:09:48 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at textFile at <console>:12)
17/10/13 23:09:48 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/10/13 23:09:48 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 1307 bytes)
17/10/13 23:09:48 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
17/10/13 23:09:49 INFO HadoopRDD: Input split: hdfs://alamps:9000/wordcount/input/test.txt:0+88
17/10/13 23:09:49 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/10/13 23:09:49 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/10/13 23:09:49 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/10/13 23:09:49 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/10/13 23:09:49 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/10/13 23:09:53 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1920 bytes result sent to driver
17/10/13 23:09:53 INFO DAGScheduler: Stage 0 (count at <console>:15) finished in 4.875 s
17/10/13 23:09:53 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4812 ms on localhost (1/1)
17/10/13 23:09:53 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/10/13 23:09:53 INFO DAGScheduler: Job 0 finished: count at <console>:15, took 5.480197 s
res2: Long = 8

[hadoop@alamps sbin]$ jps
3596 Master
3733 Worker
2558 DataNode
2748 SecondaryNameNode
3814 Jps
2884 ResourceManager
2986 NodeManager
2467 NameNode
[hadoop@alamps sbin]$ hadoop fs -ls /
Found 11 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:29 /aaa
drwxr-xr-x   - hadoop supergroup          0 2017-10-06 04:04 /external
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 09:14 /flowsum
-rw-r--r--   1 hadoop supergroup         43 2017-10-02 02:52 /hello.txt
drwxr-xr-x   - hadoop supergroup          0 2017-10-04 21:10 /index
-rw-r--r--   1 hadoop supergroup  143588167 2017-10-01 08:38 /jdk-7u65-linux-i586.tar.gz
drwx------   - hadoop supergroup          0 2017-10-05 22:43 /tmp
drwxr-xr-x   - hadoop supergroup          0 2017-10-02 06:18 /upload
drwxr-xr-x   - hadoop supergroup          0 2017-10-05 22:44 /user
drwxr-xr-x   - hadoop supergroup          0 2017-10-03 06:20 /wc
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount
cat: `/wordcount': Is a directory
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:00 /wordcount/input
drwxr-xr-x   - hadoop supergroup          0 2017-10-01 09:07 /wordcount/out
[hadoop@alamps sbin]$ hadoop fs -ls /wordcount/input
Found 1 items
-rw-r--r--   1 hadoop supergroup         88 2017-10-01 09:00 /wordcount/input/test.txt
[hadoop@alamps sbin]$ hadoop fs -cat /wordcount/input/test.txt
hello tom
hello java
hello c
hello python
hello scala
hello spark
hello baby
hello java
[hadoop@alamps sbin]$

spark-shell下有提示了，但是发现不能退格的更多相关文章

  1. 02、体验Spark shell下RDD编程

    02.体验Spark shell下RDD编程 1.Spark RDD介绍 RDD是Resilient Distributed Dataset,中文翻译是弹性分布式数据集.该类是Spark是核心类成员之 ...

  2. ubuntu下使用vi是方向键变乱码 退格键不能使用的解决方法

    ubuntu下使用vi是方向键变乱码 退格键不能使用的解决方法 转载:http://blog.csdn.net/yao_qinwei/article/details/8761777 在插入模式下,按删 ...

  3. ubuntu 下使用vi时方向键乱码，退格键不能使用

    ubuntu 下使用vi时方向键乱码,退格键不能使用的解决方法 问题表现:Ubuntu下,初始使用vi,编辑模式下使用方向键的时候,并不会使光标移动,而是在命令行中出现A B C D 之类的字母,并且 ...

  4. 在Scala IDEA for Eclipse或IDEA里程序编译实现与在Spark Shell下的对比（其实就是那么一回事）

    不多说,直接上干货! 比如,我这里拿主成分分析(PCA). 1.主成分分析(PCA)的概念介绍 主成分分析(PCA) 是一种对数据进行旋转变换的统计学方法,其本质是在线性空间中进行一个基变换,使得变换 ...

  5. Cygwin下安装vim后，vim中退格键无法正常使用

    问题描述: 在Cygwin中安装完vim后 进入vim,发现上下左右键和退格键都无法正常使用 问题分析: 首先考虑到的就是缺少vim的配置文件,首先查看/etc路径下是否有vim的配置文件 admin ...

  6. Ubuntu下使用Vi时方向键变乱码 退格键不能使用的解决方法

    要在Ubuntu下编辑一些文件,这就涉及到了vi这个编辑器了.在Ubuntu下,初始使用vi的时候有点问题,就是在编辑模式下使用方向键的时候,并不会使光标移动,而是在命令行中出现[A [B [C [D ...

  7. 解决Ubuntu 下 vi编辑器不能使用方向键和退格键问题

    转自:http://blog.csdn.net/sky101010ws/article/details/51012103 使用vi命令时,不能正常编辑文件,使用方向键时老是出现很多字母 这个问题主要是 ...

  8. 解决Ubuntu14.04下vi编辑器不能使用方向键和退格键问题

    参考:http://blog.sina.com.cn/s/blog_7d0c2fed01010zbi.html 系统:Ubuntu14.04 使用vi命令时,不能正常编辑文件,使用方向键时老是出现很多 ...

  9. ubuntu 下解决sublime v3 中文输入法时 退格键删除不了拼音的问题

    ubuntu下,sulime想要支持中文需要这样设置: 1.安装中文输入解决的github git clone https://github.com/lyfeyaj/sublime-text-imfi ...

随机推荐

  1. GlusterFS分布式存储系统中更换故障Brick的操作记录1

    前面已经介绍了GlusterFS分布式存储集群环境部署记录,现在模拟下更换故障Brick的操作: 1)GlusterFS集群系统一共有4个节点,集群信息如下: 分别在各个节点上配置hosts.同步好系 ...

  2. field, or, more generally, in a ring or even a semiring 数域、环、半环

    小结: 1.数域.环.半环 :一般化.泛化 https://en.wikipedia.org/wiki/Matrix_multiplication In mathematics, matrix mul ...

  3. 分析java的堆栈信息 内存模型

    package com.test.learnJava; public class LineNum { public static void main(String[] args) { System.o ...

  4. [development][dpdk] dpdk-pmdinfo 工具的使用

    发现了一个工具 dpdk-pmdinfo 但是文档介绍的很不清楚, http://dpdk.org/doc/guides/tools/pmdinfo.html 经过摸索, 这可能就是一个查询 dpdk ...

  5. Flink - Asynchronous I/O

    https://docs.google.com/document/d/1Lr9UYXEz6s6R_3PWg3bZQLF3upGaNEkc0rQCFSzaYDI/edit   // create the ...

  6. webstorm背景颜色更改

    一.file --> settings 二.editor-->color scheme  然后右边下拉选择 三.apply -- > ok

  7. dedecms首页去掉index.html怎么设置

    很多网友用IIS服务器建站,反映说dedecms首页默认多了一个/index.html,一般是没有这个后缀的,直接就**.com,那么如何将dedecms首页去掉index.html呢?很简单,服务器 ...

  8. 误删除innodb ibdata数据文件 文件句柄 文件描述符 proc fd

    误删除innodb ibdata数据文件  文件句柄  文件描述符  proc  fd http://www.cnblogs.com/gomysql/p/3702216.html 提示:如果不小心通过 ...

  9. Cglib动态代理实现原理

    Cglib动态代理实现方式 我们先通过一个demo看一下Cglib是如何实现动态代理的. 首先定义个服务类,有两个方法并且其中一个方法用final来修饰. public class PersonSer ...

  10. 转换区别json

    private Date EndDate ; private Instant xxxxdate; private LocalDateTime localDateTime; public static ...