一、

参考 https://www.cnblogs.com/bovenson/p/5801536.html

[root@node- test]# chown hdfs:hdfs     /root/test/*
[root@node-1 test]# chown hdfs:hdfs /root/test
[root@node-1 test]# cd /var/lib/hadoop-hdfs/
[root@node-1 hadoop-hdfs]# ls
[root@node-1 hadoop-hdfs]# mkdir /var/lib/hadoop-hdfs/test
[root@node-1 hadoop-hdfs]# mv /root/test/* /var/lib/hadoop-hdfs/test
[root@node-1 hadoop-hdfs]# su hdfs
Attempting to create directory /var/lib/hadoop-hdfs/perl5
[hdfs@node-1 ~]$ hadoop fs -put test/*.txt /user/tt/me.txt
$ hadoop fs -ls -R /user/tt
-rw-r--r-- 3 hdfs supergroup 56302 2018-01-30 20:59 /user/tt/me.txt # spark-shell --master spark://node-1:7077 --executor-memory 100M --driver-memory 1000M Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_162)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc (master = spark://node-1:7077, app id = app-20180130211525-0002).
SQL context available as sqlContext. scala> val lines = sc.textFile("hdfs://node-1:8020/user/tt/me.txt")
lines: org.apache.spark.rdd.RDD[String] = hdfs://node-1:8020/user/tt/me.txt MapPartitionsRDD[1] at textFile at <console>:27 scala> lines.count()
res0: Long = 10000 scala> lines.first()
res1: String = 4 scala>

spark 文件行数统计-命令行

二、

备注:

# spark-shell  --help
Usage: ./bin/spark-shell [options] Options:
--master MASTER_URL spark://host:port, mesos://host:port, yarn, or local.
--deploy-mode DEPLOY_MODE Whether to launch the driver program locally ("client") or
on one of the worker machines inside the cluster ("cluster")
(Default: client).
--class CLASS_NAME Your application's main class (for Java / Scala apps).
--name NAME A name of your application.
--jars JARS Comma-separated list of local jars to include on the driver
and executor classpaths.
--packages Comma-separated list of maven coordinates of jars to include
on the driver and executor classpaths. Will search the local
maven repo, then maven central and any additional remote
repositories given by --repositories. The format for the
coordinates should be groupId:artifactId:version.
--exclude-packages Comma-separated list of groupId:artifactId, to exclude while
resolving the dependencies provided in --packages to avoid
dependency conflicts.
--repositories Comma-separated list of additional remote repositories to
search for the maven coordinates given with --packages.
--py-files PY_FILES Comma-separated list of .zip, .egg, or .py files to place
on the PYTHONPATH for Python apps.
--files FILES Comma-separated list of files to be placed in the working
directory of each executor. --conf PROP=VALUE Arbitrary Spark configuration property.
--properties-file FILE Path to a file from which to load extra properties. If not
specified, this will look for conf/spark-defaults.conf. --driver-memory MEM Memory for driver (e.g. 1000M, 2G) (Default: 1024M).
--driver-java-options Extra Java options to pass to the driver.
--driver-library-path Extra library path entries to pass to the driver.
--driver-class-path Extra class path entries to pass to the driver. Note that
jars added with --jars are automatically included in the
classpath. --executor-memory MEM Memory per executor (e.g. 1000M, 2G) (Default: 1G). --proxy-user NAME User to impersonate when submitting the application. --help, -h Show this help message and exit
--verbose, -v Print additional debug output
--version, Print the version of current Spark Spark standalone with cluster deploy mode only:
--driver-cores NUM Cores for driver (Default: ). Spark standalone or Mesos with cluster deploy mode only:
--supervise If given, restarts the driver on failure.
--kill SUBMISSION_ID If given, kills the driver specified.
--status SUBMISSION_ID If given, requests the status of the driver specified. Spark standalone and Mesos only:
--total-executor-cores NUM Total cores for all executors. Spark standalone and YARN only:
--executor-cores NUM Number of cores per executor. (Default: in YARN mode,
or all available cores on the worker in standalone mode) YARN-only:
--driver-cores NUM Number of cores used by the driver, only in cluster mode
(Default: ).
--queue QUEUE_NAME The YARN queue to submit to (Default: "default").
--num-executors NUM Number of executors to launch (Default: ).
--archives ARCHIVES Comma separated list of archives to be extracted into the
working directory of each executor.
--principal PRINCIPAL Principal to be used to login to KDC, while running on
secure HDFS.
--keytab KEYTAB The full path to the file that contains the keytab for the
principal specified above. This keytab will be copied to
the node running the Application Master via the Secure
Distributed Cache, for renewing the login tickets and the
delegation tokens periodically.

spark-shell --help

CDH spark 命令行测试的更多相关文章

  1. Linux命令行测试网速speedtest.net

    Linux命令行测试网速speedtest.net 当发现上网速度变慢时,人们通常会先首先测试自己的电脑到网络服务提供商(通常被称为"最后一公里")的网络连接速度.在可用于测试宽带 ...

  2. I.MX6 Android CAN 命令行测试

    /********************************************************************* * I.MX6 Android CAN 命令行测试 * 说 ...

  3. [转]使用Linux命令行测试网速

    装speedtest-cli speedtest-cli是一个用Python编写的轻量级Linux命令行工具,在Python2.4至3.4版本下均可运行.它基于Speedtest.net的基础架构来测 ...

  4. 使用Linux命令行测试网速

    安装speedtest speedtest是一个用Python编写的轻量级Linux命令行工具,在Python2.4至3.4版本下均可运行.它基于Speedtest.net的基础架构来测量网络的上/下 ...

  5. 使用Linux命令行测试网速-----speedtest-cli

    https://github.com/sivel/speedtest-cli 当发现上网速度变慢时,人们通常会先首先测试自己的电脑到网络服务提供商(通常被称为“最后一公里”)的网络连接速度.在可用于测 ...

  6. gtest命令行测试案例

    使用gtest编写的测试案例通常本身就是一个可执行文件,因此运行起来非常方便.同时,gtest也为我们提供了一系列的运行参数(环境变量.命令行参数或代码里指定),使得我们可以对案例的执行进行一些有效的 ...

  7. monkey命令行测试

    一. 什么是Monkey monkey是google提供的一个用于稳定性与压力测试的命令行工具.monkey程序由android系统自带,位于/sdcard/system/framework/monk ...

  8. Junit 命令行测试 报错:Could not find class 理解及解决方法

    一.报错 : 『Could not find class』 下面给出三个示例比较,其中只有第一个是正确的. 1. MyComputer:bin marikobayashi$ java -cp .:./ ...

  9. laravel 命令行测试 Uncaught ReflectionException: Class config does not exist

    require __DIR__ . '/vendor/autoload.php'; $app = require_once __DIR__ . '/bootstrap/app.php'; config ...

随机推荐

  1. “美登杯”上海市高校大学生程序设计赛B. 小花梨的三角形(模拟,实现)

    题目链接:https://acm.ecnu.edu.cn/contest/173/problem/B/#report9 Problem B B . 小 花梨 的 三角形 时间限制:1000ms 空间限 ...

  2. CSS基础学习-7.CSS元素分类

  3. 数据库中char和varchar的区别

    1. char类型的长度是固定的,varchar的长度是可变的. 这就表示,存储字符串'abc',使用char(10),表示存储的字符将占10个字节(包括7个空字符) 使用varchar2(10),, ...

  4. MySQL中主键的选择与磁盘性能

    偶然看到了“Fotolog: Scaling the World\'s Largest Photo Blogging Community”,才发现很多数据库的优化其实道理都很简单,至高境界是当你面对问 ...

  5. Codevs 1213 解的个数(exgcd)

    1213 解的个数 时间限制: 1 s 空间限制: 128000 KB 题目等级 : 黄金 Gold 题目描述 Description 已知整数x,y满足如下面的条件: ax+by+c=0 p< ...

  6. USACO19JAN Redistricting

    题目链接:戳我 一个优先队列优化DP 一定要注意第二关键字的排序啊!!我真的是菜,被坑了好久qwq 设\(f[i]\)表示前i个的最小答案,从前面选择的时候第一关键字是f[j]的大小,第二关键字是要确 ...

  7. 棋盘问题 ( POJ -1321 )(简单DFS)

    转载请注明出处:https://blog.csdn.net/Mercury_Lc/article/details/82684942作者:Mercury_Lc 题目链接 题解:dfs入门,就是每个点都搜 ...

  8. mac 强行关掉php

    sudo pkill -INT -o php-fpm//重启php sudo php-fpm //mac brew安装的php可以使用这个开启brew services start homebrew/ ...

  9. 转:玩转HTML5移动页面(动效篇)

    作为一名前端,在拿到设计稿时你有两种选择: 1.快速输出静态页面 2.加上高级大气上档次狂拽炫酷屌炸天的动画让页面动起来 作为一个有志向的前端,当然是选2啦!可是需求时间又很短很短,怎么办呢? 这次就 ...

  10. 笔记本在安装Windows+Linux双系统后,进入Windows时花屏的解决办法

    问题:在笔记本安装双系统(Windows7+Ubuntu14.04)[先安装Windows,后安装Ubuntu]后,进入Windows时出现了花屏. 问题原因:笔记本只有集显,在系统启动时会先加载Ub ...