When I write PySpark code, I use Jupyter notebook to test my code before submitting a job on the cluster. In this post, I will show you how to install and run PySpark locally in Jupyter Notebook on Windows. I’ve tested this guide on a dozen Windows 7 and 10 PCs in different languages.

A. Items needed

  • Spark distribution from spark.apache.org

  • The findspark Python module, which can be installed by running python -m pip install findspark either in Windows command prompt or Git bash if Python is installed in item 2. You can find command prompt by searching cmd in the search box.

  • If you don’t have Java or your Java version is 7.x or less, download and install Java from Oracle. I recommend getting the latest JDK (current version 9.0.1).

  • If you don’t know how to unpack a .tgz file on Windows, you can download and install 7-zip on Windows to unpack the .tgz file from Spark distribution in item 1 by right-clicking on the file icon and select 7-zip > Extract Here.

B. Installing PySpark

After getting all the items in section A, let’s set up PySpark.

  • Unpack the .tgz file. For example, I unpacked with 7zip from step A6 and put mine under D:\spark\spark-2.2.1-bin-hadoop2.7

  • Move the winutils.exe downloaded from step A3 to the \bin folder of Spark distribution. For example, D:\spark\spark-2.2.1-bin-hadoop2.7\bin\winutils.exe

  • Add environment variables: the environment variables let Windows find where the files are when we start the PySpark kernel. You can find the environment variable settings by putting “environ…” in the search box.

The variables to add are, in my example,

Name Value
SPARK_HOME D:\spark\spark-2.2.1-bin-hadoop2.7
HADOOP_HOME D:\spark\spark-2.2.1-bin-hadoop2.7
PYSPARK_DRIVER_PYTHON jupyter
PYSPARK_DRIVER_PYTHON_OPTS notebook

  • In the same environment variable settings window, look for the Path or PATH variable, click edit and add D:\spark\spark-2.2.1-bin-hadoop2.7\bin to it. In Windows 7 you need to separate the values in Path with a semicolon ; between the values.

  • (Optional, if see Java related error in step C) Find the installed Java JDK folder from step A5, for example, D:\Program Files\Java\jdk1.8.0_121, and add the following environment variable

Name Value
JAVA_HOME D:\Progra~1\Java\jdk1.8.0_121

If JDK is installed under \Program Files (x86), then replace the Progra~1 part by Progra~2 instead. In my experience, this error only occurs in Windows 7, and I think it’s because Spark couldn’t parse the space in the folder name.

Edit (1/23/19): You might also find Gerard’s comment helpful: http://disq.us/p/1z5qou4

C. Running PySpark in Jupyter Notebook

To run Jupyter notebook, open Windows command prompt or Git Bash and run jupyter notebook. If you use Anaconda Navigator to open Jupyter Notebook instead, you might see a Java gateway process exited before sending the driver its port number error from PySpark in step C. Fall back to Windows cmd if it happens.

Once inside Jupyter notebook, open a Python 3 notebook

In the notebook, run the following code

import findspark
findspark.init() import pyspark # only run after findspark.init()
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate() df = spark.sql('''select 'spark' as hello ''')
df.show()

When you press run, it might trigger a Windows firewall pop-up. I pressed cancel on the pop-up as blocking the connection doesn’t affect PySpark.

If you see the following output, then you have installed PySpark on your Windows system!

如何在Windows上的Jupyter Notebook中安装和运行PySpark的更多相关文章

  1. 解决在jupyter notebook中遇到的ImportError: matplotlib is required for plotting问题

    昨天学习pandas和matplotlib的过程中, 在jupyter notebook遇到ImportError: matplotlib is required for plotting错误, 以下 ...

  2. 在jupyter notebook 中同时使用安装不同版本的python内核-从而可以进行切换

    在安装anaconda的时候,默认安装的是python3.6 但是cs231n课程作业是在py2.7环境下运行的.所以需要在jupyter notebook中安装并启用python2.7版本 方法: ...

  3. 在jupyter notebook中同时安装python2和python3

    之前讨论过在anaconda下安装多个python版本,本期来讨论下,jupyter notebook中怎样同时安装python2.7 和python3.x. 由于我之前使用的jupyter note ...

  4. Windows下的Jupyter Notebook 安装与自定义启动(图文详解)

    不多说,直接上干货! 前期博客 Windows下的Python 3.6.1的下载与安装(适合32bits和64bits)(图文详解) 这是我自定义的Python 的安装目录 (D:\SoftWare\ ...

  5. 非线性函数的最小二乘拟合及在Jupyter notebook中输入公式 [原创]

    突然有个想法,能否通过学习一阶RC电路的阶跃响应得到RC电路的结构特征——时间常数τ(即R*C).回答无疑是肯定的,但问题是怎样通过最小二乘法.正规方程,以更多的采样点数来降低信号采集噪声对τ估计值的 ...

  6. Windows下的Jupyter Notebook 安装与自定义启动

    1.Jupyter Notebook 和 pip 为了更加方便地写 Python 代码,还需要安装 Jupyter notebook. 利用 pip 安装 Jupyter notebook. 为什么要 ...

  7. windows系统下jupyter notebook使用虚拟环境

    目录 [亲测好使]windows系统下jupyter notebook使用虚拟环境 在虚拟环境中安装jupyter,并添加到jupyter kernel 参考 [未测试,但觉得比上面那方法好,因为上面 ...

  8. (转)如何在Windows上安装多个MySQL

    原文:http://www.blogjava.net/hongjunli/archive/2009/03/01/257216.html 如何在Windows上安装多个MySQL 本文以免安装版的mys ...

  9. Redis简介以及如何在Windows上安装Redis

    Redis简介 Redis是一个速度非常快的非关系型内存数据库. Redis提供了Java,C/C++,C#,PHP,JavaScript,Perl,Object-C,Python,Ruby,Erla ...

随机推荐

  1. 在linux环境下部署禅道环境

    下载禅道安装包: 1)cd /home/ 2)mkdir app 3)ls 4)cd app/ 5)pwd 6)wget +禅道地址(http://dl.cnezsoft.com/zentao/9.8 ...

  2. sudoers权限管理

    该/etc/sudoers文件的权限管理很完善,覆盖了linux中的各种命令,各种shell.编辑器等等,在此留作以后作为参考. # This file MUST be edited with the ...

  3. #学号 20175201张驰 《Java程序设计》第2周学习总结

    教材学习内容总结: 一.第二章: 1:标识符与关键字 2:基本数据类型:四种整数类型(byte.short.int.long).两种浮点数类型(float.double).一种字符类型(char).一 ...

  4. .Net新利器Rider的破解安装与使用

    准备 介绍 Rider 是 JetBrains 提供的一款用于 .Net 开发的 IDE,相对于 VS,它显得更加轻量(才 500m 左右),并且不管是提示功能还是流畅度都不逊色于 VS 且某方面可能 ...

  5. btcpool之Stratum协议

    一.简介 矿机(或挖矿软件)与矿池之间通过stratum协议通讯,它包括订阅.授权.下发难度.下发任务.提交share五种消息. 二.订阅(mining.subscribe) 矿机启动后,先和矿池建立 ...

  6. JDBC(Java Data Base Connectivity,java数据库连接)

    JDBC概述 JDBC(Java Data Base Connectivity,java数据库连接)是一种用于执行SQL语句的Java API,可以为多种关系数据库提供统一访问,它由一组用Java语言 ...

  7. AIX7.1环境打补丁缺少bash OPATCHAUTO-72049

    DB:12.1.0.2.0 RAC OS:AIX 7.1 [To patch only the GI home] su - root export UNZIPPED_PATCH_LOCATION=/o ...

  8. vue二、脚手架搭建

    1:安装nodeJs(下载一路回车) https://nodejs.org/zh-cn/ 2:检验nodeJs是否安装成功 (注意nodeJs是否添加到window路径中) 进入cmd -> n ...

  9. java中的类型转换

    java中的类型转换分为两种 自动类型转换 要实现数据的自动类型转换必须同时满足下面两个条件 两种数据类型彼此兼容 目标类型的取值范围大于原类型范围 强制类型转换 当两种数据类型彼此不兼容,或者说目标 ...

  10. json对象组按某个字段排序

    JS排序 键值对 var sortBy=function (filed,rev,primer){ rev = (rev) ? -1 : 1; return function (a, b) { a = ...