PySpark + Jupyter Lab with Docker

A ready-to-use repository for Big Data, Data Engineering and Machine Learning students working with PySpark. The goal is to provide a fully configured environment with PySpark and Jupyter Lab, without having to install Java, Spark or deal with manual setup.

What is Docker

Docker lets you run applications inside containers: isolated environments that include everything needed (operating system, Java, Spark, Python, libraries) without installing anything on your machine. In this project Docker creates a container with PySpark and Jupyter Lab already configured — you just need to start it.

To learn more: Docker overview (official documentation).

Installing Docker

Windows

1. Check that WSL2 is installed. Open PowerShell and run:

   wsl --status

   If the command shows the WSL version, you're good. If you get an error, install it by opening PowerShell as administrator:

   wsl --install

   Restart your PC if prompted. Full guide: Install WSL.

2. Download and install Docker Desktop for Windows. Alternatively, it can be installed from the Docker Desktop app on the Microsoft Store.

3. On first launch Docker Desktop will ask to use the WSL2 backend — accept.

4. Verify by opening a terminal:

   docker --version

macOS

1. Download and install Docker Desktop for Mac (choose the correct link for Intel or Apple Silicon chip).

2. Open Docker Desktop from the Applications folder.

3. Verify from the terminal:

   docker --version

Linux (Ubuntu/Debian)

You can install Docker Engine directly or Docker Desktop. The quickest way:

# Update packages and install Docker
sudo apt-get update
sudo apt-get install -y docker.io docker-compose-v2

# Add your user to the docker group (avoids using sudo every time)
sudo usermod -aG docker $USER

# Log out and back in, then verify
docker --version

For other distributions or to install Docker Desktop: Install Docker Engine.

Project structure

pyspark-docker/
├── docker-compose.yml          # Container configuration
├── Dockerfile                  # Custom image (installs requirements.txt)
├── requirements.txt            # Additional Python libraries
├── .env.example                # Configuration template (copy as .env)
├── .gitignore                  # Files ignored by Git
├── .dockerignore               # Files excluded from Docker build
├── LICENSE                     # MIT License
├── README.md                   # This file
├── notebooks/
│   └── 01_intro_pyspark.ipynb  # Example notebook
└── data/
    ├── input/
    │   └── .gitkeep            # Folder for input data
    └── output/
        └── .gitkeep            # Folder for output data (generated by notebooks)

Quick start

# 1. Clone the repository
git clone https://github.com/skateddu/pyspark-docker
cd pyspark-docker

# 2. Copy the configuration file
cp .env.example .env

# 3. Start the container (first time downloads the image, takes a few minutes)
docker compose up --build

# 4. Open Jupyter Lab in your browser
# http://localhost:8888/lab?token=pyspark

Note: the Docker terminal will display a URL like http://127.0.0.1:8888/lab?token=... with an auto-generated token. That token will not work: the container is configured to use the token defined in the .env file (pyspark by default). Always use the URL shown above, or open http://localhost:8888 and enter the token from your .env file when prompted.

Versions

The image is pinned to quay.io/jupyter/pyspark-notebook:spark-4.1.2 (Spark 4.1.2, Python 3.13), instead of floating :latest, so that everyone building this repo gets the same environment. Spark 4 is a major version bump from the 3.5.x line used when this repo was first created — the DataFrame/SQL APIs used in the example notebook are unaffected, but if you bring in older PySpark code, check it against the Spark 4.0 migration notes first.

To upgrade later, pick a new tag from the pyspark-notebook tags on quay.io, update the FROM line in Dockerfile, and rebuild with docker compose up --build.

Adding Python libraries

1. Open requirements.txt
2. Uncomment or add the libraries you need
3. Rebuild the container:

docker compose up --build

Useful commands

Command	Description
docker compose up --build	Start and rebuild the container
docker compose up -d	Start in background
docker compose down	Stop the container
docker compose logs -f	View logs in real time
docker compose restart	Restart the container

Stopping and cleaning up

To stop the container (notebooks and local data are preserved):

docker compose down

To identify the image and volumes created by this project:

# List images built by this project
docker images "pyspark-docker*"

# List Docker volumes
docker volume ls

To remove only this project's image (useful to free disk space). On the next docker compose up --build the image will be rebuilt from scratch:

docker rmi pyspark-docker-pyspark

To also remove volumes created by the container (Spark metastore, temp files):

docker compose down --volumes

Note: notebooks and data in local folders (notebooks/ and data/) are never deleted by any of these commands, since they are bind mounts (shared folders between your computer and the container). Only internal container data is removed.

Spark UI

When a SparkSession is active, you can monitor Spark jobs at:

• Spark UI: http://localhost:4040

Troubleshooting

Docker not running

Cannot connect to the Docker daemon

Start Docker Desktop and try again.

Port already in use

Bind for 0.0.0.0:8888 failed: port is already allocated

Change JUPYTER_PORT in your .env file (e.g. JUPYTER_PORT=8889).

Files not visible in the notebook

Notebooks are mounted at /home/jovyan/notebooks, data at /home/jovyan/data. Use these paths when reading or writing files from PySpark.