Run Spark cluster using Docker

docker compose up -d

Generate 20GB of data

2.1 Create a virtual environment (.venv) and activate

python3.12 -m venv .venv                                                                                                                                                                                                                     
source .venv/bin/activate

2.2 Install package

pip install -r requirements.txt

2.3 Generate TCPH data

cd data && bench tpch gen -s 10

3. Run the Spark application (shuffle, sort, broadcast join)

3.1 Spin up docker container

docker compose up -d

docker exec -it spark-master /opt/spark/bin/spark-submit --master spark://spark-master:7077 /opt/spark/work-dir/spark-warehouse/data/join.py

3.2 Generate Smaller data for Broadcast join

cd data && bench tpch gen -s 1

4. Run the Spark application (ouput bucket table)

docker exec -it spark-master /opt/spark/bin/spark-submit --master spark://spark-master:7077 /opt/spark/work-dir/spark-warehouse/data/write_bucket.py

5. Run the Spark application (bucket join)

docker exec -it spark-master /opt/spark/bin/spark-submit --master spark://spark-master:7077 /opt/spark/work-dir/spark-warehouse/data/bucket_join.py