English · Русский

twitch data analytics pipeline

An ETL project for Twitch data.

It gets streams from the Twitch API and stores the raw JSON in MongoDB. Then it cleans the data in Spark and puts it into PostgreSQL with a star schema. After that, you can look at the data in Superset.

structure

• dags/ — airflow tasks that get and process the data
• scripts/ — code that uses the twitch api and spark
• config/ — shared configuration (DAG args, Twitch and MongoDB settings)
• infra/ — docker-compose, dockerfile, postgres init (schema + seed data), requirements
• tools/ — setup script

stack

• python
• apache airflow
• apache spark (pyspark)
• mongodb (raw json landing zone)
• postgresql (star schema warehouse)
• docker, docker compose
• apache superset

data flow

1. collect (twitch_data_collector DAG, every 15 min) — pull top streams from the Twitch API and insert the raw JSON documents into MongoDB (twitch.raw_streams). Old documents are cleaned up after 7 days.
2. process (twitch_data_processor DAG, daily) — Spark reads the day's documents from MongoDB, resolves the dimensions, builds the fact rows and writes them to PostgreSQL, then refreshes the analytics materialized views.
3. explore — Superset reads the star schema and materialized views.

The PostgreSQL star schema (dimension/fact tables, indexes, materialized views) and the static dimensions (dim_dates, dim_times, dim_languages) are created automatically by infra/init-postgres.sql on first start — there are no separate migrations to run.

launch

1. make a copy of the environment file: cp env.example .env
2. add your values in .env:
   • TWITCH_CLIENT_ID
   • TWITCH_CLIENT_SECRET
   • AIRFLOW_UID
   • MONGO_URI (defaults to mongodb://mongo:27017)
3. start the project: make setup

• airflow: http://localhost:8080 (airflow / airflow)
• superset: http://localhost:8088 (admin / admin)
• mongodb: mongodb://localhost:27017 (database twitch, collection raw_streams)

commands

• make up — start the services
• make down — stop the services
• make status — check the status
• make logs — see the logs