A data engineering platform that processes millions of financial transactions, implements real-time fraud detection, and provides enterprise-grade analytics through interactive dashboards and APIs.
This is a production-ready data engineering platform that demonstrates end-to-end capabilities for financial transaction processing and fraud detection. It processes a comprehensive PaySim sample dataset and provides:
- Real-time transaction processing with fraud detection algorithms
- Interactive analytics dashboard with fraud patterns and risk metrics
- RESTful API for data access and integration
- Enterprise-grade infrastructure with monitoring and security
- Scalable data pipeline handling millions of records
- Processes 1.35M+ financial transactions from PaySim dataset
- Real-time fraud detection with risk scoring algorithms
- Transaction pattern analysis and anomaly detection
- Balance change monitoring and velocity checks
- Interactive dashboard with fraud metrics and trends
- Transaction type analysis and amount distribution
- Risk score visualization and fraud pattern identification
- Real-time data updates and historical analysis
- Production-ready architecture with Docker containerization
- Real-time streaming with Apache Kafka
- PostgreSQL database with optimized queries
- Comprehensive monitoring and alerting
- JWT authentication and role-based access control
# Clone and setup
git clone https://github.com/rohansimha02/FintechDataEngineeringPlatform.git
cd FintechDataEngineeringPlatform
cp env.example .env
# Start the platform
make up
# Access the dashboard
# http://localhost:8061Note: This repository includes a 55MB optimized sample dataset (data/dashboard_sample.csv) with 750K+ transactions for demonstration purposes. The full PaySim dataset (471MB) is excluded from the repository due to GitHub's file size limits.
The platform follows a modern data engineering architecture with real-time and batch processing capabilities:
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ PaySim Data │ │ FastAPI API │ │ Dashboard │
│ (1.35M+ txns) │ │ (Analytics) │ │ (Visualization)│
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────┐
│ Data Processing Layer │
│ ETL Pipeline + Fraud Detection │
└─────────────────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Real-time │ │ Batch ETL │ │ Analytics │
│ Processing │ │ Orchestration │ │ & ML Models │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────┐
│ Data Storage & Analytics │
│ PostgreSQL + Monitoring + Security │
└─────────────────────────────────────────────────────────────────┘
- Data Ingestion: PaySim financial transaction dataset
- Processing: ETL pipeline with fraud detection algorithms
- Storage: PostgreSQL with optimized schema
- Analytics: Interactive dashboard and REST API
- Infrastructure: Docker, monitoring, and security
- ETL Pipeline: Processes 1.35M+ transactions with fraud detection
- Real-time Processing: Kafka streaming for live transaction monitoring
- Batch Processing: Daily ETL jobs with data quality checks
- Data Transformations: dbt models for analytics and reporting
- Containerized Architecture: Docker Compose for easy deployment
- Database: PostgreSQL with optimized schema and indexing
- Streaming: Apache Kafka for real-time data processing
- Monitoring: Prometheus metrics and Grafana dashboards
- Authentication: JWT-based security with role-based access
- Data Quality: Great Expectations for validation and testing
- Orchestration: Prefect for workflow management
- Testing: Comprehensive integration and load testing
- API Documentation: Auto-generated OpenAPI docs
- One-Command Setup:
make upto start entire platform - Comprehensive Logging: Structured logging with traceability
- CI/CD Ready: Production deployment configuration
- 1.35M+ transactions processed successfully
- Real-time dashboard with interactive visualizations
- Optimized ETL pipeline for large-scale data processing
- Scalable architecture supporting millions of records
- <100ms API responses at scale with proper indexing
- Efficient database queries with proper indexing
- Real-time streaming for live transaction monitoring
- Load testing capabilities for performance validation
- Python 3.11+
- Docker (optional, for full infrastructure)
- Make (for convenience commands)
# Install dependencies
pip install -r requirements.txt
# Run the dashboard
python app/dashboard/main.py
# Or use Docker for full stack
make up# Run tests
make test
# Code formatting
make lint
# View logs
make logs- API Response Time: <100ms p95
- Transaction Processing: >1000 TPS
- Consumer Lag: <1000 messages
- Database Connections: <80% utilization
- API Performance: Request rate, latency, errors
- Data Pipeline: ETL success rate, processing time
- Infrastructure: CPU, memory, disk usage
- Business Metrics: Transaction volume, fraud rate
- JWT-based authentication with secure token management
- Role-based permissions for different user types
- API security with proper authentication headers
- Environment-based configuration for sensitive data
- Secure communication with proper encryption
- Security best practices implemented throughout
- Containerized deployment with Docker
- Environment configuration for different stages
- Database migrations and schema management
- Monitoring and alerting setup
# Local development
python app/dashboard/main.py
# Docker deployment
make up
# Production deployment
docker-compose -f docker-compose.prod.yml up -d- Analytics endpoints for transaction data access
- Authentication with JWT tokens
- OpenAPI documentation for easy integration
- Health checks and monitoring endpoints
GET /analytics/dashboard- Dashboard metricsGET /analytics/transactions- Transaction analyticsGET /analytics/fraud- Fraud detection dataPOST /auth/login- User authenticationGET /health- System health check
- Extract: Load PaySim transaction data (1.35M+ records)
- Transform: Apply fraud detection algorithms and business logic
- Load: Store processed data with analytics and risk scores
- Validate: Data quality checks and validation
- Analyze: Generate fraud patterns and insights
- Streaming data for live transaction monitoring
- Fraud detection with real-time risk scoring
- Data validation and quality checks
- Alert generation for suspicious activities
- PaySim Dataset: Realistic financial transaction data for fraud detection
- Python Stack: FastAPI, Dash, SQLAlchemy for rapid development
- PostgreSQL: ACID compliance and complex analytics capabilities
- Docker: Containerized deployment for consistency and scalability
- Data Processing: Pandas, NumPy for ETL and analytics
- Web Framework: FastAPI for high-performance API
- Dashboard: Dash and Plotly for interactive visualizations
- Infrastructure: Docker, Kafka, PostgreSQL for production readiness
- Fork the repository
- Create a feature branch
- Make changes with tests
- Run linting and tests
- Submit a pull request
- Documentation: Check the
/docsendpoint - Issues: Create GitHub issues
- Questions: Open discussions for help
MIT License - see LICENSE file for details.