README.md

Reddit RAG System

A Retrieval-Augmented Generation (RAG) system that uses Reddit comments as a knowledge base. This system allows you to ask questions and get answers based on similar Reddit comments, with links back to the original posts.

📊 Data Access

To access the Reddit data used in this project, please email: jaswinderswork@gmail.com

Data Source & Credits

The Reddit data used in this project was originally scraped and generously shared by Reddit user in their post:

🔗 What 20 Million of Reddit Comments and 30K Subreddits Look Like

This dataset contains over 20 million Reddit comments across 30,000+ subreddits, providing an incredibly rich source of community knowledge and discussions. We extend our sincere thanks to u/ehtio for making this valuable dataset publicly available and enabling projects like this RAG system.

---

🎯 What This System Does

1. Processes Reddit Comments: Takes your parquet files with Reddit data and converts them into searchable embeddings
2. Semantic Search: Finds the most relevant Reddit comments for any question
3. RAG Integration: Provides context from Reddit comments to enhance LLM responses
4. Source Attribution: Links back to original Reddit comments for transparency

📊 Data Structure

Your Reddit data should contain:

• Comments file (all_comments_merged.parquet): Reddit comments with columns:

  • id: Comment ID (for linking back to Reddit)
  • body_clean: Clean comment text
  • subreddit: Subreddit name
  • username: Reddit username
  • score: Comment score
  • created_utc: Unix timestamp
  • language: Detected language
  • negative, neutral, positive: Sentiment scores

• Users file (all_users_merged.parquet): User profiles with karma, mod status, etc.

🚀 Quick Start

1. Setup Environment

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Start Milvus (if not already running)
docker compose up -d

2. Index Comments in Bulk

# Index 100,000 comments (adjust target_count in the script if needed)
python index_reddit_comments.py

This will:

• Load the embedding model
• Process comments in batches of 1,000
• Generate embeddings for each comment
• Store them in Milvus with metadata
• Provide detailed progress logging

Expected Performance: ~100k comments in 4-5 minutes

3. Query Your Knowledge Base

# Start the interactive RAG system
python reddit_rag_system.py

Choose option 2 to search similar comments and ask questions like:

• "What's the best programming language for beginners?"
• "Python tips and tricks"
• "How to learn machine learning?"

🔧 System Architecture

User Question → LLM Query Generation → Embedding → Vector Search → Relevant Comments → Enhanced Answer + Sources

Components:

1. Vector Database: Milvus Lite for efficient similarity search
2. Embedding Model: Sentence Transformers (all-MiniLM-L6-v2)
3. Data Processing: Pandas for chunked parquet reading
4. Search: Cosine similarity with optional subreddit filtering

📁 File Structure

reddit-data/
├── all_comments_merged.parquet    # Your Reddit comments (3.7GB) - NOT in git
├── all_users_merged.parquet       # Your Reddit users (1.2MB) - NOT in git
├── reddit_rag_system.py           # Main RAG system for querying
├── index_reddit_comments.py       # Bulk indexing script
├── docker-compose.yml             # Milvus setup
├── requirements.txt               # Python dependencies
├── quick_explore.py              # Data exploration script
├── .gitignore                    # Excludes large files
└── README.md                     # This file

🛠️ Installation Requirements

• Python 3.8+
• Docker (for Milvus)
• 8GB+ RAM (for processing large files)
• Virtual environment (recommended)

📦 Dependencies

• pandas: Data processing
• pyarrow: Parquet file handling
• pymilvus: Vector database client
• sentence-transformers: Text embeddings
• torch: Deep learning backend
• tqdm: Progress bars
• psutil: Memory monitoring

🔍 Usage Examples

Bulk Indexing

from index_reddit_comments import RedditIndexer

# Initialize indexer
indexer = RedditIndexer()

# Index comments (100k default, adjust as needed)
total_processed = indexer.index_comments(
    "all_comments_merged.parquet", 
    target_count=100000, 
    batch_size=1000
)

Searching Comments

from reddit_rag_system import RedditRAGSystem

# Initialize RAG system
rag = RedditRAGSystem()

# Search for similar comments
results = rag.search_similar_comments("What's the best programming language for beginners?")

# Search with subreddit filter
results = rag.search_similar_comments("Python tips", subreddit_filter="learnprogramming")

Integration with LLM

# Get relevant comments for context
context_comments = rag.search_similar_comments(user_question, n_results=5)

# Format for LLM prompt
context = "\n\n".join([f"Reddit comment: {c['text']}" for c in context_comments])

# Send to LLM with context
llm_prompt = f"""
Based on these Reddit comments, answer the user's question:

{context}

User question: {user_question}

Sources: {[c['comment_id'] for c in context_comments]}
"""

🎯 Use Cases

1. Community Knowledge Base: Answer questions based on Reddit community wisdom
2. Content Research: Find relevant discussions on any topic
3. Sentiment Analysis: Understand community sentiment on topics
4. Trend Analysis: Track discussions over time
5. Expert Finding: Identify knowledgeable users in specific domains

🔧 Configuration

Indexing Configuration

Edit index_reddit_comments.py to adjust:

TARGET_COUNT = 100000  # Number of comments to index
BATCH_SIZE = 1000      # Batch size for processing

Embedding Model

Change the model in RedditIndexer(model_name="..."):

• all-MiniLM-L6-v2 (default, fast, 384d)
• all-mpnet-base-v2 (better quality, 768d)
• multi-qa-MiniLM-L6-cos-v1 (optimized for Q&A)

🚨 Troubleshooting

Milvus Connection Issues

# Check if Milvus is running
docker ps | grep milvus

# Restart Milvus
docker compose down
docker compose up -d

Memory Issues

• Reduce batch size: batch_size=500
• Process smaller samples: target_count=50000
• Use smaller embedding model

Indexing Errors

• Check reddit_indexing.log for detailed error information
• Ensure parquet file is not corrupted
• Verify Milvus is running and accessible

📈 Performance Tips

1. Start Small: Test with 10k samples first
2. Monitor Memory: Watch RAM usage during processing
3. Use SSD: Faster I/O for large files
4. Batch Processing: Adjust batch sizes based on your hardware
5. Index Optimization: Milvus will automatically optimize indexes

🔗 Integration with LLMs

This system is designed to work with any LLM:

# Example with OpenAI
import openai

def answer_with_reddit_context(question):
    # Get relevant Reddit comments
    comments = rag.search_similar_comments(question, n_results=3)
    
    # Format context
    context = "\n".join([c['text'] for c in comments])
    
    # Send to OpenAI
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": "Answer based on the provided Reddit comments."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
        ]
    )
    
    return response.choices[0].message.content, comments

📊 Monitoring and Logs

• Indexing logs: reddit_indexing.log - Detailed progress and performance metrics
• Collection stats: Use option 3 in reddit_rag_system.py to check indexed data
• Memory monitoring: Automatic memory usage tracking during indexing
• Performance metrics: Speed, timing breakdown, sentiment distribution

🤝 Contributing

Feel free to:

• Add new embedding models
• Improve the search algorithms
• Add more data processing features
• Create visualization tools

📄 License

This project is for educational and research purposes. Please respect Reddit's terms of service when using the data.

---

Happy building! 🚀