📜 Description
There are two severe architectural bugs actively bottlenecking the speed of document ingestion and degrading the quality of RAG retrieval:
Sequential Network Latency: In application/parser/embedding_pipeline.py, the embed_and_store_documents function iterates through chunks sequentially via store.add_texts([doc.page_content]). A document split into 1,000 chunks results in 1,000 separate synchronous HTTP requests. For remote embeddings (like OpenAI), this sequential I/O creates massive latency and artificial bottlenecks.
Semantic Context Loss: In application/parser/chunking.py, the classic_chunk strategy uses a naive token-counting cut (body_tokens[current_position:end_position]). This mathematically slices words and sentences in half and provides zero token overlap across boundaries. This destroys the semantic context the LLM relies on during retrieval, breaks sentence structures, and generates orphaned, meaningless chunks.
👟 Reproduction steps
- Upload a large 50+ page document using a remote embedding model (e.g., OpenAI).
- Observe the severe ingestion delay due to hundreds of sequential, synchronous HTTP requests in the backend logs.
- Once ingested, query the document for information that was split across a token boundary.
- Observe that the context is lost or hallucinated because the classic_chunk algorithm severed the sentence in half without any token overlap.
👍 Expected behavior
Embedding: embed_and_store_documents should utilize vector store batching (store.add_texts(batch)) to send chunks in batches (e.g., 100 at a time), radically reducing HTTP overhead and resolving the ingestion delay.
Chunking: Document chunking should utilize recursive text splitting (e.g., Langchain's RecursiveCharacterTextSplitter, which is already in requirements.txt) to respect sentence boundaries and support token overlap, ensuring context survives boundary splits.
👎 Actual Behavior with Screenshots
Ingestion of large files hangs for an exceptionally long time.
The RAG pipeline returns degraded responses because the provided chunks contain sentences cut cleanly in half by the token limiter. (No screenshots applicable as this is a backend architectural bug).
💻 Operating system
Windows
What browsers are you seeing the problem on?
Chrome
🤖 What development environment are you experiencing this bug on?
Docker
🔒 Did you set the correct environment variables in the right path? List the environment variable names (not values please!)
Yes
VECTOR_STORE
EMBEDDINGS_KEY
LLM_NAME
📃 Provide any additional context for the Bug.
I have already set up a local development environment and verified these architectural bugs. I plan to refactor the loop in embedding_pipeline.py to accumulate docs.page_content into batch arrays and update the _record_progress checkpointing to track per batch. For chunking.py, I will replace/augment classic_chunk with RecursiveCharacterTextSplitter.from_tiktoken_encoder.
📖 Relevant log output
👀 Have you spent some time to check if this bug has been raised before?
🔗 Are you willing to submit PR?
Yes, I am willing to submit a PR!
🧑⚖️ Code of Conduct
📜 Description
There are two severe architectural bugs actively bottlenecking the speed of document ingestion and degrading the quality of RAG retrieval:
Sequential Network Latency: In application/parser/embedding_pipeline.py, the embed_and_store_documents function iterates through chunks sequentially via store.add_texts([doc.page_content]). A document split into 1,000 chunks results in 1,000 separate synchronous HTTP requests. For remote embeddings (like OpenAI), this sequential I/O creates massive latency and artificial bottlenecks.
Semantic Context Loss: In application/parser/chunking.py, the classic_chunk strategy uses a naive token-counting cut (body_tokens[current_position:end_position]). This mathematically slices words and sentences in half and provides zero token overlap across boundaries. This destroys the semantic context the LLM relies on during retrieval, breaks sentence structures, and generates orphaned, meaningless chunks.
👟 Reproduction steps
👍 Expected behavior
Embedding: embed_and_store_documents should utilize vector store batching (store.add_texts(batch)) to send chunks in batches (e.g., 100 at a time), radically reducing HTTP overhead and resolving the ingestion delay.
Chunking: Document chunking should utilize recursive text splitting (e.g., Langchain's RecursiveCharacterTextSplitter, which is already in requirements.txt) to respect sentence boundaries and support token overlap, ensuring context survives boundary splits.
👎 Actual Behavior with Screenshots
Ingestion of large files hangs for an exceptionally long time.
The RAG pipeline returns degraded responses because the provided chunks contain sentences cut cleanly in half by the token limiter. (No screenshots applicable as this is a backend architectural bug).
💻 Operating system
Windows
What browsers are you seeing the problem on?
Chrome
🤖 What development environment are you experiencing this bug on?
Docker
🔒 Did you set the correct environment variables in the right path? List the environment variable names (not values please!)
Yes
VECTOR_STORE
EMBEDDINGS_KEY
LLM_NAME
📃 Provide any additional context for the Bug.
I have already set up a local development environment and verified these architectural bugs. I plan to refactor the loop in embedding_pipeline.py to accumulate docs.page_content into batch arrays and update the _record_progress checkpointing to track per batch. For chunking.py, I will replace/augment classic_chunk with RecursiveCharacterTextSplitter.from_tiktoken_encoder.
📖 Relevant log output
👀 Have you spent some time to check if this bug has been raised before?
🔗 Are you willing to submit PR?
Yes, I am willing to submit a PR!
🧑⚖️ Code of Conduct