[danny-avila]

• Refactored _prepare_documents_sync to handle document preparation synchronously, improving performance and clarity.
• Updated store_data_in_vector_db to utilize the new synchronous document preparation function, ensuring non-blocking behavior in the async context.
• Changed is_health_ok to be asynchronous, allowing for proper handling of health checks for different database types.

These changes streamline document processing and improve the responsiveness of health checks in the application.

[danny-avila]

Summary

I refactored the document processing pipeline and health check system to be non-blocking, preventing CPU-bound work from blocking the async event loop.

• Extracted document preparation logic (text splitting, cleaning, metadata assignment) into a synchronous _prepare_documents_sync() function that runs in a thread executor via loop.run_in_executor()
• Fixed a bug where len(documents) was used instead of len(docs) when generating IDs for vector store insertion; this caused a mismatch since the post-split chunk count differs from the original document count
• Converted is_health_ok() to async and added await calls to pg_health_check() and mongo_health_check()
• Changed save_upload_file_sync() to save_upload_file_async() for non-blocking file uploads
• Changed os.remove() to cleanup_temp_file_async() for non-blocking temp file cleanup

Potential Issues

1. Deprecated event loop access: asyncio.get_event_loop() is deprecated in Python 3.10+. Replace with asyncio.get_running_loop().

2. Health check dependencies: The PR assumes pg_health_check() and mongo_health_check() are already async. Verify these functions have been updated or this will break.

3. Caller updates needed: Any code calling is_health_ok() synchronously will break and needs to await it now.

Change Type

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Testing

Recommend testing:

• Upload large documents and verify the API remains responsive to concurrent requests during processing
• Test health endpoint with both Postgres and MongoDB configurations
• Verify document chunking produces correct number of vector store entries with matching IDs

Checklist

• My code adheres to this project's style guidelines
• I have performed a self-review of my own code
• I have commented in any complex areas of my code
• I have made pertinent documentation changes
• My changes do not introduce new warnings
• I have written tests demonstrating that my changes are effective or that my feature works
• Local unit tests pass with my changes
• Any changes dependent on mine have been merged and published in downstream modules

[riyadislam2605-hub]

gh repo clone danny-avila/rag_api