utf-8 encoding issue for PDFs

[caizixian]

I'm currently running the latest rag_api docker image, and have trouble with some of the PDF files.
This is the backtrace from the FastAPI backend

2025-07-21 12:42:02,569 - root - ERROR - Error during file processing: 'utf-8' codec can't encode character '\ud835' in position 416: surrogates not allowed
Traceback: Traceback (most recent call last):
  File "/app/app/routes/document_routes.py", line 419, in embed_file
    result = await store_data_in_vector_db(
  File "/app/app/routes/document_routes.py", line 274, in store_data_in_vector_db
    docs = [
  File "/app/app/routes/document_routes.py", line 280, in <listcomp>
    "digest": generate_digest(doc.page_content),
  File "/app/app/routes/document_routes.py", line 252, in generate_digest
    hash_obj = hashlib.md5(page_content.encode())
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud835' in position 416: surrogates not allowed

So the PDFs might contain invalid utf-8 characters for hashing

[danny-avila]

this may have been solved with the latest update, but can you provide a file to test this with to confirm?

[caizixian]

Unfortunately I couldn't share the file due to copyright. The problem still persists using the latest image with ID 2f812738600a. I opened a PR to address this.

See also https://github.com/cloud-gov/pages-build-container/pull/100/files

The linked blog post seems to be useful https://lucumr.pocoo.org/2013/7/2/the-updated-guide-to-unicode/

[caizixian]

OK, the problem actually wasn't properly fixed. Sorry that I thought it was an easy fix and opened a PR on the web interface. Turns out it's a bit complicated. So I set up my own build environment and gave it more testing today without having to wait for the GitHub Action to build.

Problem 1:
#199 was actually incorrect. surrogateescape has to be used with decode first and then a later encode with surrogateescape will work fine. https://docs.python.org/3/library/codecs.html#error-handlers

This is not what we want. I changed it to surrogatepass, and tested it, It solved the hashing problem.

However, the same problem now happens with storing to pgvector when using the ORM.

Problem 2:

2025-08-27 00:37:55,005 - root - WARNING - Failed to remove temporary UTF-8 file: remove: path should be string, bytes or os.PathLike, not NoneType
2025-08-27 00:38:00,261 - httpx - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
2025-08-27 00:38:01,033 - root - ERROR - Failed to store data in vector DB | File ID: c0034b07-6a8b-418d-9e26-e9f285c1e7ee | User ID: 687dfe626ae9f5ca41a918ec | Error: 'utf-8' codec can't encode character '\ud835' in position 416: surrogates not allowed | Traceback: Traceback (most recent call last):
  File "/app/app/routes/document_routes.py", line 289, in store_data_in_vector_db
    ids = await vector_store.aadd_documents(
  File "/app/app/services/vector_store/async_pg_vector.py", line 69, in aadd_documents
    return await run_in_executor(
  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/config.py", line 621, in run_in_executor
    return await asyncio.get_running_loop().run_in_executor(executor_or_config, wrapper)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/config.py", line 607, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/langchain_core/vectorstores/base.py", line 288, in add_documents
    return self.add_texts(texts, metadatas, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/langchain_community/vectorstores/pgvector.py", line 565, in add_texts
    return self.add_embeddings(
  File "/usr/local/lib/python3.10/site-packages/langchain_community/vectorstores/pgvector.py", line 542, in add_embeddings
    session.bulk_save_objects(documents)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4578, in bulk_save_objects
    self._bulk_save_mappings(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4755, in _bulk_save_mappings
    with util.safe_reraise():
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 224, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4744, in _bulk_save_mappings
    bulk_persistence._bulk_insert(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/bulk_persistence.py", line 222, in _bulk_insert
    result = persistence._emit_insert_statements(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1048, in _emit_insert_statements
    result = connection.execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1415, in execute
    return meth(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 523, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1840, in _execute_context
    return self._exec_insertmany_context(dialect, context)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2122, in _exec_insertmany_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2354, in _handle_dbapi_exception
    raise exc_info[1].with_traceback(exc_info[2])
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2114, in _exec_insertmany_context
    dialect.do_execute(
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud835' in position 416: surrogates not allowed

[caizixian]

I think LangChain uses str throughout, so this error only happens when executing the SQL statement. Not quite sure how to properly fix this at the moment.

[caizixian]

@danny-avila could you reopen the issue

[caizixian]

I've just created a new PR. I've built my own Docker image and tested it out in my local installation. It handles the previously problematic PDF file fine now.