Skip to content

fix(http): sanitize non-ASCII in URL before urllib.request to prevent latin-1 encode crash (fixes #817)#821

Open
23241a6749 wants to merge 1 commit into
mvanhorn:mainfrom
23241a6749:fix/unicode-encode-817-v3
Open

fix(http): sanitize non-ASCII in URL before urllib.request to prevent latin-1 encode crash (fixes #817)#821
23241a6749 wants to merge 1 commit into
mvanhorn:mainfrom
23241a6749:fix/unicode-encode-817-v3

Conversation

@23241a6749

Copy link
Copy Markdown
Contributor

Fixes #817

Problem

Non-Latin-script topics (Arabic, CJK, Cyrillic, etc.) cause TikTok and Instagram ScrapeCreators requests to crash with:

UnicodeEncodeError: 'latin-1' codec can't encode character '\u0627' in position 28: ordinal not in range(256)

Fix

Added a urllib.parse.quote pass on the URL string after urlencode construction in http.request(), using a safe set that preserves all valid URL structural characters. This guarantees the URL is always 100% ASCII-safe before being passed to urllib.request.Request.

safe='/:@!$&\'()*+,;=-._~%?#[]=+'

Testing

  • test_http_v3.py: 14/14 passed
  • test_grounding_v3.py: 28/28 passed
  • test_env_v3.py: 13/13 passed

@greptile-apps

greptile-apps Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR makes HTTP request URLs ASCII-safe before they reach urllib.request. The main changes are:

  • Adds urllib.parse.quote to the HTTP helper imports.
  • Applies a final whole-URL quoting pass after query parameter construction.
  • Preserves URL structural characters while encoding non-ASCII text.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
skills/last30days/scripts/lib/http.py Adds final URL quoting in request() to prevent non-ASCII URLs from crashing urllib.request.

Reviews (1): Last reviewed commit: "fix(http): sanitize non-ASCII in URL bef..." | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UnicodeEncodeError ('latin-1' codec) on Arabic-script topics — TikTok/Instagram (ScrapeCreators) requests fail

1 participant