fix: special charcater decoding from datadiff username - #31134
Conversation
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
Code Review ✅ Approved 2 resolved / 2 findingsFixes special character decoding in data-diff usernames by properly rendering the SQLAlchemy URL, adds pre-flight duplicate key validation to surface table and column errors, and fixes the Makefile target definition. No issues found. ✅ 2 resolved✅ Edge Case: Separator-less Concat can flag valid composite keys as duplicates
✅ Quality: .PHONY name doesn't match renamed slim build target
OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar | Powered by Gitar — free for open source |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Suppressed comments (2)
ingestion/src/metadata/data_quality/validations/utils.py:69
- This percent-encoding implementation uses
ord(char)and emits a single%XXsequence per character. That is not valid for non-ASCII usernames (URI percent-encoding must be based on UTF-8 bytes), and it also leaves raw%characters untouched, which can produce invalid/incomplete percent-escape sequences in the rendered URI. A more robust approach is to useurllib.parse.quoteover a UTF-8 string and explicitly control thesafeset; if some characters cannot be safely unescaped by data-diff (username is not decoded), consider encoding them and emitting a warning similar to the reserved-character warning.
def _encode_username_for_data_diff(username: str) -> str:
"""Percent-encode only what data-diff's URI parser needs to locate the userinfo boundaries."""
return "".join(f"%{ord(char):02X}" if char in USERNAME_RESERVED_CHARACTERS else char for char in username)
ingestion/src/metadata/data_quality/validations/table/sqlalchemy/tableDiff.py:676
_table_with_duplicate_keysassumestable_diff_iter.result_listis always present and iterable. Ifresult_listisNone/unset for a givenDiffResultWrapper(or if a different failure path triggers theAssertionError), this will raise a secondary exception and mask the original error. Consider guarding with a falsy check (e.g., treat missingresult_listas 'unknown table' and returnNone) before iterating.
for sign, values in table_diff_iter.result_list:
marker = (sign, tuple(values[:key_length]))
if marker in seen:
return self.runtime_params.table1 if sign == "-" else self.runtime_params.table2
seen.add(marker)
return None
| if url.password is not None: | ||
| userinfo += f":{quote(str(url.password), safe=' +')}" |
Describe your changes:
Fixes #31124
The
serviceUrlwe hand todata-diffis a canonical SQLAlchemy URL, and SQLAlchemypercent-encodes the username when it renders one.
data-diff's URI parser only decodes thepassword, host and query string — never the userinfo username — so an encoded username reaches
the driver still encoded and
user@corp.comauthenticates asuser%40corp.com.render_url_for_data_diff()renders the URL so every component survives exactly oneencode/decode round trip: the username is handed over decoded (except for
:/?#, which muststay encoded or the authority no longer parses, and which we log a warning about since
data-diffwill not decode them back), while the password stays encoded becausedata-diffdoes decode it.
TableParameter.data_diff_service_urlapplies this at the single point wherethe URL leaves us; connection dicts pass through untouched. The stored
serviceUrlisunchanged and remains a valid SQLAlchemy URL.
Two related items rode along:
data-diff's result model assumes the key is unique — it foldsrows into a
{key: sign}map and asserts each key appears at most twice. A non-unique keytherefore fails deep inside the library (
ValueError: Duplicate primary keyson joindiff, abare
AssertionErroron hashdiff), or silently inflates counts through the join fan-out.None of that tells the user which column is at fault.
_validate_key_uniqueness()now runs acount/count-distinct on the key columns first and aborts with the table, the column(s), the
row/distinct-key counts and up to 5 offending values. The check fails open: if it cannot run,
we log and continue rather than failing an otherwise valid test. The test's WHERE clause is
applied (it may itself make the key unique); sampling is not (a key unique only within one
random sample is not a key).
make build-ingestion-base-slim-localwas defined under a duplicatebuild-ingestion-base-localtarget name and so was unreachable.scripts/datamodel_generation.pynow silences the
format of 'X' not understoodwarnings for the customformatvocabularyOpenMetadata owns (
queryBuilder,utc-millisec, …) — only those, so a genuinely new ormisspelled format still surfaces — and passes
--formatters black isortexplicitly, sinceexternal formatters are becoming opt-in upstream and the post-processing depends on black's
output shape.
Type of change:
High-level design:
N/A — small change.
Tests:
Use cases covered
user@corp.com) authenticatessuccessfully, with both password and private-key auth
serviceUrlwith special characters is decoded the same waythe column and sample duplicated values instead of failing with an opaque
AssertionErrortest — the diff proceeds
Unit tests
ingestion/tests/unit/observability/data_quality/validations/test_data_diff_url.py—round-trips every URL component through
data-diff's own parser, covers the double-encodingregression, reserved-character handling, the warning path, dict pass-through, and end-to-end
Snowflake
serviceUrlconstructioningestion/tests/unit/observability/data_quality/validations/table/sqlalchemy/test_table_diff.py—asserts
connect_to_tablereceives a decoded username on both diff paths without mutating thestored URL, plus
DuplicateKeyErrormessage shapes,_validate_key_uniquenessordering/skipbehaviour, and the fail-open paths
make unit_ingestion/pytest ingestion/tests/unit/observability/data_qualityBackend integration tests
Ingestion integration tests
Not applicable — covered by unit tests against
data-diff's real URI parser.A user-overridden
serviceUrlwith special characters is decoded the same wayA table diff configured with a non-unique key column aborts with a message naming the table,
the column and sample duplicated values instead of failing with an opaque
AssertionErrorA key-uniqueness check that cannot run (permissions, unsupported dialect) does not fail the
test — the diff proceeds
Unit tests
ingestion/tests/unit/observability/data_quality/validations/test_data_diff_url.py—round-trips every URL component through
data-diff's own parser, covers the double-encodingregression, reserved-character handling, the warning path, dict pass-through, and end-to-end
Snowflake
serviceUrlconstructioningestion/tests/unit/observability/data_quality/validations/table/sqlalchemy/test_table_diff.py—asserts
connect_to_tablereceives a decoded username on both diff paths without mutating thestored URL, plus
DuplicateKeyErrormessage shapes,_validate_key_uniquenessordering/skipbehaviour, and the fail-open paths
make unit_ingestion/pytest ingestion/tests/unit/observability/data_qualityBackend integration tests
Ingestion integration tests
data-diff's real URI parser.Playwright (UI) tests
Manual testing performed
tableDifftest —previously failed authentication, now connects.
tableDiffon a table with a duplicated key column — test result isAbortedwith thecolumn name and sample duplicate values in the result message.
make generate— noformat not understoodwarnings, generated models unchanged.make build-ingestion-slim-local— builds the slim image.UI screen recording / screenshots:
Not applicable.
Checklist:
Fixes <issue-number>: <short explanation>Fixes #31124above.test_it_stops_double_encoding_the_username, referencing Double encoding in data diff #31124).Suggested PR title: Fixes #31124: stop double-encoding the username in the data-diff service URL
Two things to verify before posting — I inferred them from the diff rather than running anything: the manual test steps above (I did not run a live Snowflake
diff), and whether you want the duplicate-key detection in this PR at all, since it's a distinct behaviour change from the encoding fix and might read better as
its own issue/PR.