Skip to content

fix: support encoding of special characters in path params for gen2 f… #1911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
IzaakGough wants to merge 3 commits into
base: master
Choose a base branch
from
Open

fix: support encoding of special characters in path params for gen2 f… #1911

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d054308
fix: support encoding of special characters in path params for gen2 f…
IzaakGough Jun 17, 2026
0ac9d73
Merge branch 'master' into @invertase/fix-gen2-support-encoding-of-sp…
IzaakGough Jun 25, 2026
dac6de6
Merge branch 'master' into @invertase/fix-gen2-support-encoding-of-sp…
IzaakGough Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions spec/common/utilities/path-pattern.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,70 @@ describe("path-pattern", () => {
});
});

it("should repair UTF-8 mojibake params", () => {
const pp = new pathPattern.PathPattern("{a}/something/end");

expect(pp.extractMatches("Helvétios/something/end")).to.deep.equal({
a: "Helvétios",
});
});

it("should repair UTF-8 mojibake params for CJK text", () => {
const pp = new pathPattern.PathPattern("{a}/something/end");

expect(pp.extractMatches("中文/something/end")).to.deep.equal({
a: "中文",
});
});

it("should repair UTF-8 mojibake params for emoji", () => {
const pp = new pathPattern.PathPattern("{a}/something/end");

expect(pp.extractMatches("😀/something/end")).to.deep.equal({
a: "😀",
});
});

it("should repair UTF-8 mojibake params for Hindi text", () => {
const pp = new pathPattern.PathPattern("{a}/something/end");

expect(pp.extractMatches("हिन्दी/something/end")).to.deep.equal({
a: "हिन्दी",
});
});

it("should repair UTF-8 mojibake params for Arabic text", () => {
const pp = new pathPattern.PathPattern("{a}/something/end");

expect(pp.extractMatches("اَلْعَرَبِيَّةُ/something/end")).to.deep.equal({
a: "اَلْعَرَبِيَّةُ",
});
});

it("should preserve already decoded unicode params", () => {
const pp = new pathPattern.PathPattern("{a}/something/end");

expect(pp.extractMatches("Helvétios/something/end")).to.deep.equal({
a: "Helvétios",
});
});

it("should preserve already decoded CJK params", () => {
const pp = new pathPattern.PathPattern("{a}/something/end");

expect(pp.extractMatches("中文/something/end")).to.deep.equal({
a: "中文",
});
});

it("should repair mojibake in multi-segment captures", () => {
const pp = new pathPattern.PathPattern("prefix/{path=**}/end");

expect(pp.extractMatches("prefix/中文/😀/Helvétios/end")).to.deep.equal({
path: "中文/😀/Helvétios",
});
});

it("should safely handle malformed URL encoded params", () => {
const pp = new pathPattern.PathPattern("{a}/something/end");

Expand Down
27 changes: 25 additions & 2 deletions src/common/utilities/path-pattern.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,36 @@ export function trimParam(param: string) {
return paramNoBraces;
}

/** @internal */
const UTF8_MOJIBAKE_REGEX = /[\u00c2-\u00f4][\u0080-\u00bf]/;

/** @internal */
export function tryDecodeUtf8Mojibake(value: string): string {
if (!UTF8_MOJIBAKE_REGEX.test(value)) {
return value;
}

const bytes = Array.from(value, (char) => char.charCodeAt(0));
if (bytes.some((byte) => byte > 0xff)) {
return value;
}

try {
return new TextDecoder("utf-8", { fatal: true }).decode(Uint8Array.from(bytes));
Comment on lines +48 to +54

@gemini-code-assist gemini-code-assist Bot Jun 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation of tryDecodeUtf8Mojibake is inefficient because Array.from(value) creates an intermediate array of strings (one for each character/code point), and Uint8Array.from(bytes) creates another copy. Since this function is called for every path segment during pattern matching, we can optimize it by using a single pass with a Uint8Array and charCodeAt to avoid these allocations.

  const bytes = new Uint8Array(value.length);
  for (let i = 0; i < value.length; i++) {
    const code = value.charCodeAt(i);
    if (code > 0xff) {
      return value;
    }
    bytes[i] = code;
  }

  try {
    return new TextDecoder("utf-8", { fatal: true }).decode(bytes);

} catch (_e) {
return value;
}
}

/** @internal */
export function tryDecodeURIComponent(uri: string): string {

@CorieW CorieW Jun 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would the following work?

export function tryDecodeURIComponent(uri: string): string {
  try {
    return decodeURIComponent(uri);
  } catch (_e) {
    return tryDecodeUtf8Mojibake(decoded);
  }
}

@IzaakGough IzaakGough Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that looks cleaner

let decoded = uri;
try {
return decodeURIComponent(uri);
decoded = decodeURIComponent(uri);
} catch (_e) {
return uri;
// Leave malformed percent-encoding untouched and fall through to mojibake repair.
}
return tryDecodeUtf8Mojibake(decoded);
}

/** @hidden */
Expand Down
Loading