Skip to content

Set activeCodePage to UTF-8 in the Notepad3.exe.manifest file. - #5970

Closed
geniuszxy wants to merge 1 commit into
rizonesoft:masterfrom
geniuszxy:master
Closed

Set activeCodePage to UTF-8 in the Notepad3.exe.manifest file.#5970
geniuszxy wants to merge 1 commit into
rizonesoft:masterfrom
geniuszxy:master

Conversation

@geniuszxy

@geniuszxy geniuszxy commented Aug 12, 2026

Copy link
Copy Markdown

Fix #5963

Referring to zufuliu#168, sometimes StrTrimA() cannot handle some UTF-8 strings correctly, so I changed the code to directly check the last two chars.

I have found another way to solve this issue without touching code, referring to Use UTF-8 code pages in Windows apps, so that ANSI APIs can also handle UTF-8 strings correctly.

@hpwamr

hpwamr commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Hello @geniuszxy ,
Using Notepad3 (x64) 7.26.715.1 beta, I can't reproduce issue #5963. 🤔
I'm not sure if this PR is necessary. 🤔

@hpwamr
hpwamr requested a review from RaiKoHoff August 13, 2026 17:29
@hpwamr

hpwamr commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Again, I can't reproduce your issue... 🔍 🤔

I'm using Notepad3 (x64) 7.26.715.1 beta

You can find this beta version here: #1129
Could you please try with this Notepad3Portable_7.26.715.1_beta.paf.exe portable (PAF) version "out off the box" (keep the original settings)?

@geniuszxy

Copy link
Copy Markdown
Author

Yes, I can reproduce this issue using the PAF version you provided. This is a video record:

2026-08-14.23-05-03.mp4

@geniuszxy

Copy link
Copy Markdown
Author

I did some research on this, and as I mentioned, it seems to be related to the version of Windows.

I am using Windows 10 22H2, and the version of the kernelbase.dll is 10.0.19041.6280. I decompiled the CharNextA function using IDA, and the result is:

//KernelBase.10.0.19041.6280
LPSTR __stdcall CharNextA(LPCSTR lpsz)
{
  int v2; // eax
  LPSTR result; // rax

  if ( (unsigned __int8)IsMBToWCSExtPresent() )
    v2 = CheckDBCSEnabledExt();
  else
    v2 = NlsMbCodePageTag;
  if ( v2 && IsDBCSLeadByte(*lpsz) )
    ++lpsz;
  result = (LPSTR)(lpsz + 1);
  if ( !*lpsz )
    return (LPSTR)lpsz;
  return result;
}

I guess this IsDBCSLeadByte function here is used to determine if the previous character is >127, and if it is, the next character will also be included.

Therefore, strings like ホ\r\n (\xe3\x83\x9b\x0d\x0a) will be split into 3 characters: \xe3\x83 \x9b\x0d \x0a.

And I downloaded another kernelbase.dll with version 10.0.26100.8972 online, the decompilation result of its CharNextA function is:

//KernelBase.10.0.26100.8972
LPSTR __stdcall CharNextA(LPCSTR lpsz)
{
  int v2; // eax
  LPSTR result; // rax
  __int64 v4; // rcx

  if ( (unsigned __int8)IsMBToWCSExtPresent() )
    v2 = CheckDBCSEnabledExt();
  else
    v2 = NlsMbCodePageTag;
  if ( !v2 || gAnsiCodePage == 65001 )
  {
LABEL_4:
    result = (LPSTR)(lpsz + 1);
    if ( !*lpsz )
      return (LPSTR)lpsz;
    return result;
  }
  if ( gpACPHashN )
  {
    v4 = *(_QWORD *)(gpACPHashN + 48);
    if ( v4 && *(_WORD *)(v4 + 2i64 * *(unsigned __int8 *)lpsz) )
      ++lpsz;
    goto LABEL_4;
  }
  SetLastError_0(2u);
  result = (LPSTR)(lpsz + 1);
  if ( !*lpsz )
    return (LPSTR)lpsz;
  return result;
}

From the condition of gAnsiCodePage == 65001, it can be seen that this version only counts one character for UTF-8 code page.

@geniuszxy geniuszxy changed the title Use an old-school approach instead of StrTrimA() to ignore line-breaks Set activeCodePage to UTF-8 in the Notepad3.exe.manifest file. Aug 16, 2026
@geniuszxy

Copy link
Copy Markdown
Author

Hi @hpwamr, have you enabled "Beta: Use Unicode UTF-8 for worldwide language support" in the locale settings? This setting can solve my issue, but it can also corrupt other softwares.

@RaiKoHoff

Copy link
Copy Markdown
Collaborator

Thanks for putting this together, and I understand the appeal —  activeCodePage=UTF-8  is generally good hygiene for modern apps. After digging into how it interacts with Notepad3 specifically, though, I don't think we should merge this. Let me explain the reasoning.

What the setting actually does:  UTF-8  forces the process ANSI code page to UTF-8 (65001) on Windows 10 1903+.  GetACP() ,  CP_ACP , and  CP_THREAD_ACP  all resolve to UTF-8, and every  -A  Win32/CRT call starts treating  char*  as UTF-8. Notably, it does not touch the OEM code page ( GetOEMCP ).

Why the upside is near-zero here: Notepad3 is already a fully Unicode application. Paths and the UI use the wide ( -W ) APIs, and Scintilla is driven explicitly with  CP_UTF8  ( Encoding_SciCP ), not via  CP_ACP . So the usual benefit — making legacy  char*  code paths UTF-8-clean — buys us almost nothing, because we don't rely on the  -A  APIs for the core editing/IO work.

Why the downside is real and user-facing: The "ANSI (System)" encoding is a deliberate, user-visible feature, and its code page is derived from the process ACP at startup:

•  Encoding_InitDefaults()  →  CodePageFromCharSet(ANSI_CHARSET)  →  GetCPInfoEx(CP_THREAD_ACP, …)  (fallback  GetACP() ).
• That value is stored in  g_Encodings[CPI_ANSI_DEFAULT].uCodePage  and drives the actual  MultiByteToWideChar  /  WideCharToMultiByte  conversions on load and save.

With this manifest change, on any non-UTF-8 system (Windows-1252, Shift-JIS, GBK, etc.) the "ANSI" entry silently becomes UTF-8. Concretely:

  1. Opening a genuine legacy-codepage file as "ANSI" decodes it as UTF-8 → mojibake.
  2. Saving as "ANSI" writes UTF-8 (no BOM) instead of the system codepage → breaks round-tripping with other legacy tools.
  3. We end up internally inconsistent: "ANSI" now means UTF-8 while "OEM" still reports the real OEM codepage.
  4. Even the encoding-list label flips from e.g.  (CP-1252)  to  (CP-65001) , so the regression is visible right in the UI.

For a text editor, the "ANSI/System" code page has to reflect the machine's real legacy locale — that's the whole point of the option, and it's what users rely on for interop with legacy files and tools. Users who want UTF-8 already have first-class UTF-8 encodings in the menu, so we'd be taking on a real behavioral regression for no practical gain.

I'm happy to look at addressing that specific problem directly in a more targeted way. But as a blanket process-wide ACP override, I think we should pass on this one. Appreciate the contribution regardless!

@RaiKoHoff RaiKoHoff closed this Aug 28, 2026
RaiKoHoff added a commit to RaiKoHoff/Notepad3 that referenced this pull request Aug 28, 2026
…with code-page-independent trim/search 

All prior work (analysis of rizonesoft#5970, root-cause of rizonesoft#5963, code fixes, and verified x64 build) remains complete.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: "Sort Lines" inserts additional blank lines

3 participants