Skip to content

testdir.runpytest_subprocess() fails with UnicodeDecodeError in Windows #7623

Open

Description

@altendky

This seems similar to #298.

Original issue code: https://github.com/altendky/qtrio/tree/72863a1e769d9f942e2062904c9b6f3fb077d3b8
CI (GitHub Actions Windows): https://github.com/altendky/qtrio/pull/2/checks?check_run_id=947483753

>       result = testdir.runpytest_subprocess(timeout=4 * timeout)

C:\hostedtoolcache\windows\Python\3.8.5\x64\lib\site-packages\qtrio\_tests\test_pytest.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <encodings.utf_8.IncrementalDecoder object at 0x0000022A4FF3E520>
input = b'============================= test session starts =============================\r\nplatform win32 -- Python 3.8.5, p... qtrio._exceptions.RunnerTim...\r\n============================== 1 failed in 0.74s ==============================\r\n'
final = True

    def decode(self, input, final=False):
        # decode input (taking the buffer into account)
        data = self.buffer + input
>       (result, consumed) = self._buffer_decode(data, self.errors, final)
E       UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 9091: invalid start byte

c:\hostedtoolcache\windows\python\3.8.5\x64\lib\codecs.py:322: UnicodeDecodeError
>>> b[9050:9150]
b"quest is\r\n           *also* asynchronous \x96 you don't find out until later whether the\r\n           op"

The Trio source being included in the context resulting in the \x96: https://github.com/python-trio/trio/blob/15facd95379a496b234cc762c4ae69bf4fcdd0bf/trio/_core/_traps.py#L120

2020-08-05T02:10:08.7472418Z + python -m pip list
2020-08-05T02:10:09.3155684Z Package             Version
2020-08-05T02:10:09.3156457Z ------------------- ---------
2020-08-05T02:10:09.3156870Z async-generator     1.10
2020-08-05T02:10:09.3157158Z atomicwrites        1.4.0
2020-08-05T02:10:09.3157384Z attrs               19.3.0
2020-08-05T02:10:09.3157830Z cffi                1.14.1
2020-08-05T02:10:09.3158411Z click               7.1.2
2020-08-05T02:10:09.3159025Z colorama            0.4.3
2020-08-05T02:10:09.3159984Z coverage            5.2.1
2020-08-05T02:10:09.3160282Z decorator           4.4.2
2020-08-05T02:10:09.3161003Z idna                2.10
2020-08-05T02:10:09.3161594Z iniconfig           1.0.1
2020-08-05T02:10:09.3162319Z more-itertools      8.4.0
2020-08-05T02:10:09.3162739Z outcome             1.0.1
2020-08-05T02:10:09.3163670Z packaging           20.4
2020-08-05T02:10:09.3164165Z pip                 20.2.1
2020-08-05T02:10:09.3164901Z pluggy              0.13.1
2020-08-05T02:10:09.3165352Z py                  1.9.0
2020-08-05T02:10:09.3166163Z pycparser           2.20
2020-08-05T02:10:09.3166517Z pyparsing           2.4.7
2020-08-05T02:10:09.3167314Z PySide2             5.15.0
2020-08-05T02:10:09.3167907Z pytest              6.0.1
2020-08-05T02:10:09.3168666Z pytest-cov          2.10.0
2020-08-05T02:10:09.3169237Z pytest-faulthandler 2.0.1
2020-08-05T02:10:09.3169938Z pytest-qt           3.3.0
2020-08-05T02:10:09.3170361Z QtPy                1.9.0
2020-08-05T02:10:09.3171850Z qtrio               0.1.0+dev
2020-08-05T02:10:09.3172256Z setuptools          49.2.1
2020-08-05T02:10:09.3172457Z shiboken2           5.15.0
2020-08-05T02:10:09.3172703Z six                 1.15.0
2020-08-05T02:10:09.3172974Z sniffio             1.1.0
2020-08-05T02:10:09.3173542Z sortedcontainers    2.2.2
2020-08-05T02:10:09.3174126Z toml                0.10.1
2020-08-05T02:10:09.3175258Z trio                0.16.0
2020-08-05T02:10:09.3175712Z wheel               0.34.2

So what should be different? I think there are two pieces. First, pytest should probably attempt to control the subprocess' output (and input?) encoding. Second, pytest should not fail on decoding with only the context of the single byte value and location in the otherwise inaccessible output.

Hopefully I'll get to adding a test forcing the issue.

Activity

  1. altendky commented on Aug 5, 2020

    @altendky
    MemberAuthor

    Setting PYTHONIOENCODING='utf-8' did address the issue in this case.

  2. added a commit that references this issue on Aug 5, 2020
  3. bluetech commented on Aug 6, 2020

    @bluetech
    Member

    The internal pytest frames are hidden in the traceback, so to understand the problem, can you find out exactly where the crash happens? I'm pretty sure it will be here:

    out = f1.read().splitlines()

    but want to make sure before giving my assessment.

  4. altendky commented on Aug 6, 2020

    @altendky
    MemberAuthor

    @bluetech, looks like you predicted accurately.

    2020-08-06T23:02:12.6857589Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\pytester.py", line 1269, in run
    2020-08-06T23:02:12.6857726Z     out = f1.read().splitlines()

    https://github.com/altendky/qtrio/pull/138/checks?check_run_id=956028330
    https://gist.github.com/altendky/7564bae4d644549a9f5f2d4b3cd21bc1#file-gistfile1-txt-L1056-L1139

    full stack trace
    2020-08-06T23:02:12.6843406Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\runpy.py", line 194, in _run_module_as_main
    2020-08-06T23:02:12.6843678Z     return _run_code(code, main_globals, None,
    2020-08-06T23:02:12.6843875Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\runpy.py", line 87, in _run_code
    2020-08-06T23:02:12.6844003Z     exec(code, run_globals)
    2020-08-06T23:02:12.6844161Z   File "C:\hostedtoolcache\windows\Python\3.8.5\x64\Scripts\pytest.exe\__main__.py", line 7, in <module>
    2020-08-06T23:02:12.6844289Z     sys.exit(console_main())
    2020-08-06T23:02:12.6844565Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\config\__init__.py", line 180, in console_main
    2020-08-06T23:02:12.6844712Z     code = main()
    2020-08-06T23:02:12.6844869Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\config\__init__.py", line 157, in main
    2020-08-06T23:02:12.6845002Z     ret = config.hook.pytest_cmdline_main(
    2020-08-06T23:02:12.6845158Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    2020-08-06T23:02:12.6845360Z     return self._hookexec(self, self.get_hookimpls(), kwargs)
    2020-08-06T23:02:12.6845525Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    2020-08-06T23:02:12.6845913Z     return self._inner_hookexec(hook, methods, kwargs)
    2020-08-06T23:02:12.6846083Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    2020-08-06T23:02:12.6846225Z     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
    2020-08-06T23:02:12.6846405Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    2020-08-06T23:02:12.6846516Z     res = hook_impl.function(*args)
    2020-08-06T23:02:12.6846695Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\main.py", line 289, in pytest_cmdline_main
    2020-08-06T23:02:12.6846835Z     return wrap_session(config, _main)
    2020-08-06T23:02:12.6847001Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\main.py", line 240, in wrap_session
    2020-08-06T23:02:12.6847139Z     session.exitstatus = doit(config, session) or 0
    2020-08-06T23:02:12.6847305Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\main.py", line 296, in _main
    2020-08-06T23:02:12.6847442Z     config.hook.pytest_runtestloop(session=session)
    2020-08-06T23:02:12.6847607Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    2020-08-06T23:02:12.6847751Z     return self._hookexec(self, self.get_hookimpls(), kwargs)
    2020-08-06T23:02:12.6847917Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    2020-08-06T23:02:12.6848054Z     return self._inner_hookexec(hook, methods, kwargs)
    2020-08-06T23:02:12.6848216Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    2020-08-06T23:02:12.6848363Z     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
    2020-08-06T23:02:12.6848536Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    2020-08-06T23:02:12.6848670Z     res = hook_impl.function(*args)
    2020-08-06T23:02:12.6848843Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\main.py", line 321, in pytest_runtestloop
    2020-08-06T23:02:12.6848985Z     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
    2020-08-06T23:02:12.6849157Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    2020-08-06T23:02:12.6849299Z     return self._hookexec(self, self.get_hookimpls(), kwargs)
    2020-08-06T23:02:12.6849464Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    2020-08-06T23:02:12.6849605Z     return self._inner_hookexec(hook, methods, kwargs)
    2020-08-06T23:02:12.6849768Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    2020-08-06T23:02:12.6849908Z     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
    2020-08-06T23:02:12.6850081Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    2020-08-06T23:02:12.6850192Z     res = hook_impl.function(*args)
    2020-08-06T23:02:12.6850438Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\runner.py", line 100, in pytest_runtest_protocol
    2020-08-06T23:02:12.6850599Z     runtestprotocol(item, nextitem=nextitem)
    2020-08-06T23:02:12.6850774Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\runner.py", line 117, in runtestprotocol
    2020-08-06T23:02:12.6850912Z     reports.append(call_and_report(item, "call", log))
    2020-08-06T23:02:12.6851154Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\runner.py", line 207, in call_and_report
    2020-08-06T23:02:12.6851295Z     call = call_runtest_hook(item, when, **kwds)
    2020-08-06T23:02:12.6851472Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\runner.py", line 246, in call_runtest_hook
    2020-08-06T23:02:12.6851607Z     return CallInfo.from_call(
    2020-08-06T23:02:12.6851770Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\runner.py", line 294, in from_call
    2020-08-06T23:02:12.6851908Z     result = func()  # type: Optional[_T]
    2020-08-06T23:02:12.6852074Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\runner.py", line 247, in <lambda>
    2020-08-06T23:02:12.6852216Z     lambda: ihook(item=item, **kwds), when=when, reraise=reraise
    2020-08-06T23:02:12.6852386Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    2020-08-06T23:02:12.6852530Z     return self._hookexec(self, self.get_hookimpls(), kwargs)
    2020-08-06T23:02:12.6852697Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    2020-08-06T23:02:12.6852834Z     return self._inner_hookexec(hook, methods, kwargs)
    2020-08-06T23:02:12.6853232Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    2020-08-06T23:02:12.6853369Z     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
    2020-08-06T23:02:12.6853832Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    2020-08-06T23:02:12.6853959Z     res = hook_impl.function(*args)
    2020-08-06T23:02:12.6854123Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\runner.py", line 153, in pytest_runtest_call
    2020-08-06T23:02:12.6854249Z     item.runtest()
    2020-08-06T23:02:12.6854403Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\python.py", line 1567, in runtest
    2020-08-06T23:02:12.6854532Z     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
    2020-08-06T23:02:12.6854667Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    2020-08-06T23:02:12.6854794Z     return self._hookexec(self, self.get_hookimpls(), kwargs)
    2020-08-06T23:02:12.6854949Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    2020-08-06T23:02:12.6855079Z     return self._inner_hookexec(hook, methods, kwargs)
    2020-08-06T23:02:12.6855234Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    2020-08-06T23:02:12.6855369Z     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
    2020-08-06T23:02:12.6855529Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    2020-08-06T23:02:12.6855660Z     res = hook_impl.function(*args)
    2020-08-06T23:02:12.6855825Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\python.py", line 180, in pytest_pyfunc_call
    2020-08-06T23:02:12.6855952Z     result = testfunction(**testargs)
    2020-08-06T23:02:12.6856126Z   File "C:\hostedtoolcache\windows\Python\3.8.5\x64\lib\site-packages\qtrio\_tests\test_pytest.py", line 63, in test_overrunning_test_times_out
    2020-08-06T23:02:12.6856518Z     result = testdir.runpytest_subprocess(timeout=timeout)
    2020-08-06T23:02:12.6857022Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\pytester.py", line 1326, in runpytest_subprocess
    2020-08-06T23:02:12.6857424Z     return self.run(*args, timeout=timeout)
    2020-08-06T23:02:12.6857589Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\pytester.py", line 1269, in run
    2020-08-06T23:02:12.6857726Z     out = f1.read().splitlines()
    2020-08-06T23:02:12.6857893Z   File "C:\hostedtoolcache\windows\Python\3.8.5\x64\lib\site-packages\qtrio\_tests\test_pytest.py", line 53, in decode
    2020-08-06T23:02:12.6858101Z     traceback.print_stack()
  5. altendky commented on Aug 6, 2020

    @altendky
    MemberAuthor

    And for completeness... the pytest source in v6.0.1 that matches the reported line number.

    https://github.com/pytest-dev/pytest/blob/6.0.1/src/_pytest/pytester.py#L1269

    out = f1.read().splitlines()

  6. added
    plugin: pytesterrelated to the pytester builtin plugin
    type: bugproblem that needs to be addressed
    on Aug 7, 2020
  7. bluetech commented on Aug 7, 2020

    @bluetech
    Member

    I submitted #7627 to fix this, if you can test it that would be great. You can install the PR like this:

    pip install git+https://github.com/pytest-dev/pytest.git@refs/pull/7627/head
    
  8. altendky commented on Aug 10, 2020

    @altendky
    MemberAuthor

    @bluetech, thanks for the quick response and PR and sorry for the delayed response. Some family stuff came up. After a quick look, the results look fairly similar. I have not had a chance yet to go through your PR.

    https://github.com/altendky/qtrio/pull/138/checks?check_run_id=964765802

    https://gist.github.com/altendky/bda0c3f59f9ace5e7432c464b5fe4355#file-gistfile1-txt-L871

    2020-08-10T00:22:25.8124237Z pytest              6.0.1.dev82+g7857311fc
    

    https://gist.github.com/altendky/bda0c3f59f9ace5e7432c464b5fe4355#file-gistfile1-txt-L1078-L1091

    2020-08-10T00:23:15.3429598Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    2020-08-10T00:23:15.3429682Z 
    2020-08-10T00:23:15.3429800Z self = <encodings.utf_8.IncrementalDecoder object at 0x000002532EFA3940>
    2020-08-10T00:23:15.3429947Z input = b'============================= test session starts =============================\r\nplatform win32 -- Python 3.8.5, p... qtrio._exceptions.RunnerTim...\r\n============================== 1 failed in 0.79s ==============================\r\n'
    2020-08-10T00:23:15.3430333Z final = True
    2020-08-10T00:23:15.3430388Z 
    2020-08-10T00:23:15.3430505Z     def decode(self, input, final=False):
    2020-08-10T00:23:15.3430632Z         # decode input (taking the buffer into account)
    2020-08-10T00:23:15.3430751Z         data = self.buffer + input
    2020-08-10T00:23:15.3430862Z         try:
    2020-08-10T00:23:15.3430958Z >           (result, consumed) = self._buffer_decode(data, self.errors, final)
    2020-08-10T00:23:15.3431108Z E           UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 9096: invalid start byte
    2020-08-10T00:23:15.3431209Z 
    2020-08-10T00:23:15.3431349Z C:\hostedtoolcache\windows\Python\3.8.5\x64\lib\site-packages\qtrio\_tests\test_pytest.py:49: UnicodeDecodeError
    

    https://gist.github.com/altendky/bda0c3f59f9ace5e7432c464b5fe4355#file-gistfile1-txt-L1178-L1179

    2020-08-10T00:23:15.3478347Z   File "c:\hostedtoolcache\windows\python\3.8.5\x64\lib\site-packages\_pytest\pytester.py", line 1320, in run
    2020-08-10T00:23:15.3478492Z     out = stdout_text.read().splitlines()
    
  9. antnewman commented on Nov 11, 2026

    @antnewman
    Contributor

    Picking this up after #14963, because @kakakakacat's cp936 report there makes the
    second half of the original question concrete and the first half is still open.

    In the issue above, altendky put it as two pieces: "First, pytest should probably
    attempt to control the subprocess' output (and input?) encoding. Second, pytest
    should not fail on decoding with only the context of the single byte value and
    location in the otherwise inaccessible output." #14963 does the second piece. The
    first is still undecided, and the parenthetical "(and input?)" turns out to
    matter more than it looks.

    Where the second piece stands

    #14963 decodes the captured output with errors="backslashreplace", so the read
    no longer raises. kakakakacat confirmed on a cp936 host that it stops the crash
    and that the text comes back as escapes, and drew the distinction the rest of this
    rests on, between "doesn't crash" and "decodes correctly".

    Worth noting that this half stays necessary on every Python version, including
    3.15. A child in UTF-8 mode that prints a surrogate, which is what
    surrogateescape hands back from undecodable input, still puts a raw byte on the
    wire: print(chr(0xDCFF)) under PYTHONUTF8=1 writes 0xff, and a strict UTF-8
    read of that still fails. Children writing through sys.stdout.buffer, and
    non-Python children, are unconstrained on any version.

    What changed since 2020 on the first piece

    PEP 686 is Final and UTF-8 mode is the default from Python 3.15, which shipped in
    October 2026 and which pytest already classifies (pyproject.toml:41) and tests
    (windows-py315). On a cp1252 host, python -c "import pytest;help(pytest)" run
    with PYTHONUTF8=1, which is what 3.15 now does by itself, writes real UTF-8 and
    a strict read of it succeeds.

    So the question is narrower than it was: should pytester force an encoding for
    the 3.10 to 3.14 window, given that 3.15 fixes the Python-child case on its own?

    That window is five of the six versions pytest supports and will last several
    years, so "wait" is not free, but it is bounded.

    The shapes, and what each costs

    (a) Force PYTHONIOENCODING=utf-8 in the child. #7627's implementation, and
    what kakakakacat proposed. Measured costs: it sets the error handler on the
    child's stdout and stdin from surrogateescape to strict. stderr keeps
    backslashreplace and is unaffected. Through the public API,
    Pytester.run(sys.executable, script, stdin=b"h\xf6llo") returns ret=0 today
    with the child reading 'h\xf6llo', and ret=1 with a child-side
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 1 once the
    variable is set. A child that writes a surrogate dies at that write: lines
    already flushed survive, the offending line and everything after it are lost. (I
    said in #14963's description that such a child "loses all its output"; that was
    too strong, and the traceback survives because stderr keeps its handler.)

    (b) An encoding= parameter defaulting to None. asottile's shape in #7627:
    "I think what I'd do is grown an encoding=... parameter and default it to
    None, pass it along if non-None as PYTHONIOENCODING, and decode using either
    getpreferredencoding() (nothing passed) or encoding if passed?" Note what the
    default does: nothing is passed to the child, and the parent decodes with
    getpreferredencoding(), which is shape (c) below. Forcing becomes opt-in.
    bluetech agreed to this shape, "I'm fine with this plan", and differed only on
    the default.

    (c) Do not force the child; decode in the parent with the host's locale
    encoding.
    Raised and rejected by bluetech in #7627's own description: "I opted
    against it in order to promote consistency across machines, and for consistency
    with inprocess tests, even though it makes the tests less realistic on the host
    machine." As a standalone change this is a replacement for #14963, not an
    addition
    : it fails both of #14963's tests, in opposite directions. A child's
    genuine UTF-8 b"h\xc3\xb6llo" comes back as höllo, and the undecodable
    b"before \xf6 after" comes back as before ö after rather than raising. The
    second is the worse one, because the result is plausible text that no longer
    carries the byte values: a visible failure traded for a silent one.

    (d) Force UTF-8 mode in the child below 3.15, via PYTHONUTF8=1, so pytester
    subprocesses on 3.10 to 3.14 behave as they already do on 3.15. This was not
    available in 2020; PEP 686 dates from 2022. It fixes the Python-child case and
    keeps surrogateescape, so it avoids (a)'s stdin crash and its surrogate death.
    Its costs are wider than I first thought, and I should set them out rather than
    sell it:

    • UTF-8 mode flips locale.getpreferredencoding(False) in the child, not only
      open()'s default. That reaches subprocess(text=True), tempfile, csv,
      configparser, Path.read_text() and anything third-party that asks. The
      blast radius is much wider than stdio.
    • It changes stdin too, just less violently than (a): the child that reads
      'h\xf6llo' today reads 'h\udcf6llo', a lone surrogate in place of the
      character.
    • I said in pytester: do not fail on undecodable subprocess output #14963's description that PYTHONUTF8=1 "would mask the class of bug
      PYTHONWARNDEFAULTENCODING=1 was enabled to surface". I was wrong about the
      warning, which still fires under UTF-8 mode, and right about the behaviour: an
      unannotated open() succeeds under pytester where it would mojibake in
      reality. tox.ini:83 turns that warning on for pytest's own suite, commented
      "If we don't enable this, neither can any of our downstream users!"

    I had an argument that (d) bounds asottile's concern because the divergence
    disappears above 3.15. It does not survive inspection, so I will not press it: on
    3.15 the divergence is zero because the user's own application moved too, whereas
    under (d) on 3.13 the child is in UTF-8 mode and the application is not. (d)
    creates the test-versus-reality gap rather than forward-porting anything, and per
    the first bullet it creates a wider one than (a) does. All it limits is how long
    pytest would carry it.

    The 2020 record, both sides

    #7627 was closed unmerged in September 2020, "Going to close this for now, might
    get back to it at some point". What stalled it was not implementation: altendky
    verified it fixed the original case here, with a passing CI run.

    asottile raised two objections. First, on the hunk itself: "this makes it
    hard/impossible to test under other encoding scenarios." Then: "actually on
    second thought I don't think we should do this at all (but perhaps provide
    encoding as an option). it potentially masks a class of common encoding errors
    by waving away the default behaviour". bluetech's answer was that isolating
    pytester tests from host configuration is worth that cost and is consistent with
    in-process tests, and that the status quo is wrong regardless: "the current
    behavior is definitely wrong, it just assumes UTF-8".

    They then agreed on (b)'s shape and split on its default, bluetech preferring
    utf-8: "I'm fine with this plan, but would prefer the default to be utf-8,
    not None. In this case, I prefer the tests to be consistent/isolated over being
    realistic." asottile's reply is the last word in that review thread: "my fear is
    that someone will assume pytest is following the default behaviour, write a test
    against the pytest behaviour, and then be surprised when it's actually broken in
    reality". Nothing was reconciled, and I am not reading either comment as a
    current position five years on.

    The first objection is the one no option here answers. popen assigns
    kw["env"] = env unconditionally (src/_pytest/pytester.py:1380), so under (a)
    or (d) a user could not opt out of a forced encoding at all. #7627 changed that
    first, and testing/test_parseopt.py:349-351 carries a comment wishing for
    exactly such a keyword. Any of these shapes probably wants that keyword
    underneath it.

    One more thing that bears on bluetech's "consistency with inprocess tests" leg:
    #15059 would make captured sys.stdout/sys.stderr follow the encoding and
    error handler of the streams they replace. In-process capture is UTF-8 today, so
    the argument holds today; if #15059 merges, it inverts.

    Measurements

    Windows 10, ACP 1252, Python 3.13.6. Child stdout redirected to a file, as
    Pytester.run does. The 3.15 column is PYTHONUTF8=1 on 3.13 standing in for
    the 3.15 default, which is a proxy, not the real thing.

    lever stdout stdin getpreferredencoding strict read of help(pytest)
    none cp1252 / surrogateescape cp1252 / surrogateescape cp1252 fails, 0x93 at 300518
    PYTHONIOENCODING=utf-8 utf-8 / strict utf-8 / strict cp1252 succeeds
    PYTHONUTF8=1 utf-8 / surrogateescape utf-8 / surrogateescape utf-8 succeeds

    Which code pages are affected is not uniform, because pydoc's plain pager sends
    output through escape_stdout (_pyrepl/pager.py on 3.13+, pydoc._escape_stdout
    before), encoding with backslashreplace against sys.stdout.encoding. A code
    page that cannot represent “ turns it into an ASCII escape that decodes
    cleanly. Forcing the child's encoding: cp1252, cp936, cp932 and cp1251 reproduce
    the failure; cp437, cp850 and koi8-r do not.

    On placement, if a variable is set at all. Pytester.popen() reaches run,
    runpython, runpython_c, runpytest_subprocess, the public Testdir.popen
    (legacypath.py:220), and non-Python children such as the
    pytester.run("bash", ...) in test_argcomplete, where it would be inert but
    inherited by the python -m pytest grandchild. That last part is from reading
    test_parseopt.py rather than from a run, since the test skips on my host for
    want of bash. It does not reach spawn/spawn_pytest, which call pexpect
    directly. #7627 set it in the three Python entry points instead.

    Two notes on the test suite. No test in testing/ asserts on non-ASCII
    subprocess output: an AST scan of every test calling run, popen, runpython,
    runpython_c, runpytest_subprocess or spawn* finds only #14963's own two
    tests, plus ASCII control characters. That is not the same as nothing changing,
    and test_pydoc is the counter-example: it asserts only "MarkGenerator" in s,
    yet it fails on a cp1252 host because src/_pytest/terminal.py:126 supplies the
    bytes, so forcing UTF-8 would move it from failing to passing on such a host. The
    only stdin= byte payloads in testing/ are three occurrences of
    b"input\n2ndline", pure ASCII, so (a)'s stdin cost would not show up in CI; it
    would fall on downstream users of the documented parameter.

    Relatedly, whichever shape is chosen needs a test that sets the child's encoding
    explicitly rather than relying on the host. CI is green on main while
    test_pydoc fails on a default cp1252 box, so the CI runners are evidently not
    reproducing the condition. I have not established why and make no claim about it.

    Limits

    One cp1252 host on 3.13.6. I have not tested a cp936 host, and I have no 3.14 or
    3.15 here, so every 3.15 statement above is the PYTHONUTF8=1 proxy.
    kakakakacat reported a cp936 reproduction on 3.14 and has offered to test
    variants there, which is the thing #7627 most lacked: its author noted in the
    description that it was untested, "because getting a non UTF-8 encoding (or a
    subset of it) on a modern Linux is almost impossible".

    I do not have a recommendation between (a), (b) and (d), and I am wary of (c)
    unless it replaces #14963 deliberately. The cost I would weigh heaviest is that
    none of (a), (b) or (d) is opt-out-able as popen stands. I am happy to
    implement whichever shape and default you settle on, or to leave it to PEP 686 if
    the answer is that the remaining window is not worth the change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    plugin: pytesterrelated to the pytester builtin plugintype: bugproblem that needs to be addressed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions