Repository navigation
Description
This seems similar to #298.
Original issue code: https://github.com/altendky/qtrio/tree/72863a1e769d9f942e2062904c9b6f3fb077d3b8
CI (GitHub Actions Windows): https://github.com/altendky/qtrio/pull/2/checks?check_run_id=947483753
> result = testdir.runpytest_subprocess(timeout=4 * timeout)
C:\hostedtoolcache\windows\Python\3.8.5\x64\lib\site-packages\qtrio\_tests\test_pytest.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <encodings.utf_8.IncrementalDecoder object at 0x0000022A4FF3E520>
input = b'============================= test session starts =============================\r\nplatform win32 -- Python 3.8.5, p... qtrio._exceptions.RunnerTim...\r\n============================== 1 failed in 0.74s ==============================\r\n'
final = True
def decode(self, input, final=False):
# decode input (taking the buffer into account)
data = self.buffer + input
> (result, consumed) = self._buffer_decode(data, self.errors, final)
E UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 9091: invalid start byte
c:\hostedtoolcache\windows\python\3.8.5\x64\lib\codecs.py:322: UnicodeDecodeError>>> b[9050:9150]
b"quest is\r\n *also* asynchronous \x96 you don't find out until later whether the\r\n op"The Trio source being included in the context resulting in the \x96: https://github.com/python-trio/trio/blob/15facd95379a496b234cc762c4ae69bf4fcdd0bf/trio/_core/_traps.py#L120
2020-08-05T02:10:08.7472418Z + python -m pip list
2020-08-05T02:10:09.3155684Z Package Version
2020-08-05T02:10:09.3156457Z ------------------- ---------
2020-08-05T02:10:09.3156870Z async-generator 1.10
2020-08-05T02:10:09.3157158Z atomicwrites 1.4.0
2020-08-05T02:10:09.3157384Z attrs 19.3.0
2020-08-05T02:10:09.3157830Z cffi 1.14.1
2020-08-05T02:10:09.3158411Z click 7.1.2
2020-08-05T02:10:09.3159025Z colorama 0.4.3
2020-08-05T02:10:09.3159984Z coverage 5.2.1
2020-08-05T02:10:09.3160282Z decorator 4.4.2
2020-08-05T02:10:09.3161003Z idna 2.10
2020-08-05T02:10:09.3161594Z iniconfig 1.0.1
2020-08-05T02:10:09.3162319Z more-itertools 8.4.0
2020-08-05T02:10:09.3162739Z outcome 1.0.1
2020-08-05T02:10:09.3163670Z packaging 20.4
2020-08-05T02:10:09.3164165Z pip 20.2.1
2020-08-05T02:10:09.3164901Z pluggy 0.13.1
2020-08-05T02:10:09.3165352Z py 1.9.0
2020-08-05T02:10:09.3166163Z pycparser 2.20
2020-08-05T02:10:09.3166517Z pyparsing 2.4.7
2020-08-05T02:10:09.3167314Z PySide2 5.15.0
2020-08-05T02:10:09.3167907Z pytest 6.0.1
2020-08-05T02:10:09.3168666Z pytest-cov 2.10.0
2020-08-05T02:10:09.3169237Z pytest-faulthandler 2.0.1
2020-08-05T02:10:09.3169938Z pytest-qt 3.3.0
2020-08-05T02:10:09.3170361Z QtPy 1.9.0
2020-08-05T02:10:09.3171850Z qtrio 0.1.0+dev
2020-08-05T02:10:09.3172256Z setuptools 49.2.1
2020-08-05T02:10:09.3172457Z shiboken2 5.15.0
2020-08-05T02:10:09.3172703Z six 1.15.0
2020-08-05T02:10:09.3172974Z sniffio 1.1.0
2020-08-05T02:10:09.3173542Z sortedcontainers 2.2.2
2020-08-05T02:10:09.3174126Z toml 0.10.1
2020-08-05T02:10:09.3175258Z trio 0.16.0
2020-08-05T02:10:09.3175712Z wheel 0.34.2
So what should be different? I think there are two pieces. First, pytest should probably attempt to control the subprocess' output (and input?) encoding. Second, pytest should not fail on decoding with only the context of the single byte value and location in the otherwise inaccessible output.
Hopefully I'll get to adding a test forcing the issue.
Activity
altendky commentedon Aug 5, 2020
Setting
PYTHONIOENCODING='utf-8'did address the issue in this case.Drop PYTHONIOENCODING=utf-8
bluetech commentedon Aug 6, 2020
The internal pytest frames are hidden in the traceback, so to understand the problem, can you find out exactly where the crash happens? I'm pretty sure it will be here:
pytest/src/_pytest/pytester.py
Line 1312 in aa9905d
but want to make sure before giving my assessment.
altendky commentedon Aug 6, 2020
@bluetech, looks like you predicted accurately.
https://github.com/altendky/qtrio/pull/138/checks?check_run_id=956028330
https://gist.github.com/altendky/7564bae4d644549a9f5f2d4b3cd21bc1#file-gistfile1-txt-L1056-L1139
full stack trace
altendky commentedon Aug 6, 2020
And for completeness... the pytest source in v6.0.1 that matches the reported line number.
https://github.com/pytest-dev/pytest/blob/6.0.1/src/_pytest/pytester.py#L1269
pytest/src/_pytest/pytester.py
Line 1269 in 92af2e2
bluetech commentedon Aug 7, 2020
I submitted #7627 to fix this, if you can test it that would be great. You can install the PR like this:
altendky commentedon Aug 10, 2020
@bluetech, thanks for the quick response and PR and sorry for the delayed response. Some family stuff came up. After a quick look, the results look fairly similar. I have not had a chance yet to go through your PR.
https://github.com/altendky/qtrio/pull/138/checks?check_run_id=964765802
https://gist.github.com/altendky/bda0c3f59f9ace5e7432c464b5fe4355#file-gistfile1-txt-L871
https://gist.github.com/altendky/bda0c3f59f9ace5e7432c464b5fe4355#file-gistfile1-txt-L1078-L1091
https://gist.github.com/altendky/bda0c3f59f9ace5e7432c464b5fe4355#file-gistfile1-txt-L1178-L1179
altendky commentedon Aug 10, 2020
with: https://github.com/pytest-dev/pytest/pull/7627/files#r467657738
looks ok: https://github.com/altendky/qtrio/pull/138/checks?check_run_id=964937422
antnewman commentedon Nov 11, 2026
Picking this up after #14963, because @kakakakacat's cp936 report there makes the
second half of the original question concrete and the first half is still open.
In the issue above, altendky put it as two pieces: "First, pytest should probably
attempt to control the subprocess' output (and input?) encoding. Second, pytest
should not fail on decoding with only the context of the single byte value and
location in the otherwise inaccessible output." #14963 does the second piece. The
first is still undecided, and the parenthetical "(and input?)" turns out to
matter more than it looks.
Where the second piece stands
#14963 decodes the captured output with
errors="backslashreplace", so the readno longer raises. kakakakacat confirmed on a cp936 host that it stops the crash
and that the text comes back as escapes, and drew the distinction the rest of this
rests on, between "doesn't crash" and "decodes correctly".
Worth noting that this half stays necessary on every Python version, including
3.15. A child in UTF-8 mode that prints a surrogate, which is what
surrogateescapehands back from undecodable input, still puts a raw byte on thewire:
print(chr(0xDCFF))underPYTHONUTF8=1writes0xff, and a strict UTF-8read of that still fails. Children writing through
sys.stdout.buffer, andnon-Python children, are unconstrained on any version.
What changed since 2020 on the first piece
PEP 686 is Final and UTF-8 mode is the default from Python 3.15, which shipped in
October 2026 and which pytest already classifies (
pyproject.toml:41) and tests(
windows-py315). On a cp1252 host,python -c "import pytest;help(pytest)"runwith
PYTHONUTF8=1, which is what 3.15 now does by itself, writes real UTF-8 anda strict read of it succeeds.
So the question is narrower than it was: should pytester force an encoding for
the 3.10 to 3.14 window, given that 3.15 fixes the Python-child case on its own?
That window is five of the six versions pytest supports and will last several
years, so "wait" is not free, but it is bounded.
The shapes, and what each costs
(a) Force
PYTHONIOENCODING=utf-8in the child. #7627's implementation, andwhat kakakakacat proposed. Measured costs: it sets the error handler on the
child's stdout and stdin from
surrogateescapetostrict.stderrkeepsbackslashreplaceand is unaffected. Through the public API,Pytester.run(sys.executable, script, stdin=b"h\xf6llo")returnsret=0todaywith the child reading
'h\xf6llo', andret=1with a child-sideUnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 1once thevariable is set. A child that writes a surrogate dies at that write: lines
already flushed survive, the offending line and everything after it are lost. (I
said in #14963's description that such a child "loses all its output"; that was
too strong, and the traceback survives because
stderrkeeps its handler.)(b) An
encoding=parameter defaulting toNone. asottile's shape in #7627:"I think what I'd do is grown an
encoding=...parameter and default it toNone, pass it along if non-None as PYTHONIOENCODING, and decode using eithergetpreferredencoding()(nothing passed) orencodingif passed?" Note what thedefault does: nothing is passed to the child, and the parent decodes with
getpreferredencoding(), which is shape (c) below. Forcing becomes opt-in.bluetech agreed to this shape, "I'm fine with this plan", and differed only on
the default.
(c) Do not force the child; decode in the parent with the host's locale
encoding. Raised and rejected by bluetech in #7627's own description: "I opted
against it in order to promote consistency across machines, and for consistency
with inprocess tests, even though it makes the tests less realistic on the host
machine." As a standalone change this is a replacement for #14963, not an
addition: it fails both of #14963's tests, in opposite directions. A child's
genuine UTF-8
b"h\xc3\xb6llo"comes back ashöllo, and the undecodableb"before \xf6 after"comes back asbefore ö afterrather than raising. Thesecond is the worse one, because the result is plausible text that no longer
carries the byte values: a visible failure traded for a silent one.
(d) Force UTF-8 mode in the child below 3.15, via
PYTHONUTF8=1, so pytestersubprocesses on 3.10 to 3.14 behave as they already do on 3.15. This was not
available in 2020; PEP 686 dates from 2022. It fixes the Python-child case and
keeps
surrogateescape, so it avoids (a)'s stdin crash and its surrogate death.Its costs are wider than I first thought, and I should set them out rather than
sell it:
locale.getpreferredencoding(False)in the child, not onlyopen()'s default. That reachessubprocess(text=True),tempfile,csv,configparser,Path.read_text()and anything third-party that asks. Theblast radius is much wider than stdio.
'h\xf6llo'today reads'h\udcf6llo', a lone surrogate in place of thecharacter.
PYTHONUTF8=1"would mask the class of bugPYTHONWARNDEFAULTENCODING=1was enabled to surface". I was wrong about thewarning, which still fires under UTF-8 mode, and right about the behaviour: an
unannotated
open()succeeds under pytester where it would mojibake inreality.
tox.ini:83turns that warning on for pytest's own suite, commented"If we don't enable this, neither can any of our downstream users!"
I had an argument that (d) bounds asottile's concern because the divergence
disappears above 3.15. It does not survive inspection, so I will not press it: on
3.15 the divergence is zero because the user's own application moved too, whereas
under (d) on 3.13 the child is in UTF-8 mode and the application is not. (d)
creates the test-versus-reality gap rather than forward-porting anything, and per
the first bullet it creates a wider one than (a) does. All it limits is how long
pytest would carry it.
The 2020 record, both sides
#7627 was closed unmerged in September 2020, "Going to close this for now, might
get back to it at some point". What stalled it was not implementation: altendky
verified it fixed the original case here, with a passing CI run.
asottile raised two objections. First, on the hunk itself: "this makes it
hard/impossible to test under other encoding scenarios." Then: "actually on
second thought I don't think we should do this at all (but perhaps provide
encodingas an option). it potentially masks a class of common encoding errorsby waving away the default behaviour". bluetech's answer was that isolating
pytester tests from host configuration is worth that cost and is consistent with
in-process tests, and that the status quo is wrong regardless: "the current
behavior is definitely wrong, it just assumes UTF-8".
They then agreed on (b)'s shape and split on its default, bluetech preferring
utf-8: "I'm fine with this plan, but would prefer the default to beutf-8,not
None. In this case, I prefer the tests to be consistent/isolated over beingrealistic." asottile's reply is the last word in that review thread: "my fear is
that someone will assume pytest is following the default behaviour, write a test
against the pytest behaviour, and then be surprised when it's actually broken in
reality". Nothing was reconciled, and I am not reading either comment as a
current position five years on.
The first objection is the one no option here answers.
popenassignskw["env"] = envunconditionally (src/_pytest/pytester.py:1380), so under (a)or (d) a user could not opt out of a forced encoding at all. #7627 changed that
first, and
testing/test_parseopt.py:349-351carries a comment wishing forexactly such a keyword. Any of these shapes probably wants that keyword
underneath it.
One more thing that bears on bluetech's "consistency with inprocess tests" leg:
#15059 would make captured
sys.stdout/sys.stderrfollow the encoding anderror handler of the streams they replace. In-process capture is UTF-8 today, so
the argument holds today; if #15059 merges, it inverts.
Measurements
Windows 10,
ACP1252, Python 3.13.6. Child stdout redirected to a file, asPytester.rundoes. The 3.15 column isPYTHONUTF8=1on 3.13 standing in forthe 3.15 default, which is a proxy, not the real thing.
getpreferredencodinghelp(pytest)surrogateescapesurrogateescape0x93at 300518PYTHONIOENCODING=utf-8strictstrictPYTHONUTF8=1surrogateescapesurrogateescapeWhich code pages are affected is not uniform, because
pydoc's plain pager sendsoutput through
escape_stdout(_pyrepl/pager.pyon 3.13+,pydoc._escape_stdoutbefore), encoding with
backslashreplaceagainstsys.stdout.encoding. A codepage that cannot represent
“turns it into an ASCII escape that decodescleanly. Forcing the child's encoding: cp1252, cp936, cp932 and cp1251 reproduce
the failure; cp437, cp850 and koi8-r do not.
On placement, if a variable is set at all.
Pytester.popen()reachesrun,runpython,runpython_c,runpytest_subprocess, the publicTestdir.popen(
legacypath.py:220), and non-Python children such as thepytester.run("bash", ...)intest_argcomplete, where it would be inert butinherited by the
python -m pytestgrandchild. That last part is from readingtest_parseopt.pyrather than from a run, since the test skips on my host forwant of bash. It does not reach
spawn/spawn_pytest, which call pexpectdirectly. #7627 set it in the three Python entry points instead.
Two notes on the test suite. No test in
testing/asserts on non-ASCIIsubprocess output: an AST scan of every test calling
run,popen,runpython,runpython_c,runpytest_subprocessorspawn*finds only #14963's own twotests, plus ASCII control characters. That is not the same as nothing changing,
and
test_pydocis the counter-example: it asserts only"MarkGenerator" in s,yet it fails on a cp1252 host because
src/_pytest/terminal.py:126supplies thebytes, so forcing UTF-8 would move it from failing to passing on such a host. The
only
stdin=byte payloads intesting/are three occurrences ofb"input\n2ndline", pure ASCII, so (a)'s stdin cost would not show up in CI; itwould fall on downstream users of the documented parameter.
Relatedly, whichever shape is chosen needs a test that sets the child's encoding
explicitly rather than relying on the host. CI is green on
mainwhiletest_pydocfails on a default cp1252 box, so the CI runners are evidently notreproducing the condition. I have not established why and make no claim about it.
Limits
One cp1252 host on 3.13.6. I have not tested a cp936 host, and I have no 3.14 or
3.15 here, so every 3.15 statement above is the
PYTHONUTF8=1proxy.kakakakacat reported a cp936 reproduction on 3.14 and has offered to test
variants there, which is the thing #7627 most lacked: its author noted in the
description that it was untested, "because getting a non UTF-8 encoding (or a
subset of it) on a modern Linux is almost impossible".
I do not have a recommendation between (a), (b) and (d), and I am wary of (c)
unless it replaces #14963 deliberately. The cost I would weigh heaviest is that
none of (a), (b) or (d) is opt-out-able as
popenstands. I am happy toimplement whichever shape and default you settle on, or to leave it to PEP 686 if
the answer is that the remaining window is not worth the change.