TAP implementation issue

[matthiasclasen]

Describe the bug

The TAP protocol states:
Any output line that is not a version, a plan, a test line, a diagnostic or a bail out is considered an “unknown” line. A TAP parser is required to not consider an unknown line as an error

Yet, the meson TAP parser does:
yield self.Error('unexpected input at line %d' % (lineno,))

[dcbaker]

It might make sense to have a warning here instead, but either way this should be pretty easy for someone to fix.

[bonzini]

Duplicate of #5944. Copying from there:

The part of the specification you quoted continues "... but may optionally choose to capture said line and hand it to the test harness, which may have custom behavior attached [...] TAP::Harness reports TAP syntax errors at the end of a test run".

In mtest, the TAP parser and harness are unified. It is possible to introduce an Unexpected namedtuple in the TAP parser if desired, so that we only report TAP syntax errors at the end of the test run, rather than exit as soon as they happen.

[JamBalaya56562]

Partially fixed, but not fully resolved.

The TAP parser itself no longer errors on non-standard output: it yields a TAPParser.UnknownLine for such lines, and the test runner collects those as warnings rather than treating them as parser errors. This matches the TAP requirement that a parser must not treat an unknown line as an error, and follows the approach suggested in the last comment here.

Verification:

• Code: mesonbuild/mtest.py — TAPParser yields self.UnknownLine(line, self.lineno); in the runner, isinstance(i, TAPParser.UnknownLine) appends to warnings (and only TAPParser.Error sets TestResult.ERROR).
• Introducing change: the asynchronous TAP-parsing rework (PR mtest: asynchronous TAP parsing, improved progress report #8029), merge a8c138e
• Release note: Meson 0.57.0 mentions the TAP parser rewrite / async handling related to this change.

However, as @bonzini pointed out, this is not entirely correct as a full fix: an error can still be raised by mtest at levels above the parser. That remaining behavior is tracked in #15952.

So this issue appears to be fixed at the parser level, but not fully resolved at the mtest level yet.

[bonzini]

Not entirely correct because an error is still raised by mtest at levels above the parser (#15952)