fix(tests): skip translate eval e2e when the harness is not shipped - #2234
Open
martyanovandrey wants to merge 1 commit into
Open
fix(tests): skip translate eval e2e when the harness is not shipped#2234martyanovandrey wants to merge 1 commit into
martyanovandrey wants to merge 1 commit into
Conversation
The npm package ships tests/ (so downstream consumers can run the integration suite against their own binary) but not scripts/ and src/, which the eval harness runner needs. In that layout the spec spawned a missing script and failed with empty stdout. Skip it when scripts/translate-eval.mjs is absent; in-repo runs are unaffected.
martyanovandrey
requested review from
Yurok868 and
stenin-nikita
and removed request for
a team
August 28, 2026 12:08
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The npm package ships
tests/(so downstream consumers can run the integration suite against their own binary viaDIPLODOC_BINARY_PATH), but notscripts/andsrc/, which the eval harness runner (scripts/translate-eval.mjs, added in #2225) needs: the runner bundlessrc/commands/translate/eval/cli.tson the fly with esbuild.So any consumer running the packaged integration tests got
tests/e2e/translate-eval.spec.tswithout its runner: the spec spawned a missing script, the child died with MODULE_NOT_FOUND on stderr, and the assertion failed withexpected '' to contain 'Verdict: PASS'. This is exactly what broke the internal cli release flow on 5.57.x.Fix
Skip the spec when
scripts/translate-eval.mjsis absent next to the tests (i.e. when running from the packaged layout). In-repo runs are unaffected and still execute the full e2e.Verified both ways:
npm pack+ running the spec from the extracted tarball reports 1 skipped; the in-repo run passes.Making the harness runnable from the package (prebuilding the eval cli into
build/) would let downstream flows gate on translation quality too - left as a possible follow-up.