Repository navigation
fix(auto): skip blank lines in wav.scp and jsonl file lists - #3758
Conversation
A blank or whitespace-only line in a file list crashes prepare_data_iterator: the id/data branch indexes lines[0] on an empty split (IndexError), and the .jsonl branch feeds an empty string to json.loads (JSONDecodeError). Skip blank lines so a trailing or separator line no longer aborts the whole batch.
LauraGPT
left a comment
There was a problem hiding this comment.
Reviewed exact head b09cbbd. On current-main source, both added regressions first failed with the reported IndexError / JSONDecodeError; the merge candidate passes the file-list and byte-loading suites (84 tests and 2 subtests). Six independent boundaries also pass: blank-only lists, Unicode whitespace, CRLF/order/text preservation, malformed nonblank JSON still raising, and an unsupported extension remaining a direct input. The three approved head workflows are successful. NumPy 1.26.4 and 2.4.0 artifacts each show 122 tests with zero failures/errors/skips and this exact source SHA. The cumulative #3754/#3758/#3761 candidate passes 128 tests and 10 subtests. No blocking findings; no model weights or inference needed for this parser fix.
Summary
prepare_data_iterator()reads every line of a file list without checking whether it is empty, so a blank or whitespace-only line inwav.scp/text.txtraisesIndexError(indexinglines[0]on an emptysplit()result), and a blank line in a.jsonllist raisesjson.decoder.JSONDecodeErrorfromjson.loads(""). Both come from the same missing guard, so skipping blank lines at the top of the loop fixes the two branches with a single two-line change.Fixes #3757
Reproduction (before):
Type of change
Validation
New
test_scp_filelist_skips_blank_linesandtest_jsonl_filelist_skips_blank_linesintests/test_pcm_input_format.py: 2 failed onmainat66d7a4c2(IndexErrorandJSONDecodeError), and the whole file passes with the fix —python -m pytest -q tests/test_pcm_input_format.py→41 passed.python -m compileall funasr examples tests→ exit 0.tests/test_pcm_input_format.pyis already in the "Validate NumPy compatibility" workflow run list andfunasr/auto/auto_model.pyis in itspathsfilter, so the new tests are exercised by that job onmain.python -m compileall funasr examples testsDocs or links checked
Runtime/deployment command tested
User impact
Anyone running batch inference through a
wav.scp,text.txtorfile.jsonllist — the documented entry point formodel.generate(input=<list>)— can keep hand-edited or concatenated lists with separator lines instead of the whole batch aborting on one empty line.Notes for reviewers