Skip to content

fix(auto): skip blank lines in wav.scp and jsonl file lists - #3758

Merged
LauraGPT merged 1 commit into
modelscope:mainfrom
Lesereingrape:contrib/filelist-blank-line
Oct 8, 2026
Merged

LauraGPT merged 1 commit into
modelscope:mainfrom
Lesereingrape:contrib/filelist-blank-line

Conversation

@Lesereingrape

Copy link
Copy Markdown
Contributor

Summary

prepare_data_iterator() reads every line of a file list without checking whether it is empty, so a blank or whitespace-only line in wav.scp / text.txt raises IndexError (indexing lines[0] on an empty split() result), and a blank line in a .jsonl list raises json.decoder.JSONDecodeError from json.loads(""). Both come from the same missing guard, so skipping blank lines at the top of the loop fixes the two branches with a single two-line change.

Fixes #3757

Reproduction (before):

from funasr.auto.auto_model import prepare_data_iterator
open("wav.scp", "w", encoding="utf-8").write("utt1 /a.wav\n\nutt2 /b.wav\n")
prepare_data_iterator("wav.scp")   # IndexError: list index out of range
# after: (['utt1', 'utt2'], ['/a.wav', '/b.wav'])

Type of change

  • Bug fix
  • Documentation
  • Example or demo
  • Runtime or deployment
  • Benchmark or evaluation
  • Model/training change

Validation

  • New test_scp_filelist_skips_blank_lines and test_jsonl_filelist_skips_blank_lines in tests/test_pcm_input_format.py: 2 failed on main at 66d7a4c2 (IndexError and JSONDecodeError), and the whole file passes with the fix — python -m pytest -q tests/test_pcm_input_format.py → 41 passed.

  • python -m compileall funasr examples tests → exit 0.

  • tests/test_pcm_input_format.py is already in the "Validate NumPy compatibility" workflow run list and funasr/auto/auto_model.py is in its paths filter, so the new tests are exercised by that job on main.

  • python -m compileall funasr examples tests

  • Docs or links checked

  • Runtime/deployment command tested

User impact

Anyone running batch inference through a wav.scp, text.txt or file.jsonl list — the documented entry point for model.generate(input=<list>) — can keep hand-edited or concatenated lists with separator lines instead of the whole batch aborting on one empty line.

Notes for reviewers

  • Real entries are still parsed exactly as before; keys and data order are unchanged, and non-matching file extensions are untouched.
  • This is the practical limitation to note: a fork PR in this repository does not trigger workflow runs, so the evidence above is local, and the reproduction needs no model weights or downloads.
  • Prepared with an AI coding agent; a human maintainer at the PR author's side has not re-reviewed the diff beyond the checks listed above.

A blank or whitespace-only line in a file list crashes prepare_data_iterator:
the id/data branch indexes lines[0] on an empty split (IndexError), and the
.jsonl branch feeds an empty string to json.loads (JSONDecodeError). Skip
blank lines so a trailing or separator line no longer aborts the whole batch.

@LauraGPT LauraGPT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head b09cbbd. On current-main source, both added regressions first failed with the reported IndexError / JSONDecodeError; the merge candidate passes the file-list and byte-loading suites (84 tests and 2 subtests). Six independent boundaries also pass: blank-only lists, Unicode whitespace, CRLF/order/text preservation, malformed nonblank JSON still raising, and an unsupported extension remaining a direct input. The three approved head workflows are successful. NumPy 1.26.4 and 2.4.0 artifacts each show 122 tests with zero failures/errors/skips and this exact source SHA. The cumulative #3754/#3758/#3761 candidate passes 128 tests and 10 subtests. No blocking findings; no model weights or inference needed for this parser fix.

@LauraGPT
LauraGPT merged commit 22b4282 into modelscope:main Oct 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: blank lines in wav.scp / jsonl file lists crash prepare_data_iterator (IndexError / JSONDecodeError)

2 participants