Skip to content

Add skip_origin_metadata flag to optimize streaming startup time#8291

Open
yuxin00j wants to merge 3 commits into
huggingface:mainfrom
yuxin00j:fix-8197
Open

Add skip_origin_metadata flag to optimize streaming startup time#8291
yuxin00j wants to merge 3 commits into
huggingface:mainfrom
yuxin00j:fix-8197

Conversation

@yuxin00j

Copy link
Copy Markdown

Resolves #8197.

When streaming datasets that consist of many remote data files, the initial origin metadata fetch (_get_origin_metadata which queries ETag/mtime) can become a significant latency bottleneck.

This PR implements the accepted feature request to introduce an optional skip_origin_metadata keyword argument to bypass these checks and improve streaming startup time.

Changes:

  • Added skip_origin_metadata: bool = False to DatasetBuilder.__init__ and data resolution methods across DataFilesList and DataFilesDict.
  • When skip_origin_metadata=True, pattern resolution assigns empty tuple placeholders [()] * len(data_files) rather than making expensive network/filesystem calls.
  • The standard file-existence check is preserved during resolution; if a file doesn't exist, it correctly raises a FileNotFoundError.
  • Added validation in load_dataset: this optimization is strictly intended for streaming mode. If skip_origin_metadata=True is passed alongside
    streaming=False, a warning is logged and the flag is safely forced back to False.
  • Added tests to tests/test_data_files.py and tests/test_load.py to ensure correct behavior and error handling.

@yuxin00j yuxin00j marked this pull request as ready for review June 30, 2026 02:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request / Discussion: Skip origin_metadata fetch when streaming=True?

1 participant