Fix: preserve features after IterableDataset.add_column (GH#5752)#8297
Open
uttam12331 wants to merge 1 commit into
Open
Fix: preserve features after IterableDataset.add_column (GH#5752)#8297uttam12331 wants to merge 1 commit into
uttam12331 wants to merge 1 commit into
Conversation
add_column delegates to map() but does not pass features, causing map() to set info.features = None and losing the original schema. Fix by inferring the new column's feature type from the provided data, merging it with the existing features, and passing the result to map() so the returned dataset's .features are correctly set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After calling
.add_column()on a streamingIterableDataset, accessing.featuresreturnsNone:Root Cause
add_columncallsself.map(...)without passing thefeaturesargument. Inside_map, the new dataset's info is constructed as:This unconditionally overwrites the copied features with
None.Fix
Infer the feature type of the new column from the provided data, merge it with the existing features, and pass the combined
Featuresobject tomap():Tests
test_iterable_dataset_add_column_preserves_featureswhich explicitly checks that.featuresis notNoneand contains both original and new columns afteradd_column.test_iterable_dataset_add_columncontinues to pass.Closes #5752