-
Notifications
You must be signed in to change notification settings - Fork 66
Feature: NumPy-compliant distributed advanced indexing #938
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ClaudiaComito
wants to merge
300
commits into
main
Choose a base branch
from
914_adv-indexing-outshape-outsplit
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
300 commits
Select commit
Hold shift + click to select a range
587bc05
expand tests
ClaudiaComito 24239a1
remove redundant code
ClaudiaComito cd65b37
Implement slicing with negative step
ClaudiaComito 86e8801
test slicing with negative step
ClaudiaComito 6779010
merge branch bugs/#1057-Allgatherv-contiguity-mismatch
ClaudiaComito 3b1f46d
Fix single-element indexing within mixed-type key
ClaudiaComito 1a4bf97
Non-ordered indexing, split != 0
ClaudiaComito 9e42156
generalize negative step slicing to all splits, loss of dims
ClaudiaComito 1a310a9
loop over active ranks only when key in descending order
ClaudiaComito c2ba0d9
replace list-on-list mapping with argsort mapping for non-ordered key
ClaudiaComito f6bb5c3
replace list-on-list mapping with argsort mapping for boolean indexing
ClaudiaComito cad9975
fix advanced indexing via list, remove last key-mapping bottleneck fo…
ClaudiaComito 83e6950
fix local slices, expand tests
ClaudiaComito 28ab925
fix and test dimensional indexing
ClaudiaComito bc226fc
Fix same-dim advanced indexing, expand tests
ClaudiaComito c48c66e
[skip ci] implement single-element indexing along split axis w/ Itera…
ClaudiaComito 18329a1
[skip ci] generalize advanced indexing incl. distributed DNDarray key
ClaudiaComito f024ebb
[skip ci] Expand tests combined advanced / basic indexing
ClaudiaComito 6ae2788
[skip ci] fix advanced dimensional indexing on non-distributed array
ClaudiaComito 178d7f8
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 09e586c
fix distr advanced indexing with broadcasted shape
ClaudiaComito c56ebf4
transpose without copying
ClaudiaComito 86f704a
[skip ci] document __process_key(), clean up code
ClaudiaComito 68ead71
[skip ci] docs edits
ClaudiaComito 252995c
fix Ellipsis dimensions
ClaudiaComito c2a7e20
fix shape and split bookkeeping within advanced indexing
ClaudiaComito 235a7b8
test adv indexing on non consecutive dims
ClaudiaComito 4e936e8
abstract scalar key checks for both getitem and setitem
ClaudiaComito 8a74cd9
setitem scalar key
ClaudiaComito 8cf3ff1
DRAFT - abstraction common utilities for getitem and setitem
ClaudiaComito b45578a
handle all single-element indexing along split axis in same block
ClaudiaComito cec4bb9
resolve send/recv dimensions mismatch in a few edge cases
ClaudiaComito cc49a49
transpose self back to original shape after indexing
ClaudiaComito fe26ae8
add setitem tests
ClaudiaComito 611b46d
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito affdb60
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 7e5be66
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 6d2e369
do not index input unnecessarily for sanitation
ClaudiaComito f528356
test named split dimension for torch_proxy
ClaudiaComito 01a1140
value broadcasting abstraction
ClaudiaComito f8264a9
introduce distr sanitation for value when key is ordered
ClaudiaComito b1cd02f
keep track of original key
ClaudiaComito 31bdb34
fix value broadcasting for advanced setitem
ClaudiaComito c4d6749
match broadcasting to numpy
ClaudiaComito 5782d6e
finalize broadcast_value and fix test
ClaudiaComito 2174e84
assignment to negative slice along split axis
ClaudiaComito 782bde2
getitem: index underlying tensor with processed key in non-distr case
ClaudiaComito 084371d
setitem: test neg step slice along non-zero split axis
ClaudiaComito b1aa7aa
allow for nominal value/self split mismatch
ClaudiaComito 1c2b71e
expand test negative step along split axis
ClaudiaComito 7201a89
allow value.ndim > indexed_dims if extra dims are singletons
ClaudiaComito dfc7266
BROKEN: expand negative step tests
ClaudiaComito 8bbe242
squeeze out singleton dimensions when broadcasting value
ClaudiaComito 00a17e6
fix negative step slicing on 1 process
ClaudiaComito bdd2dd8
setitem w. dimensional indexing, add tests
ClaudiaComito 1fbd4d6
setitem w. advanced indexing on first dim
ClaudiaComito 95d3c92
setitem: test boolean indexing, local and split=0
ClaudiaComito f335aa8
fix output shape for boolean indexing w. split>0
ClaudiaComito d520ddf
setitem with non-ordered, mask-like key and non-distr value
ClaudiaComito d754a9c
allow for partial boolean indexing on first key.ndim dims of array
ClaudiaComito 5e69fe6
remove unnecessary check
ClaudiaComito 8d9849e
add tests for partial boolean indexing
ClaudiaComito 66ae371
set w. single-tensor key and non-distr value
ClaudiaComito 980e8f0
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito ae4d423
non-ordered, non-mask-like key and local value
ClaudiaComito b695e5a
broken: set up comm map for full distributed setitem
ClaudiaComito e6c1e10
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] d42f1cb
implement setitem w. distributed non-ordered key
ClaudiaComito 7868fa0
[skip ci] broken: add tests for distr value non-ordered key
ClaudiaComito f8055ff
Merge branch '914_adv-indexing-outshape-outsplit' of github.com:helmh…
ClaudiaComito 2944903
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 1bffd26
__process_key(): refactor adv indexing tensor extraction
ClaudiaComito 83842ec
working: setitem w. mask-like adv indexing, non-ordered split key
ClaudiaComito 366aaf9
adapt tests
ClaudiaComito bbe0a7b
refactor __process_key(): address boolean ind within adv ind
ClaudiaComito 24c6cd1
Merge branch '914_adv-indexing-outshape-outsplit' of github.com:helmh…
ClaudiaComito 1c47b42
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 4ee9b96
getitem: address mask-like key
ClaudiaComito 3c88f8b
Merge branch '914_adv-indexing-outshape-outsplit' of github.com:helmh…
ClaudiaComito 54db23d
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 15cce44
define nonzero_size in non-distr case
ClaudiaComito 09fb199
handle split_bookkeeping when key is mask-like
ClaudiaComito 9c8d051
fix key type mismatch in advanced indexing
ClaudiaComito 41fba0a
getitem: address n-D key along split axis, free memory
ClaudiaComito e4a90de
balance indexed array before eq()
ClaudiaComito c8967e7
remove print statements
ClaudiaComito 2d443d8
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 95eaaeb
test adv ind on non-consecutive dims
ClaudiaComito 835a13f
remove print statement
ClaudiaComito 216a1a0
setitem: mixed indexing w. shape broadcasting
ClaudiaComito b62bad2
expand tests for mixed indexing w. broadcasting
ClaudiaComito 7ea2abe
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 435ff0c
reinstate tests for specific bugs
ClaudiaComito 30efe59
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito e6679a0
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito ad96822
prep send_buffer - expand value dimension if necessary
ClaudiaComito c9d44ae
fix send_indices dims when key is not mask-like
ClaudiaComito cc70400
test split mismatch on comm.size > 1
ClaudiaComito b78de30
broadcasting assignment along split axis
ClaudiaComito a8f2d57
expand tests
ClaudiaComito 62b2142
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 6d76b0c
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 57d9dbb
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 58823f6
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 4a5ef0f
edits
ClaudiaComito 243084e
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 941c28e
get rid of torch.tensor warning
ClaudiaComito fbb3fe5
fix dimension loss
ClaudiaComito 0a120d7
add edge case for boolean mask
ClaudiaComito 9657746
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito b0bfa08
do not index scalar value
ClaudiaComito dfb0667
debugging
ClaudiaComito 2e8001a
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 39ab011
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 6d848d6
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito cae4670
Merge branch '914_adv-indexing-outshape-outsplit' of github.com:helmh…
ClaudiaComito 9d74da2
I already hate talisman after 1 day
ClaudiaComito b204589
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
Hakdag97 5a5ae6e
Fixed __setitem__ bug for unordered split_key
Hakdag97 0aa3ee0
Fixed bugs causing errors in test_getitem_boolean_fewer_dims
Hakdag97 36855d7
Bug fixes for test_setitem_edge_cases
Hakdag97 151d2b2
Further bug fixes
Hakdag97 960c5dd
All tests are running
Hakdag97 d67c5a9
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
Hakdag97 3e0e6a6
Edge case handling for test_indexing intermediate results
Hakdag97 25e1b34
Fixed test_indexing.py
Hakdag97 95c72a2
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
Hakdag97 0f09e7e
Bug fixes in function where()
Hakdag97 9956639
Merge branch '914_adv-indexing-outshape-outsplit' of github.com:helmh…
Hakdag97 aadcf35
Edge case handling for slice type keys in __getitem__
Hakdag97 b0147ce
Debugging tests for clustering - intermediate results
Hakdag97 f2e168c
Fixed edge case in indexing causing deadlock in kmedoids clustering
Hakdag97 638d1f8
Delete bug prints
Hakdag97 466c1f0
Edge case handling for keys like [:, -1], in order to fix test_basics
Hakdag97 047488c
Bug fixes for test_factories.py
Hakdag97 376cbb1
Fixed bug in test_cov (wrong balance)
Hakdag97 50f0ad1
Fixed bug in test_manipulations.py (function tile)
Hakdag97 c2ce57e
Drop tensor names in function tile
Hakdag97 595f84a
Handle edge case for test_svd and test_eigh
Hakdag97 28e46a1
Fix test_knn.py
Hakdag97 2c34b36
Added edge case neccessary for local outlier factor
Hakdag97 d790865
.
Hakdag97 b9f132e
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
Hakdag97 8a373c9
Fixed device mismatch in process_key
Hakdag97 3c9ea98
Merge branch '914_adv-indexing-outshape-outsplit' of github.com:helmh…
Hakdag97 bc6616b
Refine test_dndarray
Hakdag97 43f73c3
Handling of duplicate advanced indices
Hakdag97 1fc4f1e
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
Hakdag97 2240117
.
Hakdag97 e6256d5
Merge branch '914_adv-indexing-outshape-outsplit' of github.com:helmh…
Hakdag97 6da8259
Improved code coverage in tests
Hakdag97 3c9b989
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
mrfh92 2bfdbc5
Test debugging advanced indexing for dmd
Hakdag97 0149260
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
Hakdag97 17446a2
Fixed bug in process_key leading to failing dmd test
Hakdag97 9aa581e
Robustified edge cases in __process_key
Hakdag97 75483a7
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
Hakdag97 477aa2f
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
JuanPedroGHM 4d79b0b
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
JuanPedroGHM df6714d
chore: minor type hints improvements for dndarray.py
JuanPedroGHM 1c5090f
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 82fd6bf
Merge branch '914_adv-indexing-outshape-outsplit' of github.com:helmh…
ClaudiaComito f445652
fix attribute assignment in DNDarray calls)
ClaudiaComito 117bebb
fix position of split argument
ClaudiaComito a5c8788
add as_tuple argument for nonzero
ClaudiaComito 1afc837
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito d680e25
move helper functions out of __getitem__/__setitem__
ClaudiaComito 1108548
move most key sanitation to __process_key()
ClaudiaComito 66c23ed
unbind instead of torch.split if as_tuple
ClaudiaComito 0651339
extract distr logic unordered key from getitem
ClaudiaComito 0a748de
extract distr logic for unordered value from setitem
ClaudiaComito a945528
process_key returns namedtuple, getitem acts as dispatch
ClaudiaComito e525f28
remove redundant _is_basic_component check
ClaudiaComito 1331187
reorganize dispatching order, remove redundant checks
ClaudiaComito 15ee181
refactor setitem - dispatch to appropriate helpers
ClaudiaComito a6ccbf9
fix misidentification of adv ind as mask-like
ClaudiaComito fde5b60
disentangle local from distributed masking
ClaudiaComito 6470339
refactor distributed boolean mask fast-path
ClaudiaComito 29c9885
fast-track local bool mask in tuple key
ClaudiaComito fe15b3f
do not fast-track mask getter with split>0
ClaudiaComito 1564745
fix types call
ClaudiaComito fd36f92
distr setitem bug fixes
ClaudiaComito 0ed5d9d
fix IndexError caused by list(key) on single tensor in Alltoallv
ClaudiaComito 305accb
comment out print statements
ClaudiaComito 7778cd1
always route distr key to unordered indexing
ClaudiaComito 13df61c
expand tests
ClaudiaComito 21e1ca4
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 0cedcee
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito d5eb00b
remove property decorator from stride()
ClaudiaComito 51da43d
remove most itemsetting from test_matmul
ClaudiaComito ad7b8e5
adapt nonzero tests to as_tuple option
ClaudiaComito 1873ef6
fix nonzero tests
ClaudiaComito 0a2f383
nonzero tests
ClaudiaComito 922a41a
handle 0-d bool indexing of 0-d array
ClaudiaComito b177e7d
test 0-d bool indexing of 0-d array
ClaudiaComito 3a2aa67
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito ffa7cf7
reinstate CUDA deduplication
ClaudiaComito 426c5d6
Issue #824, negative indices and multi-dim adv ind
ClaudiaComito 8508d11
cover edge cases in #824
ClaudiaComito 4239511
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 5815c79
remove dead code
ClaudiaComito ffd87cd
comment out print statements
ClaudiaComito d745001
vectorized mapping to dest ranks
ClaudiaComito 8e68226
refactor unordered setitem, extract communication prep
ClaudiaComito 61f1740
switch unordered getitem to Alltoallv
ClaudiaComito c8589c4
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 86ecb5c
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 9feaf97
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 2d5502e
Reintegrate input sanitation in nonzero
ClaudiaComito 6af95c2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 937bc18
Apply suggestions from code review
ClaudiaComito 1ec9543
Remove edits
ClaudiaComito 6bfb650
bring back to original state
ClaudiaComito 199518a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] eaa34eb
Refactor distr_mask_fast_path
brownbaerchen 350aaf8
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 23ab286
remove legacy indexing leftovers
ClaudiaComito b3bf485
remove orphaned functions after refactoring
ClaudiaComito c803588
better name and docstring for tafkaprocessed_key
ClaudiaComito 442d83e
move resolve_indexing_state out of Class
ClaudiaComito 49406ab
rename dedup and move out of class
ClaudiaComito 61b3799
introduce Indexer type alias
ClaudiaComito 518d023
introduce Indexer type alias
ClaudiaComito ce60526
edits for readability
ClaudiaComito 3fc2666
remove dead code
ClaudiaComito 012b502
update getitem docstring
ClaudiaComito cffe445
update getitem docstring
ClaudiaComito 9623a20
update setitem docstring
ClaudiaComito 66b1a69
add docstrings for helper functions
ClaudiaComito 02a4dc1
add docstrings for helper functions
ClaudiaComito 8622766
fix split axis bookkeeping
ClaudiaComito fb7c5ba
revert split bookkeeping fix
ClaudiaComito eda3789
add indexing documentation
ClaudiaComito 75246d2
correctly track split axis in adv indexing
ClaudiaComito cfaf4d2
expand tests
ClaudiaComito 937a4cb
add indexing docs to rtd index
ClaudiaComito d4ce2aa
remove TMI
ClaudiaComito 56d3b1f
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
ClaudiaComito 4c1ddde
render hyperlinks
ClaudiaComito 324211f
remove print statements
ClaudiaComito e349b17
refactor indexing tests
ClaudiaComito 70f133f
remove redundant where(cond) heuristics
ClaudiaComito bf8cedd
fix where tests for correct tuple output
ClaudiaComito 425acd0
enforce consistent output split
ClaudiaComito 447e5ca
fix nonzero tests
ClaudiaComito a05e0ea
Remove unnecessary stuff
brownbaerchen 9f0543b
Merge branch 'main' into 914_adv-indexing-outshape-outsplit
brownbaerchen f415953
Moved changes re batchparallelclustering to their own PR #2359
brownbaerchen 76c5c85
Merge branch '914_adv-indexing-outshape-outsplit' of github.com:helmh…
brownbaerchen 102b005
Cast numpy to torch in `_resolve_indexing_state` (#2369)
brownbaerchen File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
brownbaerchen marked this conversation as resolved.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Documentation looks great!