Report the rank of each entry when scanning a rank index - #4563
Conversation
| if (!rankScanBounds.includeRankAsValue()) { | ||
| return recordCursor; | ||
| } | ||
| return recordCursor.mapPipelined(indexEntry -> rankFor(indexEntry.getKey()) |
There was a problem hiding this comment.
This is the important piece of this PR, it is admittedly expensive, for each scanned record we probe the secondary subspace, if I am not mistaken, this is about O(log(n)) I/O, per item, where n is the number of elements per partitioning group.
There are perhaps more clever ways to do this, considering that values in the primary subspace are already sorted by score, which means, in theory, we just need to know the (dense) rank of the first element, and derive the (dense) rank of the following items since the score and rank are compatibly ordered, that however requires remembering the previous rank + the previous score, which is problematic because it is sensitive to order of scanned items, which is not guaranteed with mapPipelined with number of workers > 1, and I can't find a correct way for achieving this that doesn't require reduced parallelism and/or heavy synchronization logic that would be probably less compelling than this approach, I might be wrong though.
There was a problem hiding this comment.
So in a world where you only use rankFor() the first item and then just increment, you really don't have to use mapPipelined() anymore as only the first one actually incurs I/O. In other words you could roll your own cursor, that initially farms out work using rankFor() and waits for it and then just increments the rank itself. I think that's better than doing I/O for each item (but now it's pipelined). Again, in the look-up-first-item-then-increment approach you don't need the pipelining.
080d4f4 to
23f9d00
Compare
A rank index scan cannot currently tell a caller what rank each entry holds. A BY_RANK scan converts the rank range into a score range and then performs an ordinary by-value scan, so its entries are identical to those a BY_VALUE scan produces: the rank is consumed as a bound and never reported. The rank is stored in neither the entry key nor the entry value, and a rank index leaves the value empty. This adds RankScanBounds, an IndexScanBounds accepting BY_VALUE or BY_RANK together with an includeRankAsValue option that asks RankIndexMaintainer to report each entry's rank in the otherwise empty value. The rank lookup that the RANK record function already performed is reused: it needed a record only in order to evaluate the index key against it, so it is now keyed off the index key itself, which a scan already has in hand.
23f9d00 to
9d2344e
Compare
2 participants
A scan of a rank index cannot currently tell a caller what rank each entry holds.
BY_RANKconverts the rank range into a score range and then performs an ordinary by-value scan, so its entries come back byte-for-byte identical to whatBY_VALUEyields — the rank is consumed as a bound and then discarded. It is stored in neither the entry key nor the entry value, and a rank index leaves the value empty. Anything that wants to project a rank, rather than merely filter on one, therefore has nothing to read.This adds
RankScanBounds, anIndexScanBoundsthat acceptsBY_VALUEorBY_RANKalong with anincludeRankAsValueoption askingRankIndexMaintainerto report each entry's rank in that empty value. No new lookup was needed: theRANKrecord function already computed exactly this, and it wanted a record only in order to evaluate the index key against it. Keying it off the index key instead lets a scanned entry be ranked without loading its record, and the record path reduces to producing that key first. The javadoc is explicit that the option is not free — it costs one ranked-set skip-list traversal per entry returned, so the cost grows with entries returned rather than with the range asked for.