Make agility context account for clear() size in quota - #4574
Open
ohadzeliger wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes an issue with the agility context that ignored size quota accumulation for
clearandclear(range).The actual consequence of the issue was that large merge jobs done by Lucene could not be complete: The last step of the merge is a
clear()of all the old files, and in the case where there were a few thousands of them the transaction would fail, leaving them behind. Subsequent merge jobs would keep failing trying to remove them as well.The reproduction included creating large merge operation with thousands of segments, but that kind of reproduction takes hours and so is not feasible for adding to the test suites. A much faster test that shows that many clears would fail (and pass with the fix) is used instead.
Note that the actual failure reproduced is that of
Transaction too large(Byte bound), not aTransaction too old. The assumption here is that depending on network latencies and other external factors these errors can be interchanged for the purpose of reproduction.Resolves #4575