Local version cache not updated when clearing split records

[alecgrieser]

We maintain two caches where we store the local version associated with a record. One of them is the versionMutationCache that stores a mapping from byte[] to NonnullPair<MutationType, byte[]>:

fdb-record-layer/fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/FDBRecordContext.java

Line 159 in ab19a6a

private ConcurrentNavigableMap<byte[], NonnullPair<MutationType, byte[]>> versionMutationCache;

This is used to store the "dirty" incomplete version mutations. We apply these right before commit(). The reason this is done is that the FDB API otherwise provides no way to remove mutations, so we'd wind up writing just the version if we saved and deleted a record with a version in one transaction.

But there's a second spot, the localVersionCache:

fdb-record-layer/fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/FDBRecordContext.java

Line 157 in ab19a6a

private ConcurrentNavigableMap<byte[], Integer> localVersionCache;

This is from byte[] to Integer, and it allows us to read a record written with an incomplete version and fill in the version correctly. Technically, this could be done from the versionMutationCache instead, but it would require a bit more effort. (It would require reading the key from the cache, validating the MutationType is SET_VERSIONSTAMPED_VALUE, and then parsing the version out of the mutation's value.) We already update this in several places where we clear out records, including if the record is not split--see:

fdb-record-layer/fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/FDBRecordStore.java

Lines 1786 to 1792 in 4b0143a

	if (oldHasIncompleteVersion) {
	return updateIndexesFuture.thenApply(vignore -> {
	byte[] versionKey = getSubspace().pack(recordVersionKey(primaryKey));
	context.removeLocalVersion(versionKey);
	return true;
	});
	} else {

But it's missing from KeyValueUnsplitter::clearPreviousSplitRecord:

fdb-record-layer/fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/SplitHelper.java

Line 378 in 01f706e

context.getLocalVersion(versionKey).ifPresent(localVersion -> context.removeVersionMutation(versionKey));

[alecgrieser]

Upon further reflection, this may not actually be a bug. It turns out, we do clear this out, just not in the SplitHelper. It gets cleared out in FDBRecordStore::deleteTypedRecord:

fdb-record-layer/fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/FDBRecordStore.java

Lines 1786 to 1792 in 6c7ffcd

	if (oldHasIncompleteVersion) {
	return updateIndexesFuture.thenApply(vignore -> {
	byte[] versionKey = getSubspace().pack(recordVersionKey(primaryKey));
	context.removeLocalVersion(versionKey);
	return true;
	});
	} else {

Perhaps there is still a bug if you:

1. Create a record with a version
2. Update the record with a new value but without a version

In the same transaction