Repository navigation
kgo: return cached metadata brokers with their topics - #1481
Open
koloss2001 wants to merge 1 commit into
Open
koloss2001 wants to merge 1 commit into
koloss2001 wants to merge 1 commit into
Conversation
RequestCachedMetadata copies topics from the metadata cache and brokers from the live connection table, so the two halves can come from different Metadata responses. A caller that routes from that one struct can see a partition leader that is not in Brokers.
Owner
|
Simpler alternative - wdyt? #1482 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
RequestCachedMetadatareturnsBrokersandTopicsfrom the same metadata response.When the call returns topics,
Brokers,ControllerID, andClusterIDcome from the broker list saved with those topics. A cache hit whose topics come from two responses fetches that set once more and uses that response. A request with no topics still returns the live connection table.cl.brokersis unchanged, and there is no new API.Why
We're building warpstream-go, a produce-only Kafka client on top of franz-go. Its refresh calls
RequestCachedMetadataand uses that single return value as the snapshot for the call: dialable brokers come fromBrokers, and each partition leader comes fromTopicsin the same struct. A non-negativeLeaderthat is absent fromBrokersis not a broker that response can name, so the partition cannot be routed from it.kadm.Metadatareturns the same pair. kgo's own producer does not read this struct.A Metadata response is one cluster view. These two answers are each consistent:
Response A, before node 3 is gone:
Brokers: 1, 2, 3orders, partition 0, leader 3Response B, after:
Brokers: 1, 2orders, partition 0, leader 1RequestCachedMetadatacould return a third shape, which neither response contained:Brokers: 1, 2orders, partition 0, leader 3Two ways that happened. The call fetched A and stored A's topics, then another fetch (the metadata loop,
Ping, orfetchBrokerMetadata) installed B's broker list intocl.brokersbefore the helper copied it. Or the call did not fetch: topics were still insidelimit, and a brokers-only Metadata had already replacedcl.brokersand left the topic cache alone.ControllerIDis the value that response sent, including -1.updateMetadataBrokersstill ignores -1 so the client can dial the last controller it knew. That id is not copied into this snapshot, because it may not be in this response's broker list.Happy to adjust the approach.