Add binary encoding for Cosmos query pages - #17
Closed
tvaron3 wants to merge 11 commits into
Closed
Conversation
Negotiate Cosmos binary JSON for SQL query and document read-feed operations while keeping query-plan and change-feed requests text-only. Normalize binary pages at pipeline ingest and restore the planned response format at emit so DISTINCT and streaming ORDER BY retain their existing text-based internals. Preserve the response encoding selected during planning even when execution options differ. Make emulator responses and feed item slices binary-aware, pin numeric DISTINCT invariants, and add text/binary parity coverage to emulator, fuzzer, and live split test surfaces. Update the binary encoding design and changelogs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 17d20364-4ec8-4cef-9aa2-b9bc87c18430
Capture streaming ORDER BY fingerprints before request encoding so text and binary requests share continuation tokens without invalidating existing text tokens. Add SqlQuery negotiation consistency, quiet empty query-plan handling, service-style Float64 DISTINCT coverage, and emulator assertions for request negotiation and cross-mode resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 17d20364-4ec8-4cef-9aa2-b9bc87c18430
Serialize emulator query fixtures in the same field order as the public SDK so parameterized text and binary fingerprints actually diverge when capture happens too late. Remove the vacuous already-binary fingerprint self-comparison. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 17d20364-4ec8-4cef-9aa2-b9bc87c18430
Return a typed CosmosError when streaming ORDER BY planning lacks its pre-encoding fingerprint instead of panicking across an FFI host boundary. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 17d20364-4ec8-4cef-9aa2-b9bc87c18430
Return a dedicated client-side status when streaming ORDER BY planning is missing its pre-encoding fingerprint. Export the same status through the native C ABI so FFI hosts receive a named diagnostic instead of a magic number. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 17d20364-4ec8-4cef-9aa2-b9bc87c18430
Derive pipeline-boundary binary restoration from the request-side eligibility gate so Query, SqlQuery, and ReadFeed cannot drift apart. Exhaustively pin request and restore behavior for every operation type. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 17d20364-4ec8-4cef-9aa2-b9bc87c18430
…inary-query-encoding
Keep SQL query request bodies textual while negotiating binary response pages, and normalize integral doubles when binary responses are explicitly transcoded to text.\n\nModel service number behavior in the emulator, strengthen query fuzzer and split coverage, and document the measured Gateway behavior and defensive invariants.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: 17d20364-4ec8-4cef-9aa2-b9bc87c18430
Picks up the upstream merge from the parent branch, which brought in `OFFSET`/`LIMIT`/`TOP` support (Azure#4870) and reordered the query pipeline. Conflicts: - `parse_document_page` and `retain_documents` were deleted upstream when `Distinct` moved to per-item slices; `normalize_page_body` survives and is still what makes the envelope path binary-aware. Its two tests are retargeted onto it directly, keeping the malformed-binary coverage. - `build_streaming_ordered_merge_with_fingerprint` no longer peels or applies `DISTINCT`; `build_streaming_ordered_merge_inner` owns that nesting now, so the wrapper only threads the pre-encoding fingerprint through. - `execute_plan` and `plan_operation` keep the binary transcode hooks inside the boxed futures the parent introduced. - The emulator test helper decodes both page shapes *and* both encodings. `split_feed_envelope` is now binary-aware. `Distinct` consumes raw `Bytes` pages through it, and under a negotiated-binary wire those bytes are a single `0x80` envelope that its text scan cannot read — `SkipTake` had the same exposure. Normalizing at that one choke point makes every consumer of a raw feed page binary-correct by construction, matching how `deserialize_response` already handles the per-buffer case. `CLIENT_STREAMING_ORDER_BY_FINGERPRINT_MISSING` joins the FFI `mirror_driver_sub_status!` list: upstream's new compile-time drift guard (Azure#5017) requires every `CosmosSubStatus` variant to be mirrored, and this one post-dated the guard. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: eab410d0-1b95-4c38-9c03-2e6bb4a9a758
Pin binary envelope normalization at the shared feed-splitting boundary and prevent format parity assertions from passing without a classifiable page. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 17d20364-4ec8-4cef-9aa2-b9bc87c18430
Resolve the live split-test overlap by preserving text and binary coverage alongside the parent branch's continuation diagnostics and DISTINCT row-window checks. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 17d20364-4ec8-4cef-9aa2-b9bc87c18430
Owner
Author
|
Superseded by Azure#5070, now targeting upstream main as requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Cosmos binary JSON encoding for query pages, closing the two items
BINARY_ENCODING_HLD.mdlisted under §Deferred work: Query binary negotiationand Binary feed responses.
Measured against a live account
Behavior was verified rather than inferred:
SELECT *+CosmosBinarySELECT *+CosmosBinarypayload/orderByItems)/queryplan+CosmosBinaryapplication/query+json3, not3.0Those are response-body sizes from a small probe, not RU or latency numbers.
Because the negotiation header alone produces binary pages, query request
bodies deliberately stay text — matching .NET and Java, and avoiding an emulator
change that would have rubber-stamped an unverified direction.
Design
Follows the existing contract: bytes stay in wire format,
deserialize_responseauto-detects the
0x80preamble per buffer, andrequest_text_responseis whatconverts to text.
Normalize at ingest, restore at emit. A binary page is transcoded to text
once at the feed-splitting boundary, so the query pipeline (DISTINCT, streaming
ORDER BY) works unchanged; the negotiated format is restored at the pipeline
emit boundary.
split_feed_envelopeis the single normalization choke point, soevery consumer of a raw feed page —
SkipTakeandDistinct— is binary-correctby construction.
Decoding straight to
serde_json::Value, skipping the text hop, is left as afollow-up optimization; correctness does not depend on it.