feat(query): upgrade query architecture, analytics and cursor - #2908
feat(query): upgrade query architecture, analytics and cursor#2908Ahoo-Wang wants to merge 18 commits into
Conversation
github-actions
Bot
added
area: documentation
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 5754 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2908 +/- ##
============================================
- Coverage 92.35% 86.11% -6.24%
- Complexity 6822 8670 +1848
============================================
Files 984 1089 +105
Lines 26118 39659 +13541
Branches 2684 5071 +2387
============================================
+ Hits 24121 34154 +10033
- Misses 1094 3244 +2150
- Partials 903 2261 +1358
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Ahoo-Wang
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9ddeba24fd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
github-actions
Bot
added
area: api
Ahoo-Wang
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 310b33364c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: af0d4ff590
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| private fun openPit(): Mono<PitLease> = client.openPointInTime( | ||
| OpenPointInTimeRequest.of { request -> | ||
| request.index(binding.indexName).keepAlive { keepAlive -> keepAlive.time(PIT_KEEP_ALIVE) } |
There was a problem hiding this comment.
When QueryCursorLeaseConfiguration.leaseTtl is legally configured above two minutes, a grouped snapshot analytics cursor remains valid in the lease store after this hard-coded Elasticsearch PIT has expired; resuming it after two minutes therefore fails with an incomplete/backend error despite an unexpired token. Even the default two-minute lease can outlive the PIT slightly because the lease is issued after opening and querying it, so derive keep_alive from the configured lease TTL with margin or reject incompatible TTLs.
Useful? React with 👍 / 👎.
| val frozen = session.objectValue(source, 0) | ||
| val identity = (frozen.values[Documents.ID_FIELD] as? NormalizedValue.Text)?.value | ||
| ?: mappingFailure() | ||
| val document = frozen.toLogicalDocument().apply(projection, binding.identityOutputField) |
There was a problem hiding this comment.
For every Mongo binding whose schema declares LogicalFieldType.Instant, binding validation requires MongoValueEncoding.EPOCH_MILLIS, so the driver supplies a numeric value; this materialization only reshapes the frozen document and leaves that value as NormalizedValue.Int64 rather than InstantValue (including array elements). Fresh evidence in the final tree is that the Mongo mapper still has no schema-aware decode step, so planned Mongo record queries expose the wrong logical type even though the Elasticsearch mapper was corrected.
Useful? React with 👍 / 👎.
| .map { response -> | ||
| if (response.id().isBlank() || response.shards().failed() != 0) incomplete() | ||
| PitLease(response.id()) |
There was a problem hiding this comment.
When Elasticsearch returns a nonblank PIT id together with one or more failed shards, this branch throws before Mono.usingWhen receives a PitLease, so none of its cleanup callbacks can close the contexts that were opened successfully. Repeated queries while a shard is degraded can therefore accumulate PIT resources until their keep-alive expires; construct the lease from the returned id and explicitly close it before propagating the incomplete-result failure. The record-page and lifecycle-verification PIT open paths use the same reject-before-acquisition pattern.
Useful? React with 👍 / 👎.
| is QueryCursorPosition.Analytics -> { | ||
| writeByte(ANALYTICS_POSITION) | ||
| writeInt(position.dimensionAliases.size) | ||
| position.dimensionAliases.forEach { alias -> writeBoundedUtf8(alias.value) } | ||
| writeValues(position.afterKey) | ||
| } |
There was a problem hiding this comment.
When a grouped analytics query has 33 or more dimensions, which the default admission and unrestricted planning constraints allow, this encoder writes every alias and returns a continuation token successfully, but decoding that token applies MAX_POSITION_VALUES = 32 to both the alias and after-key counts. The next request therefore rejects a framework-issued token as INVALID_CURSOR_TOKEN; either reject plans above 32 dimensions before backend execution or make the codec encode/decode bounds consistent.
Useful? React with 👍 / 👎.
Summary
This PR consolidates the complete Query Service architecture upgrade into one review surface, from execution-correctness foundations through Query/Analytics/Cursor contracts and planned MongoDB/Elasticsearch backends.
LEGACY | SHADOW | PLANNED) from semantic validation (COMPATIBLE | STRICT)Contract and compatibility
QueryServicekeeps its existing seven public methods andQueryTypekeeps its existing seven valuesGLOBAL/BYgrouping and metric field requirements with strictoneOfschemasSafety and rollout
CUTOVERis denied unless an explicit write-fence guard is suppliedValidation
./gradlew detekt build./gradlew allLocalTest allContractTest allIntegrationTestcd documentation && pnpm docs:buildConsolidation
This single PR supersedes the former stacked review PRs #2909, #2910, #2915, #2916, #2917, #2918, and #2924. Their commits are retained here. GitHub automatically marked those PRs
MERGEDwhen their head commits became contained by this consolidated base branch; none of them was merged intomainindependently.Primary review boundaries:
This also supersedes the earlier operator-by-operator approach from closed PR #2903.