diff --git a/document/design/2026-08-06-query-service-architecture-upgrade.md b/document/design/2026-08-06-query-service-architecture-upgrade.md new file mode 100644 index 00000000000..06f0dc7323b --- /dev/null +++ b/document/design/2026-08-06-query-service-architecture-upgrade.md @@ -0,0 +1,1670 @@ +# Query 服务架构升级设计 + +## 1. 决策摘要 + +Query 服务重构为一条不可绕过的应用主干: + +```mermaid +flowchart LR + HTTP[WebFlux Adapter] --> Gateway[QueryGateway] + Legacy[Legacy QueryService Adapter] --> Gateway + Internal[New In-process Caller] --> Gateway + + subgraph Application[Wow Query Application] + Gateway --> Admission[Raw Admission Guard] + Admission --> Normalizer[Query Normalizer] + Normalizer --> Policy[QueryPolicy] + Policy --> Planner[Query Planner] + Planner --> Executor[Query Executor] + Executor --> ResultPolicy[Mandatory Result Policy] + ResultPolicy --> ErrorBoundary[Error Boundary] + end + + Executor --> Router[Backend Registry / Router] + Router --> Mongo[Mongo QueryBackend] + Router --> ES[Elasticsearch QueryBackend] + Mongo --> Materializer[Result Materializer] + ES --> Materializer + Materializer --> ResultPolicy +``` + +只把以下三类能力设计为长期扩展端口: + +1. `QueryGateway`:唯一应用查询入口; +2. `QueryPolicy`:可信授权、强制条件、字段和结果约束; +3. `QueryBackend`:只执行已经验证的后端无关计划。 + +`QueryNormalizer`、`QueryPlanner`、预算计算器和路由器在模型稳定前保持框架内部具体实现,不提前开放多套 SPI。现有 `SnapshotQueryService`、`EventStreamQueryService`、HTTP Handler 和存储 `QueryServiceFactory` 保留为兼容适配器,不再同时承担应用端口和 Backend 端口。 + +## 2. 背景与根因 + +当前查询合同由 `wow-api` 的 Query DTO、`wow-query` 的 Service/Handler/Filter、WebFlux 路由以及 MongoDB/Elasticsearch 实现共同解释,存在以下结构性问题: + +- HTTP 查询经过 Handler/Filter,进程内 `QueryService` 可以直接进入存储,安全与错误处理边界不一致; +- `SnapshotQueryService` / `EventStreamQueryService` 同时表示应用服务和存储服务,类型系统不能阻止循环接线; +- Filter 可通过可变 `QueryContext` 替换整个 query/result,无法承载不可移除的授权条件; +- MongoDB 和 Elasticsearch 分别解释 `Condition`、字段、分页、投影与完整性,公共 DTO 相同但语义并不相同; +- Factory、Provider、Gateway 各自缓存同一聚合的服务实例,生命周期没有唯一 owner; +- `SHADOW` 是执行策略,`STRICT` 是校验策略,把两者放入单一 profile 会形成互斥关系,无法表达“严格校验下进行 shadow”。 + +因此本次升级不是给 Elasticsearch 增加 MongoDB 操作符别名,而是重新划分应用、语义与存储边界。 + +## 3. 目标与非目标 + +### 3.1 目标 + +1. HTTP 与进程内调用最终都进入同一个 `QueryGateway`。 +2. 原始 Query DTO 在应用边界内转换为深度不可变、后端无关的 `QueryPlan`。 +3. 授权强制条件具有 provenance,任何后续扩展都不能移除或覆盖。 +4. MongoDB 作为 `PORTABLE` 查询语义基准,Backend 只负责编译和执行同一份 Plan。 +5. 查询完整性、分页一致性、预算与错误分类由公共合同定义,Backend 不得静默降级。 +6. 保持现有记录 Query DTO、七个 `QueryService` 方法、DSL、Spring 聚合 Bean 名称与主要返回结构兼容; + 经批准的 Analytics/Cursor 契约以 additive 方式升级 JSON/OpenAPI。 +7. 将分析型聚合建模为独立的一等操作,在不污染记录查询合同的前提下统一 MongoDB/Elasticsearch 的分组、指标、桶分页与完整性语义。 +8. 支持按聚合逐步启用 planned path、shadow compare、切换和回滚。 + +### 3.2 非目标 + +- 不承诺 `MATCH` 在 MongoDB 与 Elasticsearch 间具有相同分词和相关性。 +- 不把 `RAW` 纳入跨 Backend 可移植语义。 +- 不给现有记录查询增加 cursor HTTP 协议;Analytics 使用独立、已批准的 opaque cursor 契约。 +- 不给 `QueryService` 增加第八个方法;Analytics 通过独立的 `AnalyticsQueryService` / Gateway / route additive 发布。 +- 不在查询运行时提供跨聚合、跨集合或跨索引 join;此类需求通过 Projection 物化专用 read model。 +- 应用启动不自动迁移、删除 Elasticsearch 索引或切换 alias。 +- 在核心模型尚未稳定前不拆分新的 Gradle module。 + +### 3.3 聚合术语边界 + +“聚合”在 Wow 中必须区分三种含义: + +| 术语 | 含义 | 本设计的处理 | +|---|---|---| +| DDD Aggregate 查询 | 查询某个聚合根的 Snapshot 或 EventStream read model | 现有记录查询目标 | +| 分析型聚合 | `GROUP BY`、document count、`MIN/MAX/SUM/AVG` 等统计 | 独立 `ANALYZE` 操作 | +| 联邦查询 | 跨聚合/集合/索引 join | 明确排除,改用物化 Projection | + +公开类型使用 `Analytics*`,避免 `AggregateQuery` 与 DDD Aggregate 混淆。分析型聚合共享 +`QueryGateway`、Policy、Planner 和 Backend registry,但不复用记录查询的 `PagedList`、 +`DynamicDocument` 或七方法 `QueryService` 合同。 + +## 4. 边界与职责 + +### 4.1 QueryGateway + +`QueryGateway` 是唯一应用端口,负责整个 Publisher 生命周期: + +- 每次订阅创建独立的 `QueryInvocation`; +- 读取显式、可信的 `QueryExecutionContext`; +- 完成 admission、normalization、policy、planning、execution 和结果策略; +- 覆盖同步异常、异步 Backend 异常、部分 `Flux` 后的异常和取消; +- 不缓存聚合服务、不解析物理字段、不拥有存储客户端。 + +WebFlux 是 transport adapter;现有聚合级 `SnapshotQueryService` / `EventStreamQueryService` Bean 是 legacy adapter。两者都只能委托 Gateway。兼容期保留 Handler/Filter,但它们不得再被描述为最终安全边界。 + +已批准的 `AnalyticsQueryService` 是独立、additive 的应用 adapter,同样只能委托 Gateway;它不是现有 +`QueryService` 的第八个方法,也不能直接持有 MongoDB collection 或 Elasticsearch client。 + +### 4.2 Raw Admission Guard + +Policy 不直接接收未经验证的 `Any` DTO。Admission 先执行低成本结构保护: + +- 条件深度、节点数、children 形态; +- 字段、projection、sort 数量和字符串长度; +- value、options,以及未来已绑定 immutable JSON `RAW` payload 的大小上限; +- 非法分页、负 limit 和明显溢出。 + +Admission 只防止畸形输入消耗过多资源,不决定业务授权或 Backend 语义。 + +### 4.3 Normalizer + +Normalizer 将 wire DTO 转换为 `NormalizedQuery`: + +- 完整递归解析逻辑条件和 `ELEM_MATCH` 相对字段作用域; +- 将内置 ID、tenant、owner、space、deleted 操作符映射为逻辑 `SystemField`; +- 将时间操作符基于一次订阅内冻结的 `Clock.instant()` 展开为半开区间; +- 对 List、Map 和字节值进行防御性复制,消除 `Any` 的可变性; +- 验证 projection、sort、pagination、limit 和 Native payload 形态; +- 标注用户条件来源,不混入授权条件。 + +Normalizer 产物不能包含 BSON、Elasticsearch `Query`、`_id`、`.keyword`、索引名或集合名。 + +### 4.4 QueryPolicy + +`QueryPolicy` 是响应式授权扩展端口,只返回: + +```text +Deny(reason) +Allow( + mandatoryCondition: NormalizedCondition, + fieldConstraint, + resultConstraint, + analyticsConstraint +) +``` + +Authority 必须来自经过认证的 `QueryExecutionContext`,不能直接把 Header、请求参数或任意 Reactor key 当作可信身份。Policy 只接收已经规范化的 query,并且只能通过 typed policy builder 产生 `NormalizedCondition`;不得把 wire `Condition`、`Any`、`RAW` 或 Backend Native 条件直接注入 policy decision。 + +用户条件与 `mandatoryCondition` 分开保存 provenance。Planner 必须分别对两者执行同一套字段 schema、operator、capability 和预算校验,通过后才在最终 Plan 中强制执行外层 `AND`。Filter、Backend 和 Native 查询都不能移除或覆盖该条件。 + +`analyticsConstraint` 约束可使用的 dimension、metric、having、bucket order、最大桶数以及最小桶文档数。 +最小桶文档数属于带 provenance 的强制聚合条件,必须在 Backend 内执行,不能只在响应序列化阶段删除小桶, +否则 cursor、桶数或相邻查询仍可能泄露受保护信息。 + +`LEGACY` 模式下无 authority 的内部调用是一个需要显式迁移的兼容事实,不能默认等价为系统权限。 + +### 4.5 Planner + +Planner 是框架内部确定性实现,输入为 normalized query、policy decision、逻辑字段 schema 和 validation mode,输出 `QueryPlan`。它负责: + +- operation 与 typed/dynamic result shape; +- projection、稳定排序、limit/page 与一致性要求; +- 字段 capability 和语义层级; +- Native Backend binding; +- 预算评估和兼容问题报告。 + +相同语义输入必须产生相同 Plan。deadline、当前时间、执行模式、租户凭据和动态预算不写入语义 Plan,而放在 `QueryExecutionContext/QueryExecutionOptions` 中,避免破坏计划比较、缓存和 shadow 稳定性。 + +### 4.6 QueryBackend + +Backend 只接收完整、已验证的 Plan: + +```kotlin +interface QueryBackend { + fun single(plan: SingleQueryPlan, options: QueryExecutionOptions): Mono> + fun stream(plan: StreamQueryPlan, options: QueryExecutionOptions): Flux> + fun page(plan: PageQueryPlan, options: QueryExecutionOptions): Mono> + fun count(plan: CountQueryPlan, options: QueryExecutionOptions): Mono +} + +interface AnalyticsQueryBackend { + fun analyze( + plan: AnalyticsQueryPlan, + options: QueryExecutionOptions + ): Mono +} +``` + +分析能力使用独立 Backend capability interface,避免强迫只支持记录查询的 Backend 实现空方法或在运行期 +抛出通用异常。Planner/Router 在执行前根据 capability 拒绝不支持的 operation。 + +Backend module 负责: + +- 逻辑字段到物理字段/multi-field/nested path 的映射; +- Plan 到驱动查询对象的编译; +- driver/cursor/PIT 生命周期; +- timeout、failed shard、total relation、缺失 source 等完整性校验; +- 返回独立 identity 与 payload,不直接物化 typed API 对象。 + +Backend 不再校验 wire DTO、不追加授权条件、不决定公共 projection/分页语义,也不恢复异常为成功。 + +### 4.7 Result Materializer 与结果策略 + +`ResultMaterializer` 在 Backend record 与公共结果之间隔离存储格式: + +- typed 查询需要完整信封; +- dynamic 查询可投影 payload,但 identity 由独立 metadata 承载; +- 物理 `_id`、索引名和内部字段不得泄漏; +- mandatory result policy 和 masking 在 materialization 后执行; +- partial `Flux` 必须以 error 终止,不能因为错误处理器返回 empty 而伪装完整成功。 + +不为所有结果增加通用 `QueryResult` 包装。内部 page 使用带 total relation 与 consistency 的 `BackendPage/PlannedPage`,兼容 adapter 再映射为现有 `PagedList`。 + +分析结果使用独立、深度不可变的 `AnalyticsPage`:bucket key、metric value、cursor 和 completeness +均为显式值对象。它不伪装成实体列表,不复用可变 `DynamicDocument`,也不把 bucket count 填入 +`PagedList.total`。桶总数默认不计算;未来若调用方显式请求,必须作为单独高成本 capability 预算和执行。 + +## 5. 核心模型 + +### 5.1 QueryInvocation 与执行上下文 + +`QueryInvocation` 显式表达: + +- materialized aggregate; +- document kind:`SNAPSHOT` / `EVENT_STREAM`; +- result shape:`TYPED` / `DYNAMIC` / `COUNT` / `ANALYTICS`; +- operation:`SINGLE` / `STREAM` / `PAGE` / `COUNT` / `ANALYZE`; +- 原始兼容 Query DTO。 + +`QueryExecutionContext` 显式携带可信 authority、purpose、deadline、execution mode、validation mode 和 budget。它不是任意 attribute map。 + +### 5.2 NormalizedCondition + +```text +All +Junction(AND | OR | NOR, children) +Predicate(field, operator, immutableValue, options) +ElementMatch(field, childScopeCondition) +Search(scope, text) +Native(backendId, immutableUtf8Json) +``` + +逻辑字段分为: + +```text +SystemField(IDENTITY | AGGREGATE_ID | TENANT_ID | OWNER_ID | SPACE_ID | DELETED) +Path(segments, basis = ROOT | CURRENT_ELEMENT) +``` + +嵌套元素中的 `id` 是元素相对字段,不得被错误映射为根文档 `_id`。`RAW` 只有显式 Backend binding、可信 capability 和不可变 JSON 才能进入 Plan;旧 `Bson/Query/Any` 只留在 legacy passthrough。 + +### 5.3 QueryPlan + +Plan 使用 sealed operation: + +```text +SingleQueryPlan +StreamQueryPlan(limit = Bounded | Unbounded) +PageQueryPlan(offset: Long, size, totalMode, consistency) +CountQueryPlan +AnalyticsQueryPlan +``` + +所有 Plan 共享 `QueryTarget`、用户条件与 mandatory condition 的最终外层合取、 +`RequiredCapabilities` 和 `SemanticTier`。记录型 Plan 另外共享: + +- `PlannedProjection`; +- `PlannedSort`; + +`AnalyticsQueryPlan` 不复用记录 projection/sort/pagination,改为显式包含 dimension、metric、having、 +bucket order、bucket window/cursor、missing policy、numeric policy、required consistency 和 +required completeness。预算与 deadline 仍属于 `QueryExecutionOptions`,不写入可比较的语义 Plan。 + +Plan 禁止包含: + +- `Any`、BSON 或 Elasticsearch driver 对象; +- 物理字段、集合、索引或 alias; +- mixed include/exclude; +- 未验证的负分页、Int offset 或溢出值; +- 未绑定 Backend 的 Native 条件。 + +### 5.4 逻辑字段 Schema 与 Backend binding + +`QueryFieldSchema` 只描述逻辑合同:字段类型、允许的 operator、exact/range/full-text/literal-pattern/ +sort/projection/aggregation/nested 能力及 null/missing/array 模型。 + +MongoDB/Elasticsearch 各自持有 `BackendFieldBinding`:物理字段、keyword/text multi-field、nested mapping、索引限制和 mapping version。Backend 启动校验 binding/mapping 是否满足逻辑 schema,并记录 capability digest;逻辑 schema 不直接负责生成某个 Backend 的 mapping。 + +#### 5.4.1 字符串能力模型 + +客户端和 Plan 只引用 `state.name` 这样的逻辑字段,不允许引用 `.keyword`、`.exact`、analyzer、normalizer +或 Elasticsearch field type。字符串不是单一 `STRING` capability,而是按查询语义组合: + +| 逻辑能力 | 允许的操作 | Elasticsearch 典型 binding | MongoDB 基准 | +|---|---|---|---| +| `EXACT` | `EQ/NE/IN/NOT_IN/ALL_IN` | 未分析的 `keyword` field | 原字段精确值比较 | +| `PRESENCE` | `NULL/NOT_NULL/EXISTS` | `exists` 与显式 null/missing binding | 按已声明 null/missing 模型判断 | +| `FULL_TEXT` | `MATCH` / `Search` | `text` field | 已声明 text index/search scope | +| `LITERAL_PATTERN` | `CONTAINS/STARTS_WITH/ENDS_WITH` | `keyword` 或专用 `wildcard` field | 转义用户值后的字面量 regex | +| `SORTABLE` | record sort、稳定 tie-breaker | 启用 `doc_values` 的 exact field | 原字段排序与显式 collation | +| `AGGREGATABLE` | analytics dimension | 启用 `doc_values` 的 exact field | 原字段 `$group` key | +| `PROJECTABLE` | include/exclude/materialization | `_source` 中的 source field | 原文档字段 | + +`MATCH` 不再伪装成普通字段 predicate,而规范化为 `Search(scope, text)`。`scope` 是逻辑搜索范围:MongoDB +binding 指向一个已声明的 text index,Elasticsearch binding 指向一个或多个 `text` field。MongoDB text index +不能满足任意单字段 scope 时,该 scope 在 MongoDB 上为 unsupported;`SEARCH` tier 不承诺跨 Backend analyzer、 +分词、相关性或排序等价。 + +现有 wire `Condition.MATCH(field, value)` 保持 JSON/DSL 兼容,由 Normalizer 把 `field` 解析为 schema 已声明的 +legacy search scope。找不到 scope 或 Backend 无法满足字段范围时,planned path 稳定拒绝;`COMPATIBLE` 模式 +可以带 reason/metric 回退 legacy,但不得在 planned path 中悄悄扩大为 MongoDB 全 text index。 + +首批 `PORTABLE` 字面量字符串合同只承诺 case-sensitive。`ignoreCase=true` 只有在 schema 声明大小写模型、 +MongoDB/Elasticsearch binding 具有等价 normalization/collation 且 shared TCK 证明 Unicode/边界行为后才开放; +否则稳定返回 `UnsupportedFeature`,不能把 Elasticsearch `case_insensitive` 当作 MongoDB `i` regex 的无证据 +替代。 + +字符串范围比较只在 schema 显式声明有序字符串、normalization 和 collation 时允许。否则 +`GT/LT/GTE/LTE/BETWEEN` 对字符串稳定拒绝,避免把 Backend 默认字典序误认为公共合同。 + +#### 5.4.2 Elasticsearch 字符串 binding + +需要全文检索和精确能力的同一逻辑字段使用 multi-field,但 sub-field 名称仍是 Backend 私有实现,例如: + +```text +logical field: state.name + +ElasticsearchStringBinding( + sourceField = state.name, + searchField = state.name, + exactField = state.name.exact, + literalField = state.name.exact, + sortField = state.name.exact, + groupField = state.name.exact +) +``` + +Planner 只产生 required capability;Elasticsearch compiler 必须按 binding 精确选择字段: + +- `EQ/IN` 及其否定、排序和分组只使用 exact/sort/group field;不得对 analyzed `text` 执行 `term`; +- `MATCH` 只使用 search field;不得自动回退到 keyword; +- `CONTAINS/STARTS_WITH/ENDS_WITH` 只使用 literal field,并转义 `\\`、`*`、`?` 等模式字符;不得用 + `match_phrase` 替代字面量子串; +- projection/materialization 始终读取 logical source field,不返回或泄漏 multi-field; +- 缺少所需 binding、mapping 类型不符或 `doc_values` 不满足时,在执行前返回 `UnsupportedFeature` 或 + readiness failure,不按字段名猜测 `.keyword`。 + +推荐 mapping 由字段用途决定,而不是使用一个全局 dynamic string template: + +| 字段用途 | 推荐 mapping | +|---|---| +| ID、code、status、tag | `keyword` | +| name/title,既检索又精确排序/分组 | `text` + exact `keyword` multi-field | +| description/content,仅全文检索 | `text` | +| 高频 grep-like、前导 wildcard 的机器生成大文本 | 经 benchmark 后声明专用 `wildcard` literal field | + +`ignore_above` 是完整性边界,不只是 mapping 参数。超过限制的值仍可能存在于 `_source`,却不进入 exact +field,导致精确查询、排序和聚合静默漏数。只有满足以下全部条件,Backend 才能发布对应 exact/sort/group +capability: + +1. logical schema 声明并在写入端执行可索引的字符/UTF-8 byte 长度约束; +2. `ignore_above`、Lucene term 限制与该约束一致; +3. readiness 对现有 generation 完成 mapping 和数据审计,不存在会被忽略的历史值; +4. 新增 multi-field 后已重建/回填历史文档;只更新 template/mapping 不算完成。 + +旧索引的动态 `.keyword` 只有通过上述 readiness 后才能临时绑定。默认动态 mapping、Snapshot/EventStream +不同 template 或字段名恰好存在都不能成为 capability 证据。 + +#### 5.4.3 其他字段类型 + +- numeric、instant/date、boolean 使用对应逻辑类型与物理 binding,不通过字符串 multi-field 模拟; +- object 本身不能参与 exact/range/sort/group,必须选择有 schema 的叶子字段; +- array 沿用元素类型能力,并显式声明 missing/null/empty/multi-valued 模型; +- `ELEM_MATCH` 需要 element-relative schema;Elasticsearch 必须具有对应 `nestedPath`,普通 object array + mapping 不能冒充 nested capability。 + +Mongo baseline 下 `null` 是查询语义而不是普通字段类型:`EQ null`/`IN [..., null]` 包含 missing,`NE null` +只匹配存在且非 null,`NOT_IN` 仍包含 missing;详见 MongoDB 官方 +[null/missing 查询](https://www.mongodb.com/docs/manual/tutorial/query-for-null-fields/) 与 +[`$nin`](https://www.mongodb.com/docs/manual/reference/operator/query/nin/) 合同。因此 exact 集合允许 canonical +`Null` operand,由 Compiler 按 presence/nullability 明确展开;range/BETWEEN 一律拒绝 `Null`,不得借 BSON type +order 冒充 portable 数值/时间范围语义。 + +## 6. 语义、分页与完整性 + +### 6.1 语义层级 + +| 层级 | 合同 | 路由与对比 | +|---|---|---| +| `PORTABLE` | 以 MongoDB 行为为基准的精确查询语义 | 可跨 Backend 路由和等价 shadow | +| `SEARCH` | `MATCH` 等全文能力 | 要求 `FULL_TEXT` capability;只比较结构与错误,不承诺分词等价 | +| `NATIVE` | `RAW` 等 Backend 原生能力 | 必须绑定 Backend;禁止自动跨 Backend 路由和等价判断 | + +`CONTAINS`、`STARTS_WITH`、`ENDS_WITH` 是字面量字符串合同,不允许 Elasticsearch 用 analyzed `match_phrase` 代替。 + +### 6.2 Projection + +- strict typed 查询只接受 `Projection.ALL`; +- compatible typed projection 只有在 legacy Backend 能直接承载 typed result 且证明投影映射等价时才允许 fallback。P2-C 的 + immutable `BackendRecord` 过渡 seam 尚不具备该证明,因此 non-`ALL` typed projection 在两种 validation mode 都稳定拒绝, + 禁止先裁剪动态文档再伪装成完整 typed result; +- dynamic 查询允许 include 或 exclude,二者同时非空必须拒绝; +- Backend 永远独立返回 identity,最终 logical projection 不泄漏物理字段。 + +### 6.3 排序与分页 + +- `index >= 1`、`size > 0`; +- offset 使用 `Math.multiplyExact((index - 1).toLong(), size.toLong())`; +- strict page 必须具有唯一稳定排序,缺少 identity 时由 Planner 追加逻辑 identity; +- offset page 只支持预算允许的窗口,深分页后续使用独立 cursor 协议; +- page 是 Backend 的单个 SPI 操作,返回 total relation 与 consistency;Executor 不允许静默降低一致性。 + +MongoDB `SAME_INPUT` 使用一条 aggregation 观察流:匹配记录后追加不读取集合的 `$documents` sentinel, +再由 `$setWindowFields` 同时计算记录位置与 total,最后分别输出当前页记录和一条 total row。该结构不二次读取集合, +也不把整页记录封装进单个 BSON document;records 与 total 因而来自同一 observed input。它不把 MongoDB +`local` read concern 冒充 point-in-time snapshot;若未来声明 `SNAPSHOT`,仍必须具有匹配的 read concern/transaction +能力。窗口排序、`allowDiskUse`、stage 内存与目标数据规模仍需 explain/profile 签署。Elasticsearch exact page 必须 +校验 `total.relation == Eq`。 + +### 6.4 List limit + +现有 `limit=0` 继续表示 `Unbounded`,不能静默映射为 Elasticsearch 的固定 result window。策略只能显式允许或返回 `BudgetExceeded`。Elasticsearch planned backend 使用 PIT + `search_after`,并在 complete/error/cancel 时关闭 PIT。 + +### 6.5 完整性与错误 + +公共错误模型至少区分: + +- `InvalidQuery`; +- `InvalidCursor`; +- `AccessDenied`; +- `UnsupportedFeature`; +- `BudgetExceeded`; +- `BackendUnavailable`; +- `BackendTimeout`; +- `IncompleteResult`; +- `MappingFailure`。 + +Elasticsearch Backend 必须拒绝 timeout、failed shards、缺失 `_source` 和要求 exact 时的非 `Eq` total。MongoDB 与 Elasticsearch 的 driver exception 统一映射,但根因保留为 cause。 + +### 6.6 分析型聚合合同 + +分析请求先规范化为独立模型: + +```text +AnalyticsQueryPlan( + target, + preFilter = userCondition AND mandatoryCondition, + grouping: Global | By(NonEmptyList), + metrics: NonEmptyList, + having: AnalyticsCondition = All, + bucketOrder, + bucketWindow: First | After(cursor), + missingPolicy, + numericPolicy, + requiredConsistency, + requiredCompleteness, + requiredCapabilities +) +``` + +- `preFilter` 只引用 read-model 字段,在分组前执行; +- `having` 使用独立 AST,只能引用 dimension 或 metric alias,不能混入 `NormalizedCondition`、`RAW` + 或物理字段; +- `Global` 表示无 `GROUP BY` 的全局统计,只产生一个 global bucket;`By` 才要求至少一个 dimension; +- dimension/metric alias 在一次请求中唯一,并在 Planner 阶段绑定逻辑 schema; +- bucket order 是分析结果合同,不等价于记录 `sort`; +- cursor 是不透明值,绑定 plan digest、target、稳定 key order 和 Backend paging state;调用方不能自行拼装; +- 成功结果的 completeness 为 `Exact`,或在调用方显式允许近似时为 + `Approximate(errorBound?, warnings)`;要求 exact 时任何近似、timeout 或分片失败都返回 + `IncompleteResult`。 + +结果模型: + +```text +AnalyticsPage( + buckets: List)>, + nextCursor: AnalyticsCursor?, + consistency: Eventual | Snapshot, + completeness: Exact | Approximate, + warnings +) +``` + +`Global` grouping 最多返回一个 bucket 且没有 next cursor。`By` grouping 的 `nextCursor == null` 只表示本次 +查询已无更多桶,不承诺已计算桶总数。exact bucket pagination 只能按唯一、稳定的 dimension key 顺序进行; +按 metric 排序的全局 top-N 不是同一能力。 + +`completeness` 与 `consistency` 正交:`Exact` 表示每次请求没有已知近似或部分结果,不表示多次 cursor +请求观察到同一数据快照。`Snapshot` 才承诺跨页输入集合固定;Backend 不具备该能力时必须在执行前拒绝, +不能用 `Eventual` 静默替代。 + +第一批 `PORTABLE` 合同收敛为: + +- 单个 `QueryTarget`,优先只开放 `SNAPSHOT`; +- 支持无 dimension 的 global aggregation;分组时只允许根级、单值、非 nested 的 scalar dimension,支持单 key 和复合 key; +- `DOCUMENT_COUNT`;`MIN/MAX` 用于 schema 已声明的 numeric/instant 字段,`SUM/AVG` 仅用于 numeric 字段; + 只要 metric 涉及 numeric,就必须提供显式 precision、scale、type promotion、rounding 与 overflow policy; +- `EXCLUDE` 或 `AS_NULL_BUCKET` missing policy;后者按 MongoDB 基准把 missing 与显式 null 合为同一桶; +- 按 dimension key 的 exact cursor 分页;首批顺序固定为 binary collation、null-first,cursor key 使用 dimension + 输出类型的 canonical value,不复用 predicate numeric widening;第一批跨 Backend 合同只承诺 `Eventual` + consistency; +- numeric promotion 首批固定为 `Decimal128`,precision 为 `1..34`;`Global` bucket window canonicalize 为 + `limit=1`,调用方传入的更大 limit 不改变 Plan 或 fingerprint。 + +以下能力先标记为 unsupported,而不是由 Backend 猜测或降级: + +- array/multi-valued、nested、自动 unwind 和跨文档 join; +- metric-sorted paging、全局 top-N、cardinality、percentile、pipeline metric; +- portable `having`;模型先保留,但首个 Elasticsearch exact path 只接受 `All`; +- locale collation、calendar interval/timezone; +- 无精度策略的 `Decimal128`、超过安全表示范围的整数与浮点精确相等比较。 + +`EVENT_STREAM` 的统计粒度必须显式定义。现有 QueryTarget 的一个文档表示一个 +`DomainEventStream`,`ANALYZE` 不会隐式展开其中的 event array。按单个 domain event 统计应先投影到专用 +read model;待 EventStream grain 和 mapping fixture 独立验证后再声明相应 capability。 + +### 6.7 MongoDB 基准与 Elasticsearch 编译策略 + +MongoDB planned backend 是 portable aggregation 的语义基准。基本 pipeline 顺序为: + +```text +$match(userCondition AND mandatoryCondition) + -> $group(dimensions, metrics) + -> $match(mandatoryHaving AND userHaving) + -> keyset cursor predicate + -> $sort(stable dimension key) + -> $limit(bucketLimit + 1) +``` + +首批 portable path 的 `userHaving=All`。`allowDiskUse`、`maxTimeMS`、扫描文档预算、最大 dimension/metric +数量、最大候选桶数和返回桶数必须由 options/policy 显式控制。`$group` 是 blocking stage;是否允许落盘是 +部署策略,不是 Backend 可以自行开启的透明优化。只有未来显式请求 bucket total 时才考虑 `$facet/$count`, +且必须单独评估内存、文档大小和重复扫描成本。 + +首批 Mongo analytics cursor 只声明 `Eventual` consistency。除非后续证明并封装可跨 HTTP 请求安全恢复、 +有界保留且可释放的 snapshot/session 机制,否则 `requiredConsistency=Snapshot` 必须返回 +`UnsupportedFeature`,不能因为单次 aggregation 命令内部读取一致就宣称跨页快照一致。 + +Elasticsearch exact portable path 使用 `composite` aggregation: + +- dimension 绑定到满足 exact/doc-values 的字段; +- 使用响应返回的 `after_key` 生成 cursor,不能从最后一个 bucket key 自行推导; +- `requiredConsistency=Snapshot` 时使用 PIT;cursor 保存每次响应返回的最新 PIT id。无 next cursor 的 + terminal page 立即关闭,error/cancel 尽力关闭,调用方停止翻页时由短 keep-alive/expiry 有界回收; +- timeout、failed shards、无法解析的 `after_key` 或 metric 精度不满足 Plan 时失败关闭; +- `terms` aggregation 的 doc count/sub-aggregation 可能近似,只能进入显式允许近似的非 portable capability, + 不能冒充 MongoDB exact 结果; +- `composite` 不能满足首批 portable 的 metric-sorted 全局分页或 pipeline `having`,因此 Planner 直接拒绝。 +- policy 要求 `minBucketDocumentCount` 时等价于 mandatory having;首批 composite path 不支持时必须拒绝或 + 路由到满足能力的 Backend,不能在返回后过滤。 + +MongoDB 与 Elasticsearch 共享同一组 fixtures/TCK,至少比较 bucket key、顺序、metric type/value、 +missing/null、cursor replay、completeness 和错误类别。仅比较 JSON 形状不构成语义等价证据。 + +### 6.8 聚合安全、预算与可观测性 + +- mandatory pre-filter 必须在分组前进入最终 Plan,Native payload 不能绕开; +- Policy 分别约束 filter field、dimension field、metric field、having alias、bucket order 和输出 metric; +- 强制最小桶文档数在 Backend 内执行,并保留 policy provenance; +- admission 限制 dimension/metric/having 节点、alias、cursor 和 payload 大小; +- execution budget 至少覆盖 deadline、扫描文档数、候选桶数、返回桶数、内存/落盘策略和跨页次数; +- cursor 首页把完整 execution budget ceiling 写入签名 lease;continuation 可逐项收紧但不得删除或放宽 ceiling, + 因此 security-context digest 可以保持身份/用途/模式绑定而不阻止合法收紧; +- 指标记录 operation、target、semantic tier、capability、bucket 数、扫描/执行耗时、completeness、fallback + 原因和 budget rejection;不得把 dimension key 或 metric 原值写入低基数标签/普通日志。 + +## 7. 兼容与发布模式 + +执行策略与语义校验是两个独立维度: + +```text +QueryExecutionMode = LEGACY | SHADOW | PLANNED +QueryValidationMode = COMPATIBLE | STRICT +``` + +这样可以表达 `SHADOW + STRICT`,也可以在 `PLANNED + COMPATIBLE` 下对无法规范化的历史请求显式 fallback。禁止散落 `strictEnabled`、`shadowEnabled` 等布尔开关。 + +SHADOW 始终返回 legacy 结果,比较 planned 的: + +- 错误类别; +- identity 集合与顺序; +- exact/lower-bound total; +- null/missing/array 行为; +- 完整性与延迟。 + +现有系统没有 legacy analytics path,因此 `ANALYZE` 不适用 `LEGACY/SHADOW`,只能在 `PLANNED` 下执行。 +公开发布前使用 TCK、离线 fixtures 和内部 dual-backend probe 比较 bucket key/order、metric type/value、 +cursor termination 与 completeness;发布后若需要非 serving Backend 对比,应增加独立的 backend comparison +option,而不是伪造 legacy 结果。近似查询只记录误差元数据和结构差异,不能据此宣称与 MongoDB exact 等价。 + +`SEARCH/NATIVE` 只记录差异,不判定跨 Backend 等价。任何 fallback 都必须产生原因和指标,不能默默发生。 + +兼容阶段不能静默修改: + +- Query DTO JSON/OpenAPI、七个 QueryService 方法或聚合 Bean 名; +- `limit=0` 语义; +- legacy 排序、typed projection、NoOp 和 HTTP status; +- 自动索引迁移、alias 切换或旧索引删除。 + +## 8. 生命周期与模块归属 + +### 8.1 生命周期 + +- `QueryGateway`、Normalizer、Planner、Policy chain 和 Backend registry 为单例、无请求级可变状态; +- `QueryInvocation`、normalized query、Plan 和 execution context 每订阅创建; +- Backend registry 是 planned Backend 实例的唯一生命周期 owner,key 至少包含 materialized aggregate、document kind 和 backend id; +- 旧 Factory 的缓存行为在兼容期保留,但不再叠加 Provider/Gateway cache; +- record cursor、analytics cursor、PIT/session 默认是每次执行资源;需要跨请求的 PIT/session 只能通过 + 有有效期的 cursor lease 显式转移所有权,并在 terminal page/error/cancel 时尽力释放,在调用方停止翻页时 + 依靠短 expiry 有界回收; +- analytics cursor 必须有版本、有效期和 plan/target binding;升级后无法安全恢复时返回明确的 cursor 错误, + 不能从错误位置继续。 + +### 8.2 模块 + +| 模块 | 职责 | +|---|---| +| `wow-api` | 现有 wire DTO、JSON/OpenAPI 和 DSL 合同;稳定后 additive 引入 analytics wire contract | +| `wow-query` | Gateway、Plan/Normalizer/Planner、Policy、错误、analytics model、experimental Backend SPI | +| `wow-spring` | 现有聚合 QueryService 兼容 adapter 与 Bean name/generic 注册 | +| `wow-spring-boot-starter` | 默认装配、Backend registry/routing 和模式配置 | +| `wow-webflux` | HTTP、authority、request/response adapter;显式依赖 `wow-query` | +| `wow-mongo` | Mongo compiler、binding、record/analytics Backend、materializer | +| `wow-elasticsearch` | ES compiler、mapping 校验、PIT record/composite analytics Backend、materializer | +| `wow-tck` | Mongo 基准 fixtures、计划 golden、record/analytics 双 Backend 语义对比 | + +Backend SPI 稳定前放在 experimental package 并增加兼容测试,不新建 Gradle module。 + +## 9. Elasticsearch 索引生命周期 + +逻辑名保持稳定 alias,物理索引使用 mapping version 与 generation: + +```text +wow...snapshot-v0002-000001 +wow...es-v0002-000001 +``` + +mapping `_meta` 保存 mapping version、document kind、schema contract id 和由 binding 规范编码生成的 capability digest。 +应用启动不执行 reindex、alias 切换或删除;`CREATE` 也只能由显式运维命令触发。回填与 alias 切换是独立运维流程: + +1. 固化 logical schema 与 backend binding; +2. 创建 template 和新物理索引; +3. Snapshot 从权威事件流重建; +4. EventStream 排空 writer 或启用受控镜像写; +5. 校验 count、identity、版本连续性和 checksum; +6. 执行 SHADOW; +7. 原子切换 alias; +8. 在回滚窗口保留旧索引与 legacy backend。 + +若切换后没有向旧 EventStream 索引镜像新增写入,alias 不能直接回切。Snapshot 回滚同样需要以权威事件流重建和版本校验为准。 + +## 10. 分阶段实施计划 + +### 10.1 当前基线与完成度 + +以下状态以 stacked query-service 分支的当前源码和测试为审计基线。状态只根据可执行代码与验证结果判断, +不根据设计意图推断: + +| Phase | 当前证据 | 状态 | +|---|---|---| +| 0 | `QueryHandler` 已形成单一 defer/fail-closed 边界;EventStream factory 已使用 materialized key 与并发缓存;对应同步、异步、partial Flux、cancel、direct handle 和多订阅测试已存在 | 已实现,待 PR #2908 合并 | +| 1 | P1-A 已引入 internal `QueryInvocation`、语义代数、`QueryPlan` 与最小 analytics model;P1-B 已引入单遍 admission snapshot、全局 value/payload budget、typed rejection 与 43 operator Normalizer;P1-C 已引入 logical schema、provenance-bearing Planner、record/analytics Plan 与 canonical fingerprint;Backend compiler 与运行时接线由后续 Phase 独立实现 | 已实现,stacked Draft PR #2909/#2910/#2915 | +| 2 | P2-A 已引入 typed execution context 与 fail-closed Policy;P2-B 已引入 per-subscription Gateway、Executor、legacy lowering/attestation、绝对 deadline、typed error boundary 与受控 SHADOW;P2-C 已把 framework-managed Service factory、Registrar、Handler/WebFlux transport 接到 Gateway,并保留 legacy wiring rollback 与原七方法 ABI;公共 Query/Analytics/Cursor 与完整 Query status OpenAPI 扩展已经审批并由 golden 锁定 | P2-A/P2-B/P2-C 已实现 | +| 3 | P3-A 已提升最小 experimental Backend contract,并实现 Snapshot Mongo binding/compiler/backend/materializer、text-index readiness、storage-routed composition、per-operation profile 与真实 Mongo SHADOW 对比;P3-B 已增加 Snapshot/EventStream PAGE 单操作、独立 system/identity/deletion binding、分库 Spring route、完整 budget envelope、Mongo deadline/allowDiskUse 执行与真实 explain fixture;P3-C 已增加 Snapshot `ANALYZE` Backend contract/adapter、Mongo Global/By pipeline、numeric/missing/cursor 映射及真实 Mongo fixture。高基数 explain/concurrency/cancel、跨 Backend Analytics TCK,以及同一真实集合上的 `SHADOW -> PLANNED -> LEGACY` count 演练已完成;精确 scanned-record enforcement、生产性能签署与 unbounded stream 尚未完成 | P3-A/P3-B/P3-C vertical slice 与仓库级回滚演练已实现;Phase 3 生产性能 exit gate 未完成 | +| 4 | P4-A 已引入显式 Elasticsearch source/exact/presence/search/literal/sort/group/nested binding、mapping-version/readiness 与 analyzer/normalizer/keyword 完整性证明;P4-B 已实现 Snapshot SINGLE/bounded STREAM/PAGE/COUNT compiler、mapper、完整性/error validator、Spring storage-routed contribution和真实 Elasticsearch fixture,PAGE 已使用 PIT + search_after 并通过 >10k 实库回归,PIT expiry/cancel/closed-transport fault 均有真实 client 回归;P4-C 已实现 EVENTUAL/EXACT 的 global/grouped DocumentCount composite、服务端 after_key replay、`EXCLUDE/AS_NULL_BUCKET`,并通过 Mongo/Elasticsearch shared TCK;P5-A 现已为 grouped SNAPSHOT Analytics 接入跨请求 PIT state/lifecycle。同一真实索引上的 `SHADOW -> PLANNED -> LEGACY` count 演练已完成;unbounded、EventStream 与 Decimal128 metric 未完成 | P4-A、P4-B 与 P4-C exact-count vertical slice、仓库级回滚演练已实现;SNAPSHOT cursor 由 P5-A 闭环;Phase 4 exit gate 未完成 | +| 5 | P5-A 已实现 internal cursor envelope/codec/lease manager,并由公共 Analytics Gateway 使用经批准的持久化 store SPI;MongoDB 已提供显式初始化、固定容量 slot、unique lease id、TTL grace 与 revision CAS 的跨节点 store,真实 Mongo fixture 已证明跨实例翻页、单次消费和 HMAC key rotation。Elasticsearch grouped SNAPSHOT Analytics 已把最新 PIT id 作为 opaque Backend state 交给 target/backend keyed lease coordinator,覆盖 continuation、terminal/error/cancel、capacity failure、expired reaper 与 PIT expiry `IncompleteResult`;公共 token 不含 PIT。Starter 已提供默认关闭、显式配置、单 lifecycle owner 的有界串行 reaper。P5-B 已实现 internal migration manifest、版本/generation 命名、不可变 inventory/attestation/verification、CAS command state、幂等恢复、component/index template、显式 create、source↔destination write-alias 原子切换,以及基于独立 hidden system index 与 `_seq_no/_primary_term` 的持久化 migration repository;Snapshot/EventStream rebuild 与 verification 已形成 vertical slice。公开 Analytics contract 已 additive 落地;目标应用 probe/cutover、EventStream controlled mirror/生产 barrier 接线和旧同名 concrete index 转 managed alias尚未完成 | P5-A persistent cursor、自动 reaper 与 Elasticsearch SNAPSHOT PIT lifecycle、P5-C 公共契约已实现;P5-B vertical slice 已实现;Phase 5 exit gate 未完成 | + +关闭但未合并的 PR #2903(`agent/unify-mongo-elasticsearch-query-semantics`)包含 +`ConditionValidator`、operator fixtures 和部分双 Backend 测试。它作为 Phase 1/3 的需求与测试素材使用, +不整体 cherry-pick:其中直接修改 legacy converter/mapping 的实现必须重新落到 Normalizer、logical schema、 +Backend binding 和 shared TCK 的新边界中,避免重新形成双重语义源。 + +### 10.2 依赖关系与合并策略 + +```mermaid +flowchart LR + P0[Phase 0
Execution Correctness] --> P1A[P1-A
Semantic Kernel] + P1A --> P1B[P1-B
Admission + Normalizer] + P1B --> P1C[P1-C
Schema + Planner] + P1C --> P2A[P2-A
Context + Policy] + P2A --> P2B[P2-B
Gateway + Legacy Backend] + P2B --> P2C[P2-C
Spring/WebFlux Cutover] + P2C --> P3A[P3-A
Mongo Snapshot Record] + P3A --> P3B[P3-B
Mongo Event/Page] + P3B --> P3C[P3-C
Mongo Analytics Baseline] + P3B --> P4A[P4-A
ES Readiness] + P4A --> P4B[P4-B
ES Record] + P3C --> P4C[P4-C
ES Analytics] + P4B --> P4C + P4C --> P5A[P5-A
Cursor Lease] + P5A --> P5B[P5-B
Index Lifecycle] + P5B --> P5C[P5-C
Public Analytics API] + P5C --> P5D[P5-D
Major Cleanup] +``` + +实施与合并规则: + +1. 每个 slice 单独 PR;一个 PR 只跨越完成该 slice 必需的模块。 +2. 前置 PR 未合并时可以建立 stacked branch,但前置 PR 合并后必须 rebase 到 `main` 并重新跑本 slice 全部门槛。 +3. 纯模型、接线、Backend 行为、公开协议和数据迁移分开审查;不得用“大重构”一次切换全部边界。 +4. 每个 PR 同步更新本节的基线提交、状态和证据;“代码存在”不等于 slice 完成。 +5. 前一 slice 的 exit gate 未通过时,不让后一 slice 接管生产流量;可以并行只读研究或编写未接线的 fixture。 +6. 新依赖、新 Gradle module、公开 ABI、OpenAPI/JSON 和自动数据变更继续服从仓库的显式确认边界。 + +### 10.3 Phase 0:执行正确性基础 + +交付范围: + +- `QueryHandler` 每次订阅创建独立 context; +- 同步与异步 Backend 错误统一进入 error observer,错误处理器不能把查询恢复为成功; +- 覆盖 partial Flux、cancellation、direct `handle`、结果未订阅和多订阅隔离; +- EventStream legacy Factory 使用 materialized aggregate key 和并发安全缓存; +- 不切换 Spring Bean,不新增 Gateway/Provider cache,不宣称已形成安全边界。 + +Exit gate:`wow-query` 聚焦测试与 PR required checks 全绿;现有七方法和 Bean 行为无变化。回滚只需回退 +Phase 0 提交,不涉及配置、索引或数据。 + +### 10.4 Phase 1:纯语义模型 + +#### P1-A:Semantic Kernel + +- 仅在 `wow-query` 的 `internal` package 以 Kotlin `internal` visibility 引入 `QueryTarget`、 + `QueryInvocation`、operation/result shape、 + execution/validation mode、无 `Any` 的 `NormalizedValue/NormalizedCondition`、record/analytics plan skeleton; +- analytics grouping 建模为 `Global | By(NonEmptyList)`,支持合法的无 `GROUP BY` 全局统计; +- `QueryInvocation` 是每订阅创建、不得比较或缓存的临时 envelope;其中 legacy wire DTO 保持原样,深度不可变 + 从 admitted snapshot/normalized query 边界开始; +- normalized condition/value、analytics model 与 Plan 的所有 collection/map/bytes 在边界防御复制,不能用 + Kotlin read-only interface 冒充深度不可变; +- Kotlin `internal` 类型在 JVM 字节码中仍可能表现为 public class;它们位于 internal package 且不进入受支持 + API,兼容性声明不得表述为“JAR 严格零 ABI diff”; +- 不加 Jackson/Swagger 注解,不修改 `wow-api`、`QueryService` 或 `QueryType`,不发布 cursor codec。 + +#### P1-B:Admission 与 Normalizer + +- 实现内部具体 `RawAdmissionGuard` 与 `QueryNormalizer`,不先开放 SPI; +- admission 必须在一次有界遍历中完成校验与防御性物化,产出 immutable admitted snapshot;Normalizer + 只能读取该 snapshot,禁止校验后再次读取调用方的动态 getter、`Any`、List、Map 或 ByteArray; +- 固化深度/节点/字段/value/options 大小,递归 `AND/OR/NOR`、`ELEM_MATCH` 相对字段、system field、 + projection、sort、limit 和 page 规则; +- 每次 normalization 只读取一次 `Clock.instant()`;时间范围使用半开区间; +- 从 PR #2903 搬运有效 operator fixtures/validator cases,但不复用其 Backend-specific converter 修改; +- 返回稳定 category/path/code 的 typed rejection,测试不绑定异常文案。 + +P1-B 当前实现约束: + +- `QueryAdmissionLimits` 同时限制局部容器和整次 admission 的 condition/value node、UTF-8 payload、数字精度; + 默认值仍是 internal safety baseline,P2-A 接入运行时时再通过配置与真实流量证据校准; +- legacy `RAW` 不携带 Backend id,admission 不读取、不预算并丢弃原 driver object,只产出 `NativeUnbound` marker, + Normalizer 稳定返回 `UNSUPPORTED_FEATURE/NATIVE_BACKEND_UNBOUND`;未来已绑定 immutable JSON Native contract + 才进入 payload budget,留 P1-C/P5-A; +- mixed include/exclude 在 Normalizer 保留为 `NormalizedProjection.Mixed`,不在缺失 validation mode 时提前决定 + compatible/strict policy;P1-C Planner 根据 result shape、validation mode 和 compatibility issue 决策; +- Normalizer 以 `NormalizedDeletionScope` 保留 legacy 根条件是否显式声明删除范围;Planner 对未显式声明的 record/count + 查询加入 logical `DELETED=false`,该框架默认条件不受 user field allow-list 影响。显式 `deleted(ALL)` 保持 `All`, + final execution request 禁止再次经过会隐式追加 `ACTIVE` 的 legacy guard; +- `RawAdmissionGuard`、`RawValueSnapshotter`、`AdmissionBudget` 与 `QueryNormalizer` 均保持 internal,生产调用链、 + Spring Bean、现有 wire DTO 和 OpenAPI 不接线。 + +P1-B 验证证据: + +- admission tests 覆盖动态 getter 单读、one-shot iterable、List/Map/ByteArray 防御复制、condition/value cycle、 + hostile duplicate-key Map、局部与累计预算、稳定 key path、分页 Long offset 和 typed rejection; +- Normalizer golden tests 覆盖全部 43 个 wire operator、Mongo 空集合常量、数字 canonicalization、system field、 + 多层 `ELEM_MATCH` 相对 scope、literal pattern、projection/sort、一次 Clock、DST-safe 半开时间范围和 RAW 拒绝; +- `./gradlew :wow-query:check` 与 OpenAPI snapshot test 是该切片的退出验证;P1-C 完成前不宣称 Phase 1 exit gate + 整体满足。 + +#### P1-C:Logical Schema 与 Planner + +- 引入 `QueryFieldSchema`、logical capability、`RequiredCapabilities`、`SemanticTier`、plan fingerprint; +- 固化 `EXACT/FULL_TEXT/LITERAL_PATTERN/SORTABLE/AGGREGATABLE/PROJECTABLE` 字符串能力和 typed + `SearchScope`;Plan/公开 DTO 不得包含 `.keyword`、analyzer 或 Backend field type; +- `PlanningConstraints` 作为未来 Policy decision 的内部承接模型; +- user/mandatory condition 分别校验并保留 provenance,最终只能形成不可拆除的外层 `AND`; +- strict page 自动追加 logical identity 稳定排序;`limit=0 -> Unbounded`;offset 使用 Long 和 + `Math.multiplyExact`; +- analytics 首批能力和 unsupported 集合在 Planner 稳定拒绝;Phase 1 cursor 只固定 decoded semantic state, + token codec、签名和 expiry 留到 P5-A。 + +P1-C 当前实现约束: + +- `QuerySchemaRegistry` 按完整 `QueryTarget(context, aggregate, documentKind)` 精确注册 immutable + `QueryDocumentSchema`;schema 只包含 canonical logical field/type/operator/capability、null/missing/array 语义和 + typed `SearchScope`,logical alias 在 schema 内解析为 canonical system/path field,nested search scope 必须绑定 + 最近的 `ELEMENT_MATCH` owner,并生成与注册顺序无关的 `SchemaContractId`; +- Planner 输出 `Planned(QueryPlan)` 或 `LegacyFallback(issues, validatedMandatory)`;fallback 只能由 user-side + compatibility gap 触发,必须携带已验证 mandatory proof,mandatory schema/capability/Native 失败始终 fail closed; +- `EnforcedFilter` 分别保存 user/mandatory provenance,并固定生成不可拆除的外层 `AND`;Plan 中的 predicate、 + projection、sort 和 analytics dimension/metric 全部引用 canonical `QueryFieldId`,不再保留相对 path; +- version 1 plan fingerprint 使用显式 length-prefixed canonical encoder 与 SHA-256;包含 schema contract、target、 + operation/result shape、provenance、condition、projection/sort origin、page/limit、capability、tier 和 analytics + 语义,保留 condition/list/sort/object entry order,对 set/map canonical 排序;不包含 validation/execution mode、 + deadline、动态 planning limit、物理 binding 或 analytics cursor position;fingerprint 是 immutable Plan 内容的 + 派生值,构造方不能注入与 Plan 不一致的 digest; +- 首批 analytics 只计划 `SNAPSHOT`、portable pre-filter、`Global` 或根级单值 scalar dimension、 + `DOCUMENT_COUNT`、numeric `MIN/MAX/SUM/AVG` 与 instant `MIN/MAX`、dimension-key ascending、`Eventual + Exact` + 和 decoded exact keyset cursor;排序显式固定 binary/null-first,numeric policy 固定 Decimal128 precision 1..34, + `Global limit=1`;array/nested、having、metric order、Snapshot consistency、Approximate completeness、无 numeric + policy、超精度 policy 及错误 cursor binding 稳定 typed reject,不进入 legacy fallback; +- P1-C 仍只位于 `wow-query/internal`,不接 Gateway、Backend compiler、Spring Bean、现有 wire DTO、OpenAPI、 + cursor token codec、签名或 expiry。 + +Phase 1 exit gate: + +- golden tests 覆盖 invocation matrix、深度不可变、递归 scope、一次 Clock、projection、limit/page overflow、 + mandatory provenance、global/grouped analytics、string operator-to-capability、`SearchScope`、禁止物理字段泄漏、 + alias/capability 与 cursor binding; +- 增加 public compatibility guard,固定 `QueryService` 七方法和现有 `QueryType` 常量; +- `./gradlew :wow-query:check` 通过; +- `./gradlew :wow-openapi:test --tests "me.ahoo.wow.openapi.snapshot.OpenApiCompatibilitySnapshotTest"` 通过; +- 生产调用链、Spring Bean 和 wire DTO 零变化。回滚为删除 additive internal model,不需要运行时开关。 + +### 10.5 Phase 2:Policy、Gateway 与 legacy Backend adapter + +#### P2-A:Trusted Context 与 Policy + +- 建立 typed `QueryExecutionContext`、authority resolver 和响应式 `QueryPolicy`; +- policy builder 只能产生 normalized mandatory condition/field/result/analytics constraint; +- authority 缺失、跨租户、policy error 和 mandatory schema/capability failure 均 fail closed; +- 明确 LEGACY 内部无 authority 调用的精确 grant 与迁移负责人,禁止默认提升为 system authority。 + +P2-A 当前实现约束: + +- `QueryExecutionContextFactory` 在每次 subscription 内解析 authority;provider 同步异常、异步异常、empty、过期 + deadline 和不匹配的 legacy grant 都形成 typed rejection,不捕获创建时身份,也不 fail open;legacy grant 由受信 + adapter 固定,并精确绑定 `callerId + target + purpose + resourceScope`,调用方不能在单次 request 中自报 caller; +- route/header 中的 tenant/owner/space 只建模为 `QueryResourceScope` selector;`TenantIsolationQueryPolicy` 必须先与 + authenticated subject 或 tenant-scoped service authority 比较,匹配后才转为 mandatory condition;System authority + 必须显式提供 justification;Subject 的 owner/space grant 即使 selector 缺失也必须形成 mandatory condition 或拒绝, + 不能静默扩大到同 tenant 全量数据; +- `QueryPolicyAllowance.Builder` 默认拒绝全部 user field/search/native 访问,只能返回 normalized mandatory condition、 + 分维度 field/search/native constraint 和静态 result/stream/page/analytics constraint;mandatory Native 在 builder + 边界直接拒绝; +- Planner 在 user condition、projection、sort、search scope、native backend、analytics dimension/metric 上分别执行 + access constraint,且 access denial 不被 `COMPATIBLE` fallback 吞掉;mandatory condition 使用独立 provenance 和 + schema/capability 校验,不受 user allow-list 限制; +- typed compatible projection 的字段访问检查发生在 legacy fallback 决策前,避免 fallback 绕过 projection policy; + `MaximumRecords` 对 unbounded stream 和 oversized page fail closed,不做 clamp; +- 本 slice 仍全部位于 `wow-query/internal`,未注册 Spring Bean、未接管 `QueryHandler`/public factory、未增加公开 + error code 或 OpenAPI schema;跨 module experimental SPI、公共错误映射和生产入口接线分别留给 P2-B/P2-C。 + +P2-A exit gate: + +- 测试覆盖 authority 每订阅解析、sync/async/empty error、deadline 跨越、精确 legacy grant、Policy deny/empty/error、 + tenant/owner/space mismatch 与 selector 缺失、mandatory Native、约束深度不可变、受限 projection All/Exclude,以及 + filter/projection/sort/search/native/analytics 各 Planner 访问维度; +- `./gradlew :wow-query:check`、public compatibility guard 和 OpenAPI snapshot 通过; +- 生产调用链保持不变,回滚为删除 additive internal context/policy/constraint 模型并恢复 Planner 的默认 unrestricted + 参数,不需要运行时开关。 + +#### P2-B:Gateway、Executor 与 legacy Backend + +- 实现单一 `QueryGateway`、Executor、Backend registry/router、legacy Backend adapter 和内部 typed 错误边界; +- 跨 module 必需的 Gateway/错误类型在 P2-C 接线前使用受控 experimental opt-in 和 ABI guard; + Normalizer/Planner 仍保持 internal,公共 HTTP 错误映射与 status golden 同 P2-C 一起闭环; +- Publisher 生命周期覆盖同步、异步、partial Flux、cancel,并复用 Phase 0 已证明的错误边界; +- registry 成为 planned Backend 唯一 owner;Gateway/Provider 不再增加聚合级缓存; +- legacy adapter 只接收 Gateway 已最终确定的 immutable execution request;mandatory condition 无法无损执行时 + 必须 fail closed,不能退回原始 DTO; +- 默认仍不接管现有 Bean,先以内部 probe 验证 plan 与 legacy adapter。 + +P2-B 当前实现约束: + +- `QueryGateway` 的 `single/stream/page/count/analyze` 保持 operation-specific `Mono/Flux` cardinality;整个 + admission、normalization、authority、policy、planning、routing、Backend 和 observer 链在 subscription 内创建, + 同一个 Publisher 再订阅会重新创建 invocation 与可信上下文,不缓存 request、Plan、结果或聚合 service; +- subscription 开始后立即在同步 budget 边界内冻结 legacy wire DTO,再进入异步 authority/policy;调用方随后修改 + condition children、List、Map 或 ByteArray 不得改变该次执行。`LegacyCompilationInput` 只携带 immutable + `NormalizedQueryInvocation + QueryDocumentSchema + PlanningDecision`,类型边界不包含 `QueryInvocation`、wire + `Condition` 或 `Any`; +- `QueryBackendRegistry` 按 `QueryTarget + BackendId` 唯一注册,default route 与 Native-pinned route 均执行 + schema contract、operation、semantic tier、field/search/native capability 精确校验;planned route 缺失或运行期失败 + 永不触发 legacy fallback,只有 Planner 显式 `LegacyFallback` 可以选择兼容路径。`Planned/Shadow` route 只持有 + immutable registry 与 Plan,Backend registration 必须在每次订阅时由 registry 解析,不能直接注入 registration + 绕过唯一 owner; +- mode matrix 固定为:`LEGACY` 对 record planned/fallback 均走 legacy,`SHADOW` 只对 planned request 创建受控 probe、 + fallback 明确记录 shadow skipped,`PLANNED` 对 planned request 走 planned Backend、仅 `COMPATIBLE` fallback 走 + legacy;所有 mode 的 `LegacyFallback` 都通过独立 decision observer 上报 target/operation/mode/reasons,不能只在 + `SHADOW` 中可见。任意 `STRICT + LegacyFallback` 是内部不变量错误。Analytics 只允许 `PLANNED`,不进入 + legacy/shadow; +- `LegacyQueryCompiler` 与 `LegacyQueryBackend` 由同一个 typed binding 封装,compiler 每订阅只接收 final + immutable input,并校验 target/operation/result shape/schema contract。compiled query 必须携带绑定本次 input 的唯一 + token,并由 trusted compiler 显式 attestation 已 lowering framework deletion scope 与 validated mandatory condition; + attestation 不是对物理 Backend AST 的独立证明,生产 compiler 注册前必须通过 golden/TCK,证明物理查询同时包含 + user condition、default-active deletion 与 mandatory condition,且 unsupported Search/Native 稳定拒绝。registry 只接受 + final、受控 factory 创建的 erased binding,禁止自定义 binding 绕过 attestation。mandatory 无法无损 lowering 固定为 + `ACCESS_DENIED/$.constraints.mandatoryCondition/MANDATORY_CONDITION_UNENFORCEABLE`。P2-B 不提供跨 Backend + 通用 lowering:unbound RAW 继续在 Normalizer 前置拒绝,Search/Native 只有未来 target-specific trusted compiler + 能证明等价时才可开放,禁止原 driver object/DTO passthrough; +- planned Backend 返回独立 identity、immutable document、total relation、achieved consistency 与 completeness; + `BackendRecord.completeness` 无默认值。planned single/stream/page 对 unknown record、bounded stream max+1、非 exact + total、非 same-input page、page 实际条数与 `min(size, max(0, total-offset))` 不一致均 fail closed;analytics 同样校验 + bucket limit、alias shape、cursor arity 以及 achieved consistency/completeness 不弱于 Plan。legacy 可以显式返回 + `UNKNOWN` provenance,但不能冒充 planned exact success;page/count/analyze 的 empty Publisher 与任意 mode 的负 count + 均视为 incomplete result; +- deadline 使用 subscription 时计算的单个绝对 timer,覆盖 authority、policy 和持续出数的 Backend Flux;timer 到期 + cancel 上游并返回 `BUDGET_EXCEEDED/$.executionContext.deadline/DEADLINE_EXPIRED`,不会像逐项 `timeout` 那样被 + 每个 onNext 重置;Mono 使用单次绝对 timeout,不通过 `Flux.next()` 把正常完成误判为 cancel。SHADOW 的 planned + registry resolve、Backend error 和 deadline 全在受 supervisor 管理的 cold probe 内,planned readiness/error 不得阻断 + legacy primary;probe deadline 固定为 `min(request deadline, configured shadow cap)`,request 未提供 deadline 时仍由 + shadow cap 终止并 cancel probe。supervisor 同时接收 target/operation/tier、planned publisher 与 typed primary + value/terminal signal;submit 必须返回 `Accepted(handle)` 或 `Rejected(typed issue)`,disabled/overload/reject 与同步异常 + 都由独立 decision observer 以上报 `target/fingerprint/tier/operation + stable rejection` 的健康事件,事件不暴露可执行 + Publisher,不能静默丢失,也不能改变 legacy primary。`SEARCH/NATIVE` tier 只记录差异,不得由 supervisor 判定跨 + Backend 等价;fallback 与 unbounded stream 明确 + 上报 skipped。observer 每订阅独立计数并在 complete/error/cancel 仅终结一次,observer/shadow callback 失败不得替换 + primary 结果; +- 本 slice 的 Gateway/Backend/legacy contracts 仍在 `wow-query/internal`,不注册 Spring Bean、不改变受支持 + `QueryService` 七方法、JSON/OpenAPI 或 HTTP status;P2-C 提升最小跨 module facade 并接线所有框架托管入口。 + +P2-B exit gate: + +- tests 覆盖 route matrix、registry duplicate/missing/schema/operation/capability、legacy compiler cold/mismatch/ + mandatory fail-closed、Gateway cold 与多订阅、wire TOCTOU、sync/async error、partial Flux、cancel、authority/backend + absolute deadline、observer failure、RAW 拒绝、shadow primary 隔离及 result completeness; +- `./gradlew :wow-query:check`、public compatibility guard 和 OpenAPI snapshot 通过; +- 默认生产 Bean 和 transport 调用链保持不变,回滚为删除 additive internal execution package 与 deletion provenance + 接线,不涉及配置、索引或数据。 + +#### P2-C:Spring/WebFlux/进程内接线 + +- WebFlux、`SnapshotQueryService`、`EventStreamQueryService` legacy adapter 和新进程内调用最终全部委托 Gateway; +- 保持 Bean name、generic injection、JSON/OpenAPI、HTTP status、NoOp 和七方法 ABI; +- 默认 `LEGACY + COMPATIBLE`,fallback 带 reason/metric;`SHADOW + STRICT` 可独立配置; +- 兼容 request Filter 必须在 admission/policy 前执行;policy 后不得再替换 query。result Filter 只能在 mandatory + result policy 后执行更严格的 masking; +- 证明 Filter 不能绕过 mandatory condition;兼容 Filter 只作为 adapter hook,不再作为安全边界; +- 当前 legacy storage 的 typed 方法已经完成 `MaterializedSnapshot`/`DomainEventStream` 物化,而 P2-B Gateway + 返回带 identity/completeness 的 immutable logical document。P2-C 禁止通过 `Any`、unchecked generic cast、原 wire DTO + passthrough 或“typed -> JSON -> typed”重复序列化桥接这两个边界;必须先让 Gateway 的 legacy leaf 调用 storage + dynamic 方法,再由 target-bound materializer 统一产生 dynamic/typed public result。 + +P2-C 拆成三个可独立回滚的实现切片: + +##### P2-C1:Core facade、legacy lowering 与 materializer + +- 只提升跨 module 必需的 experimental facade、trusted authority context 和公共 typed error;Plan、Normalizer、schema、 + registry、`BackendRecord`、attestation token 继续保持 internal; +- facade 只暴露 operation-specific `Mono/Flux`,record path 以 immutable logical document 为唯一中间结果。dynamic + result 从它生成新的独立 `DynamicDocument`,typed result 由 exact `QueryTarget + result Class` 绑定的唯一 materializer + 生成;materializer 运行在 Gateway 的 absolute deadline、error boundary 与 lifecycle observer 内。materializer 缺失、 + 类型不匹配或 mapping 失败均 fail closed,不能回调 storage typed 方法绕过 Gateway; +- legacy compiler 只接收 `LegacyCompilationInput`,为 single/stream/page/count 新建 wire DTO。Snapshot 的最终 + condition 显式为 `user AND deletion AND mandatory`:`DEFAULT_ACTIVE` lowering 为 direct + `DELETED=ACTIVE`,`EXPLICIT` 追加 direct `DELETED=ALL` sentinel 以中和旧 `DeleteConditionGuard`,同时保留用户原 + deletion predicate。EventStream 没有 snapshot deletion 字段,Planner 不合成 default-active,legacy lowering 也不注入 + deletion 条件; +- compiler 必须按 target-specific dialect 处理 `ELEM_MATCH` 子字段相对/绝对路径和 legacy `MATCH` scope;未注册 dialect、 + `RAW/Native` 或不能证明等价的条件稳定拒绝。`None` 在 legacy leaf 本地短路,不能依赖 Backend 对空集合的偶然行为; +- storage dynamic projection 必须额外取 identity;raw document 先通过一次有界 snapshot,再从 frozen value 读取 identity, + 禁止先无界复制或对动态 getter/entries 做两次观察。之后按原 projection 深层恢复 Include/Exclude/Mixed 并移除内部补取 + 字段;output snapshot 有独立 value/payload budget,源 `Map/List/ByteArray` 后续变化不能影响结果; +- runtime 只接受独立类型 `QueryRawServiceSource`,不能接收与 application facade 相同的 public factory 类型。P2-C1 公开 + budget 在 P2-C1 首次只开放可证明执行的 `maxReturnedRecords`;P3-B 已扩展完整 envelope。Planner 仍只消费自身可证明 + 的 result/page 约束,其余 budget 必须由最终 Backend 显式执行或在 storage 前稳定拒绝,不能形成 fail-open 承诺; +- 首个 facade schema 只声明 target-independent system fields。未知 user path 在 `COMPATIBLE` 下形成显式 legacy + fallback,`STRICT` 下拒绝;不得伪造尚未存在的 production field/search capability。 + +##### P2-C2:Raw storage registry 与 Spring facade + +- `SnapshotQueryServiceFactoryBinding`/`EventStreamQueryServiceFactoryBinding` 继续持有 Mongo/Elasticsearch/custom raw + factory;新增不同类型的 raw registry,按 materialized aggregate/document kind 精确解析并拒绝重复 binding; +- 依赖方向固定为 `storage binding -> raw registry -> Gateway runtime -> @Primary facade factory -> Registrar/Tail`。 + Gateway 不能注入同一个 public facade factory,raw registry 也不能注册 facade,避免递归与双 `@Primary`; +- Spring 不再注册 `RoutingSnapshotQueryServiceFactory`/`RoutingEventStreamQueryServiceFactory` 作为 application Bean; + raw registry 直接保留解析后的 factory route,并且只在每个 Gateway target 初始化 legacy leaf 时调用对应 raw factory。 + 内建 Mongo/Elasticsearch route 同时解析固定 dialect;未声明 storage 的 custom binding 必须显式贡献 + `QueryLegacyDialectResolver`,不能猜测 Backend 语义; +- Registrar 保留现有 aggregate Bean name、`ResolvableType` 与七方法;framework-managed public factory、aggregate Bean、 + `QueryHandler.handle` 和七个 convenience 方法均只能得到 Gateway facade。手工构造的 Mongo/Elasticsearch concrete + service/factory 仍是受信 Backend/TCK 边界,P2-C 不破坏其构造器 ABI,也不宣称 JVM 内物理不可绕过; +- NoOp 下沉为显式 raw route,empty single/list、empty page、count 0 仍经过 admission/policy/Gateway;Gateway/facade + 不缓存 raw executor、Plan、DTO 或结果。为保持既有 factory identity contract,application factory 只缓存轻量 target facade; + 现有 raw factory cache 仍是 legacy Backend service 的唯一 owner; +- 七方法缺少显式 context 参数,facade 因此在每次 subscription 调用 `QueryCallResolver`,并校验 exact target。P2-C2 + 的默认 resolver/authority 均为空且 fail closed,不把进程内调用默认提升为 System;P2-C3 再从 trusted transport marker + 或 exact legacy grant 解析 call/authority; +- 自定义 factory 不再通过 `@ConditionalOnMissingBean` 被猜测为 application facade;必须显式注册 raw binding,保留一版 + migration adapter 和启动期诊断。 +- `QueryGatewayRuntime` 是 Spring 组合的唯一 customization owner:resolver/configuration 与每 runtime 的 authority capability + 在构造时冻结,公开 facade factory 只允许零参获取。仅注册自定义 `QueryGateway` 的部分覆盖会启动失败;需要替换执行栈时必须 + 提供完整 runtime,避免 direct Gateway 与 facade 使用不同实例。 + +##### P2-C3:Trusted transport、Filter 分相与公共错误 + +- WebFlux 只把 authenticated application context 转成 trusted authority;path/header/CoSec tenant/owner/space 只能形成 + frozen `QueryResourceScope` selector,不能建立 principal;missing/empty/error authority 与 selector mismatch 均在 storage + publisher 前 fail closed; +- POST query route 与两个 GET load route 都必须写入同一 typed transport marker。存在 transport marker 时绝不降级为 + legacy grant;进程内兼容调用只允许预注册且精确匹配 `caller + target + purpose + mode + resourceScope` 的 legacy grant; +- legacy request Filter 只能在 admission 前重写隔离 query,不得设置/替换 result;policy 后 result extension 改为一入一出、 + 不可替换 Publisher/identity/cardinality/total/cursor 的 masker。未声明 phase 的第三方旧 Filter 在 Gateway wiring 下启动失败; +- internal rejection 在 facade 边界映射为稳定公共 Query error,复用 `DefaultErrorInfo` envelope,不新增 JSON 字段、不泄漏 + policy/backend cause。锁定 400/403/408/429/502/503/504/500 status matrix;JSON Flux partial error 与 SSE 已提交 + 200 后的 transport 语义单独 golden,不把 Publisher 正确性误报为 HTTP 已闭环; +- 保留一个版本的显式 legacy wiring 回滚开关并记录告警/指标;授权、policy、mandatory、schema、lowering 或 mapping 失败 + 绝不能自动切回旧链。 + +P2-C3 已实现合同: + +- `QueryWebTransportResolvers` 以一个原子 trusted resolver 同时解析 call 与 authority。所有 Snapshot/EventStream 的 + single/list/page/count route 与两个 GET load route 都写入同一种 Reactor transport marker;marker 固化 exact target、 + `QueryType`、purpose 与 tenant/owner/space selector。默认 `QueryWebAuthorityResolver` 返回 empty 并 fail closed,应用必须从 + 已认证的 principal/security context 贡献 authority,禁止从 path/header/CoSec selector 反推身份; +- `CompositeQueryTrustedContextResolver` 按 Spring order 对 `QueryTrustedContextRequest` 原子解析 call 与 authority;同一次 + facade subscription 选中的 resolver 必须同时给出两者,禁止 A resolver 的 call 与 B resolver 的 authority 混合。解析出的 + authority 只通过每个 `QueryGatewayRuntime` 独有的对象 capability 在受控 Reactor context 中交给 Gateway;resolver 在 runtime + 构造时冻结,公开 factory 不接受替换 resolver,同名字符串或另一 runtime/channel 的对象都不能伪造。Web marker 存在但 authority 缺失时稳定返回 + `ACCESS_DENIED / $.executionContext.authority / AUTHORITY_REQUIRED`,即使 Reactor context 同时存在 legacy caller marker 也不降级; +- 进程内迁移使用 `QueryLegacyContextResolver` 的预注册 `QueryLegacyGrant`。caller marker 只选择固定 grant,不能修改 + `target + purpose + executionMode + resourceScope`;不存在或不精确匹配稳定返回 + `ACCESS_DENIED / $.executionContext.legacyGrant / LEGACY_CALLER_NOT_ALLOWED`,不提升为 System; +- request chain 只运行显式实现 `PreAdmissionQueryFilter` 的 Filter,随后丢弃任何提前写入的 result,再进入唯一 Gateway tail。 + policy 后只运行内建 masking Filter;一入一出 mapper 不能绕过 Gateway source 或改变 cardinality/page envelope,dynamic masker + 不能新增/改写 `id/aggregateId/tenantId/ownerId/spaceId`。未声明 phase 的第三方 `QueryFilter` 启动失败。旧 `AbacQueryFilter` + 仅以 deprecated pre-admission 兼容桥保留,不再被视为安全边界;授权必须迁移为 mandatory policy constraint; +- `QueryExecutionException` 继续复用 `DefaultErrorInfo` 与 `bindingErrors(name=path,msg=code)`。functional/global WebFlux 使用同一 + 映射:Invalid/Cursor/Unsupported=400,AccessDenied=403,deadline=408,其他 BudgetExceeded=429,Incomplete=502, + BackendUnavailable=503,BackendTimeout=504,Mapping/Internal=500;不增加 OpenAPI error schema 字段。P2-C3 只锁定运行时 + 映射。经公共 Query/Analytics/Cursor 契约升级审批后,全部 query/load operation 已统一声明 + 400/403/408/429/502/503/504/500 response,并继续复用 `DefaultErrorInfo` 与 `Wow-Error-Code`;该 additive OpenAPI diff + 已由 route contract test 与更新后的 compatibility snapshot 锁定; +- 一版紧急回滚属性为 `wow.query.gateway.legacy-wiring-rollback=true`。它显式停用 Gateway facade,并让 application factory + 直接委托独立 raw registry;启动持续记录 warning,计数器 `wow.query.gateway.legacy.wiring.rollback` 增加一次。该开关不由 + 任何运行时错误自动触发,不恢复旧的双 `@Primary` routing factory,也不改变 raw factory cache owner;启用期间 admission、 + policy 与生命周期保护均被绕过,只能作为限时迁移措施;属性值不是精确 `true/false` 时启动失败,禁止 typo 静默绕过。 + +P2-C exit gate: + +- core golden 覆盖 deletion 五态、portable condition、两级 `ELEM_MATCH` dialect、`MATCH`/`RAW`、identity 补取与 projection + 恢复、dynamic/typed materialization、结果深度不可变、NoOp、fallback reason 和每订阅 cold 行为; +- Spring context 覆盖 no-storage/Mongo/Elasticsearch/mixed/custom/duplicate binding,证明唯一 application factory、raw registry + 无 facade、无循环、Bean name/generic injection 与七方法 ABI 不变; +- 公开入口矩阵覆盖 aggregate Service、public factory、`QueryHandler.handle`、七 convenience 方法与全部 WebFlux routes, + storage probe 证明每 subscription 只在 Gateway 后触达一次; +- 安全矩阵覆盖无 authority、provider empty/error、跨 tenant/owner/space、伪造 Header、Filter 替换 query/result、Native、 + mandatory lowering failure,全部 storage 零调用; +- `:wow-query:check`、`:wow-spring:check`、`:wow-spring-boot-starter:check`、`:wow-webflux:check`、Java/reflection ABI + guard 与经审批更新的 OpenAPI snapshot 通过;全部 Query/Analytics/load route 的运行时 status response 已由统一组件与矩阵测试 + 锁定。P2-C1/P2-C2/ + P2-C3 均保持独立 commit/PR 或可单独 revert 的 commit 边界。 + +Phase 2 exit gate:P2-C 三个切片全部完成后,所有 framework-managed 公开入口调用链必须经过 Gateway;安全测试覆盖 +直接 Service 调用、HTTP、Filter 重写、Native 条件、无 authority 和跨租户;Spring context/Java compatibility/OpenAPI +全绿。任意代码手工构造 concrete storage service 或直接使用 driver 属于受信 Backend 边界,只有下一 major 收窄 public +constructor/SPI 才能物理禁止。运行时回滚使用 `LEGACY` execution mode;如果 Gateway wiring 本身失败,显式 legacy +wiring rollback 只允许在一个迁移版本内启用并持续记录告警/指标。 + +### 10.6 Phase 3:MongoDB planned path 与语义基准 + +#### P3-A:Snapshot record vertical slice + +- 将 Backend 必需的 Plan/value/record 与 SPI 从 Phase 1 internal model 最小化提升到受控 experimental opt-in; + Normalizer、Planner、policy implementation 不对 Backend module 公开; +- 实现 `MongoFieldBinding`、record plan compiler、`MongoQueryBackend` 和 materializer; +- `MongoFieldBinding` 分别声明 value path、text search scope 与 collation;`MATCH` 只能进入已声明 text index, + literal string 必须转义用户值,`ignoreCase` 在 shared TCK 完成前保持 unsupported; +- compiler 只接收 validated Plan,禁止调用 wire/legacy converter 或接收 RAW/Bson; +- 先支持 Snapshot 的 single、bounded stream、count,再扩展 page; +- logical identity、tenant/owner/deleted、projection 和 stable sort 只从 binding 解析物理字段。 + +P3-A 当前实现约束: + +- `LEGACY` target 不执行 planned readiness I/O;只有配置了非 `LEGACY` record operation 的 target 才按精确 storage route + 选择 planned source,不能以 Mongo Bean 存在推断所有 target 使用 Mongo; +- `SHADOW` 遇到已配置但未 ready 的 binding 时继续返回 legacy primary,并通过受控 probe 上报 + `BACKEND_UNAVAILABLE/$.backend/BACKEND_NOT_READY`;`PLANNED` 在启动阶段 fail closed; +- P3-A 的最小 contribution 只声明 `SINGLE`、`COUNT` 和 `BOUNDED_ONLY` stream;Native/RAW、`ignoreCase` 和无法 + exact 编码的 Decimal/Instant 在 driver I/O 前稳定拒绝; +- text search 仅接受 binding 精确声明、readiness 验证通过的 collection-wide root text index,并要求 simple/binary + collation;一个 ready contribution 必须覆盖 logical schema 的全部字段,user path 不得与 framework system path + 冲突,collection namespace 必须与 binding 精确一致; +- materializer 对 Mongo source 只做一次有界冻结,再仅从 binding 重建 logical document;projection 内部补取 identity + 后重新应用 logical projection,不能泄漏 `_id`、未声明的顶层物理字段或被排除的 identity; +- bounded shadow supervisor 有独立上限,planned registry resolve、deadline 和 Backend error 均留在 cold probe 内, + readiness、overload 或 planned failure 不得替换 legacy primary。 + +#### P3-B:EventStream、page、一致性与预算 + +- 增加 EventStream record binding,保持一个 document 等于一个 `DomainEventStream`; +- page 是 Backend 单操作;Mongo `SAME_INPUT` 使用单个 matched input、内存 sentinel 与 window accumulator, + `SNAPSHOT` 只有 read concern 能力满足时开放; +- unbounded stream、deadline、cancel、driver error、mapping failure 和资源释放全部显式; +- 预算覆盖扫描、offset、返回记录、stage、内存/落盘;性能结论必须有 integration fixture/explain。 + +P3-B 当前实现约束: + +- experimental record SPI 已增加 `BackendPageQueryPlan` 与 immutable `BackendPage`;Mongo PAGE 使用单次 + `$match` 输入,借助不指定 collection 的 `$unionWith + $documents` sentinel 和 `$setWindowFields` 同时计算 + position/total,再分别输出记录与 total row;它不二次读取 collection、不构造 page-sized BSON array,返回 + `EXACT` total 和 `SAME_INPUT`,但不把 `local` read concern 冒充 point-in-time snapshot; +- Snapshot 与 EventStream 共用 validated compiler/backend,但 binding 分别固定 system path:Snapshot identity/ + aggregateId 为 `_id` 且包含 deleted;EventStream identity 为 `_id -> id`、aggregateId 保留独立字段且禁止 deleted; +- Snapshot/EventStream planned source 按既有 storage route 分别绑定 snapshot database/event-stream database,未选中 + Mongo 的 target 不读取 Mongo database/readiness; +- PAGE、SINGLE、bounded STREAM、COUNT 共享 mandatory/projection/sort/value 编译与 immutable mapper;deadline、cancel、 + max-returned 及 Backend error 继续由 Gateway/Executor 的绝对生命周期边界统一执行; +- `QueryExecutionBudget`/`QueryBackendExecutionOptions` 已完整携带 scan/return/page/bucket/cursor/disk budget; + `maxReturnedRecords` 与 `maxPageWindow` 在 Planner 和 Mongo Backend 双重校验,高级 budget 在 legacy storage 前拒绝; + Mongo 对 find/aggregate/count 使用绝对 deadline 派生的 `maxTime`,find/page 显式设置 `allowDiskUse`; +- Mongo 目前不能以单次普通查询精确限制 `totalDocsExamined`,因此 `maxScannedRecords` 在 driver I/O 前返回 unsupported, + 不以 result limit 或预跑第二次 explain 冒充 scan budget。真实 Testcontainers explain fixture 已证明代表性 tenant/deleted/ + identity PAGE 走 `IXSCAN` 且无 `COLLSCAN`;空结果、越界页和两条各 8 MiB 记录的 page fixture 证明 total sentinel + 与 16 MiB BSON document 边界不依赖 page array,但这些证据仍不是生产数据分布的性能签署; +- unbounded stream 仍保持 `BACKEND_OPERATION_UNSUPPORTED`。精确 scanned-record enforcement 与目标应用 explain/profile + 阈值签署完成前,生产 `PLANNED` 必须继续使用 operation-scoped rollout。 + +#### P3-C:Mongo analytics baseline + +- 首批只开放 Snapshot、`Global` 或根级单/复合 scalar dimension、`DOCUMENT_COUNT`、 + `MIN/MAX/SUM/AVG` 的显式 numeric policy、`EXCLUDE/AS_NULL_BUCKET` 和 key-order cursor; +- pipeline 强制 `$match(user AND mandatory)` 在 `$group` 前,mandatory having 在 group 后; +- 默认不计算 bucket total;`Eventual` 是首批跨页 consistency,`Snapshot` capability 未证明前稳定拒绝; +- keyset cursor 不能掩盖每页重跑 `$group` 的成本,必须由扫描/跨页预算和真实 explain 约束。 + +P3-C 当前实现约束: + +- experimental Backend contract 已增加深度不可变的 `BackendAnalyticsQueryPlan`、bucket/page result 与独立 + `AnalyticsQueryBackend`;internal adapter 只做 validated Plan/options/result 的类型转换,不向 Mongo 暴露 + Normalizer、Planner、wire DTO 或 legacy converter; +- Snapshot Mongo contribution 原子声明 `ANALYZE` 并注册 analytics Backend;EventStream contribution 明确不声明 + `ANALYZE`,避免把一个 `DomainEventStream` document 误报为 domain-event 粒度; +- Mongo compiler 只接受 `PORTABLE + EVENTUAL + EXACT + having=All`,Global 强制 limit=1 且无 cursor;By 只接受 + root scalar dimension、binary collation、null-first 与 dimension-key ascending;所有 dimension/metric 必须从同一 + logical binding 证明 `AGGREGATABLE`,cursor key 继续按 dimension canonical type 校验; +- pipeline 固定为 `$match(user AND mandatory)`、dimension missing filter、`$group`、keyset cursor、稳定 key sort、 + `limit+1`;`EXCLUDE` 在 group 前排除 missing/null,`AS_NULL_BUCKET` 使用 `$ifNull` 合并二者。Global 空输入由 + Backend 合成为一个 bucket:document count/sum 为零,min/max/avg 为 null; +- numeric metric 使用显式 Decimal128 policy;`SUM/AVG` 通过 `$toDecimal` 聚合,所有 numeric metric 在 materialize + 时按 rounding/scale/precision 校验,超出 policy 或 BSON 可表示范围返回 mapping failure,不静默截断; +- analytics Backend 对 deadline 派生 `maxTime`,显式传递 `allowDiskUse`,并在 I/O 前双检最大返回桶数;当前 Mongo + 单次 aggregation 无法精确执行 `maxScannedRecords`、`maxCandidateBuckets` 或 `maxCursorPages`,这些 budget 稳定 + unsupported,不以结果 limit 或额外预查询冒充; +- unit/golden 已覆盖 enforced filter、Global/复合 By pipeline、missing policy、canonical cursor、Decimal128 rounding、 + Global 空桶、预算零 I/O 与 Snapshot/EventStream 注册矩阵;Testcontainers 已覆盖 mandatory tenant/deleted、 + Global count/min/max/sum/avg、空输入、missing/null bucket、257 个高基数桶的完整 key cursor replay、跨页并发插入的 + `EVENTUAL` 可见性、Decimal128 aggregation overflow 的 fail-closed mapping,以及 2,000 documents 高基数 + aggregation `executionStats` 使用声明的 tenant/deleted/group index 且无 `COLLSCAN`。目标应用 profile/阈值签署、 + Mongo aggregate Publisher 的剩余 deadline→`maxTime` 与 cancel 传播也已有 driver-probe 回归。目标应用生产 profile/ + 阈值签署与真实服务端 timeout/kill 场景仍属于 Phase 3 exit gate,不构成当前生产性能签署。shared analytics TCK + 已明确锁定 capability 差异:Mongo 对 Decimal128 policy 执行 exact `MIN/MAX/SUM/AVG`,Elasticsearch 对合法 + `Int64 AGGREGATABLE` numeric plan 在 I/O 前稳定 `UNSUPPORTED`,不得用 `double` 冒充 exact portable capability。 + +Phase 3 exit gate:Mongo integration 与 shared TCK 固化 portable record/analytics 基准;覆盖 null/missing/array、 +数字提升/溢出、稳定排序、`limit=0 > 10,000`、cursor replay、并发写、budget/deadline/cancel 和 mandatory +顺序。按 DDD Aggregate 先 `SHADOW` record path;安全/完整性差异必须为零,性能阈值由目标应用基于基准显式 +签署后才切 `PLANNED`。回滚到该 Aggregate 的 legacy Backend,不迁移或删除数据。 + +### 10.7 Phase 4:Elasticsearch planned path + +#### P4-A:Binding readiness 与完整性 validator + +- 实现 `ElasticsearchFieldBinding`、mapping version/capability digest 校验和 readiness; +- 分别验证 source/search/exact/literal/sort/group field,不允许 compiler 通过追加 `.keyword` 猜测; +- 验证 exact keyword/doc-values、text analyzer、literal keyword/wildcard、`ignore_above` 与数据长度审计、 + numeric/date metric、nested path、所有目标 generation 的 mapping 一致性; +- timeout、failed shard、partial total、缺失 source、未知 aggregation response 全部 fail closed; +- readiness 未通过时不注册 capability,不接业务流量。 + +#### P4-B:Record planned path + +- 实现 bounded single/page/count/stream compiler、materializer 与错误映射; +- exact、full-text、literal string、nested scope、projection、stable sort 只按所需 capability/binding 编译; +- `MATCH` 只使用 search binding;literal string 转义模式字符且禁止 `match_phrase`;首批 portable + `ignoreCase=true` 在等价 normalization/collation TCK 完成前保持 unsupported; +- `limit=0` 使用 PIT + `search_after`,以 `usingWhen` 等价生命周期覆盖 complete/error/cancel; +- 不再保留 legacy “缺失 source 静默跳过”行为,planned path 返回 `IncompleteResult`。 + +P4-A/P4-B 当前实现约束: + +- Snapshot binding 必须精确绑定目标 Aggregate 的标准索引/alias 名,并在启动 readiness 阶段读取该名字解析出的每个 + concrete mapping;所有 generation 都必须携带一致的 `wow_query_mapping_version`,任一 generation 缺字段、类型、 + `doc_values`、nested、analyzer、normalizer、`ignore_above` 或历史长度审计证明即不注册 Backend capability; +- exact/search/literal/sort/group/source/presence 是相互独立的显式物理角色,compiler 不追加 `.keyword`。字符串 exact + 首批只接受无 normalizer 的 binary keyword;全文字段必须显式声明并同时匹配 analyzer 与 search analyzer;nullable + presence 使用独立、已验证的 boolean presence marker,禁止可能与合法业务字符串碰撞的 `null_value` sentinel; +- 当前 record vertical slice 已覆盖 Snapshot `SINGLE`、bounded `STREAM`、`PAGE`、`COUNT`,mandatory filter、logical + projection、stable sort、literal wildcard 转义、显式 search scope 与 nested path 均只从 binding 编译;timeout、failed + shards、非 `Eq` total、缺失 `_source`/identity、`_ignored` 与 mapper 异常 fail closed; +- direct bounded `STREAM` 只接受 `1..10_000`;大于 `10_000` 在 Elasticsearch I/O 前稳定拒绝,`limit=0` + unbounded stream 仍保持 unsupported,不能退化为固定 result window; +- Spring planned source 只在目标 Aggregate 的既有 storage route 实际选择 Elasticsearch 且 execution profile 非 + `LEGACY` 时检查 mapping;Mongo/Elasticsearch mixed routing 不以某个 client Bean 的存在推断默认 Backend; +- `PAGE` 使用每订阅独立 PIT + stable sort + `search_after` 有界推进,响应轮换 PIT id 时更新租约,并在 + complete/error/cancel 三条路径关闭;真实 Elasticsearch fixture 已跨越 10,000 result window。`maxCursorPages` 与 + `maxPageWindow` 双重限制内部请求数与窗口;PIT 404 expiry 通过真实 transport 归一为 `IncompleteResult`,并保持 + 最新 lease cleanup;`search_context_missing_exception` 的结构化 root cause 与 `usingWhen` 成功后 cleanup 包装都稳定归类为 + `IncompleteResult`,cleanup 失败不会被外层误报成 `BackendUnavailable`;真实 client 回归同时覆盖 cancel cleanup 与 closed + transport 的 `BackendUnavailable` 分类; +- composite Analytics 真实 fixture 已以 257 个额外 dimension key、31 bucket/page 完整 replay 全部 key,证明 after-key 无缺口、 + 无重复且在有界页数内终止;跨页插入排序位于 cursor 之后的新 bucket 可在下一页观察到,同时结果继续显式声明 + `EVENTUAL + EXACT`,不冒充 snapshot consistency; +- `limit=0` unbounded stream、EventStream,以及 composite 的 Decimal128 metric 尚未完成;grouped SNAPSHOT cursor + 已在 P5-A 通过持久化 lease + PIT state/lifecycle 补齐,因此 + P4-B/P4-C exit gate 未通过,生产只能继续 operation-scoped `SHADOW`,不得把该 vertical slice 视为 Phase 4 完成。 + +#### P4-C:Composite analytics + +- Snapshot/root scalar 使用 composite + response `after_key`;Eventual 先行; +- 当前 exact-count vertical slice 只发布 `DocumentCount`、root scalar、`EXCLUDE/AS_NULL_BUCKET` missing、 + `EVENTUAL + EXACT`;global count 必须取得 `track_total_hits` 的 `eq` 关系,grouped count 使用 composite + bucket 的精确 `doc_count`;Mongo/Elasticsearch 已运行同一 shared TCK,覆盖 mandatory pre-filter、稳定 key 顺序与 + cursor replay、有界终止、显式 null 与 missing 合并为同一 canonical null bucket,以及 bucket/scan/deadline budget 的 + fail-closed 分类;Elasticsearch 只使用响应 `after_key`,因此允许最后一个非空页继续返回 opaque cursor,再以一个空页终止, + 禁止从最后一个 bucket 自行推导 cursor; +- Elasticsearch 的 `sum/avg/min/max` 响应值经 double 表示,在没有额外精度证明前不得冒充 Decimal128 exact; + `AS_NULL_BUCKET` 只能使用与 presence sentinel 分离、无 `null_value` 的显式 group binding,并要求历史值审计证明; + compiler 固定 `missing_bucket=true`、null-first,cursor 把 Elasticsearch null key 还原为 canonical `Null`,从而按 + MongoDB 基准把显式 null 与 missing 合并为一个桶; +- `requiredConsistency=Snapshot` 由 Phase 5 把最新 PIT id 作为 opaque Backend state 纳入持久化 lease;只有注册 + target/backend keyed lifecycle closer 的 Backend 才能执行,否则在 storage-I/O 前拒绝; +- `terms`、metric-sorted top-N、pipeline having 不进入 exact portable path; +- 与 MongoDB 运行同一 fixtures/TCK;只比较结构不能作为语义等价证据。 + +Phase 4 exit gate:mapping mismatch readiness fail、record/analytics portable TCK、>10k、PIT complete/error/cancel/ +expiry、after-key、failed shard、timeout、numeric precision 和 concurrent write consistency 全部有 integration +证据。先内部 dual-backend probe,再按 Aggregate/target planned route;回滚路由到 Mongo/legacy,并关闭或等待 +短 TTL 回收现有 PIT,不切 alias、不删除索引。 + +仓库级切换/回滚演练记录(2026-08-09): + +| Backend | 可执行 fixture | 演练结果 | +|---|---|---| +| MongoDB | `MongoSnapshotRecordQueryBackendIntegrationTest.gateway Mongo rehearsal should shadow cut over and roll back against one collection` | 同一 Testcontainers 集合上 SHADOW=`MATCH`;PLANNED 结果一致且 legacy raw 调用数不增加;切回 LEGACY 后 raw 调用恢复 | +| Elasticsearch | `ElasticsearchSnapshotRecordQueryBackendIntegrationTest.gateway Elasticsearch rehearsal should shadow cut over and roll back against one index` | 同一 Testcontainers 索引上 SHADOW=`MATCH`;PLANNED 结果一致且 legacy raw 调用数不增加;切回 LEGACY 后 raw 调用恢复 | +| Elasticsearch Analytics | 同 fixture 的 composite replay、并发写 EVENTUAL、跨页 PIT SNAPSHOT、PIT expiry/cancel 用例 | 服务端 `after_key`、opaque PIT state、terminal/error/cancel 清理和 `IncompleteResult` 分类均由真实 client 验证 | +| Spring 示例目标 | `QueryGatewayAutoConfigurationTest.example order should rehearse storage routed shadow cut over and rollback through one facade` | 使用真实 `example-service/order` 聚合元数据和 Mongo storage route;LEGACY 只调用 raw,SHADOW raw/planned 各一次且 `MATCH`,PLANNED 不再调用 raw;三个模式复用同一个 Gateway facade | + +以上证明仓库内 vertical slice 可切换、可回滚,不替代目标生产应用的 mapping inventory、性能阈值、迁移窗口与 +运维签署;后者仍是 Phase 3/4/5 的生产 exit gate。目标应用必须按 +[Query 服务目标应用发布与回滚 Runbook](./2026-08-09-query-service-application-rollout-runbook.md) 固定 authority、 +Schema/Binding、operation profile、阈值和签署证据。 + +### 10.8 Phase 5:cursor、索引生命周期与公开 analytics contract + +#### P5-A:Cursor envelope 与 lease + +- 稳定 record/analytics cursor version、target、plan fingerprint、mapping generation、sort/group key、 + Backend state、expiry 和完整性保护; +- 跨请求 PIT/session 通过有期限 lease 转移所有权;terminal/error/cancel 尽力关闭,客户端遗弃由短 TTL 回收; +- tamper、version/target/plan/order mismatch 和 expiry 映射为 `InvalidCursor`; +- cursor token 不泄漏 PIT id、物理索引或 policy 数据。 + +P5-A 当前实现边界: + +- token 是固定二进制 format version、独立 signing-key id、256-bit 随机 lease id、expiry 与 HMAC-SHA256,不把 target、plan fingerprint、 + group/sort key、mapping generation digest 或 Backend state 放入客户端 token;上述 envelope 全部保存在有界服务端 + registry,避免把 Base64 当成加密; +- expiry 在签名与持久化前统一截断到毫秒精度,确保 HMAC envelope 与 Mongo BSON Date 往返值一致;Backend state + 使用同一份配置限额完成 runtime 校验与 codec 编解码,默认 `4_096` bytes,公共硬上限 `1 MiB`; +- internal codec 已实现最多 4 把 key 的有界 key ring:current key 只签发,previous key 只验签;key material 防御复制,重复、未知或 + 已退役 key id 稳定拒绝且不会消费 lease。运维仍必须在 `maxCursorTtl` 之后才能移除 previous key;公共 + `QueryCursorLeaseConfiguration` 只接受显式 store 与 key ring,不从普通请求/header 推导密钥; +- envelope 额外保存 security-context SHA-256 digest,绑定 canonical authority type/principal/grants、purpose、resource scope + 与 policy constraint。`acquire` 必须提交 expected target、plan fingerprint、mapping generation 与 security-context binding; + 任一 mismatch 在原子移除前返回 `INVALID_CURSOR_BINDING`,且不能消费合法主体的 lease; +- binding 验证后 `acquire` 原子移除 entry,防止重放和并发双重消费;调用方取得 ownership 后只能向同一 Backend 转移为下一 token, + 或在 terminal/error/cancel 幂等关闭一次;遗弃 entry 由 TTL reaper 转交 Backend closer,cleanup/observer 失败隔离; +- registry 对 entry 数、最大 TTL 和 Backend state bytes 设硬预算;tamper、未知/version mismatch/replay、expiry 与 + Backend owner mismatch 均为稳定 typed rejection; +- `MongoQueryCursorLeaseStore` 使用固定数量 slot 实现严格容量上限,以 unique lease id 区分 token collision,并以 store revision + 条件删除转移唯一 ownership;`scanExpired` 按 lease id 有界 keyset 扫描。TTL 只作用于 `expiresAt + retentionGrace`,给 framework + reaper 留出关闭 Backend state 的窗口,随后才由 MongoDB 作为最终遗弃清理。集合和 unique/TTL index 只能通过显式 + `ensureIndexes()` 管理操作初始化,Starter 不会因发现 MongoDB 就自动启用 cursor 或隐式执行 DDL; +- 公共 grouped Analytics Gateway 已在同一条 admission/policy/planner/executor 流水线中签发和接续 opaque wire token, + 校验 target、Plan fingerprint、mapping generation、security-context 与 Backend 后才执行 revision CAS。真实 Mongo fixture 已证明 + 两个 runtime/两个 client 之间接续、previous-key 验签、新 key 签发、未知 key 不消费和 replay 拒绝; +- experimental Backend SPI 以 defensive-copy 的 `BackendAnalyticsCursorState` 传递物理 continuation state,并以 + `AnalyticsQueryCursorLifecycle` 幂等关闭;Gateway 仅接受与 exact `QueryTarget + BackendId` 匹配的 closer,禁止同 Backend id + 跨 target 误清理; +- Elasticsearch grouped SNAPSHOT Analytics 首页打开 PIT、后续页使用服务端 lease 中的最新 PIT id,并在 terminal、error、cancel、 + lease capacity failure 与 expired reaper 路径关闭。真实 Elasticsearch fixture 已证明并发写不进入同一 PIT continuation,关闭后续页 + 稳定归类为 `IncompleteResult`;公共 cursor 从不携带 PIT id; +- `QueryGatewayRuntime.reapExpiredQueryCursors(batchSize)` 提供有界单批运维入口。Starter 的 reaper 默认关闭;只有显式配置 + `wow.query.cursor.reaper.enabled=true` 且存在 `QueryCursorLeaseConfiguration` 时,才由单一 `SmartLifecycle` 串行调度。 + 每轮受 batch size 与 max-batches 双重上限约束,运行重叠被丢弃,单轮错误不终止后续周期。framework 不在普通查询请求中 + 隐式清理,Mongo TTL grace 继续作为最终安全网;不使用 Starter 时仍可由外部运维 scheduler 调用单批入口。 + +#### P5-B:版本化索引与显式 cutover 工具 + +- 支持 component/index template、`_meta` mapping version/capability digest、physical generation 与 stable alias; +- template 按字段用途显式生成 keyword、text + exact multi-field、text 或经批准的 wildcard;禁止全局 + “所有 string 都是 keyword”规则,也不依赖 Elasticsearch 默认动态 `.keyword`; +- 工具只执行显式 `VALIDATE/CREATE/REBUILD/VERIFY/CUTOVER/ROLLBACK` 命令,应用启动不 reindex/cutover/delete; +- Snapshot 从权威事件流重建;EventStream 必须 pause/drain 或受控镜像写; +- cutover 前校验 count、identity、version continuity、checksum、record/analytics probe;旧 generation 保留 + `max cursor TTL + rollback window`。 + +P5-B 当前实现边界: + +- physical 名固定为 `-v<4-digit-mapping-version>-<6-digit-generation>`;manifest 精确绑定 target、 + schema contract、由 planned binding 规范编码生成的 capability digest、source/destination、重建策略、cursor TTL + 与 rollback window,并固定 checksum algorithm 与 probe suite id; +- `NEW -> VALIDATED -> CREATED -> REBUILT -> VERIFIED -> CUTOVER -> ROLLBACK_VERIFIED -> ROLLED_BACK` + 是唯一合法状态序列。每个外部动作前先以 revision CAS 认领 command,失败后只允许同一 command id 恢复; + 另一个并发 command fail closed; +- `VERIFY` 在 cutover 前证明 destination,在 cutover 后重新证明 source,后者不是复用旧报告。报告必须同时满足 + count、identity/content checksum、version continuity、authoritative/indexed watermark、record/analytics probe 零差异; +- `CREATE` 先幂等写 component template 和 composed index template,再显式创建 physical index并回读 `_meta`; + alias 切换使用同一个 `_aliases` 请求执行 `remove(source, must_exist=true) + add(destination, is_write_index=true)`, + 完成后再次读取 alias;无任何 delete index 行为; +- Snapshot destination verification 与 alias 切换之间仍存在 writer 继续写 source 的窗口,因此 CUTOVER 默认由 + `ElasticsearchSnapshotCutoverGuard.DENY` fail closed。只有目标应用提供受信 pause/drain 或 controlled-mirror attestation + 才能执行 alias transition;仓库测试 allow guard 不是生产实现,本次任务不执行真实切换; +- Snapshot rebuild 由 EventStore authority port 全量重放;Snapshot verification authority 端复用相同有序分页,physical + 端用 exact generation PIT + `aggregateId` keyset 扫描,并以 `CANONICAL_DOCUMENT_SHA256_V1` 生成 count、identity/content + checksum。顶层 `snapshotTime` 作为重建时易变元数据被排除,其他 logical document 内容全部进入摘要;严格拒绝 + partial shard、非 exact/不稳定 total、identity/source/version/顺序错误和超预算值; +- EventStream pause-and-drain rebuild 通过窄 barrier port 在复制前后取得同一 non-negative authority watermark,按 aggregate + 分页读取完整 EventStore history,并严格校验 named aggregate、aggregate 顺序、stream version 连续性、event sequence/last + 与 body size,再幂等覆盖 exact physical generation。authority verification 复用同一有序扫描,physical verification 使用 PIT + 与 `(aggregateId,version)` keyset,并以 `CANONICAL_EVENT_STREAM_SHA256_V1` 比较完整 canonical document;真实 + Elasticsearch fixture 已证明 authority/physical count、identity/content checksum 与 watermark 一致。外部生产 writer 的 + pause/drain、indexed-watermark 持久化和 barrier attestation 仍未接线;`EVENT_STREAM_CONTROLLED_MIRROR` 继续稳定拒绝; +- 目标应用 probe 已有 internal bounded runner:manifest suite id、exact `QueryTarget` 与 `SchemaContractId` 精确匹配, + probe id 唯一且 canonical 排序、总数硬限制为 + 256;每个 record/analytics probe 分别从 authority 与 exact physical generation 取得完整的 + `resultCount + resultChecksum` evidence,empty/incomplete/上游 error 一律 `VERIFICATION_FAILED`,mismatch 分类计数。 + 具体 probe catalog/evaluator 仍由目标应用窄 port 提供;管理内核不允许用旧索引 `_reindex` 冒充权威重建; +- `ReactiveElasticsearchIndexLifecycleRepository` 使用显式创建的 `.wow-query-index-lifecycle-v1` hidden system index; + repository document/mapping format 固定为 `v1`,当前有界 lifecycle payload codec 为 `v2`,旧 payload 不做隐式升级; + strict mapping 只保存 format version、migration id、revision 与 payload;状态写入用 document create,更新用 + `_seq_no + _primary_term` compare-and-set。duplicate create 与 stale CAS 返回空,由 Executor 执行既有幂等/冲突协议; + 损坏、越界、跨 migration payload 以 `REPOSITORY_CORRUPTED` fail closed。真实 Elasticsearch 容器已验证重复 + `ensureIndex()`、跨 repository 实例重载与 stale CAS 不覆盖; +- `InMemoryElasticsearchIndexLifecycleRepository` 仍只用于单进程验证;生产 repository 不自动注册,也不在应用启动时 + 隐式创建 system index。管理入口必须先显式 `ensureIndex()`,再注册 manifest/执行命令; +- Elasticsearch 不允许 alias 与同名 concrete index 共存。当前工具要求 source 已是受管 generation/write alias; + 既有 `wow...snapshot|es` concrete index 到 managed alias 的一次性转换涉及停写、备份、权威 + 重建及删除/改名限制,必须另行审批 runbook 和演练,工具不会静默删除旧 concrete index。 + +`EVENT_STREAM_CONTROLLED_MIRROR` 当前继续 fail closed。安全实现至少需要权威 EventStore 提供全局单调 watermark 与 +`scanAsOf(watermark)`,并在 destination 达到同一 watermark 时原子取得 exact physical PIT;或在 mirror 历史回填后进入 +短暂 pause-and-drain finalization。现有 `EventStore` 只有 aggregate/version 读取,EventStream physical document 也没有全局 +offset,无法证明不停写期间 authority checksum 与稍后打开的 PIT 属于同一快照,因此不得用“两个当前水位相等”冒充完成。 + +#### P5-C:公开 Analytics API(公共契约已批准) + +- additive 引入 `AnalyticsQueryService`、独立 wire DTO、HTTP route、DSL 和 OpenAPI; +- 不修改七方法 `QueryService`,不复用 `PagedList/DynamicDocument`; +- 发布文档明确 portable/unsupported、numeric/missing/consistency/completeness、budget 和 cursor 合同; +- Java/Kotlin/JSON/OpenAPI compatibility 与客户端生成链路一起验证。 + +以下公共契约已获批准并按 additive 方式落地;Mongo/Elasticsearch 的全部 capability、生产级 store 实现与目标应用 +cutover 仍需各自 readiness/TCK 证据,不能因公共 route 已存在就宣称 Backend 已支持: + +| 模块 | 新增边界 | 约束 | +|---|---|---| +| `wow-api` | `me.ahoo.wow.api.query.analytics` request/result DTO | 只有逻辑字段、显式 discriminator、无 Backend/driver/`Any` result | +| `wow-query` | `AnalyticsQueryService`、独立 `AnalyticsQueryGateway`、cursor store SPI | 不修改 `QueryService` 七方法和现有 `QueryGateway` JVM descriptor | +| `wow-spring` | `..AnalyticsQueryService` target-bound Bean | 首批只注册 Snapshot/document grain;EventStream 不自动 unwind | +| `wow-webflux` | 现有 tenant/owner variant 下新增 `POST .../snapshot/analyze` | 使用同一 trusted context、policy、deadline/error boundary | +| `wow-openapi` | request/page/value/cursor schema 与 Query error responses | 属于经批准的 additive snapshot 变化,随后重新生成客户端 | + +服务边界固定为: + +```kotlin +interface AnalyticsQueryService : NamedAggregateDecorator { + fun analyze(query: AnalyticsQuery): Mono +} + +interface AnalyticsQueryGateway { + fun analyze(call: QueryCall, query: AnalyticsQuery): Mono +} +``` + +独立 `AnalyticsQueryGateway` 避免给当前 experimental `QueryGateway` 增加新的 abstract JVM method;runtime 实例可以同时实现 +两个端口,但 authority、admission、policy、planner、executor 仍只有一条内部流水线。 + +首版 request wire shape: + +```json +{ + "condition": { "operator": "ALL" }, + "grouping": { + "kind": "BY", + "dimensions": [ + { "alias": "status", "field": "state.status", "missingPolicy": "EXCLUDE" } + ] + }, + "metrics": [ + { "alias": "count", "kind": "DOCUMENT_COUNT" }, + { "alias": "total", "kind": "SUM", "field": "state.total" } + ], + "window": { "limit": 100, "cursor": null }, + "numericPolicy": { + "promotion": "DECIMAL128", + "precision": 34, + "scale": 2, + "roundingMode": "HALF_EVEN", + "overflowPolicy": "REJECT" + }, + "consistency": "EVENTUAL", + "completeness": "EXACT" +} +``` + +- `grouping.kind=GLOBAL` 要求 dimensions 为空、limit 为 1、cursor 为空;`BY` 要求 1..N 个唯一 alias; +- metric 首版仅 `DOCUMENT_COUNT/MIN/MAX/SUM/AVERAGE`。`DOCUMENT_COUNT` 禁止 field,其他 metric 必须有逻辑 field; +- public 首版不暴露 `having`、metric sort、approximate bucket total 或 Backend-specific option;未形成 portable 合同的能力不进入 + wire schema,而不是用可选字段接收后静默忽略; +- bucket order 固定为 dimensions declaration order 的 binary/null-first ascending,不接受 `.keyword`、physical path、analyzer + 名或 collation 名; +- `condition` 复用现有 Mongo-baseline `Condition` wire,但在同一次 subscription 内立即 admission/snapshot;policy mandatory + condition 仍保留独立 provenance; +- cursor 是不超过 256 字符的 opaque URL-safe token;client 不得读取、拼接 after-key 或 PIT state。 + +首版 result 不复用 mutable `DynamicDocument`,也不直接以 JSON number 暴露 `Int64/Decimal`: + +```json +{ + "buckets": [ + { + "keys": { "status": { "type": "TEXT", "value": "PAID" } }, + "metrics": { + "count": { "type": "INT64", "value": "42" }, + "total": { "type": "DECIMAL", "value": "120.50" } + } + } + ], + "nextCursor": null, + "consistency": "EVENTUAL", + "completeness": "EXACT" +} +``` + +`AnalyticsValue.type` 首版只允许 `NULL/BOOLEAN/TEXT/INT64/DECIMAL/INSTANT`;`value` 为 nullable canonical string, +`NULL` 必须为 null,Boolean 为小写,Int64/Decimal 为十进制规范串,Instant 为 ISO-8601。这样 OpenAPI/JavaScript/Java/Kotlin +不会因 IEEE-754 或 JSON parser 自动提升丢失精度。keys/metrics 以 alias 索引,容器 defensive-copy 且不可变。 + +持久化 cursor SPI 不暴露 `NormalizedValue`、Plan、PIT 或 policy:framework 先把完整 envelope 编码为有界、版本化、完整性保护的 +opaque bytes,再交给 experimental `QueryCursorLeaseStore`。最小原子合同为: + +```kotlin +interface QueryCursorLeaseStore { + fun create(entry: QueryCursorLeaseEntry): Mono + fun load(id: QueryCursorLeaseId): Mono + fun compareAndDelete(expected: StoredQueryCursorLease): Mono + fun scanExpired(before: Instant, afterId: QueryCursorLeaseId?, limit: Int): Flux +} +``` + +- `create` 区分 `CREATED/COLLISION/CAPACITY_EXCEEDED`;不得 last-write-wins; +- `load` 只读后,framework 先校验 token expiry 与 expected target/plan/mapping/security binding,再调用 compare-and-delete;错误 + authority 即使持有有效 token 也不能消费 lease; +- `compareAndDelete` 必须按 store revision 原子转移唯一 ownership;只有 winner 可返回下一页或关闭 Backend state; +- reaper 对有界 keyset scan 的每条 expired entry 也执行 compare-and-delete,成功者才清理 PIT; +- entry 只包含随机 lease id、expiry、payload format、opaque bytes 和 store revision,所有 byte array defensive-copy; +- store 必须支持 TTL、容量上限、跨节点一致的原子删除与可运维 namespace。进程内 manager 不能作为多实例默认; +- token HMAC key ring 至少 256 bit,token 带 key/version id;current key 签发,previous keys 只在 `maxCursorTtl` 内验签,移除 + 旧 key 前必须等待其全部 token 过期。 + +兼容与发布门禁: + +- 原 `QueryService` 七方法、`QueryType` 七值、Bean 名、HTTP route 与 error JSON 不变; +- 新 Analytics route/OpenAPI schema、opaque Cursor 与 Query status response 是经审批的 additive diff;仓库 golden、 + `wow-apiclient` 与 Java/Kotlin compile fixture 必须通过,下游 Fetcher/其他 SDK 在发布流程中从更新后的 OpenAPI 重新生成; +- `INVALID_CURSOR`、`UNSUPPORTED_FEATURE`、`BUDGET_EXCEEDED` 与 Backend failure 使用现有 Query error category/status 矩阵, + 不新增另一个错误 envelope; +- Aggregate 没有 Snapshot analytics schema/backend readiness 时稳定 `UNSUPPORTED_FEATURE` 且 storage zero-I/O;不能因 route + 存在就猜 mapping/capability; +- 完成标准包括 EVENTUAL 多页 cursor replay、SNAPSHOT PIT complete/error/cancel/expiry、跨节点 lease acquisition、key rotation、 + Mongo/Elasticsearch shared TCK、OpenAPI/client golden 与目标应用授权/预算负测。 + +#### P5-D:主版本清理 + +- 只有所有目标 Aggregate 完成 planned cutover、fallback 指标归零、回滚演练通过并经过弃用周期后, + 才移除 legacy Filter/converter/service 和 temporary wiring; +- breaking removal 单独主版本 PR,附调用方、反射/Spring binding、序列化和迁移审计。 + +Phase 5 exit gate:cursor 安全/租约测试、索引迁移与回滚演练、公开 ABI/OpenAPI/client compatibility、 +双 Backend portable TCK、全量 build 与目标应用 shadow/cutover 证据齐全。EventStream 未镜像旧 generation 时, +回滚必须从权威事件流追平,不能只切 alias。 + +### 10.9 已锁定决策与待决事项 + +已锁定、后续 slice 不得自行改写的决策: + +- analytics 使用 `Grouping.Global | Grouping.By`,不强迫全局统计伪造 dimension; +- Phase 1 模型保持 Kotlin internal;跨 module API 只在实际需要时最小化提升为 experimental opt-in; +- PR #2903 只作为 validator/TCK 素材,不整体合并 legacy Backend 修改; +- MongoDB 是 portable expected baseline;Elasticsearch mapping 不满足时 unsupported/readiness fail; +- 客户端和 Plan 只使用逻辑字段;text/keyword/multi-field 由字符串 capability 与 Backend binding 决定; +- 首批 portable literal string 仅 case-sensitive;`ignoreCase` 在等价 normalization/collation TCK 前 unsupported; +- 全局 dynamic keyword template 和硬编码 `.keyword` 都不是长期方案;exact capability 必须证明长度完整性; +- 首批 analytics 是 Snapshot/document grain;EventStream 不自动 unwind; +- Phase 1 只固定 cursor semantic state,token codec/签名/lease 属于 P5-A; +- `Exact/Approximate` 与 `Eventual/Snapshot` 是正交维度; +- bucket total 默认不计算;跨聚合 join 使用物化 Projection。 + +以下事项必须在指定 slice 前用代码实验/benchmark/运维约束决定,当前文档不伪造默认值: + +| 待决事项 | 最晚决策点 | 所需证据 | +|---|---|---| +| admission/budget 默认上限 | P1-B/P2-A | 现有 DTO 分布、边界测试、目标应用配置需求 | +| Mongo page window pipeline 的目标规模性能阈值 | Mongo target 进入 `PLANNED` 前 | 目标数据分布 explain/profile、window sort/落盘指标与 latency/内存预算;语义实现已不使用 `$facet` 或 collection re-read | +| Mongo analytics cursor predicate 是否可安全下推 | P3-C | missing/null/collation 等价测试与 explain;否则保持 group 后过滤并限制预算 | +| numeric policy 的 portable type/range | P3-C | Mongo int/long/double/Decimal128 与 ES numeric metric 双 Backend TCK | +| 各 Aggregate 字符串 capability、search scope、长度、analyzer/normalizer/collation | P1-C/P4-A | 查询用例、实际 mapping、现有值长度/`_ignored` 审计与双 Backend fixtures | +| cursor token 完整性方案与 lease 是否需要 server-side state | P5-A | threat model、token 大小、key rotation、PIT TTL/资源实验 | +| ES template v2 与现有 concrete index 到 alias 的迁移步骤 | P5-B | 当前 template/index inventory、rebuild/checksum rehearsal、回滚窗口 | +| 每 Aggregate 的 shadow/cutover 性能阈值 | P3/P4 rollout 前 | 可重复 benchmark/metrics;安全和 exact semantic 差异阈值固定为零 | + +这些待决事项不阻塞 P1-A;进入对应 slice 前若证据仍缺失,该 capability 保持 unsupported,不能用临时默认 +值静默发布。 + +## 11. 当前 PR 的处理 + +PR #2908 只保留 Phase 0 的执行正确性修复和本设计文档。以下过早抽象已经撤回,后续 slice 也不得在 +Plan/Policy/Backend contract 形成前以其他名称重新引入: + +- `SnapshotQueryGateway*` / `EventStreamQueryGateway*`; +- 名为 Backend、实际返回旧 `QueryService` 的 Provider; +- Gateway/Provider/Factory 三层缓存; +- Spring Registrar 与 Web 文档中“Bean 已切到 Gateway”的声明; +- 为上述临时层保留的自动配置与 ABI bridge。 + +历史 PR #2908 只评审 Analytics portable scope、cursor、precision、missing、security 和 completeness 合同, +没有提前发布公开 DTO 或 Backend SPI。当前实现已按 Phase 1-5 的顺序完成内部 Plan/Policy、experimental Backend SPI、 +MongoDB/Elasticsearch vertical slice 与经批准的 additive 公共契约;这一历史切片边界不再表示当前能力状态。 + +## 12. 验证与完成审计 + +### 12.1 分层验证矩阵 + +| 层 | 必须证明 | 权威证据 | +|---|---|---| +| Public contract | Query DTO/JSON/OpenAPI、七方法 `QueryService`、`QueryType`、Bean name/generic injection、Kotlin/Java 调用保持兼容 | reflection/Java compile guard、OpenAPI golden、Spring context tests、required CI | +| Semantic model | invocation matrix、深度不可变、一次 Clock、field/search scope、字符串 capability、projection/limit/page、global/grouped analytics、provenance、fingerprint | `wow-query` unit/golden tests | +| Gateway/Policy | HTTP 与直接 Service 调用同路;authority/policy/mandatory/result constraint 不可绕过;错误 fail closed | Gateway contract tests、WebFlux tests、direct in-process tests、security negative cases | +| Backend compiler | 只接受 Plan;source/search/exact/literal/sort/group/nested binding 正确;禁止 RAW/driver object 和物理字段猜测;unsupported 稳定拒绝 | compiler golden 与 architecture tests | +| MongoDB | record/page/analytics、null/missing/array、numeric、deadline/cancel、read concern、预算和 pipeline 顺序 | Mongo container integration、shared TCK、必要时 explain/profile | +| Elasticsearch | mapping readiness、完整性、record/PIT/composite、after-key、failed shard/timeout、mapping generation | Elasticsearch container integration、shared TCK、PIT/resource tests | +| Cross Backend | portable literal/exact string、identity/order/total/error 与 bucket key/order/metric type/value/completeness 等价 | 同一 logical fixture 的双 Backend TCK;MongoDB 是 expected baseline | +| Operations | mode rollout、fallback/shadow 指标、index rebuild/checksum、alias cutover、cursor lease、rollback | rehearsal logs、指标快照、迁移清单和目标应用验证记录 | + +本地最低命令随 slice 扩展,而不是用一个窄测试代替全局结论: + +```text +./gradlew :wow-query:check +./gradlew :wow-openapi:test --tests "me.ahoo.wow.openapi.snapshot.OpenApiCompatibilitySnapshotTest" +./gradlew :wow-mongo:integrationTest +./gradlew :wow-elasticsearch:integrationTest +./gradlew allLocalTest allContractTest allIntegrationTest +./gradlew detekt build +``` + +容器、网络或外部系统使某个命令无法运行时,必须记录 exact command、失败边界和缺失证据;不能把 +“环境未验证”报告为行为成功。 + +### 12.2 目标—证据完成矩阵 + +| 目标 | 完成所需证据 | +|---|---| +| 1. 所有入口进入 Gateway | HTTP、Snapshot/EventStream Service 和新进程内调用的调用链/测试都命中同一 Gateway;不存在 storage factory 直达路径 | +| 2. 深度不可变 Plan | Normalizer golden 证明对原 List/Map/ByteArray 后续修改不敏感;Plan 类型不含 `Any`、BSON、ES Query 或物理字段 | +| 3. Mandatory provenance | user/mandatory 分别校验,Backend 只能取得最终外层 AND;Filter/Native/direct caller 的绕过测试全部拒绝 | +| 4. Mongo portable baseline | Mongo record/analytics TCK 给出 expected semantic results;ES 对同一 fixtures 全部满足或显式 unsupported | +| 5. 完整性/一致性/预算/错误 | partial shard/source、timeout、overflow、deep page、budget、cancel、Eventual/Snapshot 与 exact/approximate 分支都有正反测试 | +| 6. 兼容性 | DTO/OpenAPI golden、七方法反射 guard、Java compile、Spring Bean/generic、HTTP status/NoOp/legacy fallback 全部通过 | +| 7. Analytics 一等能力 | Global/By、metrics、missing/numeric、cursor/completeness/result policy、Mongo/ES Backend 与公开独立 Service/HTTP/DSL 均有证据 | +| 8. 渐进发布与回滚 | 每 Aggregate 的 mode、shadow/fallback reason、阈值、Mongo/ES route、index generation、cutover 与 rollback rehearsal 有实际记录 | + +### 12.3 整体 Definition of Done + +只有 12.2 的八行都具备当前、直接、覆盖相应范围的证据,并且以下条件同时成立,才可宣称本设计完整实现: + +- Phase 0-5 所有 slice 已合并,工作树与发布分支无未说明差异; +- 本地分层命令和远端 required checks 全绿,无 retry 后偶然通过; +- 所有 portable/unsupported 边界与公开文档一致,无 Backend 静默 fallback 或近似降级; +- 至少一个仓库内可运行的 Snapshot Aggregate/示例服务完成 Mongo planned shadow/cutover/rollback 演练; +- 至少一个具备合格 mapping 的 Snapshot Aggregate/示例服务完成 Elasticsearch planned/composite/PIT 演练; +- EventStream 的 record path 已验证,event-level analytics 仍明确 unsupported 时不得包装为完成; +- 没有待处理的安全绕过、数据迁移、cursor 资源泄漏或兼容性 finding。 + +单个 Phase、PR、测试任务或绿色 CI 只能证明对应 slice,不能替代上述整体完成审计。 + +### 12.4 当前完成审计(2026-08-09) + +本节区分“仓库内可重复验证的实现证据”和“必须由目标应用、真实数据与发布环境提供的运营证据”。 +前者通过不代表后者已经完成,也不能据此执行生产 planned cutover。 + +| 目标 | 当前直接证据 | 当前判定 | +|---|---|---| +| 1. 所有受支持入口进入 Gateway | Spring aggregate Service/Factory、Handler/WebFlux route 与 direct facade 的 vertical tests;raw storage registry 与 facade 类型隔离;恶意 Filter、缺 authority、selector mismatch 均在 storage 前拒绝 | 仓库级已证明 | +| 2. 深度不可变 Plan | admission/normalization 的 getter-once、one-shot Iterable、List/Map/ByteArray 防御复制、canonical fingerprint 与 public Backend contract architecture tests | 仓库级已证明 | +| 3. Mandatory provenance | policy mandatory 与 user condition 分离验证;最终 `EnforcedFilter` 外层 AND;legacy attestation、Native/Filter/direct caller 绕过负测 | 仓库级已证明 | +| 4. Mongo portable baseline | Mongo planned record/analytics integration、shared record/analytics TCK;Elasticsearch 对同一 portable fixtures 的 integration 与 unsupported/readiness 负测 | 仓库级已证明 | +| 5. 完整性、Consistency、Budget 与错误 | page/stream/analytics envelope、partial/timeout/mapping、deadline/cancel、EVENTUAL cursor、Elasticsearch PIT SNAPSHOT、lease/reaper/key rotation tests | 仓库级已证明;目标数据规模与性能预算仍待实测 | +| 6. 兼容性 | 原 `QueryService` 七方法/`QueryType` 七值 reflection 与 Java fixture;Spring Bean/generic;HTTP status/ErrorInfo;经批准的 Analytics/Cursor/OpenAPI/client additive golden | 仓库级已证明 | +| 7. Analytics 一等能力 | 独立 public Analytics DTO/Service/Gateway/HTTP/DSL;Global/By、metric、missing/numeric、cursor/completeness;Mongo/Elasticsearch shared TCK | 仓库级已证明 | +| 8. 渐进发布与回滚 | Mongo 与 Elasticsearch integration 中的 LEGACY→SHADOW→PLANNED→LEGACY;Spring example `order` target 经同一 facade/storage route 的 mode rehearsal;目标应用 Runbook | 仓库演练已证明;生产未完成 | + +当前仓库门禁结果: + +```text +./gradlew :wow-spring-boot-starter:check PASS +pnpm --dir documentation docs:build PASS +./gradlew allLocalTest allContractTest allIntegrationTest PASS +./gradlew detekt build PASS +git diff --check PASS +``` + +因此当前不能宣称 12.3 的整体 Definition of Done 已完成,原因不是仓库内已知测试失败,而是以下证据只能在 +真实目标应用或发布流程中产生: + +- 当前工作树尚未形成已合并、远端 required checks 全绿的发布提交; +- 目标应用尚未提供受认证的 `QueryWebAuthorityResolver`/direct-call grant,框架保持 fail closed,禁止自动提升为 + `System` authority; +- 目标 Aggregate 的真实 schema、Mongo/Elasticsearch binding、mapping/index inventory、历史值长度与 `_ignored` + 审计尚未签署; +- 生产或等价预发布环境尚未留下 SHADOW 差异、fallback reason、deadline/budget、性能阈值和资源使用快照; +- 同名 Elasticsearch concrete index 到 alias 的转换以及 EventStream generation mirror/watermark 仍属于需审批的 + 数据迁移边界,未执行时不得把回滚描述为无损。 + +目标应用必须按 +[Query 服务目标应用发布与回滚 Runbook](./2026-08-09-query-service-application-rollout-runbook.md) +补齐 authority、schema/binding、preflight、LEGACY 基线、SHADOW 观察、PLANNED cutover、rollback 和 operations sign-off。 +任何安全、完整性、cursor 资源、mapping readiness 或 portable semantic 差异非零,立即按第 13 节停止条件回到 +`LEGACY`,不得用 silent fallback 掩盖。 + +## 13. 风险登记与统一停止条件 + +| 风险 | 触发条件 | 控制与证明 | +|---|---|---| +| 双重语义源 | 直接 cherry-pick PR #2903 的 legacy converter,同时新增 Normalizer/Planner | 只迁移 fixtures/validator contract;planned compiler 禁止调用 legacy converter;architecture test 搜索依赖 | +| internal model 意外成为 ABI | Phase 1 类型 public,或被 `wow-api`/OpenAPI 引用 | 使用 Kotlin `internal` visibility,并禁止公开签名引用;public reflection/OpenAPI golden;仓库增加 ABI guard | +| mandatory 条件被绕过 | direct Service、Filter rewrite、Native payload 或 Backend 自行追加条件 | 所有入口统一 Gateway;typed policy builder;外层 AND provenance;负向安全测试 | +| Mongo/ES 语义漂移 | 相同 DTO 分别由两个 converter 解释 | 单一 Plan + Backend binding;Mongo expected TCK;mapping readiness;unsupported fail closed | +| 字符串 mapping 猜测 | compiler 硬编码 `.keyword`、依赖默认 dynamic mapping 或全局 string-as-keyword | logical string capability;显式 source/search/exact/literal/sort/group binding;mapping digest/readiness | +| exact 字段静默漏数 | 历史值或新值超过 `ignore_above`,仍在 `_source` 但未被索引 | schema/write 长度约束;历史数据审计;重建/回填;不满足时撤销 exact/sort/group capability | +| `Exact` 被误当 `Snapshot` | cursor 多页期间有并发写 | completeness/consistency 分离;Eventual 明示;Snapshot capability 不满足时执行前拒绝 | +| Mongo aggregation 成本失控 | 高基数 `$group/$sort`、每页重扫、自动落盘 | maxTime/scan/bucket/page budget;allowDiskUse policy;真实 fixture explain/profile;无通用性能承诺 | +| PIT/search context 泄漏 | client 停止翻页、error/cancel、旧 cursor | 短 TTL lease、最新 PIT id、terminal/error/cancel close、nodes stats/资源测试 | +| mapping/alias 迁移不可回滚 | concrete index 直接替换、EventStream 未镜像、旧 generation 过早删除 | 显式 rebuild/verify/cutover;旧 generation 保留;EventStream pause/drain/dual write;rollback rehearsal | +| EventStream 粒度误报 | 把一个 `DomainEventStream` document count 当作 domain-event count | target/grain capability 显式;event-level analytics 走专用 Projection;TCK 固化 document shape | +| numeric precision 漂移 | Long 超过安全范围、Decimal128/double、全 null/missing、overflow | logical numeric policy/type promotion;双 Backend type/value TCK;不满足时 unsupported | +| 生命周期/缓存重复 owner | Factory、Provider、Gateway、registry 同时缓存 Backend/Service | registry 是 planned Backend 唯一 owner;缓存 identity/concurrency tests;Phase 2 接线审计 | +| 大 PR 无法审查或回滚 | 模型、接线、Backend、public API、迁移混在同一 PR | 执行 10.2 slice/stack 规则;每 slice 独立 exit gate 与 rollback | + +出现以下任一情况必须停止 planned rollout,而不是增加 fallback 掩盖问题: + +- mandatory/result policy 可能被绕过; +- partial/approximate 结果被作为 exact success 返回; +- portable TCK 出现无法解释的 identity/order/metric/type 差异; +- fallback 没有 reason/metric,或 budget/deadline 被 Backend 忽略; +- mapping digest/readiness 不一致; +- exact/sort/group binding 存在 `_ignored` 或超长未索引值,或 search/literal field 类型与声明不符; +- cursor/PIT 无法有界释放; +- index rebuild/checksum/version continuity 或 rollback rehearsal 未通过。 + +停止后 record path 按 Aggregate 回到 `LEGACY` 或已验证的 Mongo planned route;Elasticsearch 不执行 alias +cutover/删除。已经写入新 EventStream generation 且旧 generation 未镜像时,以权威事件流补齐后再决定回切, +禁止只改路由伪造数据回滚。 + +## 14. 后端语义依据 + +- [MongoDB `$group`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/group/):blocking stage 与内存/落盘边界; +- [MongoDB `$avg`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/avg/):numeric、missing、array 与返回类型语义; +- [Elasticsearch composite aggregation](https://www.elastic.co/docs/reference/aggregations/search-aggregations-bucket-composite-aggregation): + bucket 分页、`after_key`、source ordering 与 pipeline aggregation 限制; +- [Elasticsearch terms aggregation](https://www.elastic.co/docs/reference/aggregations/search-aggregations-bucket-terms-aggregation): + doc count/sub-aggregation 近似性和 error bound; +- [Elasticsearch PIT](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time): + 跨请求 index state、一致性、keep-alive 与资源成本。 +- [Elasticsearch multi-fields](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/multi-fields): + 同一 source value 的全文、精确、排序和聚合多种索引方式,以及新增 multi-field 后的历史数据回填边界; +- [Elasticsearch keyword type family](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/keyword): + keyword/text/wildcard 适用场景、doc values、`ignore_above` 与动态 `.keyword` 的完整性限制; +- [MongoDB `$regex`](https://www.mongodb.com/docs/manual/reference/operator/query/regex/) 与 + [MongoDB `$text`](https://www.mongodb.com/docs/manual/reference/operator/query/text/):字面量模式与 text index 搜索边界。 diff --git a/document/design/2026-08-08-elasticsearch-query-index-migration-runbook.md b/document/design/2026-08-08-elasticsearch-query-index-migration-runbook.md new file mode 100644 index 00000000000..4672ea26128 --- /dev/null +++ b/document/design/2026-08-08-elasticsearch-query-index-migration-runbook.md @@ -0,0 +1,194 @@ +# Elasticsearch 查询索引迁移与回滚 Runbook + +## 1. 适用范围 + +本 Runbook 只适用于 Query Gateway 版本化索引生命周期工具管理的 Elasticsearch 索引: + +- stable alias 已指向一个受管 physical generation; +- physical index 名符合 `-v<4-digit-mapping-version>-<6-digit-generation>`; +- Snapshot 使用 `SNAPSHOT_FROM_EVENT_STREAM` 权威重建; +- EventStream 仅使用经过审批、由外部 write-fence/barrier 提供稳定 watermark 的 pause-and-drain; +- mapping、logical schema 与 planned binding 已生成一致的 `schemaContractId` 和 `capabilityDigest`。 + +工具不会在应用启动时创建 system index、创建 template/physical index、重建数据、切换 alias 或删除旧索引。 +所有动作必须由显式管理命令触发。 + +以下场景不在当前工具的安全执行范围内: + +- stable 名当前是 concrete index,而不是 alias; +- source generation 不是 manifest 声明的 exact physical index; +- 计划依赖未验证的动态 `.keyword`、analyzer/normalizer、nested mapping 或 text index; +- migration 未使用与 document kind 匹配的 manifest 固定 checksum,目标应用尚未配置可重复的 + record/analytics probe suite,或 EventStream 没有可证明停写并排空的生产 barrier; +- EventStream 选择 `EVENT_STREAM_CONTROLLED_MIRROR`;当前 internal vertical slice 会稳定拒绝该策略; +- 需要删除、改名或覆盖现有索引。 + +上述任一条件成立时必须停止。特别是 concrete index 到同名 alias 的转换,Elasticsearch 不允许两者同时存在, +需要独立审批的停写、备份、权威重建、删除/改名和恢复演练,不能复用本 Runbook 静默处理。 + +## 2. 不变量与职责 + +| 边界 | 不变量 | +|---|---| +| Manifest | 精确绑定 target、mapping version、generation、schema contract、capability digest、source physical、重建策略、checksum algorithm、probe suite、cursor TTL 与 rollback window | +| Repository | `.wow-query-index-lifecycle-v1` 由管理工具显式创建;repository document/mapping format 为 `v1`,有界 lifecycle payload codec 为 `v2`;document create 注册,`_seq_no + _primary_term` CAS 更新;旧/损坏/跨 migration payload fail closed,不做隐式迁移 | +| Template | component template 先于 composed index template;`allow_auto_create=false`;mapping `_meta` 必须与 manifest 完全一致 | +| Rebuild | Snapshot 由框架按 AggregateId 分页并从权威 EventStore 全量重放,幂等写 exact physical generation;禁止把旧索引 `_reindex` 当作权威重建 | +| Verify | authority 与 exact physical index 使用 document-kind-specific canonical checksum;count、identity/content checksum、version continuity、watermark、record/analytics probe 必须全部满足,差异阈值固定为零 | +| Cutover | destination verification 之后仍必须由目标应用提供受信的写栅栏。Snapshot 默认 guard 为 deny;只有 pause/drain 或受控 mirror 证明写入窗口已封闭,才允许单个 `_aliases` 请求完成切换并回读确认 | +| Retention | source 至少保留 `maxCursorTtl + rollbackWindow`;工具不删除 generation | +| Rollback | cutover 后必须对 source 做一次新的 verification;旧报告不能复用 | + +## 3. 迁移前检查 + +在变更单中固定以下输入,执行期间不得动态改写: + +1. `migrationId`、每一步唯一 `commandId`、目标 Aggregate 与 document kind; +2. source alias、source physical、destination physical; +3. mapping version/generation、`schemaContractId`、`capabilityDigest`; +4. Snapshot 权威事件流范围或 EventStream pause/drain/mirror 方案; +5. `maxCursorTtl`、`rollbackWindow`、最早允许删除旧 generation 的时间; +6. manifest `verificationContract`:Snapshot 使用 `CANONICAL_DOCUMENT_SHA256_V1`,EventStream 使用 + `CANONICAL_EVENT_STREAM_SHA256_V1`,并固定经过审批的 record/analytics probe suite id; +7. Query execution profile 的当前值、SHADOW 观测窗口、回退负责人; +8. Elasticsearch 集群备份/恢复证据与变更窗口。 + +执行只读预检: + +- alias 恰好有一个未过滤、无 routing 的 write index; +- source physical 存在并与 alias target 相同; +- destination 不存在,或其 mapping attestation 与 manifest 完全相同; +- planned binding readiness 已通过,未出现 `_ignored`、超长 exact 值或 mapping drift; +- lifecycle repository system index mapping/document format 为 strict `v1`,payload codec 必须为当前 `v2`; +- 没有另一个 active command;本次 command 的 `expectedRevision` 等于持久化 state revision。 +- Snapshot CUTOVER 已配置目标应用受信的 `ElasticsearchSnapshotCutoverGuard`;默认 `DENY` 会稳定返回 + `CUTOVER_FENCE_REQUIRED`。仅有 destination verification、SHADOW 零差异或一次 checksum 相等都不能替代写栅栏。 + +任一预检失败不得进入 CREATE。 + +## 4. 标准执行序列 + +每一步先执行 dry-run/plan,人工核对 `from`、`to`、`expectedRevision`、target 与 physical index,再执行命令。 +同一步重试必须复用原 `commandId`;不得用新 command 绕过 active command。 + +### 4.1 初始化 repository 与注册 manifest + +1. 显式执行 repository `ensureIndex()`;重复执行必须幂等; +2. 注册 immutable manifest;相同 `migrationId` 绑定不同 manifest 必须失败; +3. 保存返回的 state revision、Elasticsearch `_seq_no` 与 `_primary_term` 到变更证据。 + +### 4.2 VALIDATE + +读取 alias 与所有受管 generation,确认 source/destination attestation。状态必须从 `NEW` 进入 `VALIDATED`。 + +### 4.3 CREATE + +幂等写 versioned component/index template,显式创建 destination physical index并回读 `_meta`。状态必须从 +`VALIDATED` 进入 `CREATED`。禁止把 alias 加到 destination。 + +### 4.4 REBUILD + +按 manifest 策略从权威源写 destination: + +- Snapshot:使用 EventStore authoritative rebuilder 按 aggregate identity/version 重放全部事件;source page 必须严格递增且不超过请求上限,结果通过 version guard 幂等写入 manifest exact physical index; +- EventStream pause/drain:外部系统先停止新写并排空已接受写入;工具在复制前后读取同一 barrier watermark,只有 + watermark 稳定、aggregate/page/version/event-body 契约完整时才签发 receipt; +- EventStream controlled mirror:当前未实现,工具稳定拒绝,不得用 pause/drain 实现冒充。 + +receipt 必须记录 authoritative/indexed watermark 与完成时间。状态从 `CREATED` 进入 `REBUILT`。 + +### 4.5 VERIFY destination + +针对 destination 重新计算完整 verification: + +- expected/actual count; +- identity checksum; +- content checksum; +- aggregate/event version continuity; +- authoritative/indexed watermark; +- record probe mismatch count; +- analytics probe mismatch count。 + +全部满足后状态从 `REBUILT` 进入 `VERIFIED`。任何差异都保持 alias 指向 source。 + +Snapshot 的 `CANONICAL_DOCUMENT_SHA256_V1` 固定以下语义:EventStore 全量重放结果与 exact physical index 都按 +`aggregateId` 严格递增扫描;object key 排序、array 顺序保留、数学等价 number 规范化,identity 与每条 canonical +document 分别做 length-prefixed SHA-256;只排除重建时生成的顶层 `snapshotTime`。physical 端在 PIT 内使用 +`aggregateId > lastIdentity` keyset 分页,拒绝 partial shard、非 exact/不稳定 total、identity/source 不一致、缺失或 +负 version、重复/乱序和超预算 document。算法或 probe suite 与 manifest 不一致时 verification fail closed。 + +EventStream 的 `CANONICAL_EVENT_STREAM_SHA256_V1` 固定以下语义:authority 端按 aggregate id 分页读取完整 +`DomainEventStream` 历史,physical 端在 PIT 内按 `(aggregateId, version)` keyset 扫描;两端都要求 aggregate +严格递增、每个 aggregate 从初始版本开始且版本按 event count 连续、document id 等于 `aggregateId-version`、body +非空,并对完整 canonical document 生成 identity/content checksum。authority 与 destination watermark 必须分别在扫描 +前后保持稳定且最终相等。工具只验证外部 barrier 的证明,不负责主动暂停生产 writer。 + +Probe suite 精确绑定 manifest 固定的 suite id、`QueryTarget` 与 `SchemaContractId`,且只接受预注册的 probe id;不接受 +临时 wire query、driver object 或 `Any` payload。 +internal bounded runner 最多执行 256 个 canonical probe,分别从 authority 与 exact physical generation 取得完整、不可变的 +`resultCount + resultChecksum` evidence;任一 empty、incomplete、executor error 或 typed error 都统一为 verification failure。 +record 与 analytics mismatch 分开计数且阈值都固定为零。具体 probe catalog、authority evaluator 和 physical evaluator 必须由 +目标应用提供并经过评审;仅有 runner 不构成目标应用验收证据。 + +### 4.6 CUTOVER + +`VERIFIED` 只证明 verification 时刻 destination 与权威状态一致,不会阻止 Snapshot writer 在 verification 与 alias 切换之间 +继续写 source。执行 CUTOVER 前,外部系统必须暂停/排空目标 Snapshot 写入,或启用经过审批、可证明 destination 已追平的 +controlled mirror;随后由目标应用实现的 `ElasticsearchSnapshotCutoverGuard` 对 exact manifest 与 verification 签发一次性 +许可。仓库默认 guard 是 `DENY`,本次代码/PR 没有生产 guard,也不授权任何真实 alias 切换。 + +只有 guard 成功后,才在批准窗口执行一次 CAS alias transition。完成后: + +1. 回读 alias,确认唯一 write index 是 destination; +2. 保存 transition timestamp 与 `retainedSourceUntil`; +3. Query profile 先保持 SHADOW,核对 error/diff/incomplete 指标; +4. 不删除、不封闭 source generation。 + +状态从 `VERIFIED` 进入 `CUTOVER`。 + +### 4.7 VERIFY source for rollback + +cutover 后对 source 重新执行 verification,证明它仍满足当前权威状态。只有新报告通过,状态才从 `CUTOVER` +进入 `ROLLBACK_VERIFIED`。EventStream 若未持续 mirror,通常不能通过此步骤,也不得声称可直接回切。 + +## 5. 回滚 + +触发条件包括语义差异、incomplete result、mapping failure、性能阈值越界或业务验收失败。 + +1. 先把目标 Query execution profile 回到 `LEGACY`,停止 planned 业务流量; +2. 确认 state 为 `ROLLBACK_VERIFIED`,source verification 是 cutover 后的新报告; +3. 执行 `ROLLBACK` CAS alias transition:remove destination,add source 为唯一 write index; +4. 回读 alias并执行 smoke/probe; +5. destination 保留供审计,不删除; +6. 若 source 不再新鲜,禁止直接回切,改为从权威源新建 generation。 + +`wow.query.gateway.legacy-wiring-rollback` 会绕过新的 admission/policy,不是常规索引回滚手段,只能作为单独审批、 +限时的框架 wiring 应急开关。 + +## 6. 故障恢复 + +| 故障 | 处理 | +|---|---| +| 管理进程在外部动作前退出 | 读取持久化 state;若无 active command,按 revision 重新 plan | +| 管理进程在外部动作后、state 完成前退出 | 使用原 command id 重试;外部 port 必须幂等,Executor 恢复同一 active command | +| stale CAS | 重新 load state;不得覆盖;检查是否由另一管理实例推进 | +| repository payload 损坏 | `REPOSITORY_CORRUPTED` 停止;保留原文档与审计日志,禁止手工改 payload 后继续 | +| alias 与 expected source 不一致 | `ALIAS_CONFLICT` 停止;先确认是否存在带外变更 | +| template/mapping attestation 不一致 | 停止;新建 mapping version/generation,不覆盖现有 destination | +| verification 失败 | 保持/恢复 LEGACY;保留 destination 调查,修复权威 rebuild 后以新 command 重新执行 | +| Snapshot cutover 未配置写栅栏 | `CUTOVER_FENCE_REQUIRED` 停止;不得用测试 allow guard、人工跳过或重复命令绕过 | + +## 7. 证据与完成标准 + +每次演练/生产迁移必须保留: + +- immutable manifest 与所有 command plan/result; +- repository revision、`_seq_no/_primary_term` 演进; +- template、mapping `_meta`、alias before/after; +- rebuild receipt 与完整 verification; +- SHADOW error/diff/incomplete 指标、延迟与资源快照; +- cutover、rollback rehearsal 与 `retainedSourceUntil`; +- 目标应用负责人签署。 + +未同时满足“权威重建、零差异 verification、真实 alias cutover/rollback 演练、回退证据”时,P5-B 只能标记为 +管理内核/持久化 vertical slice 已实现,不能标记为生产迁移闭环完成。 diff --git a/document/design/2026-08-09-query-service-application-rollout-runbook.md b/document/design/2026-08-09-query-service-application-rollout-runbook.md new file mode 100644 index 00000000000..339fd3488d5 --- /dev/null +++ b/document/design/2026-08-09-query-service-application-rollout-runbook.md @@ -0,0 +1,164 @@ +# Query 服务目标应用发布与回滚 Runbook + +## 1. 适用范围 + +本 Runbook 用于把一个精确的 `QueryTarget(context, aggregate, documentKind)` 从兼容执行逐步切到 +`QueryGateway` planned Backend。它覆盖记录查询与 Snapshot Analytics 的应用接线、SHADOW、PLANNED 和回滚, +不替代 Elasticsearch 索引生命周期 Runbook,也不授权任何生产写入、索引删除或 alias 切换。 + +每次发布只允许选择明确的 `target + operation`。禁止用全局开关同时切换所有 Aggregate;禁止把 +`wow.query.gateway.legacy-wiring-rollback` 当作普通回滚手段,该开关会绕过 admission、policy 和生命周期保护。 + +以下条件任一成立时必须停止: + +- 应用没有从已认证上下文生成的 `QueryAuthority`,或仍把 path/header/调用方自报 id 当作 authority; +- 目标没有经过评审的逻辑 `QueryDocumentSchema` 和精确 Backend binding; +- Backend readiness、mapping/index、Mongo collation 或 text/keyword 完整性证据缺失; +- mandatory tenant/owner/space/deleted 字段无法在 Backend 中无损执行; +- 目标操作会落入 silent fallback、近似 total、partial result 或未受控 unbounded stream; +- Elasticsearch stable 名仍是同名 concrete index,却计划直接创建同名 alias; +- EventStream 迁移依赖尚未实现的 `EVENT_STREAM_CONTROLLED_MIRROR` 或不可证明的生产 watermark。 + +## 2. 发布前固定输入 + +变更单必须固定以下内容,执行期间不得动态修改: + +| 输入 | 必填证据 | +|---|---| +| Target | `contextName`、`aggregateName`、`documentKind` | +| Operations | `SINGLE/STREAM/PAGE/COUNT/ANALYZE` 的独立清单 | +| Schema | `schemaContractId`、字段类型/operator/capability、search scope | +| Backend binding | `backendId`、`capabilityDigest`、物理 namespace/index generation、readiness report | +| Authority | resolver/provider 代码版本、tenant/owner/space grant 负测、缺失/错误 authority 的 storage-zero 证据 | +| Budget | deadline、returned/scanned/page/candidate bucket/cursor page 上限及目标应用阈值 | +| Cursor | store owner、HMAC current/previous key id、TTL、容量、reaper owner;无 cursor 的操作写 `N/A` | +| Probe suite | 固定 suite id、canonical query/input、expected identity/order/value/total/completeness | +| Observability | shadow outcome、fallback reason、Backend error、deadline、latency、resource 和 cursor 指标面板 | +| Rollback | profile 回切负责人、目标恢复时间、索引/lease 处置、证据保存位置 | + +安全与 exact semantic 阈值固定为零差异,不能由目标应用放宽。延迟、吞吐、CPU、内存、扫描量等性能阈值必须由目标 +应用基于可重复负载签署,框架不提供伪造的通用默认值。 + +## 3. 应用接线预检 + +### 3.1 可信上下文 + +1. HTTP 使用 `QueryWebAuthorityResolver` 从认证结果生成 authority;tenant/owner/space 只是 selector; +2. 新进程内调用使用显式 `QueryCall` 与 `QueryAuthorityResolver`; +3. 兼容七方法只能选择预注册的 exact `QueryLegacyGrant`;grant 必须同时固定 target、purpose、mode 和 scope; +4. authority provider 的 empty/error、selector mismatch、跨 tenant、无权限字段和 Native 请求全部在 storage 前拒绝; +5. 不允许默认 `System` authority,不允许从可猜字符串或公开 Reactor context key 注入 trusted authority。 + +### 3.2 Schema 与 Backend binding + +1. Schema 只包含逻辑字段,不包含 `.keyword`、Mongo `_id`、ES index 名或 driver 对象; +2. binding 覆盖 Schema 的每个可执行字段,并固定 system field、value encoding、collation/analyzer/normalizer、nested owner; +3. contribution 的 target、schema contract、capability digest 与当前 storage route 精确一致; +4. Mongo/Elasticsearch readiness 在应用启动或显式管理步骤中 fail closed;未 ready 不能广告 capability; +5. record/analytics compiler 只接收 validated Plan,拒绝 RAW/未绑定 Native/未声明 Search; +6. identity 独立返回,logical projection 不泄漏 Backend 为执行而补取的物理字段。 + +### 3.3 运行时与资源 + +1. `QueryExecutionProfiles` 默认保持 `LEGACY + COMPATIBLE`,只为本次 target/operation 添加 override; +2. SHADOW 必须配置有界 supervisor 和 observation sink;拒绝、饱和、超时不能静默丢失; + `QueryShadowObserver` 与 `QueryRuntimeHealthObserver` 都必须实际接入,后者固定记录 fallback、shadow supervisor failure + 与 cursor cleanup failure 的低基数 reason code; +3. Snapshot cursor 需要匹配 target/backend 的持久化 lease store 和 lifecycle closer; + 首次签发的完整 budget ceiling 会进入签名 lease,continuation 只能保持或收紧,不能通过下一页放宽扫描、返回、窗口、 + bucket、cursor page 或 `allowDiskUse`; +4. cursor reaper 只能有一个 lifecycle owner;Mongo TTL 是遗弃 lease 的最终安全网,不是正常关闭路径; +5. Elasticsearch PIT、Mongo Publisher、deadline 与 cancel 的资源释放回归必须在目标版本上通过。 + +## 4. 分阶段执行 + +### 4.1 LEGACY 基线 + +- 保持目标 operation 为 `LEGACY`; +- 运行固定 probe suite,记录结果 checksum、错误分类、p50/p95/p99、扫描量和资源基线; +- 证明所有入口已经经过 Gateway,即使最终由 legacy Backend 执行; +- 清空未知 fallback reason;无法规范化的已知请求必须登记 owner 和迁移方案。 + +### 4.2 SHADOW + +- 只把一个 target/operation 改为 `SHADOW`;validation 建议先 `STRICT`; +- 返回值始终来自 legacy primary;planned probe 只能由有界 supervisor 订阅; +- 观察窗口必须覆盖正常、空结果、边界分页、mandatory scope、超时、取消和错误请求; +- `VALUE_MISMATCH/PROBE_ERROR/INCOMPLETE_RESULT/MAPPING_FAILURE` 阈值为零; +- `SKIPPED/SATURATED` 必须有原因并低于目标应用签署阈值,否则不得进入 PLANNED。 + +### 4.3 PLANNED canary + +- 只切已完成 SHADOW 签署的 operation;不要把未支持的 PAGE/ANALYZE/unbounded 一并切换; +- planned route missing/not-ready、schema/mapping generation drift 必须在 storage 前失败,禁止自动回 legacy; +- canary 期间同时验证 HTTP、聚合 Bean、新进程内 Gateway 和后台任务入口; +- 对 Snapshot Analytics 验证 EVENTUAL continuation 与 SNAPSHOT PIT continuation、terminal/error/cancel/expiry; +- 达到应用签署的最短窗口与样本量后,才扩大流量或下一个 operation。 + +### 4.4 完成切换 + +- 每个 operation 单独保存 `LEGACY -> SHADOW -> PLANNED` 的时间、配置 diff、probe 结果和指标快照; +- fallback reason 必须为零;unsupported operation 继续显式 LEGACY/unsupported,不包装成完成; +- 保留 legacy Backend 和旧索引 generation 至少一个已批准的 rollback window; +- Elasticsearch alias cutover 另按 + [Elasticsearch 查询索引迁移与回滚 Runbook](./2026-08-08-elasticsearch-query-index-migration-runbook.md) 执行。 + +## 5. 回滚 + +触发条件包括任何语义/安全差异、incomplete result、mapping failure、cursor 泄漏、deadline/cancel 失效、性能阈值越界或 +目标应用验收失败。 + +1. 先把精确 target/operation profile 回到 `LEGACY`; +2. 证明新请求不再命中 planned Backend,raw legacy 调用恢复; +3. 等待或显式关闭已存在的 PIT/cursor lease,不删除仍在 TTL/rollback window 内的 state; +4. 如果同时发生 Elasticsearch alias 迁移,只能在 source 完成 cutover 后新 verification 时回切; +5. 保存 planned error、shadow observation、profile diff、cursor/index 状态和回滚完成时间; +6. 只有 Gateway 本身阻断所有兼容流量且经过单独审批时,才允许限时使用 legacy wiring rollback;恢复后立即关闭。 + +## 6. 签署记录模板 + +```yaml +queryRolloutEvidence: + target: + contextName: "" + aggregateName: "" + documentKind: SNAPSHOT + operations: [] + artifactVersion: "" + gitCommit: "" + schemaContractId: "" + backendId: "" + capabilityDigest: "" + mappingGeneration: "" + authorityReview: "" + probeSuiteId: "" + phases: + legacyBaseline: { startedAt: "", completedAt: "", evidence: "" } + shadow: { startedAt: "", completedAt: "", evidence: "" } + plannedCanary: { startedAt: "", completedAt: "", evidence: "" } + rollbackRehearsal: { startedAt: "", completedAt: "", evidence: "" } + semanticMismatchCount: 0 + securityMismatchCount: 0 + incompleteResultCount: 0 + fallbackCount: 0 + performanceThresholds: "application-approved evidence link" + cursorAndIndexState: "evidence link or N/A" + rollbackOwner: "" + applicationOwnerApproval: "" + securityApproval: "" + operationsApproval: "" +``` + +空字段、未附 evidence 的零值、只引用单元测试、或仅写“CI 通过”都不能作为生产签署。 + +## 7. 当前明确未闭环的边界 + +- `EVENT_STREAM_CONTROLLED_MIRROR`:现有 EventStore 没有全局单调 watermark 与 `scanAsOf(watermark)`,继续 fail closed; +- 同名 concrete Elasticsearch index 到 managed alias:需要停写、备份、权威重建及删除/改名审批;工具不会自动执行; +- Snapshot alias CUTOVER 默认由 `ElasticsearchSnapshotCutoverGuard.DENY` 拒绝;目标应用没有受信 pause/drain 或 controlled + mirror 证明时,`CUTOVER_FENCE_REQUIRED` 是预期结果,不得临时替换为测试 allow guard; +- Mongo 精确 scanned-record enforcement、目标数据分布性能阈值、ES mapping inventory:必须由目标应用实测; +- EventStream analytics、unbounded stream 和未证明的 string/search capability:保持 unsupported。 + +仓库内的真实 Mongo/Elasticsearch 容器演练和 `example-service/order` Spring mode 回归只证明框架机制可执行,不能替代 +本模板要求的生产 authority、数据、mapping、流量与负责人签署。 diff --git a/documentation/docs/en/guide/query.md b/documentation/docs/en/guide/query.md index 96ee319abd5..8c4a717c308 100644 --- a/documentation/docs/en/guide/query.md +++ b/documentation/docs/en/guide/query.md @@ -260,11 +260,15 @@ pagedQuery { ## Rewrite Query +`PreAdmissionQueryFilter` is only a compatibility hook for rewriting query input. Results written in this phase are +discarded, and appended conditions remain user conditions. Do not use it as a tenant, ABAC, or authorization boundary; +security constraints must be emitted as mandatory conditions by the Query Gateway policy. + ```kotlin @Component @Order(ORDER_FIRST) @FilterType(SnapshotQueryHandler::class) -class DataFilterSnapshotQueryFilter : SnapshotQueryFilter { +class DataFilterSnapshotQueryFilter : SnapshotQueryFilter, PreAdmissionQueryFilter { override fun filter( context: QueryContext<*, *>, @@ -295,6 +299,35 @@ This means developers usually only need to focus on writing domain models to com The examples below query the `sales-order` aggregate for `tenant-1`. All four requests describe the same synthetic snapshot, so their conditions and response counts stay consistent. +::: warning Query endpoints deny anonymous access by default +The Query Gateway never treats a path, `Wow-Space-Id`, or another request header as trusted identity evidence. An +application must implement `QueryWebAuthorityResolver` and return `Mono` from an authenticated +principal/security context. The default resolver is empty, so a query without authentication integration returns +`403 Query.ACCESS_DENIED.AUTHORITY_REQUIRED`. The curl examples below assume the application resolves the +`Authorization` credential to the corresponding tenant/owner/space grants. + +```kotlin +fun interface QueryAuthorityService { + fun resolveAuthenticated(request: ServerRequest): Mono +} + +@Bean +fun queryWebAuthorityResolver(authorityService: QueryAuthorityService): QueryWebAuthorityResolver = + QueryWebAuthorityResolver { request -> + authorityService.resolveAuthenticated(request.request) + } +``` + +Tenant, owner, and space remain resource selectors. The resolver must compare them with authenticated authority and +deny a conflict instead of silently falling back to personal scope. +::: + +Generated Snapshot/Event query, load, and Analytics endpoints declare one Query failure contract: `400` for an +invalid query, cursor, or unsupported capability; `403` for denied access; `408` for an expired deadline; `429` for +an exceeded budget; `502` for an incomplete result; `503` for an unavailable backend; `504` for a backend timeout; +and `500` for mapping or internal failures. Responses continue to use the `DefaultErrorInfo` JSON body and the +`Wow-Error-Code` header; Analytics/Cursor does not introduce a second error envelope. + ![Query Service](../../public/images/query/open-api-query.png) ### Paged Query @@ -305,6 +338,7 @@ The examples below query the `sales-order` aggregate for `tenant-1`. All four re curl -X 'POST' \ 'http://localhost:8080/tenant/tenant-1/sales-order/snapshot/paged' \ -H 'accept: application/json' \ + -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'Wow-Space-Id: space-1' \ -d '{ @@ -360,6 +394,7 @@ eq("state.status", "CREATED") curl -X 'POST' \ 'http://localhost:8080/tenant/tenant-1/sales-order/snapshot/list' \ -H 'accept: application/json' \ + -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'Wow-Space-Id: space-1' \ -d '{ @@ -403,6 +438,7 @@ eq("state.status", "CREATED") curl -X 'POST' \ 'http://localhost:8080/tenant/tenant-1/sales-order/snapshot/count' \ -H 'accept: application/json' \ + -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'Wow-Space-Id: space-1' \ -d '{ @@ -427,6 +463,7 @@ eq("state.status", "CREATED") curl -X 'POST' \ 'http://localhost:8080/tenant/tenant-1/sales-order/snapshot/single' \ -H 'accept: application/json' \ + -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'Wow-Space-Id: space-1' \ -d '{ @@ -454,6 +491,146 @@ eq("state.status", "CREATED") ::: +### Analytics Query + +Analytics uses separate `AnalyticsQuery` / `AnalyticsPage` contracts and does not add a method to the seven-method +`QueryService`. Every field is logical; `.keyword`, index names, Mongo/Elasticsearch native queries, and backend options +are not part of the public request. `Int64`, `Decimal`, and `Instant` results use typed canonical strings so JavaScript +and JSON number parsing cannot lose precision. + +This release contains an approved upgrade of the public Query, Analytics, and Cursor contracts marked with +`@ExperimentalQueryGatewayApi` / `@ExperimentalQueryCursorApi`. Applications using those experimental APIs must be +recompiled and migrated to the current constructors and budget fields. The stable seven-method `QueryService` and the +seven `QueryType` values remain unchanged. OpenAPI now fixes the conditional Analytics shape with `oneOf`: `GLOBAL` +requires empty dimensions, `limit=1`, and no cursor; `BY` requires at least one dimension; `DOCUMENT_COUNT` has no field, +while every other metric requires one. + +```shell +curl -X POST \ + 'http://localhost:8080/tenant/tenant-1/sales-order/snapshot/analyze' \ + -H 'Authorization: Bearer ' \ + -H 'Content-Type: application/json' \ + -d '{ + "condition": { "operator": "ALL" }, + "grouping": { + "kind": "BY", + "dimensions": [ + { "alias": "status", "field": "state.status", "missingPolicy": "EXCLUDE" } + ] + }, + "metrics": [ + { "alias": "count", "kind": "DOCUMENT_COUNT" }, + { "alias": "total", "kind": "SUM", "field": "state.total" } + ], + "window": { "limit": 100 }, + "numericPolicy": { "scale": 2 }, + "consistency": "EVENTUAL", + "completeness": "EXACT" + }' +``` + +```json +{ + "buckets": [ + { + "keys": { "status": { "type": "TEXT", "value": "PAID" } }, + "metrics": { + "count": { "type": "INT64", "value": "42" }, + "total": { "type": "DECIMAL", "value": "120.50" } + } + } + ], + "nextCursor": null, + "consistency": "EVENTUAL", + "completeness": "EXACT" +} +``` + +`grouping.kind=GLOBAL` requires `dimensions=[]`, `window.limit=1`, and no cursor. A `BY` query's `nextCursor` is an +opaque URL-safe token of at most 256 characters. A client may only copy it into the next request's `window.cursor`; it +must never parse, modify, or manufacture the token. The server validates the target, plan fingerprint, mapping +generation, authority/security binding, and backend before deleting the lease with CAS. Wrong authority or mapping +does not consume the cursor, and only one request may acquire a cursor page. + +When the first page issues a cursor, the server also stores the complete `QueryExecutionBudget` ceiling in the signed +lease. A continuation may only preserve or tighten `maxReturnedRecords`, `maxScannedRecords`, `maxPageWindow`, candidate +and returned bucket limits, `maxCursorPages`, and `allowDiskUse`; it cannot remove or relax an initial bound. An attempted +relaxation returns `CURSOR_BUDGET_RELAXATION_NOT_ALLOWED` without consuming the lease. + +Lease expiry is normalized to millisecond precision before HMAC signing and persistence so a Mongo BSON Date round trip +cannot diverge from the signed envelope. Backend continuation state is limited to `4_096` bytes by default, and the public +hard maximum for `QueryCursorLeaseConfiguration.maxBackendStateBytes` is `1 MiB`; runtime validation and the cursor codec +use the same configured value. + +A multi-instance deployment must provide a shared `QueryCursorLeaseStore`. The MongoDB implementation uses bounded +slots, a unique lease id, revision CAS, and a grace-delayed TTL. It never creates the collection or indexes implicitly at +application startup; initialize them through an explicit, controlled operation: + +```kotlin +val store = MongoQueryCursorLeaseStore( + cursorDatabase, + MongoQueryCursorLeaseStoreOptions( + maxEntries = 65_536, + retentionGrace = Duration.ofMinutes(5), + ), +) + +// Run once from a migration/operations entry point, never from a normal query request. +store.ensureIndexes().block() + +@Bean +fun queryCursorLeaseConfiguration( + signingKeys: QueryCursorSigningKeys, // Build from a managed Secret; never log or commit key material. +): QueryCursorLeaseConfiguration = QueryCursorLeaseConfiguration( + store = store, + signingKeys = signingKeys, + maxBackendStateBytes = 4_096, +) +``` + +MongoDB TTL applies to `expiresAt + retentionGrace`: the framework reaper first attempts revision CAS and Backend-state +cleanup, while TTL remains the final safety net for abandoned leases. No scheduler is created by default. After configuring +the shared store, explicitly enable the Starter's single-owner, bounded serial reaper when desired: + +```yaml +wow: + query: + cursor: + reaper: + enabled: true + initial-delay: 30s + interval: 1m + batch-size: 100 + max-batches-per-run: 10 +``` + +Enabling the reaper without a `QueryCursorLeaseConfiguration` fails application startup. Each run processes at most +`batch-size * max-batches-per-run` leases, never overlaps the previous run, and isolates a store/closer failure so the next +scheduled run remains active. Operations code may still invoke `QueryGatewayRuntime.reapExpiredQueryCursors(batchSize)` +explicitly when Starter scheduling is not used. Normal query requests never trigger cleanup or DDL. During HMAC rotation, +issue with the new `current` key and keep the old key in `previous` until `maxCursorTtl` has elapsed. + +Mongo Analytics currently declares `EVENTUAL + EXACT`. Elasticsearch grouped Analytics supports `SNAPSHOT + EXACT` when +its mapping/readiness checks pass and a shared cursor store is configured. The PIT id is stored only as opaque server-side +lease state and never appears in the client token; terminal, error, cancellation, capacity rejection, and expired-reaper +paths all attempt to close the PIT. Without a lifecycle closer registered for the exact `QueryTarget + BackendId`, the +request is rejected before storage access. + +For record queries, Mongo planned `PAGE` derives the page and exact total from one matched input plus an in-memory sentinel +and window accumulator. It neither rereads the collection nor packs the page into one BSON document; `SAME_INPUT` is not a +point-in-time snapshot claim. Elasticsearch planned direct `STREAM` accepts only `limit=1..10_000`; a larger limit is +rejected before Elasticsearch I/O, while `limit=0` unbounded streaming remains unsupported instead of silently becoming a +fixed result window. + +Spring also registers a `..AnalyticsQueryService` bean. Direct process calls still require trusted +context, while HTTP calls reuse the query route's authenticated authority. If an aggregate has no matching Analytics +schema/backend readiness, the request is rejected before storage is accessed. + +Enabling any `SHADOW` profile also requires a bounded `QueryShadowConfiguration`, a `QueryShadowObserver`, and a +`QueryRuntimeHealthObserver`; a missing observer fails runtime startup. A health observation contains only target, +operation, kind, and a stable reason code. It deliberately excludes authority, query values, and backend causes and must +feed bounded low-cardinality metrics/alerts instead of becoming a silent fallback. + ## Query Service Registrar @@ -461,6 +638,34 @@ eq("state.status", "CREATED") `SnapshotQueryServiceRegistrar` is used to automatically register all local aggregate root query services into the `Spring` container. Developers can obtain the corresponding `SnapshotQueryService` from the `BeanFactory` using the specified `Bean Name`. +The seven compatibility methods on an aggregate query bean have no explicit context parameter, so they are never +promoted to `System` authority automatically. New code should prefer `QueryGateway` with an explicit `QueryCall`. If a +migration still uses the aggregate bean, register an exact `QueryLegacyGrant` and select it at subscription time with +`withLegacyQueryCaller`. The caller marker can only select a pre-bound +`target + purpose + executionMode + resourceScope`; it cannot broaden the grant: + +```kotlin +@Bean +fun queryLegacyContextResolver(): QueryLegacyContextResolver = QueryLegacyContextResolver( + listOf( + QueryLegacyGrant( + callerId = "order-read-model", + target = QueryTarget( + MaterializedNamedAggregate("example", "order"), + QueryDocumentKind.SNAPSHOT, + ), + purpose = QueryPurpose("order-read-model"), + executionMode = QueryExecutionMode.LEGACY, + resourceScope = QueryResourceScope(tenantId = "tenant-1"), + ), + ), +) +``` + +Without trusted context, the compatibility bean returns `QUERY_CALL_REQUIRED`. The emergency migration switch +`wow.query.gateway.legacy-wiring-rollback=true` is temporary: it bypasses admission, policy, and lifecycle protection, +is supported for one migration version only, and must never be an automatic fallback for authorization or query errors. + > `Bean Name` naming convention: `Aggregate Root Name + ".SnapshotQueryService"`. Usage examples: @@ -476,7 +681,10 @@ class OrderService( condition { id(id) } - }.query(queryService).toState().throwNotFoundIfEmpty() + }.query(queryService) + .withLegacyQueryCaller("order-read-model") + .toState() + .throwNotFoundIfEmpty() } } ``` diff --git a/documentation/docs/zh/guide/query.md b/documentation/docs/zh/guide/query.md index d8a6d8fcaba..8e6531a22b3 100644 --- a/documentation/docs/zh/guide/query.md +++ b/documentation/docs/zh/guide/query.md @@ -260,11 +260,14 @@ pagedQuery { ## 重写查询 +`PreAdmissionQueryFilter` 只适合兼容性的查询改写。该阶段写入的结果会被丢弃,追加的条件仍属于用户条件, +不能作为租户隔离、ABAC 或其他授权边界;安全约束必须由 Query Gateway policy 生成 mandatory condition。 + ```kotlin @Component @Order(ORDER_FIRST) @FilterType(SnapshotQueryHandler::class) -class DataFilterSnapshotQueryFilter : SnapshotQueryFilter { +class DataFilterSnapshotQueryFilter : SnapshotQueryFilter, PreAdmissionQueryFilter { override fun filter( context: QueryContext<*, *>, @@ -295,6 +298,32 @@ class DataFilterSnapshotQueryFilter : SnapshotQueryFilter { 以下示例查询 `tenant-1` 的 `sales-order` 聚合。四个请求都描述同一条模拟快照,因此查询条件与响应数量保持一致。 +::: warning 查询端点默认拒绝匿名访问 +Query Gateway 不会把 path、`Wow-Space-Id` 或其他请求 Header 当作可信身份。应用必须实现 +`QueryWebAuthorityResolver`,从已经认证的 principal/security context 返回 `Mono`;默认 resolver +返回 empty,因此未接入认证的查询会得到 `403 Query.ACCESS_DENIED.AUTHORITY_REQUIRED`。下面的 curl 假设应用已经把 +`Authorization` 凭据解析为相应 tenant/owner/space grant。 + +```kotlin +fun interface QueryAuthorityService { + fun resolveAuthenticated(request: ServerRequest): Mono +} + +@Bean +fun queryWebAuthorityResolver(authorityService: QueryAuthorityService): QueryWebAuthorityResolver = + QueryWebAuthorityResolver { request -> + authorityService.resolveAuthenticated(request.request) + } +``` + +tenant/owner/space 仍然只是资源 selector;resolver 必须将它们与已认证 authority 比较,冲突时拒绝,不能降级为个人范围。 +::: + +生成的 Snapshot/Event 查询、load 与 Analytics 端点统一声明 Query 错误响应:`400`(无效查询、Cursor 或不支持的能力)、 +`403`(拒绝访问)、`408`(deadline)、`429`(预算超限)、`502`(结果不完整)、`503`(Backend 不可用)、 +`504`(Backend 超时)和 `500`(映射或内部错误)。响应继续使用 `DefaultErrorInfo` JSON 与 `Wow-Error-Code` Header, +不会为 Analytics/Cursor 引入第二套错误 envelope。 + ![Query Service](../../public/images/query/open-api-query.png) ### 分页查询 @@ -305,6 +334,7 @@ class DataFilterSnapshotQueryFilter : SnapshotQueryFilter { curl -X 'POST' \ 'http://localhost:8080/tenant/tenant-1/sales-order/snapshot/paged' \ -H 'accept: application/json' \ + -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'Wow-Space-Id: space-1' \ -d '{ @@ -360,6 +390,7 @@ eq("state.status", "CREATED") curl -X 'POST' \ 'http://localhost:8080/tenant/tenant-1/sales-order/snapshot/list' \ -H 'accept: application/json' \ + -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'Wow-Space-Id: space-1' \ -d '{ @@ -403,6 +434,7 @@ eq("state.status", "CREATED") curl -X 'POST' \ 'http://localhost:8080/tenant/tenant-1/sales-order/snapshot/count' \ -H 'accept: application/json' \ + -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'Wow-Space-Id: space-1' \ -d '{ @@ -427,6 +459,7 @@ eq("state.status", "CREATED") curl -X 'POST' \ 'http://localhost:8080/tenant/tenant-1/sales-order/snapshot/single' \ -H 'accept: application/json' \ + -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'Wow-Space-Id: space-1' \ -d '{ @@ -454,11 +487,163 @@ eq("state.status", "CREATED") ::: +### 分析查询(Analytics) + +分析查询使用独立的 `AnalyticsQuery` / `AnalyticsPage` 契约,不修改 `QueryService` 的七个兼容方法。字段均为逻辑字段; +`.keyword`、索引名、Mongo/Elasticsearch 原生查询及 Backend option 不属于公共请求。`Int64`、`Decimal` 与 `Instant` +结果使用带类型的规范字符串,避免 JavaScript/JSON number 精度损失。 + +本版本经审批升级了标记为 `@ExperimentalQueryGatewayApi` / `@ExperimentalQueryCursorApi` 的公共 Query、Analytics、Cursor +契约;使用这些实验 API 的应用需要重新编译,并迁移到当前构造器和 budget 字段。稳定的 `QueryService` 七方法与 +`QueryType` 七个值保持不变。OpenAPI 使用条件 `oneOf` 固定 Analytics 形状:`GLOBAL` 只能有空 dimensions、`limit=1` +且不能带 cursor;`BY` 至少一个 dimension;`DOCUMENT_COUNT` metric 不接受 field,其他 metric 必须带 field。 + +```shell +curl -X POST \ + 'http://localhost:8080/tenant/tenant-1/sales-order/snapshot/analyze' \ + -H 'Authorization: Bearer ' \ + -H 'Content-Type: application/json' \ + -d '{ + "condition": { "operator": "ALL" }, + "grouping": { + "kind": "BY", + "dimensions": [ + { "alias": "status", "field": "state.status", "missingPolicy": "EXCLUDE" } + ] + }, + "metrics": [ + { "alias": "count", "kind": "DOCUMENT_COUNT" }, + { "alias": "total", "kind": "SUM", "field": "state.total" } + ], + "window": { "limit": 100 }, + "numericPolicy": { "scale": 2 }, + "consistency": "EVENTUAL", + "completeness": "EXACT" + }' +``` + +```json +{ + "buckets": [ + { + "keys": { "status": { "type": "TEXT", "value": "PAID" } }, + "metrics": { + "count": { "type": "INT64", "value": "42" }, + "total": { "type": "DECIMAL", "value": "120.50" } + } + } + ], + "nextCursor": null, + "consistency": "EVENTUAL", + "completeness": "EXACT" +} +``` + +`grouping.kind=GLOBAL` 要求 `dimensions=[]`、`window.limit=1` 且不带 cursor。`BY` 的 `nextCursor` 是不超过 +256 字符的 opaque URL-safe token;客户端只能原样放入下一次 `window.cursor`,不得解析、修改或自行构造。 +服务端在校验 target、Plan fingerprint、mapping generation、authority/security binding 和 Backend 后,才以 CAS 删除 lease; +错误 authority 或 mapping 不会消费 cursor,同一 cursor 只允许一个请求成功取得下一页。 + +第一页签发 cursor 时,服务端还会把完整 `QueryExecutionBudget` ceiling 写入签名 lease。后续页只能保持或收紧 +`maxReturnedRecords`、`maxScannedRecords`、`maxPageWindow`、候选/返回 bucket 数、`maxCursorPages` 与 +`allowDiskUse`,不能删除或放宽首次请求的限制;尝试放宽会以 `CURSOR_BUDGET_RELAXATION_NOT_ALLOWED` 拒绝且不消费 lease。 + +lease expiry 在 HMAC 签名和持久化前统一规范到毫秒精度,避免 Mongo BSON Date 往返后与签名 envelope 不一致。 +Backend continuation state 默认最多 `4_096` bytes,`QueryCursorLeaseConfiguration.maxBackendStateBytes` 的公共硬上限为 +`1 MiB`;runtime 校验和 cursor codec 使用同一配置值,不能通过持久化层绕过。 + +多实例部署必须提供共享 `QueryCursorLeaseStore`。MongoDB 实现使用固定容量 slot、唯一 lease id、revision CAS 和带 grace 的 TTL; +集合与索引不会在应用启动时隐式创建,必须先在受控运维步骤中显式执行 `ensureIndexes()`: + +```kotlin +val store = MongoQueryCursorLeaseStore( + cursorDatabase, + MongoQueryCursorLeaseStoreOptions( + maxEntries = 65_536, + retentionGrace = Duration.ofMinutes(5), + ), +) + +// 由迁移/运维入口执行一次,不要放进普通查询请求。 +store.ensureIndexes().block() + +@Bean +fun queryCursorLeaseConfiguration( + signingKeys: QueryCursorSigningKeys, // 从受管 Secret 构造,禁止写入源码或普通配置日志。 +): QueryCursorLeaseConfiguration = QueryCursorLeaseConfiguration( + store = store, + signingKeys = signingKeys, + maxBackendStateBytes = 4_096, +) +``` + +Mongo TTL 使用 `expiresAt + retentionGrace`;framework reaper 会先尝试 revision CAS 并关闭 Backend state,TTL 只是遗弃 lease 的最终 +安全网。默认不创建调度器;配置共享 store 后,可以显式启用 Starter 的单 owner、有界串行 reaper: + +```yaml +wow: + query: + cursor: + reaper: + enabled: true + initial-delay: 30s + interval: 1m + batch-size: 100 + max-batches-per-run: 10 +``` + +启用 reaper 但未提供 `QueryCursorLeaseConfiguration` 会使应用启动失败。每个周期最多处理 +`batch-size * max-batches-per-run` 条 lease,前一轮未完成时不会并发启动下一轮;单轮 store/closer 错误被隔离并在下一周期重试。 +不使用 Starter 调度时,运维入口仍可显式调用 `QueryGatewayRuntime.reapExpiredQueryCursors(batchSize)`。普通查询请求永远不会触发 +清理或 DDL。轮换 HMAC key 时,新 key 作为 `current`,旧 key 仅放在 `previous`;必须等待 `maxCursorTtl` 后才能删除旧 key。 + +Mongo Analytics 目前声明 `EVENTUAL + EXACT`。Elasticsearch grouped Analytics 在 Backend mapping/readiness 通过且配置共享 cursor +store 时支持 `SNAPSHOT + EXACT`:PIT id 只作为服务端 opaque lease state 保存,客户端 token 不包含 PIT;terminal、error、cancel、 +容量拒绝与过期 reaper 都会尽力关闭 PIT。缺少与精确 `QueryTarget + BackendId` 匹配的 lifecycle closer 时,请求会在访问存储前拒绝。 + +记录查询方面,Mongo planned `PAGE` 由单个 matched input、内存 sentinel 与 window accumulator 同时产生当前页和 exact total, +不会二次读取集合,也不会把整页记录打包进一个 BSON document;`SAME_INPUT` 不等同于 point-in-time snapshot。 +Elasticsearch planned direct `STREAM` 只接受 `limit=1..10_000`;更大的 limit 会在访问 Elasticsearch 前拒绝,`limit=0` +unbounded stream 当前仍不支持,不能静默退化为固定 result window。 + +Spring 同时注册 `..AnalyticsQueryService` Bean。直接进程内调用仍必须提供 trusted context;HTTP +调用复用查询路由的认证 authority。Aggregate 未注册匹配的 Analytics schema/backend readiness 时会稳定拒绝且不会访问存储。 + +启用任一 `SHADOW` profile 时,必须同时提供有界的 `QueryShadowConfiguration`、`QueryShadowObserver` 和 +`QueryRuntimeHealthObserver`;缺少 observer 会使 runtime 启动失败。health observation 只包含 target、operation、kind 与稳定 +reason code,不包含 authority、查询值或 Backend cause,必须接入有界、低基数的指标/告警,而不能作为静默 fallback。 + ## 查询服务注册器 `SnapshotQueryServiceRegistrar` 用于自动将所有本地聚合根查询服务注册到 `Spring` 容器中。 开发者可以通过指定的 `Bean Name` 从 `BeanFactory` 中获取相应的 `SnapshotQueryService`。 +聚合查询 Bean 的七个兼容方法没有显式 context 参数,因此默认不会自动提升为 `System` authority。新代码优先直接调用 +`QueryGateway` 并显式传入 `QueryCall`;迁移期如果继续使用聚合 Bean,必须预注册精确的 `QueryLegacyGrant`,并在订阅时用 +`withLegacyQueryCaller` 选择它。caller marker 只能选择已经固定的 `target + purpose + executionMode + resourceScope`,不能扩大授权: + +```kotlin +@Bean +fun queryLegacyContextResolver(): QueryLegacyContextResolver = QueryLegacyContextResolver( + listOf( + QueryLegacyGrant( + callerId = "order-read-model", + target = QueryTarget( + MaterializedNamedAggregate("example", "order"), + QueryDocumentKind.SNAPSHOT, + ), + purpose = QueryPurpose("order-read-model"), + executionMode = QueryExecutionMode.LEGACY, + resourceScope = QueryResourceScope(tenantId = "tenant-1"), + ), + ), +) +``` + +未提供 trusted context 时,兼容 Bean 稳定返回 `QUERY_CALL_REQUIRED`。紧急迁移回滚只能临时设置 +`wow.query.gateway.legacy-wiring-rollback=true`;该开关会绕过 admission、policy 和生命周期保护,只支持一个迁移版本, +不能作为授权或查询失败时的自动 fallback。 + > `Bean Name` 命名规则:`聚合根名称 + ".SnapshotQueryService"`。 使用案例: @@ -474,7 +659,10 @@ class OrderService( condition { id(id) } - }.query(queryService).toState().throwNotFoundIfEmpty() + }.query(queryService) + .withLegacyQueryCaller("order-read-model") + .toState() + .throwNotFoundIfEmpty() } } ``` diff --git a/test/wow-tck/src/main/kotlin/me/ahoo/wow/tck/query/PlannedAnalyticsQueryBackendSpec.kt b/test/wow-tck/src/main/kotlin/me/ahoo/wow/tck/query/PlannedAnalyticsQueryBackendSpec.kt new file mode 100644 index 00000000000..e859b2c3bf5 --- /dev/null +++ b/test/wow-tck/src/main/kotlin/me/ahoo/wow/tck/query/PlannedAnalyticsQueryBackendSpec.kt @@ -0,0 +1,175 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + +package me.ahoo.wow.tck.query + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.AnalyticsQueryBackend +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import org.junit.jupiter.api.Test +import java.time.Instant +import java.util.Collections +import java.util.LinkedHashMap +import java.util.function.Consumer + +sealed interface ExactNumericAnalyticsExpectation { + class Supported(metrics: Map) : ExactNumericAnalyticsExpectation { + val metrics: Map = Collections.unmodifiableMap(LinkedHashMap(metrics)) + } + + data object Unsupported : ExactNumericAnalyticsExpectation +} + +/** Shared exact-count contract for portable MongoDB and Elasticsearch analytics backends. */ +interface PlannedAnalyticsQueryBackendSpec { + val analyticsBackend: AnalyticsQueryBackend + + val analyticsOptions: QueryBackendExecutionOptions + + val expectedGlobalCount: Long + + val expectedUnrestrictedGlobalCount: Long + + val expectedFirstKey: NormalizedValue + + val expectedSecondKey: NormalizedValue + + val expectedNullBucketCount: Long + + val dimensionAlias: AnalyticsAlias + + val countAlias: AnalyticsAlias + + val exactNumericAnalyticsExpectation: ExactNumericAnalyticsExpectation + + fun globalCountPlan(): BackendAnalyticsQueryPlan + + fun unrestrictedGlobalCountPlan(): BackendAnalyticsQueryPlan + + fun groupedCountPlan(afterKey: List?, limit: Int = 1): BackendAnalyticsQueryPlan + + fun nullBucketCountPlan(): BackendAnalyticsQueryPlan + + fun exactNumericMetricPlan(): BackendAnalyticsQueryPlan + + @Test + fun `portable analytics global count should be exact and eventual`() { + val page = analyticsBackend.analyze(globalCountPlan(), analyticsOptions).block()!! + + page.buckets.assert().hasSize(1) + page.buckets.single().keys.assert().isEmpty() + page.buckets.single().metrics.assert() + .containsEntry(countAlias, NormalizedValue.Int64(expectedGlobalCount)) + page.afterKey.assert().isNull() + page.consistency.assert().isEqualTo(BackendAnalyticsConsistency.EVENTUAL) + page.completeness.assert().isEqualTo(BackendAnalyticsCompleteness.EXACT) + } + + @Test + fun `portable analytics mandatory filter should exclude unauthorized and deleted documents`() { + val restricted = analyticsBackend.analyze(globalCountPlan(), analyticsOptions).block()!! + val unrestricted = analyticsBackend.analyze(unrestrictedGlobalCountPlan(), analyticsOptions).block()!! + + restricted.buckets.single().metrics.assert() + .containsEntry(countAlias, NormalizedValue.Int64(expectedGlobalCount)) + unrestricted.buckets.single().metrics.assert() + .containsEntry(countAlias, NormalizedValue.Int64(expectedUnrestrictedGlobalCount)) + } + + @Test + fun `portable analytics grouped cursor should replay in stable key order`() { + val first = analyticsBackend.analyze(groupedCountPlan(null), analyticsOptions).block()!! + + first.buckets.assert().hasSize(1) + first.buckets.single().keys.assert().containsEntry(dimensionAlias, expectedFirstKey) + first.afterKey.assert().isNotNull() + + val second = analyticsBackend.analyze(groupedCountPlan(first.afterKey), analyticsOptions).block()!! + second.buckets.assert().hasSize(1) + second.buckets.single().keys.assert().containsEntry(dimensionAlias, expectedSecondKey) + second.afterKey?.let { afterKey -> + val terminal = analyticsBackend.analyze(groupedCountPlan(afterKey), analyticsOptions).block()!! + terminal.buckets.assert().isEmpty() + terminal.afterKey.assert().isNull() + } + } + + @Test + fun `portable analytics should coalesce missing and explicit null into one bucket`() { + val page = analyticsBackend.analyze(nullBucketCountPlan(), analyticsOptions).block()!! + + page.buckets.assert().hasSize(1) + page.buckets.single().keys.values.assert().containsExactly(NormalizedValue.Null) + page.buckets.single().metrics.assert() + .containsEntry(countAlias, NormalizedValue.Int64(expectedNullBucketCount)) + } + + @Test + fun `exact numeric analytics capability should execute precisely or reject explicitly`() { + when (val expectation = exactNumericAnalyticsExpectation) { + is ExactNumericAnalyticsExpectation.Supported -> { + val page = analyticsBackend.analyze(exactNumericMetricPlan(), analyticsOptions).block()!! + page.buckets.assert().hasSize(1) + expectation.metrics.forEach { (alias, value) -> + page.buckets.single().metrics.assert().containsEntry(alias, value) + } + } + + ExactNumericAnalyticsExpectation.Unsupported -> { + assertThrownBy { + analyticsBackend.analyze(exactNumericMetricPlan(), analyticsOptions).block() + }.satisfies( + Consumer { error -> error.kind.assert().isEqualTo(QueryBackendFailureKind.UNSUPPORTED) }, + ) + } + } + } + + @Test + fun `portable analytics budgets and expired deadlines should fail closed`() { + assertBackendFailure(QueryBackendFailureKind.BUDGET_EXCEEDED) { + analyticsBackend.analyze( + groupedCountPlan(null, limit = 2), + analyticsOptions.copy(maxReturnedBuckets = 1), + ).block() + } + assertBackendFailure(QueryBackendFailureKind.UNSUPPORTED) { + analyticsBackend.analyze( + globalCountPlan(), + analyticsOptions.copy(maxScannedRecords = 1), + ).block() + } + assertBackendFailure(QueryBackendFailureKind.TIMEOUT) { + analyticsBackend.analyze( + globalCountPlan(), + analyticsOptions.copy(deadline = Instant.EPOCH), + ).block() + } + } + + private fun assertBackendFailure(kind: QueryBackendFailureKind, action: () -> Unit) { + assertThrownBy(action).satisfies( + Consumer { error -> error.kind.assert().isEqualTo(kind) }, + ) + } +} diff --git a/test/wow-tck/src/main/kotlin/me/ahoo/wow/tck/query/PlannedRecordQueryBackendSpec.kt b/test/wow-tck/src/main/kotlin/me/ahoo/wow/tck/query/PlannedRecordQueryBackendSpec.kt new file mode 100644 index 00000000000..5e26e55de6d --- /dev/null +++ b/test/wow-tck/src/main/kotlin/me/ahoo/wow/tck/query/PlannedRecordQueryBackendSpec.kt @@ -0,0 +1,62 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + +package me.ahoo.wow.tck.query + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendPageConsistency +import me.ahoo.wow.query.backend.BackendPageQueryPlan +import me.ahoo.wow.query.backend.BackendRecordCompleteness +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.BackendTotalRelation +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.RecordQueryBackend +import org.junit.jupiter.api.Test + +/** Shared record-result contract for portable MongoDB and Elasticsearch snapshot backends. */ +interface PlannedRecordQueryBackendSpec { + val recordBackend: RecordQueryBackend + + val recordOptions: QueryBackendExecutionOptions + + val expectedRecordIdentities: List + + fun portableCountPlan(): BackendCountQueryPlan + + fun portableStreamPlan(): BackendStreamQueryPlan + + fun portableSecondPagePlan(): BackendPageQueryPlan + + @Test + fun `portable record count and bounded stream should agree`() { + recordBackend.count(portableCountPlan(), recordOptions).block().assert() + .isEqualTo(expectedRecordIdentities.size.toLong()) + + val records = recordBackend.stream(portableStreamPlan(), recordOptions).collectList().block()!! + records.map { record -> record.identity }.assert().containsExactly(*expectedRecordIdentities.toTypedArray()) + records.forEach { record -> record.completeness.assert().isEqualTo(BackendRecordCompleteness.COMPLETE) } + } + + @Test + fun `portable record page should preserve exact total and same input consistency`() { + val page = recordBackend.page(portableSecondPagePlan(), recordOptions).block()!! + + page.total.assert().isEqualTo(expectedRecordIdentities.size.toLong()) + page.totalRelation.assert().isEqualTo(BackendTotalRelation.EXACT) + page.consistency.assert().isEqualTo(BackendPageConsistency.SAME_INPUT) + page.records.single().identity.assert().isEqualTo(expectedRecordIdentities[1]) + } +} diff --git a/wow-api/src/main/kotlin/me/ahoo/wow/api/query/analytics/AnalyticsQuery.kt b/wow-api/src/main/kotlin/me/ahoo/wow/api/query/analytics/AnalyticsQuery.kt new file mode 100644 index 00000000000..b10dda27750 --- /dev/null +++ b/wow-api/src/main/kotlin/me/ahoo/wow/api/query/analytics/AnalyticsQuery.kt @@ -0,0 +1,380 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.api.query.analytics + +import com.fasterxml.jackson.annotation.JsonCreator +import com.fasterxml.jackson.annotation.JsonInclude +import com.fasterxml.jackson.annotation.JsonValue +import io.swagger.v3.oas.annotations.media.ArraySchema +import io.swagger.v3.oas.annotations.media.Schema +import me.ahoo.wow.api.query.Condition +import java.math.BigDecimal +import java.math.RoundingMode +import java.time.Instant +import java.util.Collections +import java.util.LinkedHashMap + +/** An opaque, bounded continuation token. Clients must not inspect or modify its contents. */ +@Schema(type = "string", maxLength = AnalyticsCursor.MAX_LENGTH, pattern = AnalyticsCursor.PATTERN) +class AnalyticsCursor +@JsonCreator(mode = JsonCreator.Mode.DELEGATING) +constructor( + @get:JsonValue + val value: String, +) { + init { + require(value.isNotBlank()) { "Analytics cursor must not be blank." } + require(value.length <= MAX_LENGTH) { "Analytics cursor must not exceed $MAX_LENGTH characters." } + require(URL_SAFE.matches(value)) { "Analytics cursor must be URL-safe without padding." } + } + + override fun equals(other: Any?): Boolean = this === other || other is AnalyticsCursor && value == other.value + + override fun hashCode(): Int = value.hashCode() + + override fun toString(): String = value + + companion object { + const val MAX_LENGTH: Int = 256 + const val PATTERN: String = "^[A-Za-z0-9._-]+$" + private val URL_SAFE = Regex(PATTERN) + } +} + +enum class AnalyticsGroupingKind { + GLOBAL, + BY, +} + +enum class AnalyticsMissingPolicy { + EXCLUDE, + AS_NULL_BUCKET, +} + +data class AnalyticsDimension( + @field:Schema(minLength = 1, maxLength = MAX_ALIAS_LENGTH) + val alias: String, + @field:Schema(minLength = 1, maxLength = MAX_FIELD_LENGTH) + val field: String, + val missingPolicy: AnalyticsMissingPolicy = AnalyticsMissingPolicy.EXCLUDE, +) { + init { + requireAnalyticsAlias(alias) + requireLogicalField(field) + } +} + +/** Grouping is explicit rather than inferred from an empty dimension list. */ +class AnalyticsGrouping( + val kind: AnalyticsGroupingKind, + dimensions: List = emptyList(), +) { + val dimensions: List = immutableList(dimensions) + + init { + when (kind) { + AnalyticsGroupingKind.GLOBAL -> require(this.dimensions.isEmpty()) { + "Global analytics must not declare dimensions." + } + + AnalyticsGroupingKind.BY -> require(this.dimensions.isNotEmpty()) { + "Grouped analytics must declare at least one dimension." + } + } + require(this.dimensions.map(AnalyticsDimension::alias).distinct().size == this.dimensions.size) { + "Analytics dimension aliases must be unique." + } + } + + override fun equals(other: Any?): Boolean = + this === other || other is AnalyticsGrouping && kind == other.kind && dimensions == other.dimensions + + override fun hashCode(): Int = 31 * kind.hashCode() + dimensions.hashCode() + + companion object { + @JvmStatic + fun global(): AnalyticsGrouping = AnalyticsGrouping(AnalyticsGroupingKind.GLOBAL) + + @JvmStatic + fun by(dimensions: List): AnalyticsGrouping = + AnalyticsGrouping(AnalyticsGroupingKind.BY, dimensions) + } +} + +enum class AnalyticsMetricKind { + DOCUMENT_COUNT, + MIN, + MAX, + SUM, + AVERAGE, +} + +data class AnalyticsMetric( + @field:Schema(minLength = 1, maxLength = MAX_ALIAS_LENGTH) + val alias: String, + val kind: AnalyticsMetricKind, + @get:JsonInclude(JsonInclude.Include.NON_NULL) + @field:Schema(minLength = 1, maxLength = MAX_FIELD_LENGTH) + val field: String? = null, +) { + init { + requireAnalyticsAlias(alias) + when (kind) { + AnalyticsMetricKind.DOCUMENT_COUNT -> require(field == null) { + "Document-count metric must not declare a field." + } + + AnalyticsMetricKind.MIN, + AnalyticsMetricKind.MAX, + AnalyticsMetricKind.SUM, + AnalyticsMetricKind.AVERAGE, + -> requireLogicalField(requireNotNull(field) { "Analytics metric field is required." }) + } + } +} + +data class AnalyticsBucketWindow( + @field:Schema(minimum = "1") + val limit: Int, + @get:JsonInclude(JsonInclude.Include.NON_NULL) + val cursor: AnalyticsCursor? = null, +) { + init { + require(limit > 0) { "Analytics bucket limit must be positive." } + } +} + +enum class AnalyticsConsistency { + EVENTUAL, + SNAPSHOT, +} + +enum class AnalyticsCompleteness { + EXACT, +} + +enum class AnalyticsNumericPromotion { + DECIMAL128, +} + +enum class AnalyticsOverflowPolicy { + REJECT, +} + +data class AnalyticsNumericPolicy( + val promotion: AnalyticsNumericPromotion = AnalyticsNumericPromotion.DECIMAL128, + @field:Schema(minimum = "1", maximum = "34") + val precision: Int = DECIMAL128_PRECISION, + @field:Schema(minimum = "0", maximum = "34") + val scale: Int, + val roundingMode: RoundingMode = RoundingMode.HALF_EVEN, + val overflowPolicy: AnalyticsOverflowPolicy = AnalyticsOverflowPolicy.REJECT, +) { + init { + require(precision in 1..DECIMAL128_PRECISION) { + "Analytics Decimal128 precision must be between 1 and $DECIMAL128_PRECISION." + } + require(scale in 0..precision) { "Analytics numeric scale must be between zero and precision." } + } + + private companion object { + const val DECIMAL128_PRECISION = 34 + } +} + +/** Public analytics request. Backend names, physical fields and native options are deliberately absent. */ +class AnalyticsQuery( + val condition: Condition = Condition.ALL, + val grouping: AnalyticsGrouping, + metrics: List, + val window: AnalyticsBucketWindow, + val numericPolicy: AnalyticsNumericPolicy? = null, + val consistency: AnalyticsConsistency = AnalyticsConsistency.EVENTUAL, + val completeness: AnalyticsCompleteness = AnalyticsCompleteness.EXACT, +) { + @field:ArraySchema(minItems = 1) + val metrics: List = immutableList(metrics) + + init { + require(this.metrics.isNotEmpty()) { "Analytics metrics must not be empty." } + val aliases = grouping.dimensions.map(AnalyticsDimension::alias) + this.metrics.map(AnalyticsMetric::alias) + require(aliases.distinct().size == aliases.size) { "Analytics aliases must be unique." } + if (grouping.kind == AnalyticsGroupingKind.GLOBAL) { + require(window.limit == 1 && window.cursor == null) { + "Global analytics must use limit one without a cursor." + } + } + } + + override fun equals(other: Any?): Boolean = + this === other || + other is AnalyticsQuery && + condition == other.condition && + grouping == other.grouping && + metrics == other.metrics && + window == other.window && + numericPolicy == other.numericPolicy && + consistency == other.consistency && + completeness == other.completeness + + override fun hashCode(): Int { + var result = condition.hashCode() + result = 31 * result + grouping.hashCode() + result = 31 * result + metrics.hashCode() + result = 31 * result + window.hashCode() + result = 31 * result + (numericPolicy?.hashCode() ?: 0) + result = 31 * result + consistency.hashCode() + result = 31 * result + completeness.hashCode() + return result + } +} + +enum class AnalyticsValueType { + NULL, + BOOLEAN, + TEXT, + INT64, + DECIMAL, + INSTANT, +} + +/** Lossless JSON value representation for analytics keys and metrics. */ +data class AnalyticsValue( + val type: AnalyticsValueType, + @get:JsonInclude(JsonInclude.Include.ALWAYS) + val value: String?, +) { + init { + when (type) { + AnalyticsValueType.NULL -> require(value == null) { "Null analytics value must not carry text." } + AnalyticsValueType.BOOLEAN -> require(value == "true" || value == "false") { + "Boolean analytics value must be lowercase true or false." + } + + AnalyticsValueType.TEXT -> require(value != null) { "Text analytics value is required." } + AnalyticsValueType.INT64 -> requireCanonicalInt64(requireNotNull(value)) + AnalyticsValueType.DECIMAL -> requireCanonicalDecimal(requireNotNull(value)) + AnalyticsValueType.INSTANT -> requireCanonicalInstant(requireNotNull(value)) + } + } + + companion object { + @JvmStatic + fun nullValue(): AnalyticsValue = AnalyticsValue(AnalyticsValueType.NULL, null) + + @JvmStatic + fun of(value: Boolean): AnalyticsValue = AnalyticsValue(AnalyticsValueType.BOOLEAN, value.toString()) + + @JvmStatic + fun of(value: String): AnalyticsValue = AnalyticsValue(AnalyticsValueType.TEXT, value) + + @JvmStatic + fun of(value: Long): AnalyticsValue = AnalyticsValue(AnalyticsValueType.INT64, value.toString()) + + @JvmStatic + fun of(value: BigDecimal): AnalyticsValue = + AnalyticsValue(AnalyticsValueType.DECIMAL, value.toPlainString()) + + @JvmStatic + fun of(value: Instant): AnalyticsValue = AnalyticsValue(AnalyticsValueType.INSTANT, value.toString()) + } +} + +class AnalyticsBucket( + keys: Map, + metrics: Map, +) { + val keys: Map = immutableAliasMap(keys) + val metrics: Map = immutableAliasMap(metrics) + + override fun equals(other: Any?): Boolean = + this === other || other is AnalyticsBucket && keys == other.keys && metrics == other.metrics + + override fun hashCode(): Int = 31 * keys.hashCode() + metrics.hashCode() +} + +class AnalyticsPage( + buckets: List, + val nextCursor: AnalyticsCursor?, + val consistency: AnalyticsConsistency, + val completeness: AnalyticsCompleteness, +) { + val buckets: List = immutableList(buckets) + + override fun equals(other: Any?): Boolean = + this === other || + other is AnalyticsPage && + buckets == other.buckets && + nextCursor == other.nextCursor && + consistency == other.consistency && + completeness == other.completeness + + override fun hashCode(): Int { + var result = buckets.hashCode() + result = 31 * result + (nextCursor?.hashCode() ?: 0) + result = 31 * result + consistency.hashCode() + result = 31 * result + completeness.hashCode() + return result + } +} + +private fun requireAnalyticsAlias(value: String) { + require(value.isNotBlank()) { "Analytics alias must not be blank." } + require(value.length <= MAX_ALIAS_LENGTH) { "Analytics alias must not exceed $MAX_ALIAS_LENGTH characters." } + require(value.none(Char::isISOControl)) { "Analytics alias must not contain control characters." } + require('.' !in value && '$' !in value) { "Analytics alias must be a safe result field name." } +} + +private fun requireLogicalField(value: String) { + require(value.isNotBlank()) { "Analytics logical field must not be blank." } + require( + value.length <= MAX_FIELD_LENGTH + ) { "Analytics logical field must not exceed $MAX_FIELD_LENGTH characters." } + require(value.none(Char::isISOControl)) { "Analytics logical field must not contain control characters." } + require(value.split('.').all(String::isNotBlank)) { "Analytics logical field must not contain empty segments." } + require(value.split('.').none { segment -> segment.startsWith('$') }) { + "Analytics logical field must not contain physical operator segments." + } +} + +private fun requireCanonicalInt64(value: String) { + val parsed = value.toLongOrNull() + require(parsed != null && parsed.toString() == value) { "Analytics Int64 value must be canonical decimal text." } +} + +private fun requireCanonicalDecimal(value: String) { + val parsed = value.toBigDecimalOrNull() + require(parsed != null && parsed.toPlainString() == value) { + "Analytics Decimal value must be canonical non-exponent decimal text." + } +} + +private fun requireCanonicalInstant(value: String) { + val parsed = runCatching { Instant.parse(value) }.getOrNull() + require(parsed != null && parsed.toString() == value) { "Analytics Instant value must be canonical ISO-8601 text." } +} + +private fun immutableList(values: List): List = Collections.unmodifiableList(ArrayList(values)) + +private fun immutableAliasMap(values: Map): Map { + values.keys.forEach(::requireAnalyticsAlias) + val copy = LinkedHashMap(values.size) + values.entries.sortedBy(Map.Entry::key).forEach { entry -> + copy[entry.key] = entry.value + } + return Collections.unmodifiableMap(copy) +} + +private const val MAX_ALIAS_LENGTH = 128 +private const val MAX_FIELD_LENGTH = 512 diff --git a/wow-api/src/test/kotlin/me/ahoo/wow/api/query/analytics/AnalyticsQueryTest.kt b/wow-api/src/test/kotlin/me/ahoo/wow/api/query/analytics/AnalyticsQueryTest.kt new file mode 100644 index 00000000000..8d16530bc9c --- /dev/null +++ b/wow-api/src/test/kotlin/me/ahoo/wow/api/query/analytics/AnalyticsQueryTest.kt @@ -0,0 +1,120 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.api.query.analytics + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.api.query.Condition +import org.junit.jupiter.api.Test +import java.math.BigDecimal +import java.time.Instant + +class AnalyticsQueryTest { + @Test + fun `query should freeze collection boundaries and preserve value semantics`() { + val dimensions = mutableListOf(AnalyticsDimension("status", "state.status")) + val metrics = mutableListOf(AnalyticsMetric("count", AnalyticsMetricKind.DOCUMENT_COUNT)) + val query = AnalyticsQuery( + grouping = AnalyticsGrouping.by(dimensions), + metrics = metrics, + window = AnalyticsBucketWindow(100), + ) + dimensions.clear() + metrics.clear() + + query.grouping.dimensions.assert().hasSize(1) + query.metrics.assert().hasSize(1) + query.assert().isEqualTo( + AnalyticsQuery( + condition = Condition.ALL, + grouping = AnalyticsGrouping.by(listOf(AnalyticsDimension("status", "state.status"))), + metrics = listOf(AnalyticsMetric("count", AnalyticsMetricKind.DOCUMENT_COUNT)), + window = AnalyticsBucketWindow(100), + ), + ) + @Suppress("UNCHECKED_CAST") + assertThrownBy { + (query.metrics as MutableList).clear() + } + } + + @Test + fun `request invariants should reject ambiguous grouping metrics and cursors`() { + assertThrownBy { + AnalyticsGrouping(AnalyticsGroupingKind.GLOBAL, listOf(AnalyticsDimension("a", "state.a"))) + } + assertThrownBy { AnalyticsGrouping.by(emptyList()) } + assertThrownBy { + AnalyticsMetric("count", AnalyticsMetricKind.DOCUMENT_COUNT, "state.count") + } + assertThrownBy { AnalyticsMetric("sum", AnalyticsMetricKind.SUM) } + assertThrownBy { + AnalyticsQuery( + grouping = AnalyticsGrouping.global(), + metrics = listOf(AnalyticsMetric("count", AnalyticsMetricKind.DOCUMENT_COUNT)), + window = AnalyticsBucketWindow(2), + ) + } + assertThrownBy { AnalyticsCursor("a=") } + assertThrownBy { AnalyticsCursor("a".repeat(AnalyticsCursor.MAX_LENGTH + 1)) } + } + + @Test + fun `analytics values should be lossless canonical strings`() { + AnalyticsValue.nullValue().assert().isEqualTo(AnalyticsValue(AnalyticsValueType.NULL, null)) + AnalyticsValue.of(true).assert().isEqualTo(AnalyticsValue(AnalyticsValueType.BOOLEAN, "true")) + AnalyticsValue.of(Long.MAX_VALUE).value.assert().isEqualTo(Long.MAX_VALUE.toString()) + AnalyticsValue.of(BigDecimal("120.50")).value.assert().isEqualTo("120.50") + AnalyticsValue.of(Instant.parse("2026-08-09T00:00:00Z")).value.assert() + .isEqualTo("2026-08-09T00:00:00Z") + + listOf("01", "+1", "-0").forEach { invalid -> + assertThrownBy { AnalyticsValue(AnalyticsValueType.INT64, invalid) } + } + listOf("01.0", "1E+2", "-0.00").forEach { invalid -> + assertThrownBy { AnalyticsValue(AnalyticsValueType.DECIMAL, invalid) } + } + assertThrownBy { + AnalyticsValue(AnalyticsValueType.INSTANT, "2026-08-09T08:00:00+08:00") + } + } + + @Test + fun `page should freeze and canonicalize alias maps`() { + val keys = linkedMapOf( + "z" to AnalyticsValue.of("last"), + "a" to AnalyticsValue.of("first"), + ) + val metrics = linkedMapOf("count" to AnalyticsValue.of(2L)) + val bucket = AnalyticsBucket(keys, metrics) + val buckets = mutableListOf(bucket) + val page = AnalyticsPage( + buckets, + AnalyticsCursor("next_page"), + AnalyticsConsistency.EVENTUAL, + AnalyticsCompleteness.EXACT, + ) + keys.clear() + metrics.clear() + buckets.clear() + + bucket.keys.keys.assert().containsExactly("a", "z") + bucket.metrics.assert().containsKey("count") + page.buckets.assert().containsExactly(bucket) + @Suppress("UNCHECKED_CAST") + assertThrownBy { + (bucket.keys as MutableMap).clear() + } + } +} diff --git a/wow-apiclient/src/main/kotlin/me/ahoo/wow/apiclient/query/SnapshotAnalyticsQueryApi.kt b/wow-apiclient/src/main/kotlin/me/ahoo/wow/apiclient/query/SnapshotAnalyticsQueryApi.kt new file mode 100644 index 00000000000..491d88f0a78 --- /dev/null +++ b/wow-apiclient/src/main/kotlin/me/ahoo/wow/apiclient/query/SnapshotAnalyticsQueryApi.kt @@ -0,0 +1,36 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.apiclient.query + +import me.ahoo.wow.api.query.analytics.AnalyticsPage +import me.ahoo.wow.api.query.analytics.AnalyticsQuery +import org.springframework.web.bind.annotation.RequestBody +import org.springframework.web.service.annotation.PostExchange +import reactor.core.publisher.Mono + +const val SNAPSHOT_ANALYTICS_RESOURCE_NAME = "$SNAPSHOT_RESOURCE_NAME/analyze" + +/** Additive analytics client contract; existing SnapshotQueryApi implementations do not inherit this method. */ +interface SnapshotAnalyticsQueryApi : SnapshotQueryApi { + @PostExchange(SNAPSHOT_ANALYTICS_RESOURCE_NAME) + fun analyze(@RequestBody query: AnalyticsQuery): R +} + +interface ReactiveSnapshotAnalyticsQueryApi : SnapshotAnalyticsQueryApi> + +interface SynchronousSnapshotAnalyticsQueryApi : SnapshotAnalyticsQueryApi + +fun AnalyticsQuery.analyze(api: ReactiveSnapshotAnalyticsQueryApi): Mono = api.analyze(this) + +fun AnalyticsQuery.analyze(api: SynchronousSnapshotAnalyticsQueryApi): AnalyticsPage = api.analyze(this) diff --git a/wow-apiclient/src/test/kotlin/me/ahoo/wow/apiclient/query/SnapshotAnalyticsQueryApiTest.kt b/wow-apiclient/src/test/kotlin/me/ahoo/wow/apiclient/query/SnapshotAnalyticsQueryApiTest.kt new file mode 100644 index 00000000000..2c709fff0c8 --- /dev/null +++ b/wow-apiclient/src/test/kotlin/me/ahoo/wow/apiclient/query/SnapshotAnalyticsQueryApiTest.kt @@ -0,0 +1,59 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.apiclient.query + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.api.query.analytics.AnalyticsBucketWindow +import me.ahoo.wow.api.query.analytics.AnalyticsCompleteness +import me.ahoo.wow.api.query.analytics.AnalyticsConsistency +import me.ahoo.wow.api.query.analytics.AnalyticsGrouping +import me.ahoo.wow.api.query.analytics.AnalyticsMetric +import me.ahoo.wow.api.query.analytics.AnalyticsMetricKind +import me.ahoo.wow.api.query.analytics.AnalyticsPage +import me.ahoo.wow.api.query.analytics.AnalyticsQuery +import org.junit.jupiter.api.Test +import org.springframework.web.service.annotation.PostExchange +import reactor.core.publisher.Mono +import reactor.kotlin.test.test + +class SnapshotAnalyticsQueryApiTest { + @Test + fun `should keep analytics client separate from legacy snapshot query interface`() { + SnapshotAnalyticsQueryApi::class.java.getMethod("analyze", AnalyticsQuery::class.java) + .getAnnotation(PostExchange::class.java) + .value.assert().isEqualTo("snapshot/analyze") + SnapshotQueryApi::class.java.methods.map { it.name }.assert().doesNotContain("analyze") + } + + @Test + fun `should delegate reactive analytics extension`() { + val expected = AnalyticsPage( + buckets = emptyList(), + nextCursor = null, + consistency = AnalyticsConsistency.EVENTUAL, + completeness = AnalyticsCompleteness.EXACT, + ) + val api = object : ReactiveSnapshotAnalyticsQueryApi { + override fun analyze(query: AnalyticsQuery): Mono = Mono.just(expected) + } + + query().analyze(api).test().expectNext(expected).verifyComplete() + } + + private fun query(): AnalyticsQuery = AnalyticsQuery( + grouping = AnalyticsGrouping.global(), + metrics = listOf(AnalyticsMetric("count", AnalyticsMetricKind.DOCUMENT_COUNT)), + window = AnalyticsBucketWindow(1), + ) +} diff --git a/wow-elasticsearch/src/integrationTest/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleIntegrationTest.kt b/wow-elasticsearch/src/integrationTest/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleIntegrationTest.kt new file mode 100644 index 00000000000..ec2cec81c50 --- /dev/null +++ b/wow-elasticsearch/src/integrationTest/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleIntegrationTest.kt @@ -0,0 +1,393 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import co.elastic.clients.elasticsearch._types.Refresh +import co.elastic.clients.elasticsearch._types.mapping.Property +import co.elastic.clients.elasticsearch._types.mapping.TypeMapping +import co.elastic.clients.elasticsearch.core.IndexRequest +import co.elastic.clients.json.JsonData +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.event.DomainEventStream +import me.ahoo.wow.eventsourcing.AggregateIdScanner +import me.ahoo.wow.elasticsearch.ReactiveElasticsearchClients +import me.ahoo.wow.eventsourcing.snapshot.SimpleSnapshot +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.modeling.aggregateId +import me.ahoo.wow.modeling.state.ConstructorStateAggregateFactory +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.tck.container.ElasticsearchTestFixture +import me.ahoo.wow.tck.event.MockDomainEventStreams +import me.ahoo.wow.tck.mock.MOCK_AGGREGATE_METADATA +import me.ahoo.wow.tck.mock.MockStateAggregate +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.extension.RegisterExtension +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset +import java.util.function.Consumer + +class ElasticsearchIndexLifecycleIntegrationTest { + @JvmField + @RegisterExtension + val elasticsearch = ElasticsearchTestFixture() + + @Test + fun `versioned templates create an attested generation and alias cutover rollback are atomic`() { + val client = ReactiveElasticsearchClients.createReactiveElasticsearchClient(elasticsearch) + client.indices().create { request -> request.index(SOURCE.value).mappings(sourceMapping()) }.block() + client.indices().updateAliases { request -> + request.actions { action -> + action.add { add -> + add.index(SOURCE.value).alias(MANIFEST.names.alias.value).isWriteIndex(true) + } + } + }.block() + val admin = ReactiveElasticsearchIndexAdminClient(client, CLOCK) + + val initial = admin.inspect(MANIFEST).block()!! + MANIFEST.validate(initial) + admin.create(MANIFEST, ElasticsearchVersionedIndexTemplate(MANIFEST, destinationMapping())).block()!! + val created = admin.inspect(MANIFEST).block()!! + created.indices[DESTINATION].assert().isEqualTo(MANIFEST.destinationAttestation) + + admin.compareAndSetAlias(MANIFEST, SOURCE, DESTINATION).block()!! + admin.inspect(MANIFEST).block()!!.aliasTarget.assert().isEqualTo(DESTINATION) + + admin.compareAndSetAlias(MANIFEST, DESTINATION, SOURCE).block()!! + admin.inspect(MANIFEST).block()!!.aliasTarget.assert().isEqualTo(SOURCE) + client.indices().exists { request -> request.index(SOURCE.value) }.block()!!.value().assert().isTrue() + client.indices().exists { request -> request.index(DESTINATION.value) }.block()!!.value().assert().isTrue() + } + + @Test + fun `durable repository should persist state and enforce elasticsearch compare and set tokens`() { + val client = ReactiveElasticsearchClients.createReactiveElasticsearchClient(elasticsearch) + val repository = ReactiveElasticsearchIndexLifecycleRepository(client, REPOSITORY_INDEX) + repository.ensureIndex().block() + repository.ensureIndex().block() + + val initial = ElasticsearchIndexMigrationState.initial(MANIFEST) + val created = repository.create(initial).block()!! + repository.create(initial).block().assert().isNull() + + val reloaded = ReactiveElasticsearchIndexLifecycleRepository(client, REPOSITORY_INDEX) + .load(MANIFEST.id) + .block()!! + reloaded.state.assert().isEqualTo(initial) + reloaded.version.assert().isInstanceOf(ElasticsearchIndexLifecycleRepositoryVersion.Elasticsearch::class.java) + + val command = ElasticsearchIndexLifecycleCommand( + ElasticsearchIndexLifecycleCommandId("validate-command"), + MANIFEST.id, + ElasticsearchIndexLifecycleCommandType.VALIDATE, + expectedRevision = initial.revision, + ) + val claimed = initial.claim(command, NOW) + val updated = repository.compareAndSet(reloaded, claimed).block()!! + updated.state.assert().isEqualTo(claimed) + repository.compareAndSet(created, claimed).block().assert().isNull() + repository.load(MANIFEST.id).block()!!.state.assert().isEqualTo(claimed) + + val corruptId = ElasticsearchIndexMigrationId("corrupt-lifecycle-state") + client.index( + IndexRequest.of> { request -> + request.index(REPOSITORY_INDEX) + .id(corruptId.value) + .refresh(Refresh.True) + .document( + linkedMapOf( + "formatVersion" to 1, + "migrationId" to corruptId.value, + "revision" to 0L, + "payload" to "not-base64", + ), + ) + }, + ).block() + assertThrownBy { + repository.load(corruptId).block() + }.satisfies( + Consumer { error -> + error.code.assert().isEqualTo(ElasticsearchIndexLifecycleErrorCode.REPOSITORY_CORRUPTED) + }, + ) + } + + @Test + fun `authoritative snapshot rebuild should write replayed states to the exact physical generation`() { + val client = ReactiveElasticsearchClients.createReactiveElasticsearchClient(elasticsearch) + client.indices().create { request -> request.index(REBUILD_MANIFEST.names.physical.value) }.block() + val first = stateAggregate("001", 1) + val second = stateAggregate("002", 2) + val source = ElasticsearchAuthoritativeSnapshotSource { _, afterId, _ -> + when (afterId) { + AggregateIdScanner.FIRST_ID -> Flux.just(first, second) + else -> Flux.empty() + } + } + val rebuilder = EventStoreElasticsearchSnapshotIndexRebuilder( + source, + ReactiveElasticsearchPhysicalSnapshotWriter(client, Refresh.True), + CLOCK, + ElasticsearchSnapshotRebuildOptions(scanPageSize = 2, writeConcurrency = 2), + ) + + val receipt = rebuilder.rebuild( + ElasticsearchIndexLifecycleCommandId("integration-rebuild"), + REBUILD_MANIFEST, + ).block()!! + + receipt.physicalIndex.assert().isEqualTo(REBUILD_MANIFEST.names.physical) + client.count { request -> request.index(REBUILD_MANIFEST.names.physical.value) } + .block()!!.count().assert().isEqualTo(2) + listOf("001" to 1, "002" to 2).forEach { (id, version) -> + val stored = client.get( + { request -> request.index(REBUILD_MANIFEST.names.physical.value).id(id) }, + Map::class.java, + ).block()!!.source()!! + stored[MessageRecords.VERSION].assert().isEqualTo(version) + } + } + + @Test + fun `authoritative and exact physical snapshot sources should produce identical canonical evidence`() { + val client = ReactiveElasticsearchClients.createReactiveElasticsearchClient(elasticsearch) + client.indices().create { request -> + request.index(VERIFY_MANIFEST.names.physical.value).mappings(sourceMapping()) + }.block() + val first = stateAggregate("001", 1) + val second = stateAggregate("002", 2) + val source = ElasticsearchAuthoritativeSnapshotSource { _, afterId, _ -> + when (afterId) { + AggregateIdScanner.FIRST_ID -> Flux.just(first) + "001" -> Flux.just(second) + else -> Flux.empty() + } + } + val writer = ReactiveElasticsearchPhysicalSnapshotWriter(client, Refresh.True) + listOf(first, second).forEach { aggregate -> + writer.write( + VERIFY_MANIFEST.names.physical, + SimpleSnapshot(aggregate, NOW.toEpochMilli()), + ).block() + } + val options = SnapshotElasticsearchIndexVerificationOptions(pageSize = 1) + + val expected = EventStoreSnapshotVerificationSource(source, options) + .capture(VERIFY_COMMAND, VERIFY_MANIFEST) + .block()!! + val actual = ReactiveElasticsearchSnapshotVerificationSource(client, options) + .inspect(VERIFY_COMMAND, VERIFY_MANIFEST, VERIFY_MANIFEST.names.physical) + .block()!! + + actual.count.assert().isEqualTo(expected.count) + actual.identityChecksum.assert().isEqualTo(expected.identityChecksum) + actual.contentChecksum.assert().isEqualTo(expected.contentChecksum) + actual.versionContinuity.assert().isTrue() + } + + @Test + fun `drained event stream authority and exact physical generation should produce identical evidence`() { + val client = ReactiveElasticsearchClients.createReactiveElasticsearchClient(elasticsearch) + client.indices().create { request -> + request.index(EVENT_VERIFY_MANIFEST.names.physical.value).mappings(eventStreamMapping()) + }.block() + val streams = listOf( + eventStream("001", aggregateVersion = 0, eventCount = 2), + eventStream("001", aggregateVersion = 2, eventCount = 1), + eventStream("002", aggregateVersion = 0, eventCount = 1), + ) + val source = ElasticsearchAuthoritativeEventStreamSource { _, afterId, _ -> + when (afterId) { + AggregateIdScanner.FIRST_ID -> Flux.fromIterable(streams.take(2)) + "001" -> Flux.just(streams.last()) + else -> Flux.empty() + } + } + val writer = ReactiveElasticsearchPhysicalEventStreamWriter(client, Refresh.True) + streams.forEach { stream -> + writer.write(EVENT_VERIFY_MANIFEST.names.physical, stream).block() + writer.write(EVENT_VERIFY_MANIFEST.names.physical, stream).block() + } + val options = EventStreamElasticsearchIndexVerificationOptions( + aggregateScanPageSize = 1, + physicalPageSize = 1, + ) + val barrier = ElasticsearchEventStreamMigrationBarrier { _, _ -> Mono.just(42L) } + val indexedWatermark = ElasticsearchEventStreamIndexedWatermarkSource { _, _, physical -> + physical.assert().isEqualTo(EVENT_VERIFY_MANIFEST.names.physical) + Mono.just(42L) + } + + val expected = EventStoreEventStreamVerificationSource(source, barrier, options) + .capture(EVENT_VERIFY_COMMAND, EVENT_VERIFY_MANIFEST) + .block()!! + val actual = ReactiveElasticsearchEventStreamVerificationSource(client, indexedWatermark, options) + .inspect(EVENT_VERIFY_COMMAND, EVENT_VERIFY_MANIFEST, EVENT_VERIFY_MANIFEST.names.physical) + .block()!! + + actual.count.assert().isEqualTo(3) + actual.count.assert().isEqualTo(expected.count) + actual.identityChecksum.assert().isEqualTo(expected.identityChecksum) + actual.contentChecksum.assert().isEqualTo(expected.contentChecksum) + actual.watermark.assert().isEqualTo(expected.watermark) + actual.versionContinuity.assert().isTrue() + } + + private fun stateAggregate(id: String, version: Int) = ConstructorStateAggregateFactory.create( + MOCK_AGGREGATE_METADATA.state, + MOCK_AGGREGATE_METADATA.aggregateId(id, "tenant-1"), + MockStateAggregate(id), + version, + ) + + private fun sourceMapping(): TypeMapping = TypeMapping.of { mapping -> + mapping.meta(MAPPING_VERSION, JsonData.of("v0001")) + .meta(DOCUMENT_KIND, JsonData.of(QueryDocumentKind.SNAPSHOT.name)) + .meta(SCHEMA_CONTRACT, JsonData.of("0".repeat(64))) + .meta(CAPABILITY_DIGEST, JsonData.of("c".repeat(64))) + .properties("aggregateId", keyword()) + } + + private fun destinationMapping(): TypeMapping = TypeMapping.of { mapping -> + mapping.meta(MAPPING_VERSION, JsonData.of(MANIFEST.mappingVersion.tag)) + .meta(DOCUMENT_KIND, JsonData.of(MANIFEST.target.documentKind.name)) + .meta(SCHEMA_CONTRACT, JsonData.of(MANIFEST.schemaContractId.value)) + .meta(CAPABILITY_DIGEST, JsonData.of(MANIFEST.capabilityDigest.value)) + .properties("aggregateId", keyword()) + } + + private fun eventStreamMapping(): TypeMapping = TypeMapping.of { mapping -> + mapping.properties(MessageRecords.AGGREGATE_ID, keyword()) + .properties(MessageRecords.VERSION) { property -> property.integer { integer -> integer } } + } + + private fun eventStream( + id: String, + aggregateVersion: Int, + eventCount: Int, + ): DomainEventStream = MockDomainEventStreams.generateEventStream( + MOCK_AGGREGATE_METADATA.aggregateId(id, "tenant-1"), + aggregateVersion = aggregateVersion, + eventCount = eventCount, + ) + + private fun keyword(): Property = Property.of { property -> property.keyword { keyword -> keyword } } + + private companion object { + const val MAPPING_VERSION = "wow_query_mapping_version" + const val DOCUMENT_KIND = "wow_query_document_kind" + const val SCHEMA_CONTRACT = "wow_query_schema_contract_id" + const val CAPABILITY_DIGEST = "wow_query_capability_digest" + const val REPOSITORY_INDEX = ".wow-query-index-lifecycle-integration-v1" + val NOW: Instant = Instant.parse("2026-08-08T04:00:00Z") + val CLOCK: Clock = Clock.fixed(NOW, ZoneOffset.UTC) + val TARGET = QueryTarget( + MaterializedNamedAggregate("lifecycle", "order"), + QueryDocumentKind.SNAPSHOT, + ) + val SOURCE = ElasticsearchPhysicalIndex("wow.lifecycle.order.snapshot-v0001-000001") + val DESTINATION = ElasticsearchPhysicalIndex("wow.lifecycle.order.snapshot-v0002-000007") + val VERIFICATION_CONTRACT = ElasticsearchIndexVerificationContract( + ElasticsearchIndexChecksumAlgorithm.CANONICAL_DOCUMENT_SHA256_V1, + ElasticsearchIndexProbeSuiteId("integration-probes-v1"), + ) + val VERIFY_COMMAND = ElasticsearchIndexLifecycleCommandId("verify-canonical-snapshot") + val MANIFEST = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("lifecycle-order-snapshot-v2-g7"), + TARGET, + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(7), + SchemaContractId("1".repeat(64)), + ElasticsearchIndexCapabilityDigest("a".repeat(64)), + SOURCE, + ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM, + VERIFICATION_CONTRACT, + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + val REBUILD_TARGET = QueryTarget( + MOCK_AGGREGATE_METADATA.namedAggregate, + QueryDocumentKind.SNAPSHOT, + ) + val REBUILD_MANIFEST = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("mock-snapshot-v2-g8"), + REBUILD_TARGET, + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(8), + SchemaContractId("5".repeat(64)), + ElasticsearchIndexCapabilityDigest("6".repeat(64)), + ElasticsearchIndexNames.of( + REBUILD_TARGET, + ElasticsearchIndexMappingVersion(1), + ElasticsearchIndexGeneration(1), + ).physical, + ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM, + VERIFICATION_CONTRACT, + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + val VERIFY_MANIFEST = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("mock-snapshot-v2-g9"), + REBUILD_TARGET, + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(9), + SchemaContractId("5".repeat(64)), + ElasticsearchIndexCapabilityDigest("6".repeat(64)), + REBUILD_MANIFEST.sourcePhysicalIndex, + ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM, + VERIFICATION_CONTRACT, + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + val EVENT_VERIFY_TARGET = QueryTarget( + MOCK_AGGREGATE_METADATA.namedAggregate, + QueryDocumentKind.EVENT_STREAM, + ) + val EVENT_VERIFY_MANIFEST = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("mock-event-stream-v2-g10"), + EVENT_VERIFY_TARGET, + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(10), + SchemaContractId("7".repeat(64)), + ElasticsearchIndexCapabilityDigest("8".repeat(64)), + ElasticsearchIndexNames.of( + EVENT_VERIFY_TARGET, + ElasticsearchIndexMappingVersion(1), + ElasticsearchIndexGeneration(1), + ).physical, + ElasticsearchIndexRebuildStrategy.EVENT_STREAM_PAUSE_AND_DRAIN, + ElasticsearchIndexVerificationContract( + ElasticsearchIndexChecksumAlgorithm.CANONICAL_EVENT_STREAM_SHA256_V1, + ElasticsearchIndexProbeSuiteId("integration-event-stream-probes-v1"), + ), + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + val EVENT_VERIFY_COMMAND = ElasticsearchIndexLifecycleCommandId("verify-canonical-event-stream") + } +} diff --git a/wow-elasticsearch/src/integrationTest/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotRecordQueryBackendIntegrationTest.kt b/wow-elasticsearch/src/integrationTest/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotRecordQueryBackendIntegrationTest.kt new file mode 100644 index 00000000000..dad587538da --- /dev/null +++ b/wow-elasticsearch/src/integrationTest/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotRecordQueryBackendIntegrationTest.kt @@ -0,0 +1,978 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.planned + +import co.elastic.clients.elasticsearch._types.mapping.Property +import co.elastic.clients.elasticsearch.core.ClosePointInTimeRequest +import co.elastic.clients.elasticsearch.core.ClosePointInTimeResponse +import co.elastic.clients.elasticsearch.core.CountRequest +import co.elastic.clients.elasticsearch.core.SearchRequest +import co.elastic.clients.elasticsearch.core.bulk.BulkOperation +import co.elastic.clients.elasticsearch.core.search.ResponseBody +import co.elastic.clients.json.JsonData +import me.ahoo.test.asserts.assert +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DynamicDocument +import me.ahoo.wow.api.query.IListQuery +import me.ahoo.wow.api.query.IPagedQuery +import me.ahoo.wow.api.query.ISingleQuery +import me.ahoo.wow.api.query.PagedList +import me.ahoo.wow.elasticsearch.ReactiveElasticsearchClients +import me.ahoo.wow.elasticsearch.query.snapshot.SnapshotConditionConverter +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.AnalyticsQueryBackend +import me.ahoo.wow.query.backend.AnalyticsQueryCursorLifecycle +import me.ahoo.wow.query.backend.BackendAnalyticsBucketOrder +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsCondition +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsDimension +import me.ahoo.wow.query.backend.BackendAnalyticsGrouping +import me.ahoo.wow.query.backend.BackendAnalyticsMetric +import me.ahoo.wow.query.backend.BackendAnalyticsMissingPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNullPlacement +import me.ahoo.wow.query.backend.BackendAnalyticsNumericPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNumericPromotion +import me.ahoo.wow.query.backend.BackendAnalyticsOverflowPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsPageWindow +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.BackendAnalyticsTextCollation +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPageConsistency +import me.ahoo.wow.query.backend.BackendPageQueryPlan +import me.ahoo.wow.query.backend.BackendPageWindow +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.BackendRequiredCapabilities +import me.ahoo.wow.query.backend.BackendRequiredConsistency +import me.ahoo.wow.query.backend.BackendSingleQueryPlan +import me.ahoo.wow.query.backend.BackendSort +import me.ahoo.wow.query.backend.BackendSortOrigin +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.BackendTotalMode +import me.ahoo.wow.query.backend.BackendTotalRelation +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.JunctionOperator +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedSortDirection +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryBackendComposition +import me.ahoo.wow.query.backend.RecordQueryBackend +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.QuerySearchScopeDefinition +import me.ahoo.wow.query.backend.RecordResultShape +import me.ahoo.wow.query.backend.SearchScopeId +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryAuthority +import me.ahoo.wow.query.gateway.QueryAuthorityResolver +import me.ahoo.wow.query.gateway.QueryCall +import me.ahoo.wow.query.gateway.QueryElementPathMode +import me.ahoo.wow.query.gateway.QueryExecutionMode +import me.ahoo.wow.query.gateway.QueryExecutionProfile +import me.ahoo.wow.query.gateway.QueryExecutionProfiles +import me.ahoo.wow.query.gateway.QueryGatewayRuntime +import me.ahoo.wow.query.gateway.QueryLegacyDialect +import me.ahoo.wow.query.gateway.QueryLegacyDialectResolver +import me.ahoo.wow.query.gateway.QueryMatchScopeMode +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryOperationProfileKey +import me.ahoo.wow.query.gateway.QueryPurpose +import me.ahoo.wow.query.gateway.QueryRawServiceSource +import me.ahoo.wow.query.gateway.QueryRuntimeHealthObserver +import me.ahoo.wow.query.gateway.QueryShadowObservation +import me.ahoo.wow.query.gateway.QueryShadowObserver +import me.ahoo.wow.query.gateway.QueryShadowOutcome +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.gateway.QueryValidationMode +import me.ahoo.wow.query.event.NoOpEventStreamQueryServiceFactory +import me.ahoo.wow.query.snapshot.SnapshotQueryService +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.serialization.state.StateAggregateRecords +import me.ahoo.wow.tck.container.ElasticsearchTestFixture +import me.ahoo.wow.tck.query.ExactNumericAnalyticsExpectation +import me.ahoo.wow.tck.query.PlannedAnalyticsQueryBackendSpec +import me.ahoo.wow.tck.query.PlannedRecordQueryBackendSpec +import org.junit.jupiter.api.BeforeEach +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.extension.RegisterExtension +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Mono +import reactor.core.publisher.Sinks +import reactor.core.publisher.Flux +import reactor.test.StepVerifier +import java.math.RoundingMode +import java.time.Duration +import java.time.Instant +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit +import java.util.concurrent.atomic.AtomicInteger + +class ElasticsearchSnapshotRecordQueryBackendIntegrationTest : + PlannedAnalyticsQueryBackendSpec, + PlannedRecordQueryBackendSpec { + @JvmField + @RegisterExtension + val elasticsearch = ElasticsearchTestFixture() + + private lateinit var client: ReactiveElasticsearchClient + private lateinit var indexName: String + private lateinit var binding: ElasticsearchSnapshotQueryBinding + + override val analyticsBackend: AnalyticsQueryBackend + get() = binding.prepareContribution(client).block()!!.analyticsBackend!! + override val analyticsOptions: QueryBackendExecutionOptions + get() = OPTIONS.copy(maxReturnedBuckets = 10) + override val expectedGlobalCount: Long = 2 + override val expectedUnrestrictedGlobalCount: Long = 4 + override val expectedFirstKey: NormalizedValue = NormalizedValue.Text("alice") + override val expectedSecondKey: NormalizedValue = NormalizedValue.Text("carol") + override val expectedNullBucketCount: Long = 2 + override val dimensionAlias: AnalyticsAlias = NAME_ALIAS + override val countAlias: AnalyticsAlias = COUNT_ALIAS + override val exactNumericAnalyticsExpectation: ExactNumericAnalyticsExpectation = + ExactNumericAnalyticsExpectation.Unsupported + override val recordBackend: RecordQueryBackend + get() = binding.prepareContribution(client).block()!!.backend + override val recordOptions: QueryBackendExecutionOptions + get() = OPTIONS + override val expectedRecordIdentities: List = listOf("order-1", "order-4") + + override fun globalCountPlan(): BackendAnalyticsQueryPlan = globalAnalyticsPlan() + + override fun unrestrictedGlobalCountPlan(): BackendAnalyticsQueryPlan = globalAnalyticsPlan( + BackendEnforcedFilter(BackendPlannedCondition.All, BackendPlannedCondition.All), + ) + + override fun groupedCountPlan(afterKey: List?, limit: Int): BackendAnalyticsQueryPlan = + groupedAnalyticsPlan(afterKey, limit) + + override fun nullBucketCountPlan(): BackendAnalyticsQueryPlan = BackendAnalyticsQueryPlan( + target, + schema.contractId, + BackendEnforcedFilter(BackendPlannedCondition.All, mandatory()), + BackendAnalyticsGrouping.By( + listOf(BackendAnalyticsDimension(NOTE_ALIAS, note, BackendAnalyticsMissingPolicy.AS_NULL_BUCKET)), + ), + listOf(BackendAnalyticsMetric.DocumentCount(COUNT_ALIAS)), + BackendAnalyticsCondition.All, + BackendAnalyticsBucketOrder.DimensionKeyAscending( + BackendAnalyticsNullPlacement.FIRST, + BackendAnalyticsTextCollation.BINARY, + ), + BackendAnalyticsPageWindow(10), + null, + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("7".repeat(64)), + ) + + override fun exactNumericMetricPlan(): BackendAnalyticsQueryPlan = BackendAnalyticsQueryPlan( + target, + schema.contractId, + BackendEnforcedFilter(BackendPlannedCondition.All, mandatory()), + BackendAnalyticsGrouping.Global, + listOf( + BackendAnalyticsMetric.Min(AnalyticsAlias("minimum"), amount), + BackendAnalyticsMetric.Max(AnalyticsAlias("maximum"), amount), + BackendAnalyticsMetric.Sum(AnalyticsAlias("total"), amount), + BackendAnalyticsMetric.Average(AnalyticsAlias("average"), amount), + ), + BackendAnalyticsCondition.All, + BackendAnalyticsBucketOrder.Global, + BackendAnalyticsPageWindow(1), + NUMERIC_POLICY, + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("8".repeat(64)), + ) + + override fun portableCountPlan(): BackendCountQueryPlan = countPlan( + BackendEnforcedFilter(BackendPlannedCondition.All, mandatory()), + ) + + override fun portableStreamPlan(): BackendStreamQueryPlan = streamPlan( + BackendEnforcedFilter(BackendPlannedCondition.All, mandatory()), + ) + + override fun portableSecondPagePlan(): BackendPageQueryPlan = pagePlan( + BackendEnforcedFilter(BackendPlannedCondition.All, mandatory()), + ) + + @BeforeEach + fun setup() { + client = ReactiveElasticsearchClients.createReactiveElasticsearchClient(elasticsearch) + indexName = "wow.sales.order.snapshot" + binding = createBinding(indexName) + if (client.indices().exists { exists -> exists.index(indexName) }.block()!!.value()) { + client.indices().delete { delete -> delete.index(indexName) }.block() + } + client.indices().create { create -> + create.index(indexName).mappings { mapping -> + mapping.meta(MAPPING_VERSION_META, JsonData.of(MAPPING_VERSION)) + .meta(DOCUMENT_KIND_META, JsonData.of(QueryDocumentKind.SNAPSHOT.name)) + .meta(SCHEMA_CONTRACT_META, JsonData.of(schema.contractId.value)) + .meta(CAPABILITY_DIGEST_META, JsonData.of(binding.prepared.capabilityDigest)) + .properties(MessageRecords.AGGREGATE_ID, keyword()) + .properties(MessageRecords.TENANT_ID, keyword()) + .properties(StateAggregateRecords.DELETED, Property.of { it.boolean_ { boolean -> boolean } }) + .properties( + "state", + Property.of { state -> + state.`object` { objectField -> + objectField + .properties( + "name", + Property.of { text -> + text.text { definition -> + definition.analyzer("standard").searchAnalyzer("standard") + .fields("exact", keyword()) + } + }, + ) + .properties("note", keyword()) + .properties("notePresent", Property.of { it.boolean_ { boolean -> boolean } }) + .properties("noteGroup", keyword()) + .properties("amount", Property.of { number -> number.long_ { it } }) + } + }, + ) + } + }.block() + documents().forEach { (id, document) -> + client.index> { index -> + index.index(indexName).id(id).document(document) + }.block() + } + client.indices().refresh { refresh -> refresh.index(indexName) }.block() + } + + @Test + fun `planned Elasticsearch should enforce mandatory filter and return complete record shapes`() { + val backend = binding.prepareContribution(client).block()!!.backend + val filter = BackendEnforcedFilter( + predicate(name, PredicateOperator.EQ, NormalizedValue.Text("alice")), + mandatory(), + ) + val single = backend.single(singlePlan(filter), OPTIONS).block()!! + + single.identity.assert().isEqualTo("order-1") + single.document.values.keys.assert().containsExactly("state") + val state = single.document.values["state"] as NormalizedValue.ObjectValue + state.values.assert().containsEntry("name", NormalizedValue.Text("alice")) + + val all = BackendEnforcedFilter(BackendPlannedCondition.All, mandatory()) + val stream = backend.stream(streamPlan(all), OPTIONS).collectList().block()!! + stream.map { record -> record.identity }.assert().containsExactly("order-1", "order-4") + + val page = backend.page(pagePlan(all), OPTIONS).block()!! + page.total.assert().isEqualTo(2) + page.totalRelation.assert().isEqualTo(BackendTotalRelation.EXACT) + page.consistency.assert().isEqualTo(BackendPageConsistency.SAME_INPUT) + page.records.single().identity.assert().isEqualTo("order-4") + + backend.count(countPlan(all), OPTIONS).block().assert().isEqualTo(2) + } + + @Test + fun `gateway Elasticsearch rehearsal should shadow cut over and roll back against one index`() { + val contribution = binding.prepareContribution(client).block()!! + val raw = LegacyElasticsearchCountQueryService(target.namedAggregate, client, indexName) + val observation = arrayOfNulls(1) + val observed = CountDownLatch(1) + val call = QueryCall(target, QueryPurpose("elasticsearch-rollout-rehearsal")) + + fun gateway( + mode: QueryExecutionMode, + shadowObserver: QueryShadowObserver = QueryShadowObserver.NONE, + ) = QueryGatewayRuntime.create( + namedAggregates = listOf(target.namedAggregate), + backendComposition = QueryBackendComposition( + listOf(contribution), + mapOf(target to contribution.backendId), + ), + rawServiceSource = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate): SnapshotQueryService<*> = raw + + override fun eventStream(namedAggregate: NamedAggregate) = + NoOpEventStreamQueryServiceFactory.create(namedAggregate) + }, + dialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.ROOT_QUALIFIED, QueryMatchScopeMode.FIELD) + }, + authorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.System("integration-test", "elasticsearch-rollout-rehearsal")) + }, + executionProfiles = QueryExecutionProfiles( + operationProfiles = mapOf( + QueryOperationProfileKey(target, QueryOperation.COUNT) to + QueryExecutionProfile(mode, QueryValidationMode.STRICT), + ), + ), + shadowObserver = shadowObserver, + runtimeHealthObserver = QueryRuntimeHealthObserver { }, + ).gateway + + gateway( + QueryExecutionMode.SHADOW, + QueryShadowObserver { current -> + observation[0] = current + observed.countDown() + }, + ).count(call, Condition.ALL).block().assert().isEqualTo(3) + observed.await(5, TimeUnit.SECONDS).assert().isTrue() + observation.single()!!.outcome.assert().isEqualTo(QueryShadowOutcome.MATCH) + raw.countInvocations.get().assert().isEqualTo(1) + + gateway(QueryExecutionMode.PLANNED) + .count(call, Condition.ALL).block().assert().isEqualTo(3) + raw.countInvocations.get().assert().isEqualTo(1) + + gateway(QueryExecutionMode.LEGACY) + .count(call, Condition.ALL).block().assert().isEqualTo(3) + raw.countInvocations.get().assert().isEqualTo(2) + } + + @Test + fun `planned Elasticsearch should preserve null missing literal and search bindings`() { + val backend = binding.prepareContribution(client).block()!!.backend + val nullOrMissing = BackendEnforcedFilter( + predicate(note, PredicateOperator.EQ, NormalizedValue.Null), + mandatory(), + ) + backend.count(countPlan(nullOrMissing), OPTIONS).block().assert().isEqualTo(2) + val explicitOrNonNull = BackendEnforcedFilter( + predicate(note, PredicateOperator.EXISTS, NormalizedValue.BooleanValue(true)), + mandatory(), + ) + backend.count(countPlan(explicitOrNonNull), OPTIONS).block().assert().isEqualTo(1) + val missingOnly = BackendEnforcedFilter( + predicate(note, PredicateOperator.EXISTS, NormalizedValue.BooleanValue(false)), + mandatory(), + ) + backend.count(countPlan(missingOnly), OPTIONS).block().assert().isEqualTo(1) + + val literal = BackendEnforcedFilter( + predicate(name, PredicateOperator.CONTAINS, NormalizedValue.Text("li")), + mandatory(), + ) + backend.count(countPlan(literal), OPTIONS).block().assert().isEqualTo(1) + + val search = BackendEnforcedFilter( + BackendPlannedCondition.Search(scope, "alice"), + mandatory(), + ) + backend.count(countPlan(search, SemanticTier.SEARCH), OPTIONS).block().assert().isEqualTo(1) + } + + @Test + fun `planned Elasticsearch page should cross the ten thousand window with PIT search after`() { + (0..10_000).chunked(500).forEach { batch -> + val response = client.bulk { bulk -> + bulk.operations( + batch.map { index -> + val id = "bulk-${index.toString().padStart(5, '0')}" + BulkOperation.of { operation -> + operation.index { request -> + request.index(indexName) + .id(id) + .document(document(id, "tenant-1", false, "bulk", MISSING, index.toLong())) + } + } + }, + ) + }.block()!! + response.errors().assert().isFalse() + } + client.indices().refresh { refresh -> refresh.index(indexName) }.block() + val backend = binding.prepareContribution(client).block()!!.backend + val options = OPTIONS.copy(maxPageWindow = 10_001, maxCursorPages = 11) + + val page = backend.page( + pagePlan(BackendEnforcedFilter(BackendPlannedCondition.All, mandatory()), 10_000, 1), + options, + ).block()!! + + page.records.single().identity.assert().isEqualTo("bulk-10000") + page.total.assert().isEqualTo(10_003) + } + + @Test + fun `planned Elasticsearch page should fail incomplete when the real PIT expires`() { + val backend = ElasticsearchSnapshotRecordQueryBackend(ExpiringPitClient(client), binding.prepared) + val error = runCatching { + backend.page( + pagePlan(BackendEnforcedFilter(BackendPlannedCondition.All, mandatory()), 0, 1), + OPTIONS, + ).block() + }.exceptionOrNull() as QueryBackendException + + error.kind.assert().isEqualTo(QueryBackendFailureKind.INCOMPLETE_RESULT) + } + + @Test + fun `planned Elasticsearch page cancellation should close the real PIT`() { + val cancellingClient = NeverCompletingPitClient(client) + val backend = ElasticsearchSnapshotRecordQueryBackend(cancellingClient, binding.prepared) + + StepVerifier.create( + backend.page( + pagePlan(BackendEnforcedFilter(BackendPlannedCondition.All, mandatory()), 0, 1), + OPTIONS, + ), + ).thenAwait(Duration.ofMillis(100)) + .thenCancel() + .verify() + + cancellingClient.closedPit.asMono().block(Duration.ofSeconds(5)).assert().isNotBlank() + } + + @Test + fun `planned Elasticsearch page should classify a closed real transport as unavailable`() { + val unavailableClient = ReactiveElasticsearchClients.createReactiveElasticsearchClient(elasticsearch) + unavailableClient.close() + val backend = ElasticsearchSnapshotRecordQueryBackend(unavailableClient, binding.prepared) + + val error = runCatching { + backend.page( + pagePlan(BackendEnforcedFilter(BackendPlannedCondition.All, mandatory()), 0, 1), + OPTIONS, + ).block() + }.exceptionOrNull() as QueryBackendException + + error.kind.assert().isEqualTo(QueryBackendFailureKind.UNAVAILABLE) + } + + @Test + fun `planned Elasticsearch composite analytics should preserve mandatory filter order and response cursor`() { + val analytics = binding.prepareContribution(client).block()!!.analyticsBackend!! + val first = analytics.analyze(groupedAnalyticsPlan(), OPTIONS.copy(maxReturnedBuckets = 1)).block()!! + + first.buckets.single().keys[NAME_ALIAS].assert().isEqualTo(NormalizedValue.Text("alice")) + first.buckets.single().metrics[COUNT_ALIAS].assert().isEqualTo(NormalizedValue.Int64(1)) + first.afterKey!!.assert().containsExactly(NormalizedValue.Text("alice")) + + val second = analytics.analyze( + groupedAnalyticsPlan(first.afterKey), + OPTIONS.copy(maxReturnedBuckets = 1), + ).block()!! + second.buckets.single().keys[NAME_ALIAS].assert().isEqualTo(NormalizedValue.Text("carol")) + second.buckets.single().metrics[COUNT_ALIAS].assert().isEqualTo(NormalizedValue.Int64(1)) + + val global = analytics.analyze(globalAnalyticsPlan(), OPTIONS.copy(maxReturnedBuckets = 1)).block()!! + global.buckets.single().metrics[COUNT_ALIAS].assert().isEqualTo(NormalizedValue.Int64(2)) + } + + @Test + fun `planned Elasticsearch analytics should replay every high cardinality bucket without gaps or duplicates`() { + val generated = (0 until 257).map { index -> "key-${index.toString().padStart(3, '0')}" } + val response = client.bulk { bulk -> + bulk.operations( + generated.mapIndexed { index, value -> + val id = "analytics-$value" + BulkOperation.of { operation -> + operation.index { request -> + request.index(indexName) + .id(id) + .document(document(id, "tenant-1", false, value, MISSING, index.toLong())) + } + } + }, + ) + }.block()!! + response.errors().assert().isFalse() + client.indices().refresh { refresh -> refresh.index(indexName) }.block() + val analytics = binding.prepareContribution(client).block()!!.analyticsBackend!! + val actual = mutableListOf() + var afterKey: List? = null + var pageCount = 0 + + do { + val page = analytics.analyze( + groupedAnalyticsPlan(afterKey, HIGH_CARDINALITY_PAGE_SIZE), + OPTIONS.copy(maxReturnedBuckets = HIGH_CARDINALITY_PAGE_SIZE), + ).block()!! + page.consistency.assert().isEqualTo(BackendAnalyticsConsistency.EVENTUAL) + page.completeness.assert().isEqualTo(BackendAnalyticsCompleteness.EXACT) + actual += page.buckets.map { bucket -> + (bucket.keys.getValue(NAME_ALIAS) as NormalizedValue.Text).value + } + afterKey = page.afterKey + pageCount++ + } while (afterKey != null && pageCount < MAX_HIGH_CARDINALITY_PAGES) + + actual.assert().containsExactlyElementsOf(listOf("alice", "carol") + generated) + actual.distinct().assert().hasSize(actual.size) + afterKey.assert().isNull() + pageCount.assert().isLessThanOrEqualTo(MAX_HIGH_CARDINALITY_PAGES) + } + + @Test + fun `planned Elasticsearch analytics eventual cursor should observe a later concurrent bucket`() { + val analytics = binding.prepareContribution(client).block()!!.analyticsBackend!! + val first = analytics.analyze(groupedAnalyticsPlan(), OPTIONS.copy(maxReturnedBuckets = 1)).block()!! + first.buckets.single().keys.getValue(NAME_ALIAS).assert().isEqualTo(NormalizedValue.Text("alice")) + + val id = "order-concurrent" + client.index> { index -> + index.index(indexName) + .id(id) + .document(document(id, "tenant-1", false, "bob", MISSING, 5)) + }.block() + client.indices().refresh { refresh -> refresh.index(indexName) }.block() + + val second = analytics.analyze( + groupedAnalyticsPlan(first.afterKey), + OPTIONS.copy(maxReturnedBuckets = 1), + ).block()!! + second.buckets.single().keys.getValue(NAME_ALIAS).assert().isEqualTo(NormalizedValue.Text("bob")) + second.consistency.assert().isEqualTo(BackendAnalyticsConsistency.EVENTUAL) + second.completeness.assert().isEqualTo(BackendAnalyticsCompleteness.EXACT) + } + + @Test + fun `planned Elasticsearch snapshot analytics should keep one PIT across continuation pages`() { + val contribution = binding.prepareContribution(client).block()!! + val analytics = contribution.analyticsBackend!! + val first = analytics.analyze( + groupedAnalyticsPlan(consistency = BackendAnalyticsConsistency.SNAPSHOT), + OPTIONS.copy(maxReturnedBuckets = 1), + null, + ).block()!! + first.buckets.single().keys.getValue(NAME_ALIAS).assert().isEqualTo(NormalizedValue.Text("alice")) + first.cursorState.assert().isNotNull() + + val id = "order-snapshot-concurrent" + client.index> { index -> + index.index(indexName) + .id(id) + .document(document(id, "tenant-1", false, "bob", MISSING, 5)) + }.block() + client.indices().refresh { refresh -> refresh.index(indexName) }.block() + + val second = analytics.analyze( + groupedAnalyticsPlan( + first.afterKey, + consistency = BackendAnalyticsConsistency.SNAPSHOT, + ), + OPTIONS.copy(maxReturnedBuckets = 1), + first.cursorState, + ).block()!! + second.buckets.single().keys.getValue(NAME_ALIAS).assert().isEqualTo(NormalizedValue.Text("carol")) + second.consistency.assert().isEqualTo(BackendAnalyticsConsistency.SNAPSHOT) + second.cursorState.assert().isNotNull() + + (analytics as AnalyticsQueryCursorLifecycle).close(second.cursorState!!).block() + } + + @Test + fun `planned Elasticsearch snapshot analytics should classify an expired PIT as incomplete`() { + val contribution = binding.prepareContribution(client).block()!! + val analytics = contribution.analyticsBackend!! + val first = analytics.analyze( + groupedAnalyticsPlan(consistency = BackendAnalyticsConsistency.SNAPSHOT), + OPTIONS.copy(maxReturnedBuckets = 1), + null, + ).block()!! + val state = first.cursorState!! + (analytics as AnalyticsQueryCursorLifecycle).close(state).block() + + val error = runCatching { + analytics.analyze( + groupedAnalyticsPlan( + first.afterKey, + consistency = BackendAnalyticsConsistency.SNAPSHOT, + ), + OPTIONS.copy(maxReturnedBuckets = 1), + state, + ).block() + }.exceptionOrNull() as QueryBackendException + + error.kind.assert().isEqualTo(QueryBackendFailureKind.INCOMPLETE_RESULT) + } + + private fun singlePlan(filter: BackendEnforcedFilter) = BackendSingleQueryPlan( + target, + schema.contractId, + filter, + RecordResultShape.DYNAMIC, + BackendProjection.Include(listOf(name)), + emptyList(), + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("1".repeat(64)), + ) + + private fun streamPlan(filter: BackendEnforcedFilter) = BackendStreamQueryPlan( + target, + schema.contractId, + filter, + RecordResultShape.DYNAMIC, + BackendProjection.All, + stableSort(), + 10, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("2".repeat(64)), + ) + + private fun pagePlan( + filter: BackendEnforcedFilter, + offset: Long = 1, + size: Int = 1, + ) = BackendPageQueryPlan( + target, + schema.contractId, + filter, + RecordResultShape.DYNAMIC, + BackendProjection.All, + stableSort(), + BackendPageWindow(offset, size), + BackendTotalMode.EXACT, + BackendRequiredConsistency.SAME_INPUT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("3".repeat(64)), + ) + + private fun countPlan( + filter: BackendEnforcedFilter, + tier: SemanticTier = SemanticTier.PORTABLE, + ) = BackendCountQueryPlan( + target, + schema.contractId, + filter, + BackendRequiredCapabilities(), + tier, + PlanFingerprint("4".repeat(64)), + ) + + private fun groupedAnalyticsPlan( + afterKey: List? = null, + limit: Int = 1, + consistency: BackendAnalyticsConsistency = BackendAnalyticsConsistency.EVENTUAL, + ) = BackendAnalyticsQueryPlan( + target, + schema.contractId, + BackendEnforcedFilter(BackendPlannedCondition.All, mandatory()), + BackendAnalyticsGrouping.By( + listOf(BackendAnalyticsDimension(NAME_ALIAS, name, BackendAnalyticsMissingPolicy.EXCLUDE)), + ), + listOf(BackendAnalyticsMetric.DocumentCount(COUNT_ALIAS)), + BackendAnalyticsCondition.All, + BackendAnalyticsBucketOrder.DimensionKeyAscending( + BackendAnalyticsNullPlacement.FIRST, + BackendAnalyticsTextCollation.BINARY, + ), + BackendAnalyticsPageWindow(limit, afterKey), + null, + consistency, + BackendAnalyticsCompleteness.EXACT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("5".repeat(64)), + ) + + private fun globalAnalyticsPlan( + filter: BackendEnforcedFilter = BackendEnforcedFilter(BackendPlannedCondition.All, mandatory()), + ) = BackendAnalyticsQueryPlan( + target, + schema.contractId, + filter, + BackendAnalyticsGrouping.Global, + listOf(BackendAnalyticsMetric.DocumentCount(COUNT_ALIAS)), + BackendAnalyticsCondition.All, + BackendAnalyticsBucketOrder.Global, + BackendAnalyticsPageWindow(1), + null, + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("6".repeat(64)), + ) + + private fun stableSort() = listOf( + BackendSort(identity, NormalizedSortDirection.ASC, BackendSortOrigin.STABILITY_TIE_BREAKER), + ) + + private fun mandatory() = BackendPlannedCondition.Junction( + JunctionOperator.AND, + listOf( + predicate(tenant, PredicateOperator.EQ, NormalizedValue.Text("tenant-1")), + predicate(deleted, PredicateOperator.IS_FALSE), + ), + ) + + private fun predicate( + field: QueryFieldId, + operator: PredicateOperator, + value: NormalizedValue? = null, + ) = BackendPlannedCondition.Predicate(field, operator, value) + + private fun documents(): Map> = linkedMapOf( + "order-1" to document("order-1", "tenant-1", false, "alice", null, 1), + "order-2" to document("order-2", "tenant-2", false, "alice", "visible", 2), + "order-3" to document("order-3", "tenant-1", true, "alice", "deleted", 3), + "order-4" to document("order-4", "tenant-1", false, "carol", MISSING, 4), + ) + + private fun document( + id: String, + tenantId: String, + isDeleted: Boolean, + currentName: String, + currentNote: Any?, + amount: Long, + ): Map = linkedMapOf( + MessageRecords.AGGREGATE_ID to id, + MessageRecords.TENANT_ID to tenantId, + StateAggregateRecords.DELETED to isDeleted, + "state" to linkedMapOf("name" to currentName).also { state -> + state["notePresent"] = currentNote !== MISSING + if (currentNote !== MISSING) state["note"] = currentNote + if (currentNote is String) state["noteGroup"] = currentNote + state["amount"] = amount + }, + ) + + private fun createBinding(index: String) = ElasticsearchSnapshotQueryBinding( + schema, + index, + MAPPING_VERSION, + linkedMapOf( + identity to ElasticsearchFieldBinding( + MessageRecords.AGGREGATE_ID, + EXACT_SORT_PROJECT, + exactField = "_id", + sortField = MessageRecords.AGGREGATE_ID, + keywordReadiness = KEYWORD_READINESS, + ), + tenant to ElasticsearchFieldBinding( + MessageRecords.TENANT_ID, + setOf(FieldCapability.EXACT), + exactField = MessageRecords.TENANT_ID, + keywordReadiness = KEYWORD_READINESS, + ), + deleted to ElasticsearchFieldBinding( + StateAggregateRecords.DELETED, + setOf(FieldCapability.EXACT), + exactField = StateAggregateRecords.DELETED, + ), + stateField to ElasticsearchFieldBinding("state", emptySet()), + name to ElasticsearchFieldBinding( + "state.name", + setOf( + FieldCapability.EXACT, + FieldCapability.FULL_TEXT, + FieldCapability.LITERAL_PATTERN, + FieldCapability.PROJECTABLE, + FieldCapability.AGGREGATABLE, + ), + exactField = "state.name.exact", + searchField = "state.name", + searchAnalyzer = "standard", + literalField = "state.name.exact", + groupField = "state.name.exact", + groupReadiness = GROUP_READINESS, + keywordReadiness = KEYWORD_READINESS, + ), + note to ElasticsearchFieldBinding( + "state.note", + setOf(FieldCapability.EXACT, FieldCapability.PRESENCE, FieldCapability.AGGREGATABLE), + exactField = "state.note", + presenceField = "state.notePresent", + groupField = "state.noteGroup", + groupReadiness = GROUP_READINESS, + keywordReadiness = KEYWORD_READINESS, + ), + amount to ElasticsearchFieldBinding( + "state.amount", + setOf(FieldCapability.AGGREGATABLE), + groupField = "state.amount", + groupReadiness = GROUP_READINESS, + ), + ), + listOf(ElasticsearchSearchScopeBinding(scope, mapOf(name to "state.name"))), + ) + + private fun keyword(): Property = Property.of { property -> + property.keyword { keyword -> keyword.ignoreAbove(128) } + } + + private class ExpiringPitClient(delegate: ReactiveElasticsearchClient) : + ReactiveElasticsearchClient(delegate._transport(), delegate._transportOptions()) { + override fun search( + request: SearchRequest, + tDocumentClass: Class, + ): Mono> { + val pitId = request.pit()?.id() ?: return super.search(request, tDocumentClass) + val closeRequest = ClosePointInTimeRequest.of { close -> close.id(pitId) } + return closePointInTime(closeRequest).then(super.search(request, tDocumentClass)) + } + } + + private class NeverCompletingPitClient(delegate: ReactiveElasticsearchClient) : + ReactiveElasticsearchClient(delegate._transport(), delegate._transportOptions()) { + val closedPit: Sinks.One = Sinks.one() + + override fun search( + request: SearchRequest, + tDocumentClass: Class, + ): Mono> = if (request.pit() == null) { + super.search(request, tDocumentClass) + } else { + Mono.never() + } + + override fun closePointInTime(request: ClosePointInTimeRequest): Mono = + super.closePointInTime(request).doOnNext { response -> + if (response.succeeded()) closedPit.tryEmitValue(request.id()) + } + } + + private class LegacyElasticsearchCountQueryService( + override val namedAggregate: NamedAggregate, + private val client: ReactiveElasticsearchClient, + private val indexName: String, + ) : SnapshotQueryService { + override val name: String = "legacy-elasticsearch-integration-test" + val countInvocations = AtomicInteger() + + override fun single(singleQuery: ISingleQuery) = Mono.empty>() + + override fun dynamicSingle(singleQuery: ISingleQuery): Mono = Mono.empty() + + override fun list(listQuery: IListQuery) = Flux.empty>() + + override fun dynamicList(listQuery: IListQuery): Flux = Flux.empty() + + override fun paged(pagedQuery: IPagedQuery) = + Mono.just(PagedList.empty>()) + + override fun dynamicPaged(pagedQuery: IPagedQuery) = Mono.just(PagedList.empty()) + + override fun count(condition: Condition): Mono = Mono.defer { + countInvocations.incrementAndGet() + val request = CountRequest.of { count -> + count.index(indexName).query(SnapshotConditionConverter.convert(condition)) + } + client.count(request).map { response -> response.count() } + } + } + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + private val tenant = QueryFieldId.System(SystemFieldKind.TENANT_ID) + private val deleted = QueryFieldId.System(SystemFieldKind.DELETED) + private val stateField = QueryFieldId.Path(listOf("state")) + private val name = QueryFieldId.Path(listOf("state", "name")) + private val note = QueryFieldId.Path(listOf("state", "note")) + private val amount = QueryFieldId.Path(listOf("state", "amount")) + private val scope = SearchScopeId("state-name") + private val schema = QueryDocumentSchema( + target, + listOf( + field(identity, LogicalFieldType.Text, setOf(PredicateOperator.EQ), EXACT_SORT_PROJECT), + field(tenant, LogicalFieldType.Text, setOf(PredicateOperator.EQ), setOf(FieldCapability.EXACT)), + field(deleted, LogicalFieldType.Boolean, setOf(PredicateOperator.IS_FALSE), setOf(FieldCapability.EXACT)), + field(stateField, LogicalFieldType.Object), + field( + name, + LogicalFieldType.Text, + setOf(PredicateOperator.EQ, PredicateOperator.CONTAINS), + setOf( + FieldCapability.EXACT, + FieldCapability.FULL_TEXT, + FieldCapability.LITERAL_PATTERN, + FieldCapability.PROJECTABLE, + FieldCapability.AGGREGATABLE, + ), + ), + field( + note, + LogicalFieldType.Text, + setOf(PredicateOperator.EQ, PredicateOperator.EXISTS), + setOf(FieldCapability.EXACT, FieldCapability.PRESENCE, FieldCapability.AGGREGATABLE), + ), + field(amount, LogicalFieldType.Int64, capabilities = setOf(FieldCapability.AGGREGATABLE)), + ), + listOf(QuerySearchScopeDefinition(scope, null, listOf(name), listOf(name))), + ) + + private fun field( + id: QueryFieldId, + type: LogicalFieldType, + operators: Set = emptySet(), + capabilities: Set = emptySet(), + ) = QueryFieldSchema(id, type, Presence.OPTIONAL, Nullability.NULLABLE, operators, capabilities) + + private companion object { + const val MAPPING_VERSION_META = "wow_query_mapping_version" + const val DOCUMENT_KIND_META = "wow_query_document_kind" + const val SCHEMA_CONTRACT_META = "wow_query_schema_contract_id" + const val CAPABILITY_DIGEST_META = "wow_query_capability_digest" + const val MAPPING_VERSION = "order-query-v1" + const val HIGH_CARDINALITY_PAGE_SIZE = 31 + const val MAX_HIGH_CARDINALITY_PAGES = 10 + val MISSING = Any() + val KEYWORD_READINESS = ElasticsearchKeywordReadiness(128, 512, true, true) + val GROUP_READINESS = ElasticsearchGroupReadiness(historicalValuesAudited = true) + val NAME_ALIAS = AnalyticsAlias("name") + val NOTE_ALIAS = AnalyticsAlias("note") + val COUNT_ALIAS = AnalyticsAlias("count") + val NUMERIC_POLICY = BackendAnalyticsNumericPolicy( + BackendAnalyticsNumericPromotion.DECIMAL128, + 34, + 4, + RoundingMode.HALF_UP, + BackendAnalyticsOverflowPolicy.REJECT, + ) + val EXACT_SORT_PROJECT = setOf( + FieldCapability.EXACT, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + ) + val OPTIONS = QueryBackendExecutionOptions( + Instant.now().plusSeconds(300), + 10, + maxPageWindow = 10, + ) + } +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/eventsourcing/ElasticsearchSnapshotWrite.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/eventsourcing/ElasticsearchSnapshotWrite.kt index 9757ee57c33..55c5c8e11ee 100644 --- a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/eventsourcing/ElasticsearchSnapshotWrite.kt +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/eventsourcing/ElasticsearchSnapshotWrite.kt @@ -26,9 +26,14 @@ internal data class ElasticsearchSnapshotWrite( ) internal fun Snapshot<*>.toElasticsearchSnapshotWrite(): ElasticsearchSnapshotWrite { + return toElasticsearchSnapshotWrite(aggregateId.toSnapshotIndexName()) +} + +internal fun Snapshot<*>.toElasticsearchSnapshotWrite(index: String): ElasticsearchSnapshotWrite { + require(index.isNotBlank()) { "Elasticsearch snapshot write index must not be blank." } val document = toLinkedHashMap() return ElasticsearchSnapshotWrite( - index = aggregateId.toSnapshotIndexName(), + index = index, id = aggregateId.id, document = document, version = document.requiredSnapshotVersion(), diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/BoundedElasticsearchIndexMigrationProbeVerifier.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/BoundedElasticsearchIndexMigrationProbeVerifier.kt new file mode 100644 index 00000000000..b962aa68b2b --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/BoundedElasticsearchIndexMigrationProbeVerifier.kt @@ -0,0 +1,233 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryTarget +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.util.Collections + +internal const val MAX_ELASTICSEARCH_INDEX_MIGRATION_PROBES = 256 + +@JvmInline +internal value class ElasticsearchIndexProbeId(val value: String) { + init { + require(value.matches(PROBE_ID_PATTERN)) { + "Elasticsearch index probe id must match ${PROBE_ID_PATTERN.pattern}." + } + } +} + +internal enum class ElasticsearchIndexProbeKind { + RECORD, + ANALYTICS, +} + +internal data class ElasticsearchIndexMigrationProbe( + val id: ElasticsearchIndexProbeId, + val kind: ElasticsearchIndexProbeKind, +) + +/** Immutable, canonical suite of application-owned semantic probes. */ +internal class ElasticsearchIndexMigrationProbeSuite( + val id: ElasticsearchIndexProbeSuiteId, + val target: QueryTarget, + val schemaContractId: SchemaContractId, + probes: Collection, +) { + val probes: List + + init { + val snapshot = probes.toList() + require(snapshot.isNotEmpty()) { "Elasticsearch index migration probe suite must not be empty." } + require(snapshot.size <= MAX_ELASTICSEARCH_INDEX_MIGRATION_PROBES) { + "Elasticsearch index migration probe suite exceeds its probe budget." + } + require(snapshot.map(ElasticsearchIndexMigrationProbe::id).distinct().size == snapshot.size) { + "Elasticsearch index migration probe ids must be unique." + } + this.probes = Collections.unmodifiableList(snapshot.sortedBy { probe -> probe.id.value }) + } + + override fun equals(other: Any?): Boolean = + this === other || + other is ElasticsearchIndexMigrationProbeSuite && + id == other.id && + target == other.target && + schemaContractId == other.schemaContractId && + probes == other.probes + + override fun hashCode(): Int { + var result = id.hashCode() + result = 31 * result + target.hashCode() + result = 31 * result + schemaContractId.hashCode() + result = 31 * result + probes.hashCode() + return result + } +} + +/** Backend-neutral fingerprint of one fully materialized probe result. */ +internal data class ElasticsearchIndexProbeEvidence( + val resultCount: Long, + val resultChecksum: ElasticsearchIndexChecksum, + val complete: Boolean, +) { + init { + require(resultCount >= 0) { "Elasticsearch index probe result count must not be negative." } + } +} + +/** Executes pre-registered probes without exposing wire queries or driver objects to the lifecycle kernel. */ +internal interface ElasticsearchIndexMigrationProbeExecutor { + fun evaluateAuthority( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + probe: ElasticsearchIndexMigrationProbe, + ): Mono + + fun evaluatePhysical( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + probe: ElasticsearchIndexMigrationProbe, + ): Mono +} + +/** Compares a bounded suite against authority and one exact physical generation, once per subscription. */ +internal class BoundedElasticsearchIndexMigrationProbeVerifier( + private val suite: ElasticsearchIndexMigrationProbeSuite, + private val executor: ElasticsearchIndexMigrationProbeExecutor, +) : ElasticsearchIndexMigrationProbeVerifier { + override val suiteId: ElasticsearchIndexProbeSuiteId + get() = suite.id + + override fun compare( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono = Mono.defer { + if ( + suite.id != manifest.verificationContract.probeSuiteId || + suite.target != manifest.target || + suite.schemaContractId != manifest.schemaContractId + ) { + reject( + ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED, + manifest.id, + "Elasticsearch query probe suite does not match the migration target and verification contract.", + ) + } + Flux.fromIterable(suite.probes) + .concatMap { probe -> compareProbe(command, manifest, physicalIndex, probe) } + .reduce(ElasticsearchIndexProbeVerification(0, 0), ::accumulate) + }.onErrorMap { error -> normalizeProbeError(manifest, error) } + + private fun compareProbe( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + probe: ElasticsearchIndexMigrationProbe, + ): Mono = Mono.zip( + Mono.defer { executor.evaluateAuthority(command, manifest, probe) } + .normalizeExecutorError(manifest, probe, "authority") + .requireEvidence(manifest, probe, "authority"), + Mono.defer { executor.evaluatePhysical(command, manifest, physicalIndex, probe) } + .normalizeExecutorError(manifest, probe, "physical generation") + .requireEvidence(manifest, probe, "physical generation"), + ).map { tuple -> ProbeComparison(probe.kind, tuple.t1 != tuple.t2) } + + private fun Mono.normalizeExecutorError( + manifest: ElasticsearchIndexMigrationManifest, + probe: ElasticsearchIndexMigrationProbe, + source: String, + ): Mono = onErrorMap { error -> + ElasticsearchIndexLifecycleException( + ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED, + manifest.id, + "Elasticsearch $source failed while evaluating probe [${probe.id.value}].", + error, + ) + } + + private fun Mono.requireEvidence( + manifest: ElasticsearchIndexMigrationManifest, + probe: ElasticsearchIndexMigrationProbe, + source: String, + ): Mono = switchIfEmpty( + Mono.error( + ElasticsearchIndexLifecycleException( + ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED, + manifest.id, + "Elasticsearch $source returned no evidence for probe [${probe.id.value}].", + ), + ), + ).flatMap { evidence -> + if (evidence.complete) { + Mono.just(evidence) + } else { + Mono.error( + ElasticsearchIndexLifecycleException( + ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED, + manifest.id, + "Elasticsearch $source returned incomplete evidence for probe [${probe.id.value}].", + ), + ) + } + } + + private fun accumulate( + current: ElasticsearchIndexProbeVerification, + comparison: ProbeComparison, + ): ElasticsearchIndexProbeVerification { + if (!comparison.mismatch) { + return current + } + return when (comparison.kind) { + ElasticsearchIndexProbeKind.RECORD -> current.copy( + recordMismatchCount = Math.addExact(current.recordMismatchCount, 1), + ) + + ElasticsearchIndexProbeKind.ANALYTICS -> current.copy( + analyticsMismatchCount = Math.addExact(current.analyticsMismatchCount, 1), + ) + } + } +} + +private data class ProbeComparison( + val kind: ElasticsearchIndexProbeKind, + val mismatch: Boolean, +) + +private fun normalizeProbeError( + manifest: ElasticsearchIndexMigrationManifest, + error: Throwable, +): Throwable = if (error is ElasticsearchIndexLifecycleException) { + error +} else { + ElasticsearchIndexLifecycleException( + ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED, + manifest.id, + "Elasticsearch query probe evidence could not be computed.", + error, + ) +} + +private val PROBE_ID_PATTERN = Regex("[A-Za-z0-9][A-Za-z0-9._-]{0,127}") diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/CanonicalDocumentChecksum.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/CanonicalDocumentChecksum.kt new file mode 100644 index 00000000000..c8099bc0c16 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/CanonicalDocumentChecksum.kt @@ -0,0 +1,235 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import java.io.ByteArrayOutputStream +import java.io.DataOutputStream +import java.math.BigDecimal +import java.math.BigInteger +import java.security.MessageDigest +import java.util.IdentityHashMap +import java.util.LinkedHashMap + +internal fun canonicalDocumentHash( + source: Map<*, *>, + limits: SnapshotCanonicalChecksumLimits, + excludedRootKeys: Set = emptySet(), +): ByteArray { + val canonical = CanonicalDocumentSnapshotter(limits).snapshotRoot(source, excludedRootKeys) + val buffer = ByteArrayOutputStream() + DataOutputStream(buffer).use { output -> output.writeCanonical(canonical) } + return newLifecycleSha256().digest(buffer.toByteArray()) +} + +private class CanonicalDocumentSnapshotter(private val limits: SnapshotCanonicalChecksumLimits) { + private val active = IdentityHashMap() + private var nodes = 0 + private var payloadBytes = 0L + + fun snapshotRoot( + source: Map<*, *>, + excludedRootKeys: Set, + ): CanonicalValue.ObjectValue = snapshotObject(source, 0, excludedRootKeys) + + private fun snapshot(value: Any?, depth: Int): CanonicalValue = when (value) { + null -> enterNode(depth) { CanonicalValue.Null } + is Boolean -> enterNode(depth) { CanonicalValue.BooleanValue(value) } + is String -> enterNode(depth) { CanonicalValue.Text(snapshotString(value)) } + is Number -> enterNode(depth) { value.toCanonicalNumber() } + is ByteArray -> enterNode(depth) { + consumePayload(value.size.toLong()) + CanonicalValue.Bytes(value.copyOf()) + } + + is Map<*, *> -> snapshotObject(value, depth, emptySet()) + is Iterable<*> -> snapshotIterable(value, depth) + is Array<*> -> snapshotIterable(value.asIterable(), depth) + else -> throw IllegalArgumentException( + "Canonical index verification does not support value type ${value::class.qualifiedName}.", + ) + } + + private fun snapshotObject( + source: Map<*, *>, + depth: Int, + excludedKeys: Set, + ): CanonicalValue.ObjectValue = withContainer(source, depth) { + val values = LinkedHashMap() + val iterator = source.entries.iterator() + var count = 0 + while (iterator.hasNext()) { + require(++count <= limits.maxCollectionSize) { + "Canonical index verification object exceeds its field limit." + } + val entry = iterator.next() + val key = entry.key as? String + ?: throw IllegalArgumentException("Canonical index verification object keys must be strings.") + if (key in excludedKeys) continue + require(!values.containsKey(key)) { "Canonical index verification object contains a duplicate key." } + snapshotString(key) + values[key] = snapshot(entry.value, depth + 1) + } + CanonicalValue.ObjectValue(values.toSortedMap()) + } + + private fun snapshotIterable(source: Iterable<*>, depth: Int): CanonicalValue.ListValue = + withContainer(source, depth) { + val values = ArrayList() + val iterator = source.iterator() + while (iterator.hasNext()) { + require(values.size < limits.maxCollectionSize) { + "Canonical index verification list exceeds its item limit." + } + values += snapshot(iterator.next(), depth + 1) + } + CanonicalValue.ListValue(values) + } + + private fun snapshotString(value: String): String { + val bytes = value.toByteArray(Charsets.UTF_8) + require(bytes.size <= limits.maxStringBytes) { + "Canonical index verification string exceeds its byte limit." + } + consumePayload(bytes.size.toLong()) + return value + } + + private fun Number.toCanonicalNumber(): CanonicalValue.NumberValue { + val decimal = when (this) { + is BigDecimal -> this + is BigInteger -> BigDecimal(this) + is Byte, is Short, is Int, is Long -> BigDecimal.valueOf(toLong()) + is Float, is Double -> { + val value = toDouble() + require(value.isFinite()) { "Canonical index verification numbers must be finite." } + BigDecimal.valueOf(value) + } + + else -> try { + BigDecimal(toString()) + } catch (error: NumberFormatException) { + throw IllegalArgumentException("Canonical index verification number is not canonicalizable.", error) + } + }.canonical() + val unscaled = decimal.unscaledValue().toString() + consumePayload(unscaled.toByteArray(Charsets.UTF_8).size.toLong()) + return CanonicalValue.NumberValue(unscaled, decimal.scale()) + } + + private fun consumePayload(bytes: Long) { + payloadBytes = Math.addExact(payloadBytes, bytes) + require(payloadBytes <= limits.maxPayloadBytesPerDocument) { + "Canonical index verification document exceeds its payload limit." + } + } + + private inline fun enterNode(depth: Int, block: () -> T): T { + require(depth <= limits.maxDepth && ++nodes <= limits.maxNodesPerDocument) { + "Canonical index verification document exceeds its structural limits." + } + return block() + } + + private inline fun withContainer(source: Any, depth: Int, block: () -> T): T = + enterNode(depth) { + require(active.put(source, Unit) == null) { + "Canonical index verification document contains a container cycle." + } + try { + block() + } finally { + active.remove(source) + } + } +} + +private sealed interface CanonicalValue { + data object Null : CanonicalValue + data class BooleanValue(val value: Boolean) : CanonicalValue + data class Text(val value: String) : CanonicalValue + data class NumberValue(val unscaled: String, val scale: Int) : CanonicalValue + data class Bytes(val value: ByteArray) : CanonicalValue + data class ListValue(val values: List) : CanonicalValue + data class ObjectValue(val values: Map) : CanonicalValue +} + +private fun DataOutputStream.writeCanonical(value: CanonicalValue) { + when (value) { + CanonicalValue.Null -> writeByte(0) + is CanonicalValue.BooleanValue -> { + writeByte(1) + writeBoolean(value.value) + } + + is CanonicalValue.Text -> { + writeByte(2) + writeUtf8(value.value) + } + + is CanonicalValue.NumberValue -> { + writeByte(3) + writeUtf8(value.unscaled) + writeInt(value.scale) + } + + is CanonicalValue.Bytes -> { + writeByte(4) + writeInt(value.value.size) + write(value.value) + } + + is CanonicalValue.ListValue -> { + writeByte(5) + writeInt(value.values.size) + value.values.forEach { nested -> writeCanonical(nested) } + } + + is CanonicalValue.ObjectValue -> { + writeByte(6) + writeInt(value.values.size) + value.values.forEach { (key, nested) -> + writeUtf8(key) + writeCanonical(nested) + } + } + } +} + +private fun DataOutputStream.writeUtf8(value: String) { + val bytes = value.toByteArray(Charsets.UTF_8) + writeInt(bytes.size) + write(bytes) +} + +private fun BigDecimal.canonical(): BigDecimal = + if (compareTo(BigDecimal.ZERO) == 0) BigDecimal.ZERO else stripTrailingZeros() + +internal fun newLifecycleSha256(): MessageDigest = MessageDigest.getInstance("SHA-256") + +internal fun MessageDigest.updateLifecycleUtf8(value: String) = + updateLifecycleLengthPrefixed(value.toByteArray(Charsets.UTF_8)) + +internal fun MessageDigest.updateLifecycleLengthPrefixed(bytes: ByteArray) { + update( + byteArrayOf( + (bytes.size ushr 24).toByte(), + (bytes.size ushr 16).toByte(), + (bytes.size ushr 8).toByte(), + bytes.size.toByte(), + ), + ) + update(bytes) +} + +internal fun ByteArray.toLowerHex(): String = joinToString("") { byte -> "%02x".format(byte) } diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/DefaultElasticsearchIndexLifecycleOperations.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/DefaultElasticsearchIndexLifecycleOperations.kt new file mode 100644 index 00000000000..20fe5c0dea0 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/DefaultElasticsearchIndexLifecycleOperations.kt @@ -0,0 +1,79 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import reactor.core.publisher.Mono + +internal fun interface ElasticsearchAuthoritativeIndexRebuilder { + fun rebuild( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono +} + +internal fun interface ElasticsearchIndexMigrationVerifier { + fun verify( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono +} + +internal class DefaultElasticsearchIndexLifecycleOperations( + private val admin: ElasticsearchIndexAdminClient, + private val templates: ElasticsearchVersionedIndexTemplateProvider, + private val rebuilder: ElasticsearchAuthoritativeIndexRebuilder, + private val verifier: ElasticsearchIndexMigrationVerifier, +) : ElasticsearchIndexLifecycleOperations { + override fun validate( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = admin.inspect(manifest) + + override fun create( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = templates.get(manifest).flatMap { template -> + admin.create(manifest, template) + } + + override fun rebuild( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = rebuilder.rebuild(command, manifest) + + override fun verify( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono = verifier.verify(command, manifest, physicalIndex) + + override fun cutover( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = admin.compareAndSetAlias( + manifest, + manifest.sourcePhysicalIndex, + manifest.names.physical, + ) + + override fun rollback( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = admin.compareAndSetAlias( + manifest, + manifest.names.physical, + manifest.sourcePhysicalIndex, + ) +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/DefaultElasticsearchIndexMigrationVerifier.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/DefaultElasticsearchIndexMigrationVerifier.kt new file mode 100644 index 00000000000..301088772c8 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/DefaultElasticsearchIndexMigrationVerifier.kt @@ -0,0 +1,141 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import reactor.core.publisher.Mono +import java.time.Clock + +internal data class ElasticsearchAuthoritativeVerificationSnapshot( + val checksumAlgorithm: ElasticsearchIndexChecksumAlgorithm, + val count: Long, + val identityChecksum: ElasticsearchIndexChecksum, + val contentChecksum: ElasticsearchIndexChecksum, + val watermark: Long?, +) { + init { + require(count >= 0) { "Authoritative verification count must not be negative." } + require(watermark == null || watermark >= 0) { "Authoritative watermark must not be negative." } + } +} + +internal data class ElasticsearchPhysicalIndexVerificationSnapshot( + val checksumAlgorithm: ElasticsearchIndexChecksumAlgorithm, + val physicalIndex: ElasticsearchPhysicalIndex, + val count: Long, + val identityChecksum: ElasticsearchIndexChecksum, + val contentChecksum: ElasticsearchIndexChecksum, + val versionContinuity: Boolean, + val watermark: Long?, +) { + init { + require(count >= 0) { "Physical index verification count must not be negative." } + require(watermark == null || watermark >= 0) { "Indexed watermark must not be negative." } + } +} + +internal data class ElasticsearchIndexProbeVerification( + val recordMismatchCount: Long, + val analyticsMismatchCount: Long, +) { + init { + require(recordMismatchCount >= 0 && analyticsMismatchCount >= 0) { + "Elasticsearch query probe mismatch counts must not be negative." + } + } +} + +internal fun interface ElasticsearchAuthoritativeVerificationSource { + fun capture( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono +} + +internal fun interface ElasticsearchPhysicalIndexVerificationSource { + fun inspect( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono +} + +internal interface ElasticsearchIndexMigrationProbeVerifier { + val suiteId: ElasticsearchIndexProbeSuiteId + + fun compare( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono +} + +/** Combines independently computed authority, physical-index and query-probe evidence into one immutable report. */ +internal class DefaultElasticsearchIndexMigrationVerifier( + private val authority: ElasticsearchAuthoritativeVerificationSource, + private val physical: ElasticsearchPhysicalIndexVerificationSource, + private val probes: ElasticsearchIndexMigrationProbeVerifier, + private val clock: Clock, +) : ElasticsearchIndexMigrationVerifier { + override fun verify( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono = Mono.defer { + if (probes.suiteId != manifest.verificationContract.probeSuiteId) { + reject( + ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED, + manifest.id, + "Elasticsearch query probe suite does not match the migration verification contract.", + ) + } + Mono.zip( + Mono.defer { authority.capture(command, manifest) }, + Mono.defer { physical.inspect(command, manifest, physicalIndex) }, + Mono.defer { probes.compare(command, manifest, physicalIndex) }, + ).map { tuple -> + val expected = tuple.t1 + val actual = tuple.t2 + val probe = tuple.t3 + if (expected.checksumAlgorithm != manifest.verificationContract.checksumAlgorithm) { + reject( + ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED, + manifest.id, + "Authoritative checksum algorithm does not match the migration verification contract.", + ) + } + if (actual.checksumAlgorithm != manifest.verificationContract.checksumAlgorithm) { + reject( + ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED, + manifest.id, + "Physical-index checksum algorithm does not match the migration verification contract.", + ) + } + ElasticsearchIndexVerification( + actual.physicalIndex, + expected.count, + actual.count, + expected.identityChecksum, + actual.identityChecksum, + expected.contentChecksum, + actual.contentChecksum, + actual.versionContinuity, + expected.watermark, + actual.watermark, + probe.recordMismatchCount, + probe.analyticsMismatchCount, + clock.instant(), + ) + } + } +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleClient.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleClient.kt new file mode 100644 index 00000000000..32ae49458f0 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleClient.kt @@ -0,0 +1,444 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import co.elastic.clients.elasticsearch._types.ElasticsearchException +import co.elastic.clients.elasticsearch._types.mapping.TypeMapping +import co.elastic.clients.elasticsearch.cluster.PutComponentTemplateRequest +import co.elastic.clients.elasticsearch.indices.CreateIndexRequest +import co.elastic.clients.elasticsearch.indices.GetAliasRequest +import co.elastic.clients.elasticsearch.indices.GetMappingRequest +import co.elastic.clients.elasticsearch.indices.IndexSettings +import co.elastic.clients.elasticsearch.indices.PutIndexTemplateRequest +import co.elastic.clients.elasticsearch.indices.UpdateAliasesRequest +import co.elastic.clients.elasticsearch.indices.get_alias.IndexAliases +import co.elastic.clients.elasticsearch.indices.get_mapping.IndexMappingRecord +import co.elastic.clients.json.JsonData +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import org.springframework.data.elasticsearch.RestStatusException +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Mono +import java.time.Clock +import java.util.LinkedHashMap +import java.util.Optional + +internal class ElasticsearchVersionedIndexTemplate( + val manifest: ElasticsearchIndexMigrationManifest, + val mapping: TypeMapping, + val settings: IndexSettings? = null, +) { + val componentTemplateName = "${manifest.names.alias.value}-query-mapping-${manifest.mappingVersion.tag}" + val indexTemplateName = "${manifest.names.alias.value}-query-template-${manifest.mappingVersion.tag}" + val indexPattern = "${manifest.names.alias.value}-${manifest.mappingVersion.tag}-*" + + init { + requireManagedName(componentTemplateName, "Elasticsearch component template") + requireManagedName(indexTemplateName, "Elasticsearch index template") + mapping.toAttestation(manifest.id, manifest.names.physical).requireMatches(manifest) + } + + fun componentRequest(): PutComponentTemplateRequest = PutComponentTemplateRequest.of { request -> + request.name(componentTemplateName) + .version(manifest.mappingVersion.value.toLong()) + .create(false) + .meta(templateMetadata()) + .template { template -> + template.mappings(mapping).also { builder -> settings?.let(builder::settings) } + } + } + + fun indexTemplateRequest(): PutIndexTemplateRequest = PutIndexTemplateRequest.of { request -> + request.name(indexTemplateName) + .version(manifest.mappingVersion.value.toLong()) + .create(false) + .allowAutoCreate(false) + .priority(INDEX_TEMPLATE_PRIORITY) + .indexPatterns(indexPattern) + .composedOf(componentTemplateName) + .meta(templateMetadata()) + } + + private fun templateMetadata(): Map = linkedMapOf( + ELASTICSEARCH_MAPPING_VERSION_META to JsonData.of(manifest.mappingVersion.tag), + ELASTICSEARCH_DOCUMENT_KIND_META to JsonData.of(manifest.target.documentKind.name), + ELASTICSEARCH_SCHEMA_CONTRACT_META to JsonData.of(manifest.schemaContractId.value), + ELASTICSEARCH_CAPABILITY_DIGEST_META to JsonData.of(manifest.capabilityDigest.value), + ) +} + +internal fun interface ElasticsearchVersionedIndexTemplateProvider { + fun get(manifest: ElasticsearchIndexMigrationManifest): Mono +} + +internal interface ElasticsearchIndexAdminClient { + fun inspect(manifest: ElasticsearchIndexMigrationManifest): Mono + + fun ensureTemplates(template: ElasticsearchVersionedIndexTemplate): Mono + + fun create( + manifest: ElasticsearchIndexMigrationManifest, + template: ElasticsearchVersionedIndexTemplate, + ): Mono + + fun compareAndSetAlias( + manifest: ElasticsearchIndexMigrationManifest, + expected: ElasticsearchPhysicalIndex, + current: ElasticsearchPhysicalIndex, + ): Mono +} + +internal class ReactiveElasticsearchIndexAdminClient( + private val client: ReactiveElasticsearchClient, + private val clock: Clock, +) : ElasticsearchIndexAdminClient { + override fun inspect(manifest: ElasticsearchIndexMigrationManifest): Mono = + Mono.zip(inspectAlias(manifest), inspectMappings(manifest)) + .map { tuple -> + ElasticsearchIndexInventory( + manifest.names.alias, + tuple.t1.orElse(null), + tuple.t2, + clock.instant(), + ) + } + + override fun ensureTemplates(template: ElasticsearchVersionedIndexTemplate): Mono = + client.cluster().putComponentTemplate(template.componentRequest()) + .switchIfEmpty(operationFailed(template.manifest, "Component template returned no acknowledgement.")) + .flatMap { response -> + if (!response.acknowledged()) { + operationFailed(template.manifest, "Component template was not acknowledged.") + } else { + client.indices().putIndexTemplate(template.indexTemplateRequest()) + } + } + .switchIfEmpty(operationFailed(template.manifest, "Index template returned no acknowledgement.")) + .flatMap { response -> + requireAcknowledged( + response.acknowledged(), + template.manifest, + "Index template was not acknowledged.", + ) + } + + override fun create( + manifest: ElasticsearchIndexMigrationManifest, + template: ElasticsearchVersionedIndexTemplate, + ): Mono { + require(template.manifest == manifest) { "Elasticsearch versioned template must match the migration manifest." } + return ensureTemplates(template).then(inspectExact(manifest, manifest.names.physical)).flatMap { existing -> + if (existing.isPresent) { + requireAttestation(manifest, existing.get()) + } else { + client.indices().create( + CreateIndexRequest.of { request -> request.index(manifest.names.physical.value) }, + ) + .switchIfEmpty(operationFailed(manifest, "Create index returned no acknowledgement.")) + .flatMap { response -> + if ( + !response.acknowledged() || + !response.shardsAcknowledged() || + response.index() != manifest.names.physical.value + ) { + operationFailed(manifest, "Create index was not fully acknowledged.") + } else { + inspectExact(manifest, manifest.names.physical).flatMap { created -> + if (created.isEmpty) { + operationFailed(manifest, "Created index returned no mapping attestation.") + } else { + requireAttestation(manifest, created.get()) + } + } + } + } + } + } + } + + override fun compareAndSetAlias( + manifest: ElasticsearchIndexMigrationManifest, + expected: ElasticsearchPhysicalIndex, + current: ElasticsearchPhysicalIndex, + ): Mono = inspectAlias(manifest).flatMap { observedValue -> + val observed = observedValue.orElse(null) + if (observed == current) { + return@flatMap Mono.just( + ElasticsearchAliasTransition(manifest.names.alias, expected, current, clock.instant()), + ) + } + if (observed != expected) { + return@flatMap lifecycleError( + ElasticsearchIndexLifecycleErrorCode.ALIAS_CONFLICT, + manifest, + "Elasticsearch alias does not match the expected source generation.", + ) + } + val request = aliasTransitionRequest(manifest, expected, current) + client.indices().updateAliases(request) + .switchIfEmpty(operationFailed(manifest, "Alias transition returned no acknowledgement.")) + .flatMap { response -> + if (!response.acknowledged()) { + operationFailed(manifest, "Alias transition was not acknowledged.") + } else { + inspectAlias(manifest).flatMap { updated -> + if (updated.orElse(null) != current) { + lifecycleError( + ElasticsearchIndexLifecycleErrorCode.ALIAS_CONFLICT, + manifest, + "Elasticsearch alias did not converge on the requested generation.", + ) + } else { + Mono.just( + ElasticsearchAliasTransition( + manifest.names.alias, + expected, + current, + clock.instant(), + ), + ) + } + } + } + } + } + + private fun inspectAlias( + manifest: ElasticsearchIndexMigrationManifest, + ): Mono> = client.indices().getAlias( + GetAliasRequest.of { request -> + request.name(manifest.names.alias.value).allowNoIndices(true).ignoreUnavailable(true) + }, + ).map { response -> response.aliases() } + .onErrorResume(::isNotFound) { Mono.just(emptyMap()) } + .switchIfEmpty(Mono.just(emptyMap())) + .map { aliases -> Optional.ofNullable(aliases.requireSingleWriteAlias(manifest)) } + + private fun inspectMappings( + manifest: ElasticsearchIndexMigrationManifest, + ): Mono> = client.indices().getMapping( + GetMappingRequest.of { request -> + request.index("${manifest.names.alias.value}-v*").allowNoIndices(true).ignoreUnavailable(true) + }, + ).map { response -> response.mappings() } + .onErrorResume(::isNotFound) { Mono.just(emptyMap()) } + .switchIfEmpty(Mono.just(emptyMap())) + .map { mappings -> mappings.toAttestations(manifest) } + + private fun inspectExact( + manifest: ElasticsearchIndexMigrationManifest, + physical: ElasticsearchPhysicalIndex, + ): Mono> = client.indices().getMapping( + GetMappingRequest.of { request -> + request.index(physical.value).allowNoIndices(true).ignoreUnavailable(true) + }, + ).map { response -> response.mappings() } + .onErrorResume(::isNotFound) { Mono.just(emptyMap()) } + .switchIfEmpty(Mono.just(emptyMap())) + .map { mappings -> + if (mappings.isEmpty()) { + Optional.empty() + } else { + if (mappings.size != 1 || physical.value !in mappings) { + reject( + ElasticsearchIndexLifecycleErrorCode.ATTESTATION_MISMATCH, + manifest.id, + "Exact Elasticsearch index lookup returned another generation.", + ) + } + Optional.of(mappings.getValue(physical.value).mapping().toAttestation(manifest.id, physical)) + } + } + + private fun requireAttestation( + manifest: ElasticsearchIndexMigrationManifest, + actual: ElasticsearchIndexAttestation, + ): Mono { + actual.requireMatches(manifest) + return Mono.just(actual) + } +} + +internal fun aliasTransitionRequest( + manifest: ElasticsearchIndexMigrationManifest, + expected: ElasticsearchPhysicalIndex, + current: ElasticsearchPhysicalIndex, +): UpdateAliasesRequest = UpdateAliasesRequest.of { request -> + request.actions { action -> + action.remove { remove -> + remove.index(expected.value).alias(manifest.names.alias.value).mustExist(true) + } + }.actions { action -> + action.add { add -> + add.index(current.value).alias(manifest.names.alias.value).isWriteIndex(true) + } + } +} + +private fun Map.requireSingleWriteAlias( + manifest: ElasticsearchIndexMigrationManifest, +): ElasticsearchPhysicalIndex? { + if (isEmpty()) return null + if (size != 1) { + reject( + ElasticsearchIndexLifecycleErrorCode.ALIAS_CONFLICT, + manifest.id, + "Managed Elasticsearch alias resolves to more than one physical index.", + ) + } + val (physical, aliases) = entries.single() + val definition = aliases.aliases()[manifest.names.alias.value] ?: reject( + ElasticsearchIndexLifecycleErrorCode.ALIAS_CONFLICT, + manifest.id, + "Elasticsearch alias response is missing the requested alias definition.", + ) + if (!definition.isManagedWriteAlias()) { + reject( + ElasticsearchIndexLifecycleErrorCode.ALIAS_CONFLICT, + manifest.id, + "Managed Elasticsearch alias must be an unfiltered, unrouted write alias.", + ) + } + return ElasticsearchPhysicalIndex(physical) +} + +private fun co.elastic.clients.elasticsearch.indices.AliasDefinition.isManagedWriteAlias(): Boolean = + isWriteIndex() == true && + filter() == null && + indexRouting() == null && + searchRouting() == null && + routing() == null + +private fun Map.toAttestations( + manifest: ElasticsearchIndexMigrationManifest, +): Map { + val result = LinkedHashMap(size) + entries.sortedBy(Map.Entry::key).forEach { (index, record) -> + val physical = ElasticsearchPhysicalIndex(index) + result[physical] = record.mapping().toAttestation(manifest.id, physical) + } + return result +} + +private fun IndexMappingRecord.mapping(): TypeMapping = mappings() ?: item() ?: error( + "Elasticsearch mapping response has no mapping payload.", +) + +private fun TypeMapping.toAttestation( + migrationId: ElasticsearchIndexMigrationId, + physical: ElasticsearchPhysicalIndex, +): ElasticsearchIndexAttestation { + val mappingVersion = requiredMeta(migrationId, ELASTICSEARCH_MAPPING_VERSION_META) + val versionNumber = mappingVersion.removePrefix("v").toIntOrNull() ?: reject( + ElasticsearchIndexLifecycleErrorCode.ATTESTATION_MISMATCH, + migrationId, + "Elasticsearch mapping version metadata is invalid.", + ) + val documentKind = runCatching { + QueryDocumentKind.valueOf(requiredMeta(migrationId, ELASTICSEARCH_DOCUMENT_KIND_META)) + }.getOrElse { + reject( + ElasticsearchIndexLifecycleErrorCode.ATTESTATION_MISMATCH, + migrationId, + "Elasticsearch document kind metadata is invalid.", + ) + } + return try { + ElasticsearchIndexAttestation( + physical, + ElasticsearchIndexMappingVersion(versionNumber), + documentKind, + SchemaContractId(requiredMeta(migrationId, ELASTICSEARCH_SCHEMA_CONTRACT_META)), + ElasticsearchIndexCapabilityDigest(requiredMeta(migrationId, ELASTICSEARCH_CAPABILITY_DIGEST_META)), + ) + } catch (error: IllegalArgumentException) { + throw ElasticsearchIndexLifecycleException( + ElasticsearchIndexLifecycleErrorCode.ATTESTATION_MISMATCH, + migrationId, + "Elasticsearch mapping metadata is invalid.", + error, + ) + } +} + +private fun TypeMapping.requiredMeta( + migrationId: ElasticsearchIndexMigrationId, + key: String, +): String = meta()[key]?.let { value -> + runCatching { value.to(String::class.java) }.getOrElse { error -> + throw ElasticsearchIndexLifecycleException( + ElasticsearchIndexLifecycleErrorCode.ATTESTATION_MISMATCH, + migrationId, + "Elasticsearch mapping metadata [$key] is unreadable.", + error, + ) + } +} ?: reject( + ElasticsearchIndexLifecycleErrorCode.ATTESTATION_MISMATCH, + migrationId, + "Elasticsearch mapping metadata [$key] is missing.", +) + +private fun ElasticsearchIndexAttestation.requireMatches(manifest: ElasticsearchIndexMigrationManifest) { + if (this != manifest.destinationAttestation) { + reject( + ElasticsearchIndexLifecycleErrorCode.ATTESTATION_MISMATCH, + manifest.id, + "Elasticsearch mapping attestation does not match the migration manifest.", + ) + } +} + +private fun isNotFound(error: Throwable): Boolean = when (error) { + is ElasticsearchException -> error.status() == NOT_FOUND + is RestStatusException -> error.status == NOT_FOUND + else -> false +} + +private fun operationFailed( + manifest: ElasticsearchIndexMigrationManifest, + message: String, +): Mono = lifecycleError(ElasticsearchIndexLifecycleErrorCode.OPERATION_FAILED, manifest, message) + +private fun requireAcknowledged( + acknowledged: Boolean, + manifest: ElasticsearchIndexMigrationManifest, + message: String, +): Mono = if (acknowledged) Mono.empty() else operationFailed(manifest, message) + +private fun lifecycleError( + code: ElasticsearchIndexLifecycleErrorCode, + manifest: ElasticsearchIndexMigrationManifest, + message: String, +): Mono = Mono.error(ElasticsearchIndexLifecycleException(code, manifest.id, message)) + +private fun requireManagedName(value: String, label: String) { + require(value.length <= MAX_MANAGED_NAME_LENGTH && value.matches(MANAGED_NAME_PATTERN)) { + "$label is not a valid managed name." + } +} + +internal const val ELASTICSEARCH_MAPPING_VERSION_META = "wow_query_mapping_version" +internal const val ELASTICSEARCH_DOCUMENT_KIND_META = "wow_query_document_kind" +internal const val ELASTICSEARCH_SCHEMA_CONTRACT_META = "wow_query_schema_contract_id" +internal const val ELASTICSEARCH_CAPABILITY_DIGEST_META = "wow_query_capability_digest" +private const val INDEX_TEMPLATE_PRIORITY = 500L +private const val MAX_MANAGED_NAME_LENGTH = 255 +private const val NOT_FOUND = 404 +private val MANAGED_NAME_PATTERN = Regex("[a-z0-9][a-z0-9._-]*") diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleExecutor.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleExecutor.kt new file mode 100644 index 00000000000..05a8d329823 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleExecutor.kt @@ -0,0 +1,409 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.DateTimeException + +internal interface ElasticsearchIndexLifecycleRepository { + fun create(state: ElasticsearchIndexMigrationState): Mono + + fun load(id: ElasticsearchIndexMigrationId): Mono + + fun compareAndSet( + expected: ElasticsearchIndexLifecycleStoredState, + state: ElasticsearchIndexMigrationState, + ): Mono +} + +internal sealed interface ElasticsearchIndexLifecycleRepositoryVersion { + data class InMemory(val value: Long) : ElasticsearchIndexLifecycleRepositoryVersion + + data class Elasticsearch( + val sequenceNumber: Long, + val primaryTerm: Long, + ) : ElasticsearchIndexLifecycleRepositoryVersion +} + +internal data class ElasticsearchIndexLifecycleStoredState( + val state: ElasticsearchIndexMigrationState, + val version: ElasticsearchIndexLifecycleRepositoryVersion, +) + +internal interface ElasticsearchIndexLifecycleOperations { + fun validate( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono + + fun create( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono + + fun rebuild( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono + + fun verify( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono + + fun cutover( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono + + fun rollback( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono +} + +/** Trusted pause/drain or mirror boundary required before a Snapshot alias can move. */ +internal fun interface ElasticsearchSnapshotCutoverGuard { + fun awaitFence( + manifest: ElasticsearchIndexMigrationManifest, + verification: ElasticsearchIndexVerification, + ): Mono + + companion object { + val DENY: ElasticsearchSnapshotCutoverGuard = ElasticsearchSnapshotCutoverGuard { manifest, _ -> + lifecycleError( + ElasticsearchIndexLifecycleErrorCode.CUTOVER_FENCE_REQUIRED, + manifest.id, + "Snapshot cutover requires an explicit pause/drain or controlled-mirror write fence.", + ) + } + } +} + +internal class ElasticsearchIndexLifecycleExecutor( + private val repository: ElasticsearchIndexLifecycleRepository, + private val operations: ElasticsearchIndexLifecycleOperations, + private val clock: Clock, + private val snapshotCutoverGuard: ElasticsearchSnapshotCutoverGuard = ElasticsearchSnapshotCutoverGuard.DENY, +) { + fun register(manifest: ElasticsearchIndexMigrationManifest): Mono = + Mono.defer { + val initial = ElasticsearchIndexMigrationState.initial(manifest) + repository.create(initial).map(ElasticsearchIndexLifecycleStoredState::state) + .switchIfEmpty( + load(manifest.id).flatMap { existing -> + if (existing.state.manifest == manifest) { + Mono.just(existing.state) + } else { + lifecycleError( + ElasticsearchIndexLifecycleErrorCode.MIGRATION_CONFLICT, + manifest.id, + "Elasticsearch migration id is already registered with another manifest.", + ) + } + } + ) + } + + fun plan(command: ElasticsearchIndexLifecycleCommand): Mono = + Mono.defer { load(command.migrationId).map { stored -> stored.state.plan(command) } } + + fun execute(command: ElasticsearchIndexLifecycleCommand): Mono = + Mono.defer { claim(command).flatMap { state -> executeClaimed(command, state) } } + + private fun claim(command: ElasticsearchIndexLifecycleCommand): Mono = + load(command.migrationId).flatMap { stored -> + val current = stored.state + current.lastCompletedCommand?.takeIf { it.id == command.id }?.let { completed -> + if (completed.type != command.type) { + return@flatMap lifecycleError( + ElasticsearchIndexLifecycleErrorCode.COMMAND_CONFLICT, + current.manifest.id, + "Completed command id is reused for another command type.", + ) + } + return@flatMap Mono.just(current) + } + val claimed = current.claim(command, clock.instant()) + if (claimed === current) { + Mono.just(current) + } else { + repository.compareAndSet(stored, claimed) + .map(ElasticsearchIndexLifecycleStoredState::state) + .switchIfEmpty(Mono.defer { claim(command) }) + } + } + + private fun executeClaimed( + command: ElasticsearchIndexLifecycleCommand, + state: ElasticsearchIndexMigrationState, + ): Mono { + state.lastCompletedCommand?.takeIf { it.id == command.id }?.let { return Mono.just(state) } + val result = when (command.type) { + ElasticsearchIndexLifecycleCommandType.VALIDATE -> executeValidate(command, state) + ElasticsearchIndexLifecycleCommandType.CREATE -> executeCreate(command, state) + ElasticsearchIndexLifecycleCommandType.REBUILD -> executeRebuild(command, state) + ElasticsearchIndexLifecycleCommandType.VERIFY -> executeVerify(command, state) + ElasticsearchIndexLifecycleCommandType.CUTOVER -> executeCutover(command, state) + ElasticsearchIndexLifecycleCommandType.ROLLBACK -> executeRollback(command, state) + } + return result.onErrorMap { error -> + if (error is ElasticsearchIndexLifecycleException) { + error + } else { + ElasticsearchIndexLifecycleException( + ElasticsearchIndexLifecycleErrorCode.OPERATION_FAILED, + state.manifest.id, + "Elasticsearch lifecycle command ${command.type} failed.", + error, + ) + } + } + } + + private fun executeValidate( + command: ElasticsearchIndexLifecycleCommand, + state: ElasticsearchIndexMigrationState, + ): Mono = operations.validate(command.id, state.manifest) + .requireValue(command, state) + .flatMap { inventory -> + state.manifest.validate(inventory) + complete(command, state) { copy(inventory = inventory) } + } + + private fun executeCreate( + command: ElasticsearchIndexLifecycleCommand, + state: ElasticsearchIndexMigrationState, + ): Mono = operations.create(command.id, state.manifest) + .requireValue(command, state) + .flatMap { attestation -> + if (attestation != state.manifest.destinationAttestation) { + return@flatMap lifecycleError( + ElasticsearchIndexLifecycleErrorCode.ATTESTATION_MISMATCH, + state.manifest.id, + "Created Elasticsearch index attestation does not match the manifest.", + ) + } + complete(command, state) { copy(destinationAttestation = attestation) } + } + + private fun executeRebuild( + command: ElasticsearchIndexLifecycleCommand, + state: ElasticsearchIndexMigrationState, + ): Mono = operations.rebuild(command.id, state.manifest) + .requireValue(command, state) + .flatMap { receipt -> + receipt.requireMatches(state.manifest) + complete(command, state) { copy(rebuildReceipt = receipt) } + } + + private fun executeVerify( + command: ElasticsearchIndexLifecycleCommand, + state: ElasticsearchIndexMigrationState, + ): Mono { + val physical = when (state.phase) { + ElasticsearchIndexLifecyclePhase.REBUILT -> state.manifest.names.physical + ElasticsearchIndexLifecyclePhase.CUTOVER -> state.manifest.sourcePhysicalIndex + else -> return lifecycleError( + ElasticsearchIndexLifecycleErrorCode.INVALID_TRANSITION, + state.manifest.id, + "Elasticsearch verification is not allowed from ${state.phase}.", + ) + } + return operations.verify(command.id, state.manifest, physical) + .requireValue(command, state) + .flatMap { verification -> + verification.requireSatisfied(state.manifest, physical) + complete(command, state) { + when (phase) { + ElasticsearchIndexLifecyclePhase.REBUILT -> copy(destinationVerification = verification) + ElasticsearchIndexLifecyclePhase.CUTOVER -> copy(rollbackVerification = verification) + else -> error("Verification phase changed while command was active.") + } + } + } + } + + private fun executeCutover( + command: ElasticsearchIndexLifecycleCommand, + state: ElasticsearchIndexMigrationState, + ): Mono = requireCutoverFence(state) + .then(operations.cutover(command.id, state.manifest)) + .requireValue(command, state) + .flatMap { transition -> + requireTransition( + state, + transition, + state.manifest.sourcePhysicalIndex, + state.manifest.names.physical, + ) + val retainUntil = try { + transition.transitionedAt.plus(state.manifest.minimumRetention) + } catch (error: DateTimeException) { + return@flatMap Mono.error(error) + } catch (error: ArithmeticException) { + return@flatMap Mono.error(error) + } + complete(command, state) { copy(cutover = transition, retainedSourceUntil = retainUntil) } + } + + private fun requireCutoverFence(state: ElasticsearchIndexMigrationState): Mono = + if (state.manifest.target.documentKind == me.ahoo.wow.query.gateway.QueryDocumentKind.SNAPSHOT) { + snapshotCutoverGuard.awaitFence( + state.manifest, + requireNotNull(state.destinationVerification) { + "Verified lifecycle state must retain destination verification." + }, + ) + } else { + Mono.empty() + } + + private fun executeRollback( + command: ElasticsearchIndexLifecycleCommand, + state: ElasticsearchIndexMigrationState, + ): Mono = operations.rollback(command.id, state.manifest) + .requireValue(command, state) + .flatMap { transition -> + requireTransition( + state, + transition, + state.manifest.names.physical, + state.manifest.sourcePhysicalIndex, + ) + complete(command, state) { copy(rollback = transition) } + } + + private fun requireTransition( + state: ElasticsearchIndexMigrationState, + transition: ElasticsearchAliasTransition, + previous: ElasticsearchPhysicalIndex?, + current: ElasticsearchPhysicalIndex, + ) { + if ( + transition.alias != state.manifest.names.alias || + transition.previous != previous || + transition.current != current + ) { + reject( + ElasticsearchIndexLifecycleErrorCode.ALIAS_CONFLICT, + state.manifest.id, + "Elasticsearch alias transition does not match the expected compare-and-set.", + ) + } + } + + private fun complete( + command: ElasticsearchIndexLifecycleCommand, + claimed: ElasticsearchIndexMigrationState, + update: ElasticsearchIndexMigrationState.() -> ElasticsearchIndexMigrationState, + ): Mono { + val completed = claimed.complete(command, clock.instant(), update) + return load(claimed.manifest.id).flatMap { currentStored -> + if (currentStored.state != claimed) { + return@flatMap completedOrConflict(command, claimed, currentStored.state) + } + repository.compareAndSet(currentStored, completed) + .map(ElasticsearchIndexLifecycleStoredState::state) + .switchIfEmpty( + load(claimed.manifest.id).flatMap { latest -> + completedOrConflict(command, claimed, latest.state) + }, + ) + } + } + + private fun completedOrConflict( + command: ElasticsearchIndexLifecycleCommand, + claimed: ElasticsearchIndexMigrationState, + current: ElasticsearchIndexMigrationState, + ): Mono = + if (current.lastCompletedCommand?.id == command.id) { + Mono.just(current) + } else { + lifecycleError( + ElasticsearchIndexLifecycleErrorCode.STATE_CONFLICT, + claimed.manifest.id, + "Elasticsearch migration changed while completing the command.", + ) + } + + private fun Mono.requireValue( + command: ElasticsearchIndexLifecycleCommand, + state: ElasticsearchIndexMigrationState, + ): Mono = switchIfEmpty( + lifecycleError( + ElasticsearchIndexLifecycleErrorCode.OPERATION_FAILED, + state.manifest.id, + "Elasticsearch lifecycle command ${command.type} returned no result.", + ), + ) + + private fun load(id: ElasticsearchIndexMigrationId): Mono = + repository.load(id).switchIfEmpty( + lifecycleError( + ElasticsearchIndexLifecycleErrorCode.MIGRATION_NOT_FOUND, + id, + "Elasticsearch index migration [${id.value}] is not registered.", + ), + ) +} + +internal class InMemoryElasticsearchIndexLifecycleRepository : ElasticsearchIndexLifecycleRepository { + private val states = java.util.concurrent.ConcurrentHashMap< + ElasticsearchIndexMigrationId, + ElasticsearchIndexMigrationState, + >() + + override fun create(state: ElasticsearchIndexMigrationState): Mono = + Mono.defer { + if (states.putIfAbsent(state.manifest.id, state) == null) { + Mono.just(state.stored()) + } else { + Mono.empty() + } + } + + override fun load(id: ElasticsearchIndexMigrationId): Mono = + Mono.defer { Mono.justOrEmpty(states[id]?.stored()) } + + override fun compareAndSet( + expected: ElasticsearchIndexLifecycleStoredState, + state: ElasticsearchIndexMigrationState, + ): Mono = Mono.defer { + val id = expected.state.manifest.id + require(id == state.manifest.id) { "Elasticsearch migration state id must match repository key." } + val expectedVersion = expected.version as? ElasticsearchIndexLifecycleRepositoryVersion.InMemory + ?: error("In-memory repository requires an in-memory storage version.") + val updated = states.computeIfPresent(id) { _, current -> + if (current.revision == expectedVersion.value && current == expected.state) state else current + } + if (updated === state) Mono.just(state.stored()) else Mono.empty() + } + + fun get(id: ElasticsearchIndexMigrationId): ElasticsearchIndexMigrationState? = states[id] + + private fun ElasticsearchIndexMigrationState.stored() = ElasticsearchIndexLifecycleStoredState( + this, + ElasticsearchIndexLifecycleRepositoryVersion.InMemory(revision), + ) +} + +private fun lifecycleError( + code: ElasticsearchIndexLifecycleErrorCode, + id: ElasticsearchIndexMigrationId, + message: String, +): Mono = Mono.error(ElasticsearchIndexLifecycleException(code, id, message)) diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleModel.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleModel.kt new file mode 100644 index 00000000000..c446c7d2784 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleModel.kt @@ -0,0 +1,435 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.wow.elasticsearch.IndexNameConverter.toEventStreamIndexName +import me.ahoo.wow.elasticsearch.IndexNameConverter.toSnapshotIndexName +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import java.time.Duration +import java.time.Instant +import java.util.Collections +import java.util.LinkedHashMap + +@JvmInline +internal value class ElasticsearchIndexMigrationId(val value: String) { + init { + requireIdentifier(value, "Elasticsearch index migration id") + } +} + +@JvmInline +internal value class ElasticsearchIndexLifecycleCommandId(val value: String) { + init { + requireIdentifier(value, "Elasticsearch index lifecycle command id") + } +} + +@JvmInline +internal value class ElasticsearchIndexMappingVersion(val value: Int) { + init { + require(value in 1..MAX_MAPPING_VERSION) { + "Elasticsearch index mapping version must be between 1 and $MAX_MAPPING_VERSION." + } + } + + val tag: String + get() = "v%04d".format(value) +} + +@JvmInline +internal value class ElasticsearchIndexGeneration(val value: Int) { + init { + require(value in 1..MAX_GENERATION) { + "Elasticsearch index generation must be between 1 and $MAX_GENERATION." + } + } + + val tag: String + get() = "%06d".format(value) +} + +@JvmInline +internal value class ElasticsearchIndexAlias(val value: String) { + init { + requireIndexName(value, "Elasticsearch index alias") + } +} + +@JvmInline +internal value class ElasticsearchPhysicalIndex(val value: String) { + init { + requireIndexName(value, "Elasticsearch physical index") + } +} + +@JvmInline +internal value class ElasticsearchIndexCapabilityDigest(val value: String) { + init { + requireSha256(value, "Elasticsearch index capability digest") + } +} + +@JvmInline +internal value class ElasticsearchIndexChecksum(val value: String) { + init { + requireSha256(value, "Elasticsearch index checksum") + } +} + +internal enum class ElasticsearchIndexChecksumAlgorithm { + CANONICAL_DOCUMENT_SHA256_V1, + CANONICAL_EVENT_STREAM_SHA256_V1, +} + +@JvmInline +internal value class ElasticsearchIndexProbeSuiteId(val value: String) { + init { + requireIdentifier(value, "Elasticsearch index probe suite id") + } +} + +/** Pins the data checksum and query-probe semantics for the full migration lifecycle. */ +internal data class ElasticsearchIndexVerificationContract( + val checksumAlgorithm: ElasticsearchIndexChecksumAlgorithm, + val probeSuiteId: ElasticsearchIndexProbeSuiteId, +) + +internal data class ElasticsearchIndexNames( + val alias: ElasticsearchIndexAlias, + val physical: ElasticsearchPhysicalIndex, +) { + companion object { + fun of( + target: QueryTarget, + mappingVersion: ElasticsearchIndexMappingVersion, + generation: ElasticsearchIndexGeneration, + ): ElasticsearchIndexNames { + val alias = when (target.documentKind) { + QueryDocumentKind.SNAPSHOT -> target.namedAggregate.toSnapshotIndexName() + QueryDocumentKind.EVENT_STREAM -> target.namedAggregate.toEventStreamIndexName() + } + return ElasticsearchIndexNames( + ElasticsearchIndexAlias(alias), + ElasticsearchPhysicalIndex("$alias-${mappingVersion.tag}-${generation.tag}"), + ) + } + } +} + +internal enum class ElasticsearchIndexRebuildStrategy { + SNAPSHOT_FROM_EVENT_STREAM, + EVENT_STREAM_PAUSE_AND_DRAIN, + EVENT_STREAM_CONTROLLED_MIRROR, +} + +internal data class ElasticsearchIndexMigrationManifest( + val id: ElasticsearchIndexMigrationId, + val target: QueryTarget, + val mappingVersion: ElasticsearchIndexMappingVersion, + val generation: ElasticsearchIndexGeneration, + val schemaContractId: SchemaContractId, + val capabilityDigest: ElasticsearchIndexCapabilityDigest, + val sourcePhysicalIndex: ElasticsearchPhysicalIndex, + val rebuildStrategy: ElasticsearchIndexRebuildStrategy, + val verificationContract: ElasticsearchIndexVerificationContract, + val maxCursorTtl: Duration, + val rollbackWindow: Duration, +) { + val names: ElasticsearchIndexNames = ElasticsearchIndexNames.of(target, mappingVersion, generation) + val minimumRetention: Duration + val destinationAttestation: ElasticsearchIndexAttestation + + init { + require(!maxCursorTtl.isNegative && !maxCursorTtl.isZero) { + "Maximum cursor TTL must be positive." + } + require(!rollbackWindow.isNegative && !rollbackWindow.isZero) { + "Elasticsearch rollback window must be positive." + } + require(sourcePhysicalIndex != names.physical) { + "Elasticsearch migration source and destination must differ." + } + requireStrategyMatchesTarget(target.documentKind, rebuildStrategy) + requireVerificationContractMatchesTarget(target.documentKind, verificationContract) + minimumRetention = try { + maxCursorTtl.plus(rollbackWindow) + } catch (error: ArithmeticException) { + throw IllegalArgumentException("Elasticsearch index retention duration overflows.", error) + } + destinationAttestation = ElasticsearchIndexAttestation( + names.physical, + mappingVersion, + target.documentKind, + schemaContractId, + capabilityDigest, + ) + } + + fun validate(inventory: ElasticsearchIndexInventory) { + if (inventory.alias != names.alias) { + reject( + ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED, + id, + "Inventory alias [${inventory.alias.value}] does not match [${names.alias.value}].", + ) + } + if (inventory.aliasTarget != sourcePhysicalIndex) { + reject( + ElasticsearchIndexLifecycleErrorCode.ALIAS_CONFLICT, + id, + "Alias [${names.alias.value}] changed from the expected migration source.", + ) + } + if (sourcePhysicalIndex !in inventory.indices) { + reject( + ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED, + id, + "Elasticsearch migration source [${sourcePhysicalIndex.value}] is missing.", + ) + } + inventory.indices[names.physical]?.let { actual -> + if (actual != destinationAttestation) { + reject( + ElasticsearchIndexLifecycleErrorCode.ATTESTATION_MISMATCH, + id, + "Elasticsearch destination [${names.physical.value}] has incompatible mapping metadata.", + ) + } + } + } +} + +internal data class ElasticsearchIndexAttestation( + val physicalIndex: ElasticsearchPhysicalIndex, + val mappingVersion: ElasticsearchIndexMappingVersion, + val documentKind: QueryDocumentKind, + val schemaContractId: SchemaContractId, + val capabilityDigest: ElasticsearchIndexCapabilityDigest, +) + +internal class ElasticsearchIndexInventory( + val alias: ElasticsearchIndexAlias, + val aliasTarget: ElasticsearchPhysicalIndex?, + indices: Map, + val observedAt: Instant, +) { + val indices: Map + + init { + val copy = LinkedHashMap(indices.size) + indices.entries.sortedBy { entry -> entry.key.value }.forEach { (physical, attestation) -> + require(physical == attestation.physicalIndex) { + "Elasticsearch inventory key and attestation physical index must match." + } + copy[physical] = attestation + } + this.indices = Collections.unmodifiableMap(copy) + require(aliasTarget == null || aliasTarget in this.indices) { + "Elasticsearch alias target must be present in the physical index inventory." + } + } + + override fun equals(other: Any?): Boolean = + this === other || + other is ElasticsearchIndexInventory && + alias == other.alias && + aliasTarget == other.aliasTarget && + indices == other.indices && + observedAt == other.observedAt + + override fun hashCode(): Int { + var result = alias.hashCode() + result = 31 * result + (aliasTarget?.hashCode() ?: 0) + result = 31 * result + indices.hashCode() + result = 31 * result + observedAt.hashCode() + return result + } +} + +internal data class ElasticsearchIndexRebuildReceipt( + val physicalIndex: ElasticsearchPhysicalIndex, + val strategy: ElasticsearchIndexRebuildStrategy, + val authoritativeWatermark: Long?, + val indexedWatermark: Long?, + val completedAt: Instant, +) { + init { + require(authoritativeWatermark == null || authoritativeWatermark >= 0) { + "Authoritative watermark must not be negative." + } + require(indexedWatermark == null || indexedWatermark >= 0) { + "Indexed watermark must not be negative." + } + require((authoritativeWatermark == null) == (indexedWatermark == null)) { + "Rebuild authoritative and indexed watermarks must be present together." + } + } + + fun requireMatches(manifest: ElasticsearchIndexMigrationManifest) { + val eventStreamWatermarkSatisfied = manifest.target.documentKind != QueryDocumentKind.EVENT_STREAM || + authoritativeWatermark != null + val identitySatisfied = physicalIndex == manifest.names.physical && strategy == manifest.rebuildStrategy + val watermarkSatisfied = authoritativeWatermark == indexedWatermark && eventStreamWatermarkSatisfied + if ( + !identitySatisfied || + !watermarkSatisfied + ) { + reject( + ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED, + manifest.id, + "Elasticsearch rebuild receipt does not match the migration manifest.", + ) + } + } +} + +@Suppress("LongParameterList") +internal data class ElasticsearchIndexVerification( + val physicalIndex: ElasticsearchPhysicalIndex, + val expectedCount: Long, + val actualCount: Long, + val expectedIdentityChecksum: ElasticsearchIndexChecksum, + val actualIdentityChecksum: ElasticsearchIndexChecksum, + val expectedContentChecksum: ElasticsearchIndexChecksum, + val actualContentChecksum: ElasticsearchIndexChecksum, + val versionContinuity: Boolean, + val authoritativeWatermark: Long?, + val indexedWatermark: Long?, + val recordProbeMismatchCount: Long, + val analyticsProbeMismatchCount: Long, + val verifiedAt: Instant, +) { + init { + require(expectedCount >= 0 && actualCount >= 0) { "Verification counts must not be negative." } + require(recordProbeMismatchCount >= 0 && analyticsProbeMismatchCount >= 0) { + "Verification mismatch counts must not be negative." + } + require(authoritativeWatermark == null || authoritativeWatermark >= 0) { + "Authoritative watermark must not be negative." + } + require(indexedWatermark == null || indexedWatermark >= 0) { + "Indexed watermark must not be negative." + } + } + + fun requireSatisfied( + manifest: ElasticsearchIndexMigrationManifest, + expectedIndex: ElasticsearchPhysicalIndex, + ) { + val satisfied = physicalIndex == expectedIndex && + expectedCount == actualCount && + expectedIdentityChecksum == actualIdentityChecksum && + expectedContentChecksum == actualContentChecksum && + versionContinuity && + authoritativeWatermark == indexedWatermark && + (manifest.target.documentKind != QueryDocumentKind.EVENT_STREAM || authoritativeWatermark != null) && + recordProbeMismatchCount == 0L && + analyticsProbeMismatchCount == 0L + if (!satisfied) { + reject( + ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED, + manifest.id, + "Elasticsearch index verification failed for [${physicalIndex.value}].", + ) + } + } +} + +internal data class ElasticsearchAliasTransition( + val alias: ElasticsearchIndexAlias, + val previous: ElasticsearchPhysicalIndex?, + val current: ElasticsearchPhysicalIndex, + val transitionedAt: Instant, +) + +internal enum class ElasticsearchIndexLifecycleErrorCode { + MIGRATION_NOT_FOUND, + MIGRATION_CONFLICT, + COMMAND_CONFLICT, + INVALID_TRANSITION, + STATE_CONFLICT, + VALIDATION_FAILED, + ATTESTATION_MISMATCH, + VERIFICATION_FAILED, + ALIAS_CONFLICT, + CUTOVER_FENCE_REQUIRED, + OPERATION_FAILED, + REPOSITORY_CORRUPTED, +} + +internal class ElasticsearchIndexLifecycleException( + val code: ElasticsearchIndexLifecycleErrorCode, + val migrationId: ElasticsearchIndexMigrationId, + message: String, + cause: Throwable? = null, +) : RuntimeException(message, cause) + +internal fun reject( + code: ElasticsearchIndexLifecycleErrorCode, + migrationId: ElasticsearchIndexMigrationId, + message: String, +): Nothing = throw ElasticsearchIndexLifecycleException(code, migrationId, message) + +private fun requireStrategyMatchesTarget( + documentKind: QueryDocumentKind, + strategy: ElasticsearchIndexRebuildStrategy, +) { + val matches = when (documentKind) { + QueryDocumentKind.SNAPSHOT -> strategy == ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM + QueryDocumentKind.EVENT_STREAM -> strategy != ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM + } + require(matches) { "Elasticsearch rebuild strategy $strategy does not match $documentKind." } +} + +private fun requireVerificationContractMatchesTarget( + documentKind: QueryDocumentKind, + contract: ElasticsearchIndexVerificationContract, +) { + val expected = when (documentKind) { + QueryDocumentKind.SNAPSHOT -> ElasticsearchIndexChecksumAlgorithm.CANONICAL_DOCUMENT_SHA256_V1 + QueryDocumentKind.EVENT_STREAM -> ElasticsearchIndexChecksumAlgorithm.CANONICAL_EVENT_STREAM_SHA256_V1 + } + require(contract.checksumAlgorithm == expected) { + "Elasticsearch checksum algorithm ${contract.checksumAlgorithm} does not match $documentKind." + } +} + +private fun requireIdentifier(value: String, label: String) { + require(value.matches(IDENTIFIER_PATTERN)) { + "$label must match ${IDENTIFIER_PATTERN.pattern}." + } +} + +private fun requireIndexName(value: String, label: String) { + require(value.matches(INDEX_NAME_PATTERN) && ".." !in value) { + "$label is not a valid managed index name." + } +} + +private fun requireSha256(value: String, label: String) { + require(value.matches(SHA_256_PATTERN)) { "$label must be a lowercase SHA-256 hex string." } +} + +private const val MAX_MAPPING_VERSION = 9_999 +private const val MAX_GENERATION = 999_999 +private val IDENTIFIER_PATTERN = Regex("[A-Za-z0-9][A-Za-z0-9._-]{0,127}") +private val INDEX_NAME_PATTERN = Regex("[a-z0-9][a-z0-9._-]{0,254}") +private val SHA_256_PATTERN = Regex("[0-9a-f]{64}") diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleState.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleState.kt new file mode 100644 index 00000000000..9cd23c7f27f --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleState.kt @@ -0,0 +1,198 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import java.time.Instant + +internal enum class ElasticsearchIndexLifecycleCommandType { + VALIDATE, + CREATE, + REBUILD, + VERIFY, + CUTOVER, + ROLLBACK, +} + +internal enum class ElasticsearchIndexLifecyclePhase { + NEW, + VALIDATED, + CREATED, + REBUILT, + VERIFIED, + CUTOVER, + ROLLBACK_VERIFIED, + ROLLED_BACK, +} + +internal data class ElasticsearchIndexLifecycleCommand( + val id: ElasticsearchIndexLifecycleCommandId, + val migrationId: ElasticsearchIndexMigrationId, + val type: ElasticsearchIndexLifecycleCommandType, + val expectedRevision: Long, +) { + init { + require(expectedRevision >= 0) { "Expected Elasticsearch migration revision must not be negative." } + } +} + +internal data class ElasticsearchIndexActiveCommand( + val id: ElasticsearchIndexLifecycleCommandId, + val type: ElasticsearchIndexLifecycleCommandType, + val from: ElasticsearchIndexLifecyclePhase, + val to: ElasticsearchIndexLifecyclePhase, + val startedAt: Instant, +) + +internal data class ElasticsearchIndexCompletedCommand( + val id: ElasticsearchIndexLifecycleCommandId, + val type: ElasticsearchIndexLifecycleCommandType, + val completedAt: Instant, +) + +@Suppress("LongParameterList") +internal data class ElasticsearchIndexMigrationState( + val manifest: ElasticsearchIndexMigrationManifest, + val phase: ElasticsearchIndexLifecyclePhase, + val revision: Long, + val activeCommand: ElasticsearchIndexActiveCommand? = null, + val lastCompletedCommand: ElasticsearchIndexCompletedCommand? = null, + val inventory: ElasticsearchIndexInventory? = null, + val destinationAttestation: ElasticsearchIndexAttestation? = null, + val rebuildReceipt: ElasticsearchIndexRebuildReceipt? = null, + val destinationVerification: ElasticsearchIndexVerification? = null, + val cutover: ElasticsearchAliasTransition? = null, + val rollbackVerification: ElasticsearchIndexVerification? = null, + val rollback: ElasticsearchAliasTransition? = null, + val retainedSourceUntil: Instant? = null, +) { + init { + require(revision >= 0) { "Elasticsearch migration revision must not be negative." } + activeCommand?.let { command -> require(command.from == phase) } + } + + fun plan(command: ElasticsearchIndexLifecycleCommand): ElasticsearchIndexLifecycleCommandPlan { + requireMigration(command) + lastCompletedCommand?.takeIf { completed -> completed.id == command.id }?.let { completed -> + require(completed.type == command.type) + return ElasticsearchIndexLifecycleCommandPlan(command, phase, phase, resumed = true) + } + activeCommand?.let { active -> + if (active.id != command.id || active.type != command.type) { + reject( + ElasticsearchIndexLifecycleErrorCode.COMMAND_CONFLICT, + manifest.id, + "Elasticsearch migration already has an active command.", + ) + } + return ElasticsearchIndexLifecycleCommandPlan(command, active.from, active.to, resumed = true) + } + if (command.expectedRevision != revision) { + reject( + ElasticsearchIndexLifecycleErrorCode.STATE_CONFLICT, + manifest.id, + "Elasticsearch migration revision changed from ${command.expectedRevision} to $revision.", + ) + } + return ElasticsearchIndexLifecycleCommandPlan(command, phase, nextPhase(command.type), resumed = false) + } + + fun claim( + command: ElasticsearchIndexLifecycleCommand, + now: Instant, + ): ElasticsearchIndexMigrationState { + val plan = plan(command) + if (plan.resumed) return this + return copy( + revision = revision + 1, + activeCommand = ElasticsearchIndexActiveCommand(command.id, command.type, plan.from, plan.to, now), + ) + } + + fun complete( + command: ElasticsearchIndexLifecycleCommand, + now: Instant, + update: ElasticsearchIndexMigrationState.() -> ElasticsearchIndexMigrationState, + ): ElasticsearchIndexMigrationState { + val active = requireNotNull(activeCommand) { "Elasticsearch migration has no active command." } + if (active.id != command.id || active.type != command.type) { + reject( + ElasticsearchIndexLifecycleErrorCode.COMMAND_CONFLICT, + manifest.id, + "Elasticsearch migration active command does not match completion.", + ) + } + val updated = update() + return updated.copy( + phase = active.to, + revision = revision + 1, + activeCommand = null, + lastCompletedCommand = ElasticsearchIndexCompletedCommand(command.id, command.type, now), + ) + } + + private fun requireMigration(command: ElasticsearchIndexLifecycleCommand) { + if (command.migrationId != manifest.id) { + reject( + ElasticsearchIndexLifecycleErrorCode.MIGRATION_CONFLICT, + manifest.id, + "Elasticsearch lifecycle command targets another migration.", + ) + } + } + + private fun nextPhase(type: ElasticsearchIndexLifecycleCommandType): ElasticsearchIndexLifecyclePhase = + when (phase to type) { + ElasticsearchIndexLifecyclePhase.NEW to ElasticsearchIndexLifecycleCommandType.VALIDATE -> + ElasticsearchIndexLifecyclePhase.VALIDATED + + ElasticsearchIndexLifecyclePhase.VALIDATED to ElasticsearchIndexLifecycleCommandType.CREATE -> + ElasticsearchIndexLifecyclePhase.CREATED + + ElasticsearchIndexLifecyclePhase.CREATED to ElasticsearchIndexLifecycleCommandType.REBUILD -> + ElasticsearchIndexLifecyclePhase.REBUILT + + ElasticsearchIndexLifecyclePhase.REBUILT to ElasticsearchIndexLifecycleCommandType.VERIFY -> + ElasticsearchIndexLifecyclePhase.VERIFIED + + ElasticsearchIndexLifecyclePhase.VERIFIED to ElasticsearchIndexLifecycleCommandType.CUTOVER -> + ElasticsearchIndexLifecyclePhase.CUTOVER + + ElasticsearchIndexLifecyclePhase.CUTOVER to ElasticsearchIndexLifecycleCommandType.VERIFY -> { + ElasticsearchIndexLifecyclePhase.ROLLBACK_VERIFIED + } + + ElasticsearchIndexLifecyclePhase.ROLLBACK_VERIFIED to ElasticsearchIndexLifecycleCommandType.ROLLBACK -> + ElasticsearchIndexLifecyclePhase.ROLLED_BACK + + else -> invalidTransition(type) + } + + private fun invalidTransition(type: ElasticsearchIndexLifecycleCommandType): Nothing = reject( + ElasticsearchIndexLifecycleErrorCode.INVALID_TRANSITION, + manifest.id, + "Elasticsearch lifecycle command $type is not allowed from $phase.", + ) + + companion object { + fun initial(manifest: ElasticsearchIndexMigrationManifest): ElasticsearchIndexMigrationState = + ElasticsearchIndexMigrationState(manifest, ElasticsearchIndexLifecyclePhase.NEW, revision = 0) + } +} + +internal data class ElasticsearchIndexLifecycleCommandPlan( + val command: ElasticsearchIndexLifecycleCommand, + val from: ElasticsearchIndexLifecyclePhase, + val to: ElasticsearchIndexLifecyclePhase, + val resumed: Boolean, +) diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleStateCodec.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleStateCodec.kt new file mode 100644 index 00000000000..f7944788a81 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleStateCodec.kt @@ -0,0 +1,376 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryTarget +import java.io.ByteArrayInputStream +import java.io.ByteArrayOutputStream +import java.io.DataInputStream +import java.io.DataOutputStream +import java.io.EOFException +import java.io.IOException +import java.nio.ByteBuffer +import java.nio.charset.CodingErrorAction +import java.time.Duration +import java.time.Instant +import java.util.LinkedHashMap + +/** Explicit, bounded wire format for durable lifecycle state. */ +internal class ElasticsearchIndexLifecycleStateCodec { + fun encode(state: ElasticsearchIndexMigrationState): ByteArray = try { + val buffer = ByteArrayOutputStream() + DataOutputStream(buffer).use { output -> + output.writeInt(MAGIC) + output.writeInt(FORMAT_VERSION) + output.writeState(state) + } + buffer.toByteArray().also { encoded -> + if (encoded.size > MAX_PAYLOAD_BYTES) { + corrupt(state.manifest.id, "Encoded Elasticsearch lifecycle state exceeds the payload limit.") + } + } + } catch (error: ElasticsearchIndexLifecycleException) { + throw error + } catch (error: IllegalArgumentException) { + corrupt(state.manifest.id, "Failed to encode Elasticsearch lifecycle state.", error) + } catch (error: IOException) { + corrupt(state.manifest.id, "Failed to encode Elasticsearch lifecycle state.", error) + } + + fun decode( + expectedId: ElasticsearchIndexMigrationId, + encoded: ByteArray, + ): ElasticsearchIndexMigrationState { + if (encoded.size > MAX_PAYLOAD_BYTES) { + corrupt(expectedId, "Persisted Elasticsearch lifecycle state exceeds the payload limit.") + } + return try { + DataInputStream(ByteArrayInputStream(encoded)).use { input -> + if (input.readInt() != MAGIC || input.readInt() != FORMAT_VERSION) { + corrupt(expectedId, "Persisted Elasticsearch lifecycle state has an unsupported format.") + } + val state = input.readState() + if (state.manifest.id != expectedId) { + corrupt(expectedId, "Persisted Elasticsearch lifecycle state belongs to another migration.") + } + if (input.available() != 0) { + corrupt(expectedId, "Persisted Elasticsearch lifecycle state contains trailing data.") + } + state + } + } catch (error: ElasticsearchIndexLifecycleException) { + throw error + } catch (error: EOFException) { + corrupt(expectedId, "Persisted Elasticsearch lifecycle state is truncated.", error) + } catch (error: IllegalArgumentException) { + corrupt(expectedId, "Persisted Elasticsearch lifecycle state is invalid.", error) + } catch (error: IOException) { + corrupt(expectedId, "Failed to decode persisted Elasticsearch lifecycle state.", error) + } + } + + private fun DataOutputStream.writeState(state: ElasticsearchIndexMigrationState) { + writeManifest(state.manifest) + writeEnum(state.phase) + writeLong(state.revision) + writeNullable(state.activeCommand) { writeActiveCommand(it) } + writeNullable(state.lastCompletedCommand) { writeCompletedCommand(it) } + writeNullable(state.inventory) { writeInventory(it) } + writeNullable(state.destinationAttestation) { writeAttestation(it) } + writeNullable(state.rebuildReceipt) { writeRebuildReceipt(it) } + writeNullable(state.destinationVerification) { writeVerification(it) } + writeNullable(state.cutover) { writeAliasTransition(it) } + writeNullable(state.rollbackVerification) { writeVerification(it) } + writeNullable(state.rollback) { writeAliasTransition(it) } + writeNullable(state.retainedSourceUntil) { writeInstantValue(it) } + } + + private fun DataInputStream.readState(): ElasticsearchIndexMigrationState = ElasticsearchIndexMigrationState( + manifest = readManifest(), + phase = readEnum(), + revision = readLong(), + activeCommand = readNullable { readActiveCommand() }, + lastCompletedCommand = readNullable { readCompletedCommand() }, + inventory = readNullable { readInventory() }, + destinationAttestation = readNullable { readAttestation() }, + rebuildReceipt = readNullable { readRebuildReceipt() }, + destinationVerification = readNullable { readVerification() }, + cutover = readNullable { readAliasTransition() }, + rollbackVerification = readNullable { readVerification() }, + rollback = readNullable { readAliasTransition() }, + retainedSourceUntil = readNullable { readInstantValue() }, + ) + + private fun DataOutputStream.writeManifest(manifest: ElasticsearchIndexMigrationManifest) { + writeStringValue(manifest.id.value) + writeStringValue(manifest.target.namedAggregate.contextName) + writeStringValue(manifest.target.namedAggregate.aggregateName) + writeEnum(manifest.target.documentKind) + writeInt(manifest.mappingVersion.value) + writeInt(manifest.generation.value) + writeStringValue(manifest.schemaContractId.value) + writeStringValue(manifest.capabilityDigest.value) + writeStringValue(manifest.sourcePhysicalIndex.value) + writeEnum(manifest.rebuildStrategy) + writeEnum(manifest.verificationContract.checksumAlgorithm) + writeStringValue(manifest.verificationContract.probeSuiteId.value) + writeDuration(manifest.maxCursorTtl) + writeDuration(manifest.rollbackWindow) + } + + private fun DataInputStream.readManifest(): ElasticsearchIndexMigrationManifest = + ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId(readStringValue()), + QueryTarget( + MaterializedNamedAggregate(readStringValue(), readStringValue()), + readEnum(), + ), + ElasticsearchIndexMappingVersion(readInt()), + ElasticsearchIndexGeneration(readInt()), + SchemaContractId(readStringValue()), + ElasticsearchIndexCapabilityDigest(readStringValue()), + ElasticsearchPhysicalIndex(readStringValue()), + readEnum(), + ElasticsearchIndexVerificationContract( + readEnum(), + ElasticsearchIndexProbeSuiteId(readStringValue()), + ), + readDuration(), + readDuration(), + ) + + private fun DataOutputStream.writeActiveCommand(command: ElasticsearchIndexActiveCommand) { + writeStringValue(command.id.value) + writeEnum(command.type) + writeEnum(command.from) + writeEnum(command.to) + writeInstantValue(command.startedAt) + } + + private fun DataInputStream.readActiveCommand() = ElasticsearchIndexActiveCommand( + ElasticsearchIndexLifecycleCommandId(readStringValue()), + readEnum(), + readEnum(), + readEnum(), + readInstantValue(), + ) + + private fun DataOutputStream.writeCompletedCommand(command: ElasticsearchIndexCompletedCommand) { + writeStringValue(command.id.value) + writeEnum(command.type) + writeInstantValue(command.completedAt) + } + + private fun DataInputStream.readCompletedCommand() = ElasticsearchIndexCompletedCommand( + ElasticsearchIndexLifecycleCommandId(readStringValue()), + readEnum(), + readInstantValue(), + ) + + private fun DataOutputStream.writeInventory(inventory: ElasticsearchIndexInventory) { + writeStringValue(inventory.alias.value) + writeNullable(inventory.aliasTarget) { value -> writeStringValue(value.value) } + writeCollectionSize(inventory.indices.size) + inventory.indices.forEach { (physical, attestation) -> + writeStringValue(physical.value) + writeAttestation(attestation) + } + writeInstantValue(inventory.observedAt) + } + + private fun DataInputStream.readInventory(): ElasticsearchIndexInventory { + val alias = ElasticsearchIndexAlias(readStringValue()) + val aliasTarget = readNullable { ElasticsearchPhysicalIndex(readStringValue()) } + val count = readCollectionSize() + val indices = LinkedHashMap(count) + repeat(count) { + val physical = ElasticsearchPhysicalIndex(readStringValue()) + require(indices.put(physical, readAttestation()) == null) { + "Persisted Elasticsearch inventory contains a duplicate index." + } + } + return ElasticsearchIndexInventory(alias, aliasTarget, indices, readInstantValue()) + } + + private fun DataOutputStream.writeAttestation(attestation: ElasticsearchIndexAttestation) { + writeStringValue(attestation.physicalIndex.value) + writeInt(attestation.mappingVersion.value) + writeEnum(attestation.documentKind) + writeStringValue(attestation.schemaContractId.value) + writeStringValue(attestation.capabilityDigest.value) + } + + private fun DataInputStream.readAttestation() = ElasticsearchIndexAttestation( + ElasticsearchPhysicalIndex(readStringValue()), + ElasticsearchIndexMappingVersion(readInt()), + readEnum(), + SchemaContractId(readStringValue()), + ElasticsearchIndexCapabilityDigest(readStringValue()), + ) + + private fun DataOutputStream.writeRebuildReceipt(receipt: ElasticsearchIndexRebuildReceipt) { + writeStringValue(receipt.physicalIndex.value) + writeEnum(receipt.strategy) + writeNullableLong(receipt.authoritativeWatermark) + writeNullableLong(receipt.indexedWatermark) + writeInstantValue(receipt.completedAt) + } + + private fun DataInputStream.readRebuildReceipt() = ElasticsearchIndexRebuildReceipt( + ElasticsearchPhysicalIndex(readStringValue()), + readEnum(), + readNullableLong(), + readNullableLong(), + readInstantValue(), + ) + + private fun DataOutputStream.writeVerification(verification: ElasticsearchIndexVerification) { + writeStringValue(verification.physicalIndex.value) + writeLong(verification.expectedCount) + writeLong(verification.actualCount) + writeStringValue(verification.expectedIdentityChecksum.value) + writeStringValue(verification.actualIdentityChecksum.value) + writeStringValue(verification.expectedContentChecksum.value) + writeStringValue(verification.actualContentChecksum.value) + writeBoolean(verification.versionContinuity) + writeNullableLong(verification.authoritativeWatermark) + writeNullableLong(verification.indexedWatermark) + writeLong(verification.recordProbeMismatchCount) + writeLong(verification.analyticsProbeMismatchCount) + writeInstantValue(verification.verifiedAt) + } + + private fun DataInputStream.readVerification() = ElasticsearchIndexVerification( + ElasticsearchPhysicalIndex(readStringValue()), + readLong(), + readLong(), + ElasticsearchIndexChecksum(readStringValue()), + ElasticsearchIndexChecksum(readStringValue()), + ElasticsearchIndexChecksum(readStringValue()), + ElasticsearchIndexChecksum(readStringValue()), + readBoolean(), + readNullableLong(), + readNullableLong(), + readLong(), + readLong(), + readInstantValue(), + ) + + private fun DataOutputStream.writeAliasTransition(transition: ElasticsearchAliasTransition) { + writeStringValue(transition.alias.value) + writeNullable(transition.previous) { value -> writeStringValue(value.value) } + writeStringValue(transition.current.value) + writeInstantValue(transition.transitionedAt) + } + + private fun DataInputStream.readAliasTransition() = ElasticsearchAliasTransition( + ElasticsearchIndexAlias(readStringValue()), + readNullable { ElasticsearchPhysicalIndex(readStringValue()) }, + ElasticsearchPhysicalIndex(readStringValue()), + readInstantValue(), + ) + + private fun DataOutputStream.writeDuration(duration: Duration) { + writeLong(duration.seconds) + writeInt(duration.nano) + } + + private fun DataInputStream.readDuration(): Duration = Duration.ofSeconds(readLong(), readInt().toLong()) + + private fun DataOutputStream.writeInstantValue(instant: Instant) { + writeLong(instant.epochSecond) + writeInt(instant.nano) + } + + private fun DataInputStream.readInstantValue(): Instant = Instant.ofEpochSecond(readLong(), readInt().toLong()) + + private fun DataOutputStream.writeStringValue(value: String) { + val bytes = value.toByteArray(Charsets.UTF_8) + require(bytes.size <= MAX_STRING_BYTES) { "Persisted Elasticsearch lifecycle string exceeds the limit." } + writeInt(bytes.size) + write(bytes) + } + + private fun DataInputStream.readStringValue(): String { + val length = readInt() + require(length in 0..MAX_STRING_BYTES) { "Persisted Elasticsearch lifecycle string length is invalid." } + val bytes = ByteArray(length).also(::readFully) + return Charsets.UTF_8.newDecoder() + .onMalformedInput(CodingErrorAction.REPORT) + .onUnmappableCharacter(CodingErrorAction.REPORT) + .decode(ByteBuffer.wrap(bytes)) + .toString() + } + + private fun DataOutputStream.writeCollectionSize(size: Int) { + require(size in 0..MAX_COLLECTION_SIZE) { "Persisted Elasticsearch lifecycle collection is too large." } + writeInt(size) + } + + private fun DataInputStream.readCollectionSize(): Int = readInt().also { size -> + require(size in 0..MAX_COLLECTION_SIZE) { "Persisted Elasticsearch lifecycle collection size is invalid." } + } + + private inline fun DataOutputStream.writeNullable(value: T?, writer: DataOutputStream.(T) -> Unit) { + writeBoolean(value != null) + if (value != null) writer(value) + } + + private inline fun DataInputStream.readNullable(reader: DataInputStream.() -> T): T? = + if (readBoolean()) reader() else null + + private fun DataOutputStream.writeNullableLong(value: Long?) = + writeNullable(value) { nonNull -> writeLong(nonNull) } + + private fun DataInputStream.readNullableLong(): Long? = readNullable { readLong() } + + private fun DataOutputStream.writeEnum(value: Enum<*>) = writeStringValue(value.name) + + private inline fun > DataInputStream.readEnum(): T = enumValueOf(readStringValue()) + + internal companion object { + const val MAX_PAYLOAD_BYTES = 1_048_576 + private const val MAX_STRING_BYTES = 65_536 + private const val MAX_COLLECTION_SIZE = 4_096 + private const val FORMAT_VERSION = 2 + private const val MAGIC = 0x57514C53 + } +} + +private fun corrupt(id: ElasticsearchIndexMigrationId, message: String): Nothing = + throw corrupted(id, message, null) + +private fun corrupt( + id: ElasticsearchIndexMigrationId, + message: String, + cause: Throwable, +): Nothing = throw corrupted(id, message, cause) + +private fun corrupted( + id: ElasticsearchIndexMigrationId, + message: String, + cause: Throwable?, +) = ElasticsearchIndexLifecycleException( + ElasticsearchIndexLifecycleErrorCode.REPOSITORY_CORRUPTED, + id, + message, + cause, +) diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStoreElasticsearchEventStreamIndexRebuilder.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStoreElasticsearchEventStreamIndexRebuilder.kt new file mode 100644 index 00000000000..43624e09fa4 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStoreElasticsearchEventStreamIndexRebuilder.kt @@ -0,0 +1,275 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import co.elastic.clients.elasticsearch._types.Refresh +import co.elastic.clients.elasticsearch.core.IndexRequest +import me.ahoo.wow.api.Version +import me.ahoo.wow.api.event.DEFAULT_EVENT_SEQUENCE +import me.ahoo.wow.event.DomainEventStream +import me.ahoo.wow.eventsourcing.AggregateIdScanner +import me.ahoo.wow.eventsourcing.EventStore +import me.ahoo.wow.modeling.materialize +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.toLinkedHashMap +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Clock + +/** + * Reads a bounded page of complete aggregate event histories after [afterAggregateId]. + * Implementations must emit aggregates in id order and each aggregate's streams in version order. + */ +internal fun interface ElasticsearchAuthoritativeEventStreamSource { + fun scan( + target: QueryTarget, + afterAggregateId: String, + aggregateLimit: Int, + ): Flux +} + +internal class EventStoreAuthoritativeEventStreamSource( + private val eventStore: EventStore, +) : ElasticsearchAuthoritativeEventStreamSource { + override fun scan( + target: QueryTarget, + afterAggregateId: String, + aggregateLimit: Int, + ): Flux { + require(target.documentKind == QueryDocumentKind.EVENT_STREAM) { + "Authoritative EventStream scan requires an EventStream query target." + } + require(aggregateLimit > 0) { "Authoritative EventStream aggregate limit must be positive." } + return eventStore.scanAggregateId(target.namedAggregate, afterAggregateId, aggregateLimit) + .concatMap { aggregateId -> eventStore.load(aggregateId) } + } +} + +internal fun interface ElasticsearchPhysicalEventStreamWriter { + fun write(index: ElasticsearchPhysicalIndex, eventStream: DomainEventStream): Mono +} + +/** Idempotently replaces one immutable authoritative event-stream document in an exact generation. */ +internal class ReactiveElasticsearchPhysicalEventStreamWriter( + private val client: ReactiveElasticsearchClient, + private val refresh: Refresh = Refresh.WaitFor, +) : ElasticsearchPhysicalEventStreamWriter { + override fun write( + index: ElasticsearchPhysicalIndex, + eventStream: DomainEventStream, + ): Mono = Mono.defer { + val id = eventStream.eventStreamDocumentId() + val request = IndexRequest.of> { builder -> + builder.index(index.value) + .id(id) + .routing(eventStream.aggregateId.id) + .refresh(refresh) + .document(eventStream.toLinkedHashMap()) + } + client.index(request) + .switchIfEmpty(Mono.error(IllegalStateException("EventStream rebuild write returned no response."))) + .flatMap { response -> + if (response.index() != index.value || response.id() != id || response.shards().failed() != 0) { + Mono.error(IllegalStateException("EventStream rebuild write response is incomplete.")) + } else { + Mono.empty() + } + } + } +} + +/** + * Returns a global accepted-write watermark only while the target is externally paused and fully drained. + * The same watermark before and after a rebuild proves that the authority did not advance during the copy. + */ +internal fun interface ElasticsearchEventStreamMigrationBarrier { + fun checkpoint( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono +} + +internal data class ElasticsearchEventStreamRebuildOptions( + val aggregateScanPageSize: Int = 256, + val writeConcurrency: Int = 8, +) { + init { + require(aggregateScanPageSize > 0) { "EventStream rebuild aggregate page size must be positive." } + require(writeConcurrency > 0) { "EventStream rebuild write concurrency must be positive." } + } +} + +/** Rebuilds a drained EventStream target from the authoritative EventStore into one exact generation. */ +internal class EventStoreElasticsearchEventStreamIndexRebuilder( + private val source: ElasticsearchAuthoritativeEventStreamSource, + private val writer: ElasticsearchPhysicalEventStreamWriter, + private val barrier: ElasticsearchEventStreamMigrationBarrier, + private val clock: Clock, + private val options: ElasticsearchEventStreamRebuildOptions = ElasticsearchEventStreamRebuildOptions(), +) : ElasticsearchAuthoritativeIndexRebuilder { + override fun rebuild( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = Mono.defer { + requirePauseAndDrainManifest(manifest) + checkpoint(command, manifest).flatMap { before -> + scanOrderedAuthoritativeEventStreams(source, manifest.target, options.aggregateScanPageSize) + .flatMapSequential( + { eventStream -> writer.write(manifest.names.physical, eventStream) }, + options.writeConcurrency, + 1, + ).then(checkpoint(command, manifest)) + .flatMap { after -> + if (before != after) { + eventStreamLifecycleError( + ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED, + manifest.id, + "EventStream authority advanced while its migration write fence was held.", + ) + } else { + Mono.just( + ElasticsearchIndexRebuildReceipt( + manifest.names.physical, + manifest.rebuildStrategy, + before, + after, + clock.instant(), + ), + ) + } + } + } + } + + private fun checkpoint( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = Mono.defer { barrier.checkpoint(command, manifest) } + .switchIfEmpty( + eventStreamLifecycleError( + ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED, + manifest.id, + "EventStream migration write fence returned no watermark.", + ), + ).flatMap { watermark -> + if (watermark < 0) { + eventStreamLifecycleError( + ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED, + manifest.id, + "EventStream migration write fence returned a negative watermark.", + ) + } else { + Mono.just(watermark) + } + } + + private fun requirePauseAndDrainManifest(manifest: ElasticsearchIndexMigrationManifest) { + if ( + manifest.target.documentKind != QueryDocumentKind.EVENT_STREAM || + manifest.rebuildStrategy != ElasticsearchIndexRebuildStrategy.EVENT_STREAM_PAUSE_AND_DRAIN + ) { + reject( + ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED, + manifest.id, + "EventStore EventStream rebuild requires the pause-and-drain strategy.", + ) + } + } +} + +internal fun scanOrderedAuthoritativeEventStreams( + source: ElasticsearchAuthoritativeEventStreamSource, + target: QueryTarget, + aggregatePageSize: Int, +): Flux { + require(target.documentKind == QueryDocumentKind.EVENT_STREAM) { + "Authoritative EventStream scan requires an EventStream query target." + } + require(aggregatePageSize > 0) { "Authoritative EventStream aggregate page size must be positive." } + + fun scanPage(afterAggregateId: String): Flux = Flux.defer { + val validator = AuthoritativeEventStreamPageValidator(target, afterAggregateId, aggregatePageSize) + source.scan(target, afterAggregateId, aggregatePageSize) + .doOnNext(validator::accept) + .concatWith( + Flux.defer { + if (validator.aggregateCount == aggregatePageSize) { + scanPage(validator.lastAggregateId) + } else { + Flux.empty() + } + }, + ) + } + return scanPage(AggregateIdScanner.FIRST_ID) +} + +private class AuthoritativeEventStreamPageValidator( + private val target: QueryTarget, + private val afterAggregateId: String, + private val aggregateLimit: Int, +) { + var aggregateCount: Int = 0 + private set + var lastAggregateId: String = afterAggregateId + private set + private var currentAggregateId: String? = null + private var expectedVersion: Int = Version.INITIAL_VERSION + + fun accept(eventStream: DomainEventStream) { + val aggregateId = eventStream.aggregateId + check(aggregateId.namedAggregate.materialize() == target.namedAggregate) { + "Authoritative EventStream source returned another Aggregate." + } + if (aggregateId.id != currentAggregateId) { + check(aggregateId.id > lastAggregateId && ++aggregateCount <= aggregateLimit) { + "Authoritative EventStream source violated its aggregate page contract." + } + currentAggregateId = aggregateId.id + lastAggregateId = aggregateId.id + expectedVersion = Version.INITIAL_VERSION + } + check(eventStream.version == expectedVersion && eventStream.size > 0) { + "Authoritative EventStream source violated aggregate version continuity." + } + check(eventStream.body.size == eventStream.size) { + "Authoritative EventStream source returned an inconsistent event count." + } + eventStream.body.forEachIndexed { index, event -> + val expectedSequence = Math.addExact(DEFAULT_EVENT_SEQUENCE, index) + val expectedLast = index == eventStream.body.lastIndex + check( + event.aggregateId == aggregateId && + event.version == eventStream.version && + event.sequence == expectedSequence && + event.isLast == expectedLast + ) { + "Authoritative EventStream source returned an inconsistent event body." + } + } + expectedVersion = Math.addExact(eventStream.version, eventStream.size) + } +} + +internal fun DomainEventStream.eventStreamDocumentId(): String = "${aggregateId.id}-$version" + +private fun eventStreamLifecycleError( + code: ElasticsearchIndexLifecycleErrorCode, + id: ElasticsearchIndexMigrationId, + message: String, +): Mono = Mono.error(ElasticsearchIndexLifecycleException(code, id, message)) diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStoreElasticsearchSnapshotIndexRebuilder.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStoreElasticsearchSnapshotIndexRebuilder.kt new file mode 100644 index 00000000000..294e2d557d9 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStoreElasticsearchSnapshotIndexRebuilder.kt @@ -0,0 +1,207 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import co.elastic.clients.elasticsearch._types.Refresh +import me.ahoo.wow.api.modeling.AggregateId +import me.ahoo.wow.elasticsearch.eventsourcing.ElasticsearchSnapshotVersionGuardedWriter +import me.ahoo.wow.elasticsearch.eventsourcing.toElasticsearchSnapshotWrite +import me.ahoo.wow.eventsourcing.AggregateIdScanner +import me.ahoo.wow.eventsourcing.EventStore +import me.ahoo.wow.eventsourcing.EventStoreStateAggregateRepository +import me.ahoo.wow.eventsourcing.snapshot.SimpleSnapshot +import me.ahoo.wow.eventsourcing.snapshot.Snapshot +import me.ahoo.wow.modeling.materialize +import me.ahoo.wow.modeling.state.ConstructorStateAggregateFactory +import me.ahoo.wow.modeling.state.ReadOnlyStateAggregate +import me.ahoo.wow.modeling.state.StateAggregateRepository +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Clock + +internal fun interface ElasticsearchAuthoritativeSnapshotSource { + fun scan( + target: QueryTarget, + afterId: String, + limit: Int, + ): Flux> +} + +internal class EventStoreAuthoritativeSnapshotSource( + private val eventStore: EventStore, + private val repository: StateAggregateRepository = EventStoreStateAggregateRepository( + ConstructorStateAggregateFactory, + eventStore, + ), +) : ElasticsearchAuthoritativeSnapshotSource { + override fun scan( + target: QueryTarget, + afterId: String, + limit: Int, + ): Flux> { + require(target.documentKind == QueryDocumentKind.SNAPSHOT) { + "Authoritative state replay requires a Snapshot query target." + } + require(limit > 0) { "Authoritative Snapshot scan limit must be positive." } + return eventStore.scanAggregateId(target.namedAggregate, afterId, limit) + .concatMap { aggregateId -> + if (aggregateId.namedAggregate.materialize() != target.namedAggregate) { + Mono.error(IllegalStateException("Authoritative EventStore scan returned another Aggregate.")) + } else { + loadLatest(aggregateId) + } + } + } + + private fun loadLatest(aggregateId: AggregateId): Mono> = + repository.load(aggregateId).flatMap> { aggregate -> + if (aggregate.aggregateId != aggregateId || aggregate.version < 0) { + Mono.error(IllegalStateException("Authoritative event replay returned an invalid aggregate.")) + } else { + Mono.just>(aggregate) + } + }.switchIfEmpty( + Mono.error(IllegalStateException("Authoritative event replay returned no aggregate.")), + ) +} + +internal fun interface ElasticsearchPhysicalSnapshotWriter { + fun write(index: ElasticsearchPhysicalIndex, snapshot: Snapshot<*>): Mono +} + +internal class ReactiveElasticsearchPhysicalSnapshotWriter( + client: ReactiveElasticsearchClient, + refresh: Refresh = Refresh.WaitFor, +) : ElasticsearchPhysicalSnapshotWriter { + private val writer = ElasticsearchSnapshotVersionGuardedWriter(client, refresh) + + override fun write(index: ElasticsearchPhysicalIndex, snapshot: Snapshot<*>): Mono = + writer.write(snapshot.toElasticsearchSnapshotWrite(index.value)) +} + +internal data class ElasticsearchSnapshotRebuildOptions( + val scanPageSize: Int = 256, + val writeConcurrency: Int = 8, +) { + init { + require(scanPageSize > 0) { "Snapshot rebuild scan page size must be positive." } + require(writeConcurrency > 0) { "Snapshot rebuild write concurrency must be positive." } + } +} + +/** Rebuilds one Snapshot generation exclusively from the authoritative EventStore. */ +internal class EventStoreElasticsearchSnapshotIndexRebuilder( + private val source: ElasticsearchAuthoritativeSnapshotSource, + private val writer: ElasticsearchPhysicalSnapshotWriter, + private val clock: Clock, + private val options: ElasticsearchSnapshotRebuildOptions = ElasticsearchSnapshotRebuildOptions(), +) : ElasticsearchAuthoritativeIndexRebuilder { + override fun rebuild( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = Mono.defer { + requireSnapshotManifest(manifest) + scanOrderedAuthoritativeSnapshots(source, manifest.target, options.scanPageSize) + .flatMapSequential( + { aggregate -> write(manifest, aggregate) }, + options.writeConcurrency, + 1, + ).count() + .map { + ElasticsearchIndexRebuildReceipt( + manifest.names.physical, + manifest.rebuildStrategy, + authoritativeWatermark = null, + indexedWatermark = null, + completedAt = clock.instant(), + ) + } + } + + private fun write( + manifest: ElasticsearchIndexMigrationManifest, + aggregate: ReadOnlyStateAggregate<*>, + ): Mono = writeSnapshot(manifest, aggregate) + + private fun writeSnapshot( + manifest: ElasticsearchIndexMigrationManifest, + aggregate: ReadOnlyStateAggregate, + ): Mono = writer.write( + manifest.names.physical, + SimpleSnapshot(aggregate, clock.millis()), + ) + + private fun requireSnapshotManifest(manifest: ElasticsearchIndexMigrationManifest) { + if ( + manifest.target.documentKind != QueryDocumentKind.SNAPSHOT || + manifest.rebuildStrategy != ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM + ) { + reject( + ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED, + manifest.id, + "EventStore Snapshot rebuild received an incompatible migration strategy.", + ) + } + } +} + +internal fun scanOrderedAuthoritativeSnapshots( + source: ElasticsearchAuthoritativeSnapshotSource, + target: QueryTarget, + pageSize: Int, +): Flux> { + require(pageSize > 0) { "Authoritative Snapshot scan page size must be positive." } + fun scanPage(afterId: String): Flux> = Flux.defer { + source.scan(target, afterId, pageSize) + .take(pageSize.toLong() + 1) + .collectList() + .flatMapMany { page -> + validateAuthoritativeSnapshotPage(target, afterId, page, pageSize) + val current = Flux.fromIterable(page) + if (page.size < pageSize) { + current + } else { + current.concatWith(scanPage(page.last().aggregateId.id)) + } + } + } + return scanPage(AggregateIdScanner.FIRST_ID) +} + +private fun validateAuthoritativeSnapshotPage( + target: QueryTarget, + afterId: String, + page: List>, + pageSize: Int, +) { + if (page.size > pageSize) { + throw IllegalStateException("Authoritative Snapshot source exceeded the requested page size.") + } + var previous = afterId + page.forEach { aggregate -> + if ( + aggregate.aggregateId.namedAggregate.materialize() != target.namedAggregate || + aggregate.aggregateId.id <= previous || + aggregate.version < 0 + ) { + throw IllegalStateException("Authoritative Snapshot source violated its ordered page contract.") + } + previous = aggregate.aggregateId.id + } +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStreamCanonicalChecksum.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStreamCanonicalChecksum.kt new file mode 100644 index 00000000000..7ba175277fd --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStreamCanonicalChecksum.kt @@ -0,0 +1,118 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.wow.api.Version +import me.ahoo.wow.serialization.MessageRecords +import java.io.ByteArrayOutputStream +import java.io.DataOutputStream +import java.math.BigDecimal +import java.math.BigInteger + +internal data class EventStreamCanonicalChecksumEvidence( + val count: Long, + val identityChecksum: ElasticsearchIndexChecksum, + val contentChecksum: ElasticsearchIndexChecksum, +) + +/** Canonical checksum over event-stream documents ordered by aggregate id and aggregate version. */ +internal class EventStreamCanonicalChecksumAccumulator( + private val limits: SnapshotCanonicalChecksumLimits = SnapshotCanonicalChecksumLimits(), +) { + private val identityDigest = newLifecycleSha256().apply { updateLifecycleUtf8(IDENTITY_HEADER) } + private val contentDigest = newLifecycleSha256().apply { updateLifecycleUtf8(CONTENT_HEADER) } + private var previousAggregateId: String? = null + private var expectedVersion: Int = Version.INITIAL_VERSION + private var count = 0L + + fun accept(documentId: String, source: Map<*, *>) { + val aggregateId = source[MessageRecords.AGGREGATE_ID] as? String + ?: throw IllegalArgumentException("EventStream verification document has no aggregate id.") + require(aggregateId.isNotBlank()) { "EventStream verification aggregate id must not be blank." } + val version = source[MessageRecords.VERSION].requireExactEventStreamVersion() + val bodySize = source[MessageRecords.BODY].requireEventStreamBodySize() + require(documentId == "$aggregateId-$version") { + "EventStream verification identity does not match its serialized aggregate and version." + } + if (aggregateId != previousAggregateId) { + require(previousAggregateId == null || aggregateId > requireNotNull(previousAggregateId)) { + "EventStream verification aggregates must be unique and strictly ascending." + } + require(version == Version.INITIAL_VERSION) { + "EventStream verification aggregate does not start at the initial version." + } + previousAggregateId = aggregateId + expectedVersion = Version.INITIAL_VERSION + } + require(version == expectedVersion) { "EventStream verification aggregate version is not continuous." } + expectedVersion = Math.addExact(version, bodySize) + + val identity = eventStreamIdentityBytes(aggregateId, version) + val documentHash = canonicalDocumentHash(source, limits) + identityDigest.updateLifecycleLengthPrefixed(identity) + contentDigest.updateLifecycleLengthPrefixed(identity) + contentDigest.updateLifecycleLengthPrefixed(documentHash) + count = Math.addExact(count, 1) + } + + fun finish(): EventStreamCanonicalChecksumEvidence = EventStreamCanonicalChecksumEvidence( + count, + ElasticsearchIndexChecksum(identityDigest.digest().toLowerHex()), + ElasticsearchIndexChecksum(contentDigest.digest().toLowerHex()), + ) + + private companion object { + const val IDENTITY_HEADER = "wow-es-event-stream-identity-sha256-v1" + const val CONTENT_HEADER = "wow-es-event-stream-content-sha256-v1" + } +} + +private fun eventStreamIdentityBytes(aggregateId: String, version: Int): ByteArray { + val buffer = ByteArrayOutputStream() + DataOutputStream(buffer).use { output -> + val aggregateBytes = aggregateId.toByteArray(Charsets.UTF_8) + output.writeInt(aggregateBytes.size) + output.write(aggregateBytes) + output.writeInt(version) + } + return buffer.toByteArray() +} + +internal fun Any?.requireExactEventStreamVersion(): Int { + val value = try { + when (this) { + is BigDecimal -> toBigIntegerExact() + is BigInteger -> this + is Byte, is Short, is Int, is Long -> BigInteger.valueOf((this as Number).toLong()) + is Float, is Double -> BigDecimal.valueOf((this as Number).toDouble()).toBigIntegerExact() + else -> throw IllegalArgumentException("EventStream verification document has no numeric version.") + } + } catch (error: ArithmeticException) { + throw IllegalArgumentException("EventStream verification version must be an exact integer.", error) + } + require(value >= BigInteger.valueOf(Version.INITIAL_VERSION.toLong()) && value.bitLength() <= Int.SIZE_BITS - 1) { + "EventStream verification version is outside the supported range." + } + return value.toInt() +} + +private fun Any?.requireEventStreamBodySize(): Int { + val size = when (this) { + is Collection<*> -> size + is Array<*> -> size + else -> throw IllegalArgumentException("EventStream verification document body must be a materialized list.") + } + require(size > 0) { "EventStream verification document body must not be empty." } + return size +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStreamElasticsearchIndexVerificationSources.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStreamElasticsearchIndexVerificationSources.kt new file mode 100644 index 00000000000..34064bb5255 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStreamElasticsearchIndexVerificationSources.kt @@ -0,0 +1,315 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import co.elastic.clients.elasticsearch._types.FieldValue +import co.elastic.clients.elasticsearch._types.SortOrder +import co.elastic.clients.elasticsearch._types.query_dsl.Query +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.bool +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.range +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.term +import co.elastic.clients.elasticsearch.core.ClosePointInTimeRequest +import co.elastic.clients.elasticsearch.core.OpenPointInTimeRequest +import co.elastic.clients.elasticsearch.core.SearchRequest +import co.elastic.clients.elasticsearch.core.search.ResponseBody +import co.elastic.clients.elasticsearch.core.search.TotalHitsRelation +import co.elastic.clients.json.JsonData +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.serialization.toLinkedHashMap +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Mono + +internal fun interface ElasticsearchEventStreamIndexedWatermarkSource { + fun checkpoint( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono +} + +internal data class EventStreamElasticsearchIndexVerificationOptions( + val aggregateScanPageSize: Int = 256, + val physicalPageSize: Int = 1_000, + val pitKeepAlive: String = "1m", + val checksumLimits: SnapshotCanonicalChecksumLimits = SnapshotCanonicalChecksumLimits(), +) { + init { + require(aggregateScanPageSize > 0) { "EventStream authority page size must be positive." } + require(physicalPageSize > 0) { "EventStream physical page size must be positive." } + require(pitKeepAlive.isNotBlank()) { "EventStream verification PIT keep-alive must not be blank." } + } +} + +internal class EventStoreEventStreamVerificationSource( + private val source: ElasticsearchAuthoritativeEventStreamSource, + private val barrier: ElasticsearchEventStreamMigrationBarrier, + private val options: EventStreamElasticsearchIndexVerificationOptions = + EventStreamElasticsearchIndexVerificationOptions(), +) : ElasticsearchAuthoritativeVerificationSource { + override fun capture( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = Mono.defer { + requireEventStreamVerificationContract(manifest) + checkpoint(command, manifest).flatMap { before -> + val accumulator = EventStreamCanonicalChecksumAccumulator(options.checksumLimits) + scanOrderedAuthoritativeEventStreams(source, manifest.target, options.aggregateScanPageSize) + .doOnNext { eventStream -> + accumulator.accept(eventStream.eventStreamDocumentId(), eventStream.toLinkedHashMap()) + }.then(checkpoint(command, manifest)) + .map { after -> + check(before == after) { "EventStream authority advanced during verification." } + val evidence = accumulator.finish() + ElasticsearchAuthoritativeVerificationSnapshot( + manifest.verificationContract.checksumAlgorithm, + evidence.count, + evidence.identityChecksum, + evidence.contentChecksum, + before, + ) + } + } + }.mapEventStreamVerificationErrors(manifest) + + private fun checkpoint( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = Mono.defer { barrier.checkpoint(command, manifest) } + .switchIfEmpty(Mono.error(IllegalStateException("EventStream authority returned no watermark."))) + .map { watermark -> + require(watermark >= 0) { "EventStream authority watermark must not be negative." } + watermark + } +} + +internal class ReactiveElasticsearchEventStreamVerificationSource( + private val client: ReactiveElasticsearchClient, + private val indexedWatermarks: ElasticsearchEventStreamIndexedWatermarkSource, + private val options: EventStreamElasticsearchIndexVerificationOptions = + EventStreamElasticsearchIndexVerificationOptions(), +) : ElasticsearchPhysicalIndexVerificationSource { + override fun inspect( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono = Mono.defer { + requireEventStreamVerificationContract(manifest) + checkpoint(command, manifest, physicalIndex).flatMap { before -> + val accumulator = EventStreamCanonicalChecksumAccumulator(options.checksumLimits) + Mono.usingWhen( + openPit(physicalIndex), + { lease -> inspectPages(lease, accumulator) }, + ::closePit, + { lease, error -> closeAfterError(lease, error) }, + ::closeAfterCancel, + ).flatMap { total -> + checkpoint(command, manifest, physicalIndex).map { after -> + check(before == after) { "EventStream indexed watermark advanced during verification." } + val evidence = accumulator.finish() + check(evidence.count == total) { "EventStream physical PIT total changed during verification." } + ElasticsearchPhysicalIndexVerificationSnapshot( + manifest.verificationContract.checksumAlgorithm, + physicalIndex, + evidence.count, + evidence.identityChecksum, + evidence.contentChecksum, + versionContinuity = true, + watermark = before, + ) + } + } + } + }.mapEventStreamVerificationErrors(manifest) + + private fun checkpoint( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono = Mono.defer { indexedWatermarks.checkpoint(command, manifest, physicalIndex) } + .switchIfEmpty(Mono.error(IllegalStateException("EventStream destination returned no watermark."))) + .map { watermark -> + require(watermark >= 0) { "EventStream destination watermark must not be negative." } + watermark + } + + private fun openPit(physicalIndex: ElasticsearchPhysicalIndex): Mono = + client.openPointInTime( + OpenPointInTimeRequest.of { request -> + request.index(physicalIndex.value).keepAlive { keepAlive -> keepAlive.time(options.pitKeepAlive) } + }, + ).switchIfEmpty(Mono.error(IllegalStateException("EventStream verification PIT was not opened."))) + .map { response -> + if (response.id().isBlank() || response.shards().failed() != 0) { + throw IllegalStateException("EventStream verification PIT is incomplete.") + } + EventStreamPitLease(response.id()) + } + + private fun inspectPages( + lease: EventStreamPitLease, + accumulator: EventStreamCanonicalChecksumAccumulator, + ): Mono { + fun inspectPage(after: EventStreamVerificationCursor?, rootTotal: Long?, consumed: Long): Mono = + client.search(searchRequest(lease, after), Map::class.java) + .switchIfEmpty( + Mono.error(IllegalStateException("EventStream verification search returned no response.")), + ) + .flatMap { response -> + val currentTotal = validateResponse(response) + val exactRootTotal = rootTotal ?: currentTotal + check(currentTotal == exactRootTotal - consumed) { + "EventStream verification keyset total is not stable and exact." + } + response.pitId()?.takeIf(String::isNotBlank)?.let(lease::update) + val hits = response.hits().hits() + var nextCursor: EventStreamVerificationCursor? = null + hits.forEach { hit -> + if (hit.ignored().isNotEmpty()) { + throw IllegalStateException("EventStream verification hit contains ignored fields.") + } + val identity = hit.id() + ?: throw IllegalStateException("EventStream verification hit has no identity.") + val document = hit.source() + ?: throw IllegalStateException("EventStream verification hit has no source.") + accumulator.accept(identity, document) + nextCursor = document.toEventStreamCursor() + } + val nextConsumed = Math.addExact(consumed, hits.size.toLong()) + if (hits.size < options.physicalPageSize) { + Mono.just(exactRootTotal) + } else { + inspectPage(checkNotNull(nextCursor), exactRootTotal, nextConsumed) + } + } + return inspectPage(null, null, 0) + } + + private fun searchRequest( + lease: EventStreamPitLease, + after: EventStreamVerificationCursor?, + ): SearchRequest = SearchRequest.of { request -> + request.pit { pit -> pit.id(lease.id).keepAlive { keepAlive -> keepAlive.time(options.pitKeepAlive) } } + .size(options.physicalPageSize) + .allowPartialSearchResults(false) + .trackTotalHits { total -> total.enabled(true) } + .sort { sort -> + sort.field { field -> field.field(MessageRecords.AGGREGATE_ID).order(SortOrder.Asc) } + }.sort { sort -> + sort.field { field -> field.field(MessageRecords.VERSION).order(SortOrder.Asc) } + }.also { builder -> after?.let { cursor -> builder.query(cursor.afterQuery()) } } + } + + private fun validateResponse(response: ResponseBody<*>): Long { + check(!response.timedOut() && response.shards().failed() == 0) { + "EventStream verification search is incomplete." + } + val total = checkNotNull(response.hits().total()) { "EventStream verification search has no exact total." } + check(total.relation() == TotalHitsRelation.Eq) { "EventStream verification search total is not exact." } + return total.value() + } + + private fun closePit(lease: EventStreamPitLease): Mono = client.closePointInTime( + ClosePointInTimeRequest.of { request -> request.id(lease.id) }, + ).switchIfEmpty(Mono.error(IllegalStateException("EventStream verification PIT close returned no response."))) + .flatMap { response -> + if (!response.succeeded()) { + Mono.error(IllegalStateException("EventStream verification PIT was not closed.")) + } else { + Mono.empty() + } + } + + private fun closeAfterError(lease: EventStreamPitLease, original: Throwable): Mono = + closePit(lease).onErrorResume { closeError -> + original.addSuppressed(closeError) + Mono.empty() + } + + private fun closeAfterCancel(lease: EventStreamPitLease): Mono = closePit( + lease, + ).onErrorResume { Mono.empty() } +} + +private data class EventStreamVerificationCursor(val aggregateId: String, val version: Int) { + fun afterQuery(): Query = bool { outer -> + outer.should( + range { query -> + query.untyped { field -> + field.field(MessageRecords.AGGREGATE_ID).gt(JsonData.of(aggregateId)) + } + }, + bool { sameAggregate -> + sameAggregate.filter( + term { query -> + query.field(MessageRecords.AGGREGATE_ID).value(FieldValue.of(aggregateId)) + }, + range { query -> + query.untyped { field -> + field.field(MessageRecords.VERSION).gt(JsonData.of(version)) + } + }, + ) + }, + ).minimumShouldMatch("1") + } +} + +private fun Map<*, *>.toEventStreamCursor(): EventStreamVerificationCursor { + val aggregateId = this[MessageRecords.AGGREGATE_ID] as? String + ?: throw IllegalStateException("EventStream verification result has no aggregate id.") + val version = try { + this[MessageRecords.VERSION].requireExactEventStreamVersion() + } catch (error: IllegalArgumentException) { + throw IllegalStateException("EventStream verification result has no exact version.", error) + } + return EventStreamVerificationCursor(aggregateId, version) +} + +private class EventStreamPitLease(initialId: String) { + var id: String = initialId + private set + + fun update(nextId: String) { + id = nextId + } +} + +private fun requireEventStreamVerificationContract(manifest: ElasticsearchIndexMigrationManifest) { + require(manifest.target.documentKind == me.ahoo.wow.query.gateway.QueryDocumentKind.EVENT_STREAM) { + "Canonical EventStream verification requires an EventStream target." + } + require( + manifest.verificationContract.checksumAlgorithm == + ElasticsearchIndexChecksumAlgorithm.CANONICAL_EVENT_STREAM_SHA256_V1, + ) { + "Canonical EventStream verification received an unsupported checksum algorithm." + } +} + +private fun Mono.mapEventStreamVerificationErrors( + manifest: ElasticsearchIndexMigrationManifest, +): Mono = onErrorMap { error -> + if (error is ElasticsearchIndexLifecycleException) { + error + } else { + ElasticsearchIndexLifecycleException( + ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED, + manifest.id, + "Elasticsearch EventStream verification evidence could not be computed.", + error, + ) + } +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ReactiveElasticsearchIndexLifecycleRepository.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ReactiveElasticsearchIndexLifecycleRepository.kt new file mode 100644 index 00000000000..480cc428bb8 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ReactiveElasticsearchIndexLifecycleRepository.kt @@ -0,0 +1,325 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import co.elastic.clients.elasticsearch._types.ElasticsearchException +import co.elastic.clients.elasticsearch._types.OpType +import co.elastic.clients.elasticsearch._types.Refresh +import co.elastic.clients.elasticsearch._types.mapping.DynamicMapping +import co.elastic.clients.elasticsearch._types.mapping.TypeMapping +import co.elastic.clients.elasticsearch.core.GetRequest +import co.elastic.clients.elasticsearch.core.GetResponse +import co.elastic.clients.elasticsearch.core.IndexRequest +import co.elastic.clients.elasticsearch.core.IndexResponse +import co.elastic.clients.elasticsearch.indices.CreateIndexRequest +import co.elastic.clients.elasticsearch.indices.CreateIndexResponse +import co.elastic.clients.elasticsearch.indices.GetMappingRequest +import co.elastic.clients.elasticsearch.indices.IndexSettings +import co.elastic.clients.elasticsearch.indices.get_mapping.IndexMappingRecord +import co.elastic.clients.json.JsonData +import org.springframework.data.elasticsearch.RestStatusException +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Mono +import java.math.BigDecimal +import java.math.BigInteger +import java.util.Base64 + +/** Durable state repository. Creation of its system index is deliberately an explicit management operation. */ +internal class ReactiveElasticsearchIndexLifecycleRepository( + private val client: ReactiveElasticsearchClient, + private val indexName: String = DEFAULT_INDEX_NAME, + private val codec: ElasticsearchIndexLifecycleStateCodec = ElasticsearchIndexLifecycleStateCodec(), +) : ElasticsearchIndexLifecycleRepository { + init { + require(indexName.matches(SYSTEM_INDEX_PATTERN)) { "Elasticsearch lifecycle system index name is invalid." } + } + + fun ensureIndex(): Mono = Mono.defer { + client.indices().create(systemIndexRequest()) + .flatMap { response -> + if (!response.isExpected(indexName)) { + Mono.error( + IllegalStateException("Elasticsearch lifecycle system index was not fully acknowledged."), + ) + } else { + Mono.just(true) + } + } + .switchIfEmpty( + Mono.error(IllegalStateException("Elasticsearch lifecycle system index returned no response.")), + ) + .onErrorResume(::isAlreadyExists) { Mono.just(false) } + .then(Mono.defer(::validateSystemIndex)) + } + + override fun create(state: ElasticsearchIndexMigrationState): Mono = + Mono.defer { + client.index(createRequest(state)) + .map { response -> response.stored(state) } + .onErrorResume(::isConflict) { Mono.empty() } + } + + override fun load(id: ElasticsearchIndexMigrationId): Mono = + Mono.defer { + client.get( + GetRequest.of { request -> request.index(indexName).id(id.value) }, + Map::class.java, + ) + .flatMap { response -> if (response.found()) Mono.just(response.decode(id)) else Mono.empty() } + .onErrorResume(::isNotFound) { Mono.empty() } + } + + override fun compareAndSet( + expected: ElasticsearchIndexLifecycleStoredState, + state: ElasticsearchIndexMigrationState, + ): Mono = Mono.defer { + require(expected.state.manifest.id == state.manifest.id) { + "Elasticsearch migration state id must match repository key." + } + val version = expected.version as? ElasticsearchIndexLifecycleRepositoryVersion.Elasticsearch + ?: error("Elasticsearch repository requires an Elasticsearch storage version.") + client.index(compareAndSetRequest(state, version)) + .map { response -> response.stored(state) } + .onErrorResume(::isConflict) { Mono.empty() } + } + + private fun createRequest(state: ElasticsearchIndexMigrationState): IndexRequest> = + IndexRequest.of { request -> + request.index(indexName) + .id(state.manifest.id.value) + .opType(OpType.Create) + .refresh(Refresh.WaitFor) + .document(state.document()) + } + + private fun compareAndSetRequest( + state: ElasticsearchIndexMigrationState, + version: ElasticsearchIndexLifecycleRepositoryVersion.Elasticsearch, + ): IndexRequest> = IndexRequest.of { request -> + request.index(indexName) + .id(state.manifest.id.value) + .ifSeqNo(version.sequenceNumber) + .ifPrimaryTerm(version.primaryTerm) + .refresh(Refresh.WaitFor) + .document(state.document()) + } + + private fun ElasticsearchIndexMigrationState.document(): Map = linkedMapOf( + FORMAT_VERSION_FIELD to FORMAT_VERSION, + MIGRATION_ID_FIELD to manifest.id.value, + REVISION_FIELD to revision, + PAYLOAD_FIELD to Base64.getEncoder().encodeToString(codec.encode(this)), + ) + + private fun GetResponse>.decode( + expectedId: ElasticsearchIndexMigrationId, + ): ElasticsearchIndexLifecycleStoredState { + requireResponseIdentity(expectedId, index(), id()) + val source = source() ?: corrupt(expectedId, "Elasticsearch lifecycle state document has no source.") + if (source.keys != DOCUMENT_FIELDS) { + corrupt(expectedId, "Elasticsearch lifecycle state document has an invalid shape.") + } + val format = source[FORMAT_VERSION_FIELD].exactLong(expectedId, FORMAT_VERSION_FIELD) + if (format != FORMAT_VERSION.toLong()) { + corrupt(expectedId, "Elasticsearch lifecycle state document has an unsupported format version.") + } + if (source[MIGRATION_ID_FIELD] != expectedId.value) { + corrupt(expectedId, "Elasticsearch lifecycle state document belongs to another migration.") + } + val revision = source[REVISION_FIELD].exactLong(expectedId, REVISION_FIELD) + val payload = source[PAYLOAD_FIELD] as? String + ?: corrupt(expectedId, "Elasticsearch lifecycle state document payload is invalid.") + if (payload.length > MAX_BASE64_PAYLOAD_CHARS) { + corrupt(expectedId, "Elasticsearch lifecycle state document payload exceeds the limit.") + } + val decoded = try { + codec.decode(expectedId, Base64.getDecoder().decode(payload)) + } catch (error: IllegalArgumentException) { + throw corrupted(expectedId, "Elasticsearch lifecycle state document payload is not base64.", error) + } + if (decoded.revision != revision) { + corrupt(expectedId, "Elasticsearch lifecycle state revision does not match its payload.") + } + return ElasticsearchIndexLifecycleStoredState(decoded, repositoryVersion(expectedId, seqNo(), primaryTerm())) + } + + private fun IndexResponse.stored( + state: ElasticsearchIndexMigrationState, + ): ElasticsearchIndexLifecycleStoredState { + requireResponseIdentity(state.manifest.id, index(), id()) + return ElasticsearchIndexLifecycleStoredState( + state, + repositoryVersion(state.manifest.id, seqNo(), primaryTerm()), + ) + } + + private fun requireResponseIdentity( + expectedId: ElasticsearchIndexMigrationId, + actualIndex: String, + actualId: String, + ) { + if (actualIndex != indexName || actualId != expectedId.value) { + corrupt(expectedId, "Elasticsearch lifecycle repository response identity is invalid.") + } + } + + private fun repositoryVersion( + id: ElasticsearchIndexMigrationId, + sequenceNumber: Long?, + primaryTerm: Long?, + ): ElasticsearchIndexLifecycleRepositoryVersion.Elasticsearch { + if (sequenceNumber == null || primaryTerm == null) { + corrupt(id, "Elasticsearch lifecycle repository response has no concurrency token.") + } + if (sequenceNumber < 0 || primaryTerm <= 0) { + corrupt(id, "Elasticsearch lifecycle repository response has no concurrency token.") + } + return ElasticsearchIndexLifecycleRepositoryVersion.Elasticsearch(sequenceNumber, primaryTerm) + } + + private fun validateSystemIndex(): Mono = client.indices().getMapping( + GetMappingRequest.of { request -> request.index(indexName) }, + ).flatMap { response -> + val mappings = response.mappings() + if (mappings.size != 1 || indexName !in mappings) { + Mono.error(IllegalStateException("Elasticsearch lifecycle system index mapping is missing.")) + } else { + val mapping = mappings.getValue(indexName).mapping() + if (mapping.isSystemIndexMapping()) { + Mono.empty() + } else { + Mono.error(IllegalStateException("Elasticsearch lifecycle system index mapping is incompatible.")) + } + } + } + + private fun systemIndexRequest(): CreateIndexRequest = CreateIndexRequest.of { request -> + request.index(indexName) + .settings(IndexSettings.of { settings -> settings.hidden(true).numberOfShards("1") }) + .mappings(systemIndexMapping()) + } + + private fun systemIndexMapping(): TypeMapping = TypeMapping.of { mapping -> + mapping.dynamic(DynamicMapping.Strict) + .meta(REPOSITORY_FORMAT_META, JsonData.of(REPOSITORY_FORMAT)) + .properties(FORMAT_VERSION_FIELD) { property -> + property.integer { number -> number.index(false).docValues(false) } + } + .properties(MIGRATION_ID_FIELD) { property -> + property.keyword { keyword -> keyword.index(false).docValues(false) } + } + .properties(REVISION_FIELD) { property -> + property.long_ { number -> number.index(false).docValues(false) } + } + .properties(PAYLOAD_FIELD) { property -> + property.keyword { keyword -> keyword.index(false).docValues(false) } + } + } + + private fun TypeMapping.isSystemIndexMapping(): Boolean { + if (dynamic() != DynamicMapping.Strict || properties().keys != DOCUMENT_FIELDS) return false + val repositoryFormat = meta()[REPOSITORY_FORMAT_META]?.to(String::class.java) + if (repositoryFormat != REPOSITORY_FORMAT) return false + return properties().getValue(FORMAT_VERSION_FIELD).isNotIndexedInteger() && + properties().getValue(MIGRATION_ID_FIELD).isNotIndexedKeyword() && + properties().getValue(REVISION_FIELD).isNotIndexedLong() && + properties().getValue(PAYLOAD_FIELD).isNotIndexedKeyword() + } + + internal companion object { + const val DEFAULT_INDEX_NAME = ".wow-query-index-lifecycle-v1" + private const val FORMAT_VERSION = 1 + private const val REPOSITORY_FORMAT = "v1" + private const val REPOSITORY_FORMAT_META = "wow_query_lifecycle_repository_format" + private const val FORMAT_VERSION_FIELD = "formatVersion" + private const val MIGRATION_ID_FIELD = "migrationId" + private const val REVISION_FIELD = "revision" + private const val PAYLOAD_FIELD = "payload" + private const val MAX_BASE64_PAYLOAD_CHARS = + ((ElasticsearchIndexLifecycleStateCodec.MAX_PAYLOAD_BYTES + 2) / 3) * 4 + private val DOCUMENT_FIELDS = setOf( + FORMAT_VERSION_FIELD, + MIGRATION_ID_FIELD, + REVISION_FIELD, + PAYLOAD_FIELD, + ) + private val SYSTEM_INDEX_PATTERN = Regex("\\.[a-z0-9][a-z0-9._-]{0,253}") + } +} + +private fun CreateIndexResponse.isExpected(indexName: String): Boolean = + acknowledged() && shardsAcknowledged() && index() == indexName + +private fun co.elastic.clients.elasticsearch._types.mapping.Property.isNotIndexedInteger(): Boolean = + isInteger && integer().index() == false && integer().docValues() == false + +private fun co.elastic.clients.elasticsearch._types.mapping.Property.isNotIndexedKeyword(): Boolean = + isKeyword && keyword().index() == false && keyword().docValues() == false + +private fun co.elastic.clients.elasticsearch._types.mapping.Property.isNotIndexedLong(): Boolean = + isLong && long_().index() == false && long_().docValues() == false + +private fun IndexMappingRecord.mapping(): TypeMapping = mappings() ?: item() + ?: error("Elasticsearch lifecycle index mapping response has no mapping.") + +private fun Any?.exactLong( + id: ElasticsearchIndexMigrationId, + field: String, +): Long = try { + when (this) { + is Byte -> toLong() + is Short -> toLong() + is Int -> toLong() + is Long -> this + is BigInteger -> longValueExact() + is BigDecimal -> longValueExact() + else -> corrupt(id, "Elasticsearch lifecycle state field [$field] is not an exact integer.") + } +} catch (error: ArithmeticException) { + throw corrupted(id, "Elasticsearch lifecycle state field [$field] is not an exact long.", error) +} + +private fun corrupted( + id: ElasticsearchIndexMigrationId, + message: String, + cause: Throwable?, +) = ElasticsearchIndexLifecycleException( + ElasticsearchIndexLifecycleErrorCode.REPOSITORY_CORRUPTED, + id, + message, + cause, +) + +private fun corrupt(id: ElasticsearchIndexMigrationId, message: String): Nothing = + throw corrupted(id, message, null) + +private fun isConflict(error: Throwable): Boolean = when (error) { + is ElasticsearchException -> error.status() == 409 + is RestStatusException -> error.status == 409 + else -> false +} + +private fun isNotFound(error: Throwable): Boolean = when (error) { + is ElasticsearchException -> error.status() == 404 + is RestStatusException -> error.status == 404 + else -> false +} + +private fun isAlreadyExists(error: Throwable): Boolean = when (error) { + is ElasticsearchException -> + error.status() == 400 && error.response().error().type() == "resource_already_exists_exception" + + is RestStatusException -> error.status == 400 && error.message?.contains("resource_already_exists_exception") == true + else -> false +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/SnapshotCanonicalChecksum.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/SnapshotCanonicalChecksum.kt new file mode 100644 index 00000000000..b7104a1d463 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/SnapshotCanonicalChecksum.kt @@ -0,0 +1,91 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.serialization.state.SnapshotRecords +import java.math.BigDecimal +import java.math.BigInteger + +internal data class SnapshotCanonicalChecksumLimits( + val maxDepth: Int = 64, + val maxNodesPerDocument: Int = 250_000, + val maxCollectionSize: Int = 100_000, + val maxStringBytes: Int = 1_048_576, + val maxPayloadBytesPerDocument: Long = 32L * 1_048_576, +) { + init { + require(maxDepth > 0) + require(maxNodesPerDocument > 0) + require(maxCollectionSize > 0) + require(maxStringBytes > 0) + require(maxPayloadBytesPerDocument > 0) + } +} + +internal data class SnapshotCanonicalChecksumEvidence( + val count: Long, + val identityChecksum: ElasticsearchIndexChecksum, + val contentChecksum: ElasticsearchIndexChecksum, +) + +/** Ordered, bounded checksum accumulator shared by authoritative replay and physical-index inspection. */ +internal class SnapshotCanonicalChecksumAccumulator( + private val limits: SnapshotCanonicalChecksumLimits = SnapshotCanonicalChecksumLimits(), +) { + private val identityDigest = newLifecycleSha256().apply { updateLifecycleUtf8(IDENTITY_HEADER) } + private val contentDigest = newLifecycleSha256().apply { updateLifecycleUtf8(CONTENT_HEADER) } + private var previousIdentity: String? = null + private var count: Long = 0 + + fun accept(identity: String, source: Map<*, *>) { + require(identity.isNotBlank()) { "Snapshot verification identity must not be blank." } + require(previousIdentity == null || identity > requireNotNull(previousIdentity)) { + "Snapshot verification records must have unique, strictly ascending identities." + } + require(source[MessageRecords.AGGREGATE_ID] == identity) { + "Snapshot verification identity does not match the serialized aggregate id." + } + requireNonNegativeVersion(source[MessageRecords.VERSION]) + val identityBytes = identity.toByteArray(Charsets.UTF_8) + val documentHash = canonicalDocumentHash(source, limits, setOf(SnapshotRecords.SNAPSHOT_TIME)) + identityDigest.updateLifecycleLengthPrefixed(identityBytes) + contentDigest.updateLifecycleLengthPrefixed(identityBytes) + contentDigest.updateLifecycleLengthPrefixed(documentHash) + previousIdentity = identity + count = Math.addExact(count, 1) + } + + fun finish(): SnapshotCanonicalChecksumEvidence = SnapshotCanonicalChecksumEvidence( + count, + ElasticsearchIndexChecksum(identityDigest.digest().toLowerHex()), + ElasticsearchIndexChecksum(contentDigest.digest().toLowerHex()), + ) + + private companion object { + const val IDENTITY_HEADER = "wow-es-snapshot-identity-sha256-v1" + const val CONTENT_HEADER = "wow-es-snapshot-content-sha256-v1" + } +} + +private fun requireNonNegativeVersion(raw: Any?) { + val version = when (raw) { + is BigDecimal -> raw.toBigIntegerExact() + is BigInteger -> raw + is Byte, is Short, is Int, is Long -> BigInteger.valueOf((raw as Number).toLong()) + is Float, is Double -> BigDecimal.valueOf((raw as Number).toDouble()).toBigIntegerExact() + else -> throw IllegalArgumentException("Snapshot verification document has no numeric version.") + } + require(version.signum() >= 0) { "Snapshot verification version must not be negative." } +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/SnapshotElasticsearchIndexVerificationSources.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/SnapshotElasticsearchIndexVerificationSources.kt new file mode 100644 index 00000000000..f73754d0d07 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/SnapshotElasticsearchIndexVerificationSources.kt @@ -0,0 +1,249 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import co.elastic.clients.elasticsearch._types.SortOrder +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.range +import co.elastic.clients.elasticsearch.core.ClosePointInTimeRequest +import co.elastic.clients.elasticsearch.core.OpenPointInTimeRequest +import co.elastic.clients.elasticsearch.core.SearchRequest +import co.elastic.clients.elasticsearch.core.search.ResponseBody +import co.elastic.clients.elasticsearch.core.search.TotalHitsRelation +import co.elastic.clients.json.JsonData +import me.ahoo.wow.eventsourcing.snapshot.SimpleSnapshot +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.serialization.toLinkedHashMap +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Mono + +internal data class SnapshotElasticsearchIndexVerificationOptions( + val pageSize: Int = 1_000, + val pitKeepAlive: String = "1m", + val checksumLimits: SnapshotCanonicalChecksumLimits = SnapshotCanonicalChecksumLimits(), +) { + init { + require(pageSize > 0) { "Snapshot verification page size must be positive." } + require(pitKeepAlive.isNotBlank()) { "Snapshot verification PIT keep-alive must not be blank." } + } +} + +/** Computes expected Snapshot evidence from full authoritative EventStore replay. */ +internal class EventStoreSnapshotVerificationSource( + private val source: ElasticsearchAuthoritativeSnapshotSource, + private val options: SnapshotElasticsearchIndexVerificationOptions = + SnapshotElasticsearchIndexVerificationOptions(), +) : ElasticsearchAuthoritativeVerificationSource { + override fun capture( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = Mono.defer { + requireSnapshotVerificationContract(manifest) + val accumulator = SnapshotCanonicalChecksumAccumulator(options.checksumLimits) + scanOrderedAuthoritativeSnapshots(source, manifest.target, options.pageSize) + .doOnNext { aggregate -> + val snapshot = SimpleSnapshot(aggregate, snapshotTime = 0).toLinkedHashMap() + accumulator.accept(aggregate.aggregateId.id, snapshot) + }.then( + Mono.fromSupplier { + val evidence = accumulator.finish() + ElasticsearchAuthoritativeVerificationSnapshot( + manifest.verificationContract.checksumAlgorithm, + evidence.count, + evidence.identityChecksum, + evidence.contentChecksum, + watermark = null, + ) + }, + ) + }.mapVerificationErrors(manifest) +} + +/** Scans one exact physical Snapshot generation through a PIT and computes actual evidence. */ +internal class ReactiveElasticsearchSnapshotVerificationSource( + private val client: ReactiveElasticsearchClient, + private val options: SnapshotElasticsearchIndexVerificationOptions = + SnapshotElasticsearchIndexVerificationOptions(), +) : ElasticsearchPhysicalIndexVerificationSource { + override fun inspect( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono = Mono.defer { + requireSnapshotVerificationContract(manifest) + val accumulator = SnapshotCanonicalChecksumAccumulator(options.checksumLimits) + Mono.usingWhen( + openPit(physicalIndex), + { lease -> inspectPages(lease, accumulator) }, + ::closePit, + { lease, error -> closeAfterError(lease, error) }, + ::closeAfterCancel, + ).map { total -> + val evidence = accumulator.finish() + if (evidence.count != total) { + throw IllegalStateException("Physical Snapshot PIT total changed during verification.") + } + ElasticsearchPhysicalIndexVerificationSnapshot( + manifest.verificationContract.checksumAlgorithm, + physicalIndex, + evidence.count, + evidence.identityChecksum, + evidence.contentChecksum, + versionContinuity = true, + watermark = null, + ) + } + }.mapVerificationErrors(manifest) + + private fun openPit(physicalIndex: ElasticsearchPhysicalIndex): Mono = + client.openPointInTime( + OpenPointInTimeRequest.of { request -> + request.index(physicalIndex.value).keepAlive { keepAlive -> keepAlive.time(options.pitKeepAlive) } + }, + ).switchIfEmpty(Mono.error(IllegalStateException("Snapshot verification PIT was not opened."))) + .map { response -> + if (response.id().isBlank() || response.shards().failed() != 0) { + throw IllegalStateException("Snapshot verification PIT is incomplete.") + } + PitLease(response.id()) + } + + private fun inspectPages( + lease: PitLease, + accumulator: SnapshotCanonicalChecksumAccumulator, + ): Mono { + fun inspectPage(afterIdentity: String?, rootTotal: Long?, consumed: Long): Mono = + client.search(searchRequest(lease, afterIdentity), Map::class.java) + .switchIfEmpty(Mono.error(IllegalStateException("Snapshot verification search returned no response."))) + .flatMap { response -> + val currentTotal = validateResponse(response) + val exactRootTotal = rootTotal ?: currentTotal + check(currentTotal == exactRootTotal - consumed) { + "Snapshot verification keyset total is not stable and exact." + } + response.pitId()?.takeIf(String::isNotBlank)?.let(lease::update) + val hits = response.hits().hits() + hits.forEach { hit -> + if (hit.ignored().isNotEmpty()) { + throw IllegalStateException("Snapshot verification hit contains ignored fields.") + } + val identity = hit.id() + ?: throw IllegalStateException("Snapshot verification hit has no identity.") + val document = hit.source() + ?: throw IllegalStateException("Snapshot verification hit has no source.") + accumulator.accept(identity, document) + } + val nextConsumed = Math.addExact(consumed, hits.size.toLong()) + if (hits.size < options.pageSize) { + Mono.just(exactRootTotal) + } else { + val nextIdentity = hits.last().id() + ?: return@flatMap Mono.error( + IllegalStateException("Snapshot verification page has no terminal identity."), + ) + inspectPage(nextIdentity, exactRootTotal, nextConsumed) + } + } + return inspectPage(null, null, 0) + } + + private fun searchRequest(lease: PitLease, afterIdentity: String?): SearchRequest = + SearchRequest.of { request -> + request.pit { pit -> pit.id(lease.id).keepAlive { keepAlive -> keepAlive.time(options.pitKeepAlive) } } + .size(options.pageSize) + .allowPartialSearchResults(false) + .trackTotalHits { total -> total.enabled(true) } + .sort { sort -> + sort.field { field -> field.field(MessageRecords.AGGREGATE_ID).order(SortOrder.Asc) } + }.also { builder -> + afterIdentity?.let { identity -> + builder.query( + range { range -> + range.untyped { untyped -> + untyped.field(MessageRecords.AGGREGATE_ID).gt(JsonData.of(identity)) + } + }, + ) + } + } + } + + private fun validateResponse(response: ResponseBody<*>): Long { + check(!response.timedOut() && response.shards().failed() == 0) { + "Snapshot verification search is incomplete." + } + val total = checkNotNull(response.hits().total()) { + "Snapshot verification search has no exact total." + } + check(total.relation() == TotalHitsRelation.Eq) { "Snapshot verification search total is not exact." } + return total.value() + } + + private fun closePit(lease: PitLease): Mono = client.closePointInTime( + ClosePointInTimeRequest.of { request -> request.id(lease.id) }, + ).switchIfEmpty(Mono.error(IllegalStateException("Snapshot verification PIT close returned no response."))) + .flatMap { response -> + if (!response.succeeded()) { + Mono.error(IllegalStateException("Snapshot verification PIT was not closed.")) + } else { + Mono.empty() + } + } + + private fun closeAfterError(lease: PitLease, original: Throwable): Mono = + closePit(lease).onErrorResume { closeError -> + original.addSuppressed(closeError) + Mono.empty() + } + + private fun closeAfterCancel(lease: PitLease): Mono = closePit(lease).onErrorResume { Mono.empty() } + + private class PitLease(initialId: String) { + var id: String = initialId + private set + + fun update(nextId: String) { + id = nextId + } + } +} + +private fun requireSnapshotVerificationContract(manifest: ElasticsearchIndexMigrationManifest) { + require(manifest.target.documentKind == QueryDocumentKind.SNAPSHOT) { + "Canonical Snapshot verification requires a Snapshot target." + } + require( + manifest.verificationContract.checksumAlgorithm == + ElasticsearchIndexChecksumAlgorithm.CANONICAL_DOCUMENT_SHA256_V1, + ) { + "Canonical Snapshot verification received an unsupported checksum algorithm." + } +} + +private fun Mono.mapVerificationErrors( + manifest: ElasticsearchIndexMigrationManifest, +): Mono = onErrorMap { error -> + if (error is ElasticsearchIndexLifecycleException) { + error + } else { + ElasticsearchIndexLifecycleException( + ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED, + manifest.id, + "Elasticsearch Snapshot verification evidence could not be computed.", + error, + ) + } +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchAnalyticsQueryBackend.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchAnalyticsQueryBackend.kt new file mode 100644 index 00000000000..1beb2b406a5 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchAnalyticsQueryBackend.kt @@ -0,0 +1,404 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + +package me.ahoo.wow.elasticsearch.query.planned + +import co.elastic.clients.elasticsearch._types.FieldValue +import co.elastic.clients.elasticsearch._types.ShardStatistics +import co.elastic.clients.elasticsearch._types.aggregations.CompositeBucket +import co.elastic.clients.elasticsearch.core.ClosePointInTimeRequest +import co.elastic.clients.elasticsearch.core.OpenPointInTimeRequest +import co.elastic.clients.elasticsearch.core.SearchRequest +import co.elastic.clients.elasticsearch.core.search.ResponseBody +import co.elastic.clients.elasticsearch.core.search.TotalHitsRelation +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.AnalyticsQueryBackend +import me.ahoo.wow.query.backend.AnalyticsQueryCursorLifecycle +import me.ahoo.wow.query.backend.BackendAnalyticsBucket +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsCursorState +import me.ahoo.wow.query.backend.BackendAnalyticsDimension +import me.ahoo.wow.query.backend.BackendAnalyticsGrouping +import me.ahoo.wow.query.backend.BackendAnalyticsMetric +import me.ahoo.wow.query.backend.BackendAnalyticsMissingPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsPage +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Mono +import java.nio.ByteBuffer +import java.nio.charset.CodingErrorAction +import java.nio.charset.StandardCharsets +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.util.LinkedHashMap + +internal class ElasticsearchAnalyticsQueryBackend( + private val client: ReactiveElasticsearchClient, + private val binding: ElasticsearchPreparedQueryBinding, + private val clock: Clock = Clock.systemUTC(), +) : AnalyticsQueryBackend, AnalyticsQueryCursorLifecycle { + private val compiler = ElasticsearchAnalyticsQueryCompiler(binding) + + override fun analyze( + plan: BackendAnalyticsQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = analyze(plan, options, null) + + override fun analyze( + plan: BackendAnalyticsQueryPlan, + options: QueryBackendExecutionOptions, + cursorState: BackendAnalyticsCursorState?, + ): Mono = Mono.defer { + validateOptions(plan, options) + val compiled = compiler.compile(plan) + when (plan.requiredConsistency) { + BackendAnalyticsConsistency.EVENTUAL -> { + if (cursorState != null) unsupported() + client.search(searchRequest(plan, compiled, options, null), Map::class.java) + .map { response -> mapResponse(plan, response, BackendAnalyticsConsistency.EVENTUAL, null) } + } + + BackendAnalyticsConsistency.SNAPSHOT -> analyzeSnapshot(plan, compiled, options, cursorState) + } + }.onErrorMap(::mapBackendError) + + override fun close(cursorState: BackendAnalyticsCursorState): Mono = Mono.defer { + closePit(PitLease(decodePitId(cursorState))) + }.onErrorMap(::mapBackendError) + + private fun analyzeSnapshot( + plan: BackendAnalyticsQueryPlan, + compiled: ElasticsearchCompiledAnalyticsQuery, + options: QueryBackendExecutionOptions, + cursorState: BackendAnalyticsCursorState?, + ): Mono { + val resource = cursorState?.let { state -> Mono.just(PitLease(decodePitId(state))) } ?: openPit() + return Mono.usingWhen( + resource, + { lease -> + client.search(searchRequest(plan, compiled, options, lease.id), Map::class.java) + .map { response -> + response.pitId()?.takeIf(String::isNotBlank)?.let(lease::update) + val grouped = plan.grouping is BackendAnalyticsGrouping.By + val page = mapResponse( + plan, + response, + BackendAnalyticsConsistency.SNAPSHOT, + if (grouped) BackendAnalyticsCursorState(lease.id.encodeToByteArray()) else null, + ) + if (grouped) lease.transfer() + page + } + }, + ::closeUnlessTransferred, + { lease, original -> closeAfterError(lease, original) }, + ::closeAfterCancel, + ) + } + + private fun openPit(): Mono = client.openPointInTime( + OpenPointInTimeRequest.of { request -> + request.index(binding.indexName).keepAlive { keepAlive -> keepAlive.time(PIT_KEEP_ALIVE) } + }, + ).switchIfEmpty(Mono.error(QueryBackendException(QueryBackendFailureKind.UNAVAILABLE))) + .map { response -> + if (response.id().isBlank() || response.shards().failed() != 0) incomplete() + PitLease(response.id()) + } + + private fun searchRequest( + plan: BackendAnalyticsQueryPlan, + compiled: ElasticsearchCompiledAnalyticsQuery, + options: QueryBackendExecutionOptions, + pitId: String?, + ): SearchRequest = SearchRequest.of { request -> + request.query(compiled.query) + .size(0) + .allowPartialSearchResults(false) + .trackTotalHits { total -> total.enabled(plan.grouping == BackendAnalyticsGrouping.Global) } + .also { builder -> + if (pitId == null) { + builder.index(binding.indexName) + } else { + builder.pit { pit -> + pit.id(pitId).keepAlive { keepAlive -> keepAlive.time(PIT_KEEP_ALIVE) } + } + } + compiled.aggregation?.let { aggregation -> + builder.aggregations(ANALYTICS_AGGREGATION, aggregation) + } + options.remainingMillis()?.let { remaining -> builder.timeout("${remaining}ms") } + } + } + + private fun mapResponse( + plan: BackendAnalyticsQueryPlan, + response: ResponseBody<*>, + consistency: BackendAnalyticsConsistency, + cursorState: BackendAnalyticsCursorState?, + ): BackendAnalyticsPage = mapFailures { + if (response.timedOut()) timeout() + requireCompleteShards(response.shards()) + when (val grouping = plan.grouping) { + BackendAnalyticsGrouping.Global -> mapGlobal(plan, response, consistency) + is BackendAnalyticsGrouping.By -> mapComposite(plan, grouping, response, consistency, cursorState) + } + } + + private fun mapGlobal( + plan: BackendAnalyticsQueryPlan, + response: ResponseBody<*>, + consistency: BackendAnalyticsConsistency, + ): BackendAnalyticsPage { + val total = response.hits().total() ?: incomplete() + if (total.relation() != TotalHitsRelation.Eq || total.value() < 0) incomplete() + val metrics = plan.metrics.associate { metric -> + val count = metric as? BackendAnalyticsMetric.DocumentCount ?: mappingFailure() + count.alias to NormalizedValue.Int64(total.value()) + } + return BackendAnalyticsPage( + listOf(BackendAnalyticsBucket(emptyMap(), metrics)), + null, + consistency, + BackendAnalyticsCompleteness.EXACT, + ) + } + + private fun mapComposite( + plan: BackendAnalyticsQueryPlan, + grouping: BackendAnalyticsGrouping.By, + response: ResponseBody<*>, + consistency: BackendAnalyticsConsistency, + cursorState: BackendAnalyticsCursorState?, + ): BackendAnalyticsPage { + val aggregate = response.aggregations()[ANALYTICS_AGGREGATION] + if (aggregate == null || !aggregate.isComposite) mappingFailure() + val composite = aggregate.composite() + val buckets = composite.buckets().array() + if (buckets.size > plan.bucketWindow.limit) incomplete() + val mapped = buckets.map { bucket -> mapBucket(grouping.dimensions, plan.metrics, bucket) } + val afterKey = mapAfterKey(grouping.dimensions, composite.afterKey()) + return BackendAnalyticsPage( + mapped, + afterKey, + consistency, + BackendAnalyticsCompleteness.EXACT, + cursorState, + ) + } + + private fun closeUnlessTransferred(lease: PitLease): Mono = + if (lease.transferred) Mono.empty() else closePit(lease) + + private fun closeAfterError(lease: PitLease, original: Throwable): Mono = + closePit(lease).onErrorResume { closeError -> + original.addSuppressed(closeError) + Mono.empty() + } + + private fun closeAfterCancel(lease: PitLease): Mono = closePit(lease).onErrorResume { Mono.empty() } + + private fun closePit(lease: PitLease): Mono = client.closePointInTime( + ClosePointInTimeRequest.of { request -> request.id(lease.id) }, + ).switchIfEmpty( + Mono.error(QueryBackendException(QueryBackendFailureKind.INCOMPLETE_RESULT)), + ).flatMap { response -> + if (response.succeeded()) { + Mono.empty() + } else { + Mono.error(QueryBackendException(QueryBackendFailureKind.INCOMPLETE_RESULT)) + } + } + + private fun decodePitId(state: BackendAnalyticsCursorState): String = try { + val payload = state.payload() + if (payload.size > MAX_PIT_ID_BYTES) unsupported() + val decoded = StandardCharsets.UTF_8.newDecoder() + .onMalformedInput(CodingErrorAction.REPORT) + .onUnmappableCharacter(CodingErrorAction.REPORT) + .decode(ByteBuffer.wrap(payload)) + .toString() + if (decoded.isBlank()) unsupported() + decoded + } catch (error: java.nio.charset.CharacterCodingException) { + throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED, error) + } + + private fun mapBucket( + dimensions: List, + metrics: List, + bucket: CompositeBucket, + ): BackendAnalyticsBucket { + if (bucket.docCount() < 0 || bucket.key().keys != dimensions.map { it.alias.value }.toSet()) { + mappingFailure() + } + val keys = LinkedHashMap(dimensions.size) + dimensions.forEach { dimension -> + keys[dimension.alias] = decodeDimensionValue(dimension, bucket.key().getValue(dimension.alias.value)) + } + val metricValues = LinkedHashMap(metrics.size) + metrics.forEach { metric -> + val count = metric as? BackendAnalyticsMetric.DocumentCount ?: mappingFailure() + metricValues[count.alias] = NormalizedValue.Int64(bucket.docCount()) + } + return BackendAnalyticsBucket(keys, metricValues) + } + + private fun mapAfterKey( + dimensions: List, + raw: Map, + ): List? { + if (raw.isEmpty()) return null + if (raw.keys != dimensions.map { it.alias.value }.toSet()) mappingFailure() + return dimensions.map { dimension -> + decodeDimensionValue(dimension, raw.getValue(dimension.alias.value)) + } + } + + private fun decodeDimensionValue( + dimension: BackendAnalyticsDimension, + raw: FieldValue, + ): NormalizedValue { + if (raw.isNull) { + if (dimension.missingPolicy == BackendAnalyticsMissingPolicy.AS_NULL_BUCKET) { + return NormalizedValue.Null + } + mappingFailure() + } + return when (binding.schema.fields.getValue(dimension.field).type) { + LogicalFieldType.Text -> if (raw.isString) { + NormalizedValue.Text(raw.stringValue()) + } else { + mappingFailure() + } + + LogicalFieldType.Boolean -> if (raw.isBoolean) { + NormalizedValue.BooleanValue(raw.booleanValue()) + } else { + mappingFailure() + } + + LogicalFieldType.Int64 -> if (raw.isLong) { + NormalizedValue.Int64(raw.longValue()) + } else { + mappingFailure() + } + + LogicalFieldType.Instant -> if (raw.isLong) { + NormalizedValue.InstantValue(Instant.ofEpochMilli(raw.longValue())) + } else { + mappingFailure() + } + + LogicalFieldType.Decimal, + LogicalFieldType.Bytes, + LogicalFieldType.Object, + is LogicalFieldType.Array, + -> mappingFailure() + } + } + + private fun validateOptions(plan: BackendAnalyticsQueryPlan, options: QueryBackendExecutionOptions) { + if (options.maxScannedRecords != null || + options.maxCandidateBuckets != null || + options.maxCursorPages != null + ) { + unsupported() + } + options.maxReturnedBuckets?.let { maximum -> + if (plan.bucketWindow.limit > maximum) budgetExceeded() + } + options.remainingMillis() + } + + private fun QueryBackendExecutionOptions.remainingMillis(): Long? { + val currentDeadline = deadline ?: return null + val now = clock.instant() + val remaining = try { + Duration.between(now, currentDeadline).toMillis() + } catch (error: ArithmeticException) { + if (currentDeadline.isAfter(now)) { + Long.MAX_VALUE + } else { + throw QueryBackendException( + QueryBackendFailureKind.TIMEOUT, + error, + ) + } + } + if (remaining <= 0) { + timeout() + } + return remaining + } + + private fun requireCompleteShards(shards: ShardStatistics) { + if (shards.failed().toLong() != 0L) incomplete() + } + + @Suppress("TooGenericExceptionCaught") + private inline fun mapFailures(block: () -> T): T = try { + block() + } catch (error: QueryBackendException) { + throw error + } catch (error: RuntimeException) { + throw QueryBackendException(QueryBackendFailureKind.MAPPING_FAILURE, error) + } + + private fun mapBackendError(error: Throwable): Throwable = when { + error is QueryBackendException -> error + error.isMissingElasticsearchSearchContext() -> + QueryBackendException(QueryBackendFailureKind.INCOMPLETE_RESULT, error) + else -> QueryBackendException(QueryBackendFailureKind.UNAVAILABLE, error) + } + + private fun unsupported(): Nothing = throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED) + + private fun budgetExceeded(): Nothing = throw QueryBackendException(QueryBackendFailureKind.BUDGET_EXCEEDED) + + private fun timeout(): Nothing = throw QueryBackendException(QueryBackendFailureKind.TIMEOUT) + + private fun incomplete(): Nothing = throw QueryBackendException(QueryBackendFailureKind.INCOMPLETE_RESULT) + + private fun mappingFailure(): Nothing = throw QueryBackendException(QueryBackendFailureKind.MAPPING_FAILURE) + + internal companion object { + const val ANALYTICS_AGGREGATION = "wow_analytics" + const val PIT_KEEP_ALIVE = "2m" + const val MAX_PIT_ID_BYTES = 4_096 + } + + private class PitLease(initialId: String) { + var id: String = initialId + private set + var transferred: Boolean = false + private set + + fun update(value: String) { + id = value + } + + fun transfer() { + transferred = true + } + } +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchAnalyticsQueryCompiler.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchAnalyticsQueryCompiler.kt new file mode 100644 index 00000000000..cf2663a7bdc --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchAnalyticsQueryCompiler.kt @@ -0,0 +1,184 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.planned + +import co.elastic.clients.elasticsearch._types.FieldValue +import co.elastic.clients.elasticsearch._types.SortOrder +import co.elastic.clients.elasticsearch._types.aggregations.Aggregation +import co.elastic.clients.elasticsearch._types.aggregations.CompositeAggregationSource +import co.elastic.clients.elasticsearch._types.aggregations.MissingOrder +import co.elastic.clients.util.NamedValue +import me.ahoo.wow.query.backend.BackendAnalyticsBucketOrder +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsCondition +import me.ahoo.wow.query.backend.BackendAnalyticsDimension +import me.ahoo.wow.query.backend.BackendAnalyticsGrouping +import me.ahoo.wow.query.backend.BackendAnalyticsMetric +import me.ahoo.wow.query.backend.BackendAnalyticsMissingPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNullPlacement +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.BackendAnalyticsTextCollation +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.SemanticTier + +internal data class ElasticsearchCompiledAnalyticsQuery( + val query: co.elastic.clients.elasticsearch._types.query_dsl.Query, + val grouping: BackendAnalyticsGrouping, + val metrics: List, + val aggregation: Aggregation?, +) + +internal class ElasticsearchAnalyticsQueryCompiler( + private val binding: ElasticsearchPreparedQueryBinding, +) { + private val recordCompiler = ElasticsearchRecordQueryCompiler(binding) + + fun compile(plan: BackendAnalyticsQueryPlan): ElasticsearchCompiledAnalyticsQuery { + validateContract(plan) + validateMetrics(plan) + val aggregation = when (val grouping = plan.grouping) { + BackendAnalyticsGrouping.Global -> { + validateGlobal(plan) + null + } + + is BackendAnalyticsGrouping.By -> compileComposite(plan, grouping) + } + return ElasticsearchCompiledAnalyticsQuery( + recordCompiler.compileCondition(plan.filter.condition), + plan.grouping, + plan.metrics, + aggregation, + ) + } + + private fun validateContract(plan: BackendAnalyticsQueryPlan) { + if (plan.target != binding.schema.target || plan.schemaContractId != binding.schema.contractId) unsupported() + if (plan.having != BackendAnalyticsCondition.All || plan.semanticTier != SemanticTier.PORTABLE) unsupported() + if (plan.requiredCompleteness != BackendAnalyticsCompleteness.EXACT) { + unsupported() + } + } + + private fun validateMetrics(plan: BackendAnalyticsQueryPlan) { + if (plan.metrics.any { metric -> metric !is BackendAnalyticsMetric.DocumentCount }) unsupported() + if (plan.numericPolicy != null) unsupported() + } + + private fun validateGlobal(plan: BackendAnalyticsQueryPlan) { + if (plan.bucketOrder != BackendAnalyticsBucketOrder.Global || + plan.bucketWindow.limit != 1 || + plan.bucketWindow.afterKey != null + ) { + unsupported() + } + } + + private fun compileComposite( + plan: BackendAnalyticsQueryPlan, + grouping: BackendAnalyticsGrouping.By, + ): Aggregation { + val order = plan.bucketOrder as? BackendAnalyticsBucketOrder.DimensionKeyAscending ?: unsupported() + if (order.nullPlacement != BackendAnalyticsNullPlacement.FIRST || + order.textCollation != BackendAnalyticsTextCollation.BINARY + ) { + unsupported() + } + val sources = grouping.dimensions.map(::compileSource) + val after = plan.bucketWindow.afterKey?.let { values -> + if (values.size != grouping.dimensions.size) unsupported() + LinkedHashMap(values.size).also { result -> + grouping.dimensions.forEachIndexed { index, dimension -> + result[dimension.alias.value] = encodeDimensionValue(dimension, values[index]) + } + } + } + return Aggregation.of { aggregation -> + aggregation.composite { composite -> + composite.size(plan.bucketWindow.limit) + .sources(sources) + .also { builder -> after?.let(builder::after) } + } + } + } + + private fun compileSource(dimension: BackendAnalyticsDimension): NamedValue { + val field = requireGroupField(dimension.field) + return NamedValue.of( + dimension.alias.value, + CompositeAggregationSource.of { source -> + source.terms { terms -> + val missingBucket = dimension.missingPolicy == BackendAnalyticsMissingPolicy.AS_NULL_BUCKET + terms.field(field.groupField) + .order(SortOrder.Asc) + .missingBucket(missingBucket) + .also { builder -> + if (missingBucket) { + builder.missingOrder(MissingOrder.First) + } + } + } + }, + ) + } + + private fun encodeDimensionValue( + dimension: BackendAnalyticsDimension, + value: NormalizedValue, + ): FieldValue { + if (value == NormalizedValue.Null) { + if (dimension.missingPolicy == BackendAnalyticsMissingPolicy.AS_NULL_BUCKET) return FieldValue.NULL + unsupported() + } + return when (binding.schema.fields.getValue(dimension.field).type) { + LogicalFieldType.Text -> FieldValue.of((value as? NormalizedValue.Text)?.value ?: unsupported()) + LogicalFieldType.Boolean -> FieldValue.of( + (value as? NormalizedValue.BooleanValue)?.value ?: unsupported(), + ) + + LogicalFieldType.Int64 -> FieldValue.of((value as? NormalizedValue.Int64)?.value ?: unsupported()) + LogicalFieldType.Instant -> { + if (requireGroupField(dimension.field).valueEncoding != ElasticsearchValueEncoding.EPOCH_MILLIS) { + unsupported() + } + FieldValue.of((value as? NormalizedValue.InstantValue)?.value?.toEpochMilli() ?: unsupported()) + } + + LogicalFieldType.Decimal, + LogicalFieldType.Bytes, + LogicalFieldType.Object, + is LogicalFieldType.Array, + -> unsupported() + } + } + + private fun requireGroupField(field: QueryFieldId): ElasticsearchFieldBinding { + if (field is QueryFieldId.Path && binding.schema.elementOwner(field) != null) unsupported() + val fieldBinding = binding.fields[field] ?: unsupported() + if (FieldCapability.AGGREGATABLE !in fieldBinding.capabilities || fieldBinding.groupField == null) unsupported() + return fieldBinding + } + + private fun unsupported(): Nothing = throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED) +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchPitFailures.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchPitFailures.kt new file mode 100644 index 00000000000..fde81a44969 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchPitFailures.kt @@ -0,0 +1,54 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.elasticsearch.query.planned + +import co.elastic.clients.elasticsearch._types.ElasticsearchException +import co.elastic.clients.elasticsearch._types.ErrorCause +import org.springframework.data.elasticsearch.RestStatusException +import java.util.Collections +import java.util.IdentityHashMap + +internal fun Throwable.isMissingElasticsearchSearchContext(): Boolean { + val visited = Collections.newSetFromMap(IdentityHashMap()) + var current: Throwable? = this + while (current != null && visited.add(current)) { + when (current) { + is ElasticsearchException -> { + if (current.status() == NOT_FOUND || current.error().containsMissingSearchContext()) return true + } + + is RestStatusException -> if (current.status == NOT_FOUND) return true + } + current = current.cause + } + return false +} + +private fun ErrorCause.containsMissingSearchContext(): Boolean { + val pending = ArrayDeque() + val visited = Collections.newSetFromMap(IdentityHashMap()) + pending.add(this) + while (pending.isNotEmpty()) { + val current = pending.removeFirst() + if (!visited.add(current)) continue + if (current.type() == SEARCH_CONTEXT_MISSING_TYPE) return true + current.causedBy()?.let(pending::addLast) + pending.addAll(current.rootCause()) + pending.addAll(current.suppressed()) + } + return false +} + +private const val NOT_FOUND = 404 +private const val SEARCH_CONTEXT_MISSING_TYPE = "search_context_missing_exception" diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchPitPageExecutor.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchPitPageExecutor.kt new file mode 100644 index 00000000000..ab975ae40f0 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchPitPageExecutor.kt @@ -0,0 +1,294 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + +package me.ahoo.wow.elasticsearch.query.planned + +import co.elastic.clients.elasticsearch._types.FieldValue +import co.elastic.clients.elasticsearch.core.ClosePointInTimeRequest +import co.elastic.clients.elasticsearch.core.OpenPointInTimeRequest +import co.elastic.clients.elasticsearch.core.SearchRequest +import co.elastic.clients.elasticsearch.core.search.Hit +import co.elastic.clients.elasticsearch.core.search.ResponseBody +import co.elastic.clients.elasticsearch.core.search.TotalHitsRelation +import me.ahoo.wow.query.backend.BackendPage +import me.ahoo.wow.query.backend.BackendPageConsistency +import me.ahoo.wow.query.backend.BackendPageQueryPlan +import me.ahoo.wow.query.backend.BackendRecord +import me.ahoo.wow.query.backend.BackendTotalRelation +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.Duration +import java.util.Collections +import java.util.IdentityHashMap + +internal class ElasticsearchPitPageExecutor( + private val client: ReactiveElasticsearchClient, + private val binding: ElasticsearchPreparedQueryBinding, + private val compiler: ElasticsearchRecordQueryCompiler, + private val mapper: ElasticsearchSnapshotRecordMapper, + private val clock: Clock, +) { + fun execute( + plan: BackendPageQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono { + val compiled = compiler.compile(plan) + if (compiled.sort.isEmpty()) { + unsupported() + } + return Mono.usingWhen( + openPit(), + { lease -> collectPage(lease, plan, compiled, options) }, + ::closePit, + { lease, error -> closeAfterError(lease, error) }, + ::closeAfterCancel, + ).onErrorMap(::mapPitLifecycleError) + } + + private fun openPit(): Mono = client.openPointInTime( + OpenPointInTimeRequest.of { request -> + request.index(binding.indexName).keepAlive { keepAlive -> keepAlive.time(PIT_KEEP_ALIVE) } + }, + ).switchIfEmpty( + Mono.error(QueryBackendException(QueryBackendFailureKind.UNAVAILABLE)), + ).map { response -> + if (response.id().isBlank() || response.shards().failed() != 0) { + incomplete() + } + PitLease(response.id()) + } + + private fun collectPage( + lease: PitLease, + plan: BackendPageQueryPlan, + compiled: ElasticsearchCompiledRecordQuery, + options: QueryBackendExecutionOptions, + ): Mono { + val state = PageState( + lease, + plan, + compiled, + options, + options.maxCursorPages ?: DEFAULT_MAX_CURSOR_PAGES, + ) + return fetchNext(state).then(Mono.fromSupplier { state.toPage() }) + } + + private fun fetchNext(state: PageState): Mono = Mono.defer { + if (state.isComplete()) { + return@defer Mono.empty() + } + if (state.cursorPages >= state.maxCursorPages) { + budgetExceeded() + } + val batchSize = state.nextBatchSize() + client.search(searchRequest(state, batchSize), Map::class.java) + .onErrorMap(::mapPitSearchError) + .switchIfEmpty(Mono.error(QueryBackendException(QueryBackendFailureKind.INCOMPLETE_RESULT))) + .flatMap { response -> + state.accept(response) + fetchNext(state) + } + } + + private fun searchRequest(state: PageState, batchSize: Int): SearchRequest = SearchRequest.of { request -> + request.pit { pit -> + pit.id(state.lease.id).keepAlive { keepAlive -> keepAlive.time(PIT_KEEP_ALIVE) } + }.query(state.compiled.query) + .size(batchSize) + .allowPartialSearchResults(false) + .trackTotalHits { total -> total.enabled(true) } + .sort(state.compiled.sort) + .also { builder -> + state.compiled.sourceFilter?.let { sourceFilter -> + builder.source { source -> source.filter(sourceFilter) } + } + state.searchAfter?.let(builder::searchAfter) + state.options.remainingMillis()?.let { remaining -> builder.timeout("${remaining}ms") } + } + } + + private fun closePit(lease: PitLease): Mono = client.closePointInTime( + ClosePointInTimeRequest.of { request -> request.id(lease.id) }, + ) + .switchIfEmpty(Mono.error(QueryBackendException(QueryBackendFailureKind.INCOMPLETE_RESULT))) + .flatMap { response -> + if (!response.succeeded()) { + Mono.error(QueryBackendException(QueryBackendFailureKind.INCOMPLETE_RESULT)) + } else { + Mono.empty() + } + } + + private fun closeAfterError(lease: PitLease, original: Throwable): Mono = + closePit(lease).onErrorResume { closeError -> + original.addSuppressed(closeError) + Mono.empty() + } + + private fun closeAfterCancel(lease: PitLease): Mono = closePit(lease).onErrorResume { Mono.empty() } + + private fun QueryBackendExecutionOptions.remainingMillis(): Long? { + val currentDeadline = deadline ?: return null + val now = clock.instant() + val remaining = try { + Duration.between(now, currentDeadline).toMillis() + } catch (error: ArithmeticException) { + if (currentDeadline.isAfter(now)) { + Long.MAX_VALUE + } else { + throw QueryBackendException(QueryBackendFailureKind.TIMEOUT, error) + } + } + if (remaining <= 0) { + throw QueryBackendException(QueryBackendFailureKind.TIMEOUT) + } + return remaining + } + + private inner class PageState( + val lease: PitLease, + val plan: BackendPageQueryPlan, + val compiled: ElasticsearchCompiledRecordQuery, + val options: QueryBackendExecutionOptions, + val maxCursorPages: Int, + ) { + val records = mutableListOf() + var total: Long? = null + var consumed: Long = 0 + var cursorPages: Int = 0 + var searchAfter: List? = null + + fun isComplete(): Boolean = records.size == plan.page.size || total?.let { consumed >= it } == true + + fun nextBatchSize(): Int { + val endExclusive = Math.addExact(plan.page.offset, plan.page.size.toLong()) + return minOf(PIT_BATCH_SIZE.toLong(), endExclusive - consumed).toInt().coerceAtLeast(1) + } + + fun accept(response: ResponseBody>) { + validateResponse(response) + response.pitId()?.takeIf(String::isNotBlank)?.let(lease::update) + val currentTotal = response.hits().total() ?: incomplete() + if (currentTotal.relation() != TotalHitsRelation.Eq) { + incomplete() + } + if (total != null && total != currentTotal.value()) { + incomplete() + } + total = currentTotal.value() + val hits = response.hits().hits() + hits.forEachIndexed { index, hit -> + val position = consumed + index + if (position >= plan.page.offset && records.size < plan.page.size) { + records += hit.toRecord(plan) + } + } + consumed += hits.size + cursorPages++ + if (!isComplete()) { + val last = hits.lastOrNull() ?: incomplete() + if (last.sort().isEmpty()) { + incomplete() + } + searchAfter = last.sort() + } + } + + fun toPage(): BackendPage { + val exactTotal = total ?: incomplete() + val expected = minOf( + plan.page.size.toLong(), + (exactTotal - plan.page.offset).coerceAtLeast(0), + ) + if (records.size.toLong() != expected) { + incomplete() + } + return BackendPage( + records, + exactTotal, + BackendTotalRelation.EXACT, + BackendPageConsistency.SAME_INPUT, + ) + } + } + + private fun validateResponse(response: ResponseBody<*>) { + if (response.timedOut()) { + throw QueryBackendException(QueryBackendFailureKind.TIMEOUT) + } + if (response.shards().failed() != 0) { + incomplete() + } + } + + private fun mapPitSearchError(error: Throwable): Throwable = when { + error is QueryBackendException -> error + error.isMissingElasticsearchSearchContext() -> + QueryBackendException(QueryBackendFailureKind.INCOMPLETE_RESULT, error) + else -> error + } + + private fun mapPitLifecycleError(error: Throwable): Throwable { + val classified = error.findBackendFailure() ?: return error + if (classified === error) return error + return QueryBackendException(classified.kind, error) + } + + private fun Throwable.findBackendFailure(): QueryBackendException? { + val visited = Collections.newSetFromMap(IdentityHashMap()) + var current: Throwable? = this + while (current != null && visited.add(current)) { + if (current is QueryBackendException) return current + current = current.cause + } + return null + } + + @Suppress("UNCHECKED_CAST") + private fun Hit<*>.toRecord(plan: BackendPageQueryPlan): BackendRecord { + if (ignored().isNotEmpty()) { + incomplete() + } + val source = source() as? Map ?: incomplete() + val identity = id() ?: incomplete() + return mapper.map(identity, source, plan.projection) + } + + private fun unsupported(): Nothing = throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED) + + private fun budgetExceeded(): Nothing = throw QueryBackendException(QueryBackendFailureKind.BUDGET_EXCEEDED) + + private fun incomplete(): Nothing = throw QueryBackendException(QueryBackendFailureKind.INCOMPLETE_RESULT) + + private class PitLease(initialId: String) { + var id: String = initialId + private set + + fun update(newId: String) { + id = newId + } + } + + private companion object { + const val PIT_KEEP_ALIVE = "1m" + const val PIT_BATCH_SIZE = 1_000 + const val DEFAULT_MAX_CURSOR_PAGES = 1_024 + } +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchQueryBinding.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchQueryBinding.kt new file mode 100644 index 00000000000..ac01464dda2 --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchQueryBinding.kt @@ -0,0 +1,765 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.planned + +import co.elastic.clients.elasticsearch._types.mapping.DocValuesPropertyBase +import co.elastic.clients.elasticsearch._types.mapping.NumberPropertyBase +import co.elastic.clients.elasticsearch._types.mapping.Property +import co.elastic.clients.elasticsearch._types.mapping.PropertyBase +import co.elastic.clients.elasticsearch._types.mapping.TypeMapping +import me.ahoo.wow.elasticsearch.IndexNameConverter.toSnapshotIndexName +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.ExperimentalQueryBackendApi +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.RecordQueryBackendContribution +import me.ahoo.wow.query.backend.SearchScopeId +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.serialization.state.StateAggregateRecords +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Mono +import java.io.ByteArrayOutputStream +import java.io.DataOutputStream +import java.security.MessageDigest +import java.util.Collections +import java.util.LinkedHashMap + +@ExperimentalQueryBackendApi +enum class ElasticsearchValueEncoding { + DEFAULT, + EPOCH_MILLIS, +} + +@ExperimentalQueryBackendApi +data class ElasticsearchKeywordReadiness( + val maximumCharacters: Int, + val maximumUtf8Bytes: Int, + val historicalValuesAudited: Boolean, + val writeConstraintEnforced: Boolean, +) { + init { + require(maximumCharacters > 0) { "Elasticsearch keyword maximum characters must be positive." } + require(maximumUtf8Bytes > 0) { "Elasticsearch keyword maximum UTF-8 bytes must be positive." } + require(historicalValuesAudited) { + "Elasticsearch keyword capability requires a completed historical value audit." + } + require(writeConstraintEnforced) { + "Elasticsearch keyword capability requires an enforced write-side length constraint." + } + } +} + +@ExperimentalQueryBackendApi +data class ElasticsearchGroupReadiness( + val historicalValuesAudited: Boolean, +) { + init { + require(historicalValuesAudited) { + "Elasticsearch group capability requires a historical logical-value audit." + } + } +} + +@ExperimentalQueryBackendApi +class ElasticsearchFieldBinding( + val sourceField: String, + capabilities: Set, + val exactField: String? = null, + val presenceField: String? = null, + val rangeField: String? = null, + val searchField: String? = null, + val searchAnalyzer: String? = null, + val literalField: String? = null, + val sortField: String? = null, + val groupField: String? = null, + val groupReadiness: ElasticsearchGroupReadiness? = null, + val nestedPath: String? = null, + val valueEncoding: ElasticsearchValueEncoding = ElasticsearchValueEncoding.DEFAULT, + val keywordReadiness: ElasticsearchKeywordReadiness? = null, +) { + val capabilities: Set = Collections.unmodifiableSet( + LinkedHashSet(capabilities.sortedBy(FieldCapability::name)), + ) + + init { + physicalFields().forEach(::requirePhysicalPath) + requireRole(FieldCapability.EXACT, exactField) + requireRole(FieldCapability.PRESENCE, presenceField) + requireRole(FieldCapability.RANGE, rangeField) + requireRole(FieldCapability.FULL_TEXT, searchField) + require(FieldCapability.FULL_TEXT !in capabilities || !searchAnalyzer.isNullOrBlank()) { + "Elasticsearch FULL_TEXT capability requires an explicit search analyzer." + } + requireRole(FieldCapability.LITERAL_PATTERN, literalField) + requireRole(FieldCapability.SORTABLE, sortField) + requireRole(FieldCapability.AGGREGATABLE, groupField) + require(FieldCapability.AGGREGATABLE !in capabilities || groupReadiness != null) { + "Elasticsearch AGGREGATABLE capability requires explicit group readiness." + } + requireRole(FieldCapability.ELEMENT_MATCH, nestedPath) + } + + private fun requireRole(capability: FieldCapability, field: String?) { + require(capability !in capabilities || field != null) { + "Elasticsearch capability $capability requires an explicit physical field role." + } + } + + private fun physicalFields(): List = listOfNotNull( + sourceField, + exactField, + presenceField, + rangeField, + searchField, + literalField, + sortField, + groupField, + nestedPath, + ) +} + +@ExperimentalQueryBackendApi +class ElasticsearchSearchScopeBinding( + val scope: SearchScopeId, + fields: Map, +) { + val fields: Map + + init { + require(fields.isNotEmpty()) { "Elasticsearch search scope fields must not be empty." } + val copy = LinkedHashMap(fields.size) + fields.entries.sortedBy { entry -> entry.key.toString() }.forEach { entry -> + requirePhysicalPath(entry.value) + copy[entry.key] = entry.value + } + this.fields = Collections.unmodifiableMap(copy) + } +} + +@ExperimentalQueryBackendApi +class ElasticsearchQueryBackendNotReadyException( + message: String, + cause: Throwable? = null, +) : RuntimeException(message, cause) + +@ExperimentalQueryBackendApi +class ElasticsearchSnapshotQueryBinding( + val schema: QueryDocumentSchema, + val indexName: String, + val mappingVersion: String, + fields: Map, + searchScopes: Iterable = emptyList(), + val backendId: BackendId = BackendId("elasticsearch"), +) { + internal val prepared = prepareBinding(schema, indexName, mappingVersion, fields, searchScopes, backendId) + val fields: Map = prepared.fields + val searchScopes: Map = prepared.searchScopes +} + +@ExperimentalQueryBackendApi +fun ElasticsearchSnapshotQueryBinding.prepareContribution( + client: ReactiveElasticsearchClient, +): Mono = client.indices().getMapping { request -> request.index(indexName) } + .switchIfEmpty( + Mono.error( + ElasticsearchQueryBackendNotReadyException( + "Elasticsearch index [$indexName] returned no mapping response.", + ), + ), + ).onErrorMap { error -> + if (error is ElasticsearchQueryBackendNotReadyException) { + error + } else { + ElasticsearchQueryBackendNotReadyException( + "Elasticsearch mapping readiness for index [$indexName] could not be inspected.", + error, + ) + } + }.map { response -> + prepared.attestReadiness(response.mappings().mapValues { (_, record) -> record.mappings() }) + prepared.createContribution(client) + } + +internal class ElasticsearchPreparedQueryBinding( + val schema: QueryDocumentSchema, + val indexName: String, + val mappingVersion: String, + val fields: Map, + val searchScopes: Map, + val backendId: BackendId, +) { + val capabilityDigest: String = ElasticsearchCapabilityDigestEncoder.encode(this) + + fun attestReadiness(mappings: Map) { + if (mappings.isEmpty()) { + notReady("Elasticsearch index [$indexName] has no concrete mapping.") + } + mappings.entries.sortedBy(Map.Entry::key).forEach { (index, mapping) -> + attestMapping(index, mapping) + } + } + + fun createContribution(client: ReactiveElasticsearchClient): RecordQueryBackendContribution = + RecordQueryBackendContribution( + schema, + backendId, + setOf( + QueryOperation.SINGLE, + QueryOperation.STREAM, + QueryOperation.PAGE, + QueryOperation.COUNT, + QueryOperation.ANALYZE, + ), + me.ahoo.wow.query.backend.BackendStreamSupport.BOUNDED_ONLY, + buildSet { + add(SemanticTier.PORTABLE) + if (searchScopes.isNotEmpty()) add(SemanticTier.SEARCH) + }, + fields.mapValues { (_, field) -> field.capabilities }, + searchScopes.keys, + ElasticsearchSnapshotRecordQueryBackend(client, this), + ElasticsearchAnalyticsQueryBackend(client, this), + capabilityDigest, + ) + + private fun attestMapping(index: String, mapping: TypeMapping) { + val actualVersion = mapping.requiredMeta(index, ELASTICSEARCH_QUERY_MAPPING_VERSION_META) + if (actualVersion != mappingVersion) { + notReady("Elasticsearch index [$index] mapping version [$actualVersion] does not match [$mappingVersion].") + } + requireMeta(index, mapping, ELASTICSEARCH_QUERY_DOCUMENT_KIND_META, schema.target.documentKind.name) + requireMeta(index, mapping, ELASTICSEARCH_QUERY_SCHEMA_CONTRACT_META, schema.contractId.value) + requireMeta(index, mapping, ELASTICSEARCH_QUERY_CAPABILITY_DIGEST_META, capabilityDigest) + val properties = flattenProperties(mapping.properties()) + fields.forEach { (field, binding) -> attestField(index, field, binding, properties) } + searchScopes.values.forEach { scope -> + scope.fields.forEach { (field, searchField) -> + requireSearchField( + index, + searchField, + fields.getValue(field).searchAnalyzer, + properties, + ) + } + } + } + + private fun attestField( + index: String, + field: QueryFieldId, + binding: ElasticsearchFieldBinding, + properties: Map, + ) { + requireSourceField(index, binding.sourceField, properties) + binding.exactField?.let { physical -> + if (physical != ES_ID_FIELD) { + requireExactField(index, field, binding, physical, properties) + } + } + binding.presenceField?.let { physical -> + requirePresenceField(index, physical, properties) + } + binding.rangeField?.let { physical -> requireRangeField(index, field, physical, properties) } + binding.searchField?.let { physical -> + requireSearchField(index, physical, binding.searchAnalyzer, properties) + } + binding.literalField?.let { physical -> requireLiteralField(index, field, binding, physical, properties) } + binding.sortField?.let { physical -> + requireOrderedBucketField(index, field, binding, physical, properties) + } + binding.groupField?.let { physical -> + requireOrderedBucketField(index, field, binding, physical, properties) + requireNullBucketField(index, physical, properties) + } + binding.nestedPath?.let { physical -> + val property = requireProperty(index, physical, properties) + if (!property.isNested) notReady("Elasticsearch field [$physical] in index [$index] is not nested.") + } + } + + private fun requireSourceField(index: String, physical: String, properties: Map) { + if (physical != ES_ID_FIELD) requireProperty(index, physical, properties) + } + + private fun requireExactField( + index: String, + field: QueryFieldId, + binding: ElasticsearchFieldBinding, + physical: String, + properties: Map, + ) { + val property = requireProperty(index, physical, properties) + when (schema.fields.getValue(field).type.operandType()) { + LogicalFieldType.Text -> { + if (!property.isKeyword && !property.isConstantKeyword) { + notReady("Elasticsearch exact field [$physical] in index [$index] is not keyword.") + } + requireKeywordCompleteness(index, physical, property, binding) + } + + LogicalFieldType.Boolean -> if (!property.isBoolean) typeMismatch(index, physical) + LogicalFieldType.Int64 -> if (!property.isIntegralNumber()) typeMismatch(index, physical) + LogicalFieldType.Instant -> if (!property.isLong || binding.valueEncoding != ElasticsearchValueEncoding.EPOCH_MILLIS) { + typeMismatch(index, physical) + } + + LogicalFieldType.Decimal, + LogicalFieldType.Bytes, + LogicalFieldType.Object, + is LogicalFieldType.Array, + -> notReady("Elasticsearch exact field [$physical] in index [$index] has no portable binding.") + } + requireIndexed(property, index, physical) + } + + private fun requireRangeField( + index: String, + field: QueryFieldId, + physical: String, + properties: Map, + ) { + val property = requireProperty(index, physical, properties) + when (schema.fields.getValue(field).type.operandType()) { + LogicalFieldType.Int64 -> if (!property.isIntegralNumber()) typeMismatch(index, physical) + LogicalFieldType.Instant -> if (!property.isLong || fields.getValue(field).valueEncoding != + ElasticsearchValueEncoding.EPOCH_MILLIS + ) { + typeMismatch(index, physical) + } + + else -> notReady("Elasticsearch range field [$physical] in index [$index] has no portable binding.") + } + requireIndexed(property, index, physical) + } + + private fun requireLiteralField( + index: String, + field: QueryFieldId, + binding: ElasticsearchFieldBinding, + physical: String, + properties: Map, + ) { + val property = requireProperty(index, physical, properties) + if (!property.isKeyword && !property.isWildcard) { + typeMismatch(index, physical) + } + if (property.isKeyword) requireKeywordCompleteness(index, physical, property, binding) + requireIndexed(property, index, physical) + if (schema.fields.getValue(field).type.operandType() != LogicalFieldType.Text) typeMismatch(index, physical) + } + + private fun requireKeywordCompleteness( + index: String, + physical: String, + property: Property, + binding: ElasticsearchFieldBinding, + ) { + val readiness = binding.keywordReadiness + ?: notReady("Elasticsearch keyword field [$physical] in index [$index] lacks completeness attestation.") + val ignoreAbove = (property._get() as PropertyBase).ignoreAbove() + if (ignoreAbove != null && ignoreAbove < readiness.maximumCharacters) { + notReady( + "Elasticsearch keyword field [$physical] in index [$index] ignore_above[$ignoreAbove] " + + "is below the attested maximum[${readiness.maximumCharacters}].", + ) + } + if (property.isKeyword && property.keyword().normalizer() != null) { + notReady( + "Elasticsearch keyword field [$physical] in index [$index] normalizer" + + "[${property.keyword().normalizer()}] is not portable.", + ) + } + } + + private fun requireSearchField( + index: String, + physical: String, + expectedAnalyzer: String?, + properties: Map, + ) { + val property = requireProperty(index, physical, properties) + if (!property.isText) { + notReady("Elasticsearch search field [$physical] in index [$index] is not explicitly analyzed text.") + } + val actualAnalyzer = property.text().analyzer() + val actualSearchAnalyzer = property.text().searchAnalyzer() ?: actualAnalyzer + if (actualAnalyzer != expectedAnalyzer || actualSearchAnalyzer != expectedAnalyzer) { + notReady( + "Elasticsearch search field [$physical] in index [$index] analyzer[$actualAnalyzer] " + + "search_analyzer[$actualSearchAnalyzer] does not match [$expectedAnalyzer].", + ) + } + requireIndexed(property, index, physical) + } + + private fun requirePresenceField( + index: String, + physical: String, + properties: Map, + ) { + val property = requireProperty(index, physical, properties) + if (!property.isBoolean) { + notReady("Elasticsearch presence field [$physical] in index [$index] must be a boolean marker.") + } + requireIndexed(property, index, physical) + } + + private fun requireDocValuesField(index: String, physical: String, properties: Map) { + val property = requireProperty(index, physical, properties) + val docValues = property._get() as? DocValuesPropertyBase + ?: notReady("Elasticsearch field [$physical] in index [$index] has no doc values contract.") + if (docValues.docValues() == false) { + notReady("Elasticsearch field [$physical] in index [$index] disables doc values.") + } + } + + private fun requireNullBucketField(index: String, physical: String, properties: Map) { + val property = requireProperty(index, physical, properties) + val nullValue = when { + property.isKeyword -> property.keyword().nullValue() + property.isBoolean -> property.boolean_().nullValue() + property.isByte -> property.byte_().nullValue() + property.isShort -> property.short_().nullValue() + property.isInteger -> property.integer().nullValue() + property.isLong -> property.long_().nullValue() + else -> null + } + if (nullValue != null) { + notReady( + "Elasticsearch group field [$physical] in index [$index] indexes null_value[$nullValue]; " + + "missing and explicit null would form different composite buckets.", + ) + } + } + + private fun requireOrderedBucketField( + index: String, + field: QueryFieldId, + binding: ElasticsearchFieldBinding, + physical: String, + properties: Map, + ) { + val property = requireProperty(index, physical, properties) + when (schema.fields.getValue(field).type.operandType()) { + LogicalFieldType.Text -> { + if (!property.isKeyword && !property.isConstantKeyword) { + typeMismatch(index, physical) + } + requireKeywordCompleteness(index, physical, property, binding) + } + + LogicalFieldType.Boolean -> if (!property.isBoolean) typeMismatch(index, physical) + LogicalFieldType.Int64 -> if (!property.isIntegralNumber()) typeMismatch(index, physical) + LogicalFieldType.Instant -> if (!property.isLong || + binding.valueEncoding != ElasticsearchValueEncoding.EPOCH_MILLIS + ) { + typeMismatch(index, physical) + } + + LogicalFieldType.Decimal, + LogicalFieldType.Bytes, + LogicalFieldType.Object, + is LogicalFieldType.Array, + -> notReady("Elasticsearch field [$physical] in index [$index] has no portable sort/group binding.") + } + requireDocValuesField(index, physical, properties) + } + + private fun requireIndexed(property: Property, index: String, physical: String) { + val indexed = when { + property.isKeyword -> property.keyword().index() + property.isText -> property.text().index() + property.isBoolean -> property.boolean_().index() + property._get() is NumberPropertyBase -> (property._get() as NumberPropertyBase).index() + else -> notReady("Elasticsearch field [$physical] in index [$index] has no indexed leaf contract.") + } + if (indexed == false) notReady("Elasticsearch field [$physical] in index [$index] is not indexed.") + } + + private fun requireProperty(index: String, physical: String, properties: Map): Property = + properties[physical] ?: notReady("Elasticsearch field [$physical] is missing from index [$index].") + + private fun typeMismatch(index: String, physical: String): Nothing = + notReady("Elasticsearch field [$physical] in index [$index] has an incompatible mapping type.") + + private fun requireMeta( + index: String, + mapping: TypeMapping, + key: String, + expected: String, + ) { + val actual = mapping.requiredMeta(index, key) + if (actual != expected) { + notReady("Elasticsearch index [$index] metadata [$key] value [$actual] does not match [$expected].") + } + } +} + +private fun TypeMapping.requiredMeta(index: String, key: String): String = meta()[key]?.let { value -> + runCatching { value.to(String::class.java) }.getOrElse { error -> + throw ElasticsearchQueryBackendNotReadyException( + "Elasticsearch index [$index] has unreadable Query metadata [$key].", + error, + ) + } +} ?: notReady("Elasticsearch index [$index] is missing Query metadata [$key].") + +private object ElasticsearchCapabilityDigestEncoder { + fun encode(binding: ElasticsearchPreparedQueryBinding): String { + val bytes = ByteArrayOutputStream() + DataOutputStream(bytes).use { output -> + output.writeUtf8("wow-elasticsearch-query-capability-v1") + output.writeUtf8(binding.schema.target.namedAggregate.contextName) + output.writeUtf8(binding.schema.target.namedAggregate.aggregateName) + output.writeUtf8(binding.schema.target.documentKind.name) + output.writeUtf8(binding.schema.contractId.value) + output.writeUtf8(binding.backendId.value) + output.writeUtf8(binding.indexName) + output.writeInt(binding.fields.size) + binding.fields.forEach { (field, definition) -> + output.writeFieldId(field) + output.writeFieldBinding(definition) + } + output.writeInt(binding.searchScopes.size) + binding.searchScopes.forEach { (scope, definition) -> + output.writeUtf8(scope.value) + output.writeInt(definition.fields.size) + definition.fields.forEach { (field, physical) -> + output.writeFieldId(field) + output.writeUtf8(physical) + } + } + } + return MessageDigest.getInstance("SHA-256").digest(bytes.toByteArray()).toHex() + } + + private fun DataOutputStream.writeFieldBinding(binding: ElasticsearchFieldBinding) { + writeUtf8(binding.sourceField) + writeStrings(binding.capabilities.map(FieldCapability::name).sorted()) + writeOptional(binding.exactField) + writeOptional(binding.presenceField) + writeOptional(binding.rangeField) + writeOptional(binding.searchField) + writeOptional(binding.searchAnalyzer) + writeOptional(binding.literalField) + writeOptional(binding.sortField) + writeOptional(binding.groupField) + writeBoolean(binding.groupReadiness != null) + binding.groupReadiness?.let { writeBoolean(it.historicalValuesAudited) } + writeOptional(binding.nestedPath) + writeUtf8(binding.valueEncoding.name) + writeBoolean(binding.keywordReadiness != null) + binding.keywordReadiness?.let { readiness -> + writeInt(readiness.maximumCharacters) + writeInt(readiness.maximumUtf8Bytes) + writeBoolean(readiness.historicalValuesAudited) + writeBoolean(readiness.writeConstraintEnforced) + } + } + + private fun DataOutputStream.writeFieldId(field: QueryFieldId) { + when (field) { + is QueryFieldId.System -> { + writeByte(0) + writeUtf8(field.kind.name) + } + + is QueryFieldId.Path -> { + writeByte(1) + writeStrings(field.segments) + } + } + } + + private fun DataOutputStream.writeStrings(values: List) { + writeInt(values.size) + values.forEach { value -> writeUtf8(value) } + } + + private fun DataOutputStream.writeOptional(value: String?) { + writeBoolean(value != null) + value?.let { present -> writeUtf8(present) } + } + + private fun DataOutputStream.writeUtf8(value: String) { + val bytes = value.toByteArray(Charsets.UTF_8) + writeInt(bytes.size) + write(bytes) + } + + private fun ByteArray.toHex(): String = joinToString("") { byte -> "%02x".format(byte) } +} + +private fun prepareBinding( + schema: QueryDocumentSchema, + indexName: String, + mappingVersion: String, + fields: Map, + searchScopes: Iterable, + backendId: BackendId, +): ElasticsearchPreparedQueryBinding { + require(schema.target.documentKind == QueryDocumentKind.SNAPSHOT) { + "Elasticsearch Snapshot query binding requires a Snapshot target." + } + requirePhysicalPath(indexName) + require(indexName == schema.target.namedAggregate.toSnapshotIndexName()) { + "Elasticsearch Snapshot query binding index[$indexName] does not match target[${schema.target}]." + } + require(mappingVersion.isNotBlank() && mappingVersion.none(Char::isISOControl)) { + "Elasticsearch Query mapping version must not be blank or contain control characters." + } + val fieldCopy = LinkedHashMap(fields.size) + fields.entries.sortedBy { entry -> entry.key.toString() }.forEach { (field, binding) -> + val schemaField = requireNotNull(schema.fields[field]) { + "Elasticsearch field $field is not declared by the logical schema." + } + require(schemaField.capabilities.containsAll(binding.capabilities)) { + "Elasticsearch field $field overclaims the logical schema contract." + } + requireValidEncoding(schemaField.type, binding.valueEncoding) + if (field is QueryFieldId.Path) { + require(binding.sourceField == field.toString()) { + "Elasticsearch record materialization requires logical and source user paths to match." + } + } + fieldCopy[field] = binding + } + requireSystemBindings(schema, fieldCopy) + require(fieldCopy.keys == schema.fields.keys) { + "Elasticsearch query binding must cover every field in the logical schema." + } + val scopeCopy = prepareSearchScopes(schema, fieldCopy, searchScopes) + return ElasticsearchPreparedQueryBinding( + schema, + indexName, + mappingVersion, + Collections.unmodifiableMap(fieldCopy), + scopeCopy, + backendId, + ) +} + +private fun prepareSearchScopes( + schema: QueryDocumentSchema, + fields: Map, + searchScopes: Iterable, +): Map { + val prepared = LinkedHashMap() + searchScopes.forEach { scope -> + val definition = requireNotNull(schema.searchScopes[scope.scope]) { + "Elasticsearch search scope ${scope.scope} is not declared by the logical schema." + } + require(scope.fields.keys == definition.fields.toSet()) { + "Elasticsearch search scope ${scope.scope} must bind every declared field exactly once." + } + scope.fields.forEach { (field, physical) -> + val fieldBinding = requireNotNull(fields[field]) { + "Elasticsearch search field $field is not bound." + } + require(fieldBinding.searchField == physical && FieldCapability.FULL_TEXT in fieldBinding.capabilities) { + "Elasticsearch search scope ${scope.scope} must use the field FULL_TEXT role." + } + } + require(prepared.put(scope.scope, scope) == null) { + "Elasticsearch search scope ${scope.scope} must be unique." + } + } + return Collections.unmodifiableMap(prepared) +} + +private fun requireSystemBindings( + schema: QueryDocumentSchema, + fields: Map, +) { + SNAPSHOT_SYSTEM_FIELDS.forEach { (kind, physical) -> + val id = QueryFieldId.System(kind) + if (id in schema.fields && id in fields) { + val binding = fields.getValue(id) + require(binding.sourceField == physical.source && binding.exactField == physical.exact) { + "Elasticsearch Snapshot system field $kind must bind source[${physical.source}] exact[${physical.exact}]." + } + } + } + require(fields[QueryFieldId.System(SystemFieldKind.IDENTITY)]?.exactField == ES_ID_FIELD) { + "Elasticsearch Snapshot identity must bind exact queries to $ES_ID_FIELD." + } +} + +private fun requireValidEncoding(type: LogicalFieldType, encoding: ElasticsearchValueEncoding) { + val operand = type.operandType() + require( + when (operand) { + LogicalFieldType.Instant -> encoding == ElasticsearchValueEncoding.EPOCH_MILLIS + else -> encoding == ElasticsearchValueEncoding.DEFAULT + }, + ) { + "Elasticsearch value encoding $encoding does not match logical field type $type." + } +} + +private fun flattenProperties(properties: Map): Map { + val result = LinkedHashMap() + fun visit(prefix: String?, current: Map) { + current.entries.sortedBy(Map.Entry::key).forEach { (name, property) -> + val path = if (prefix == null) name else "$prefix.$name" + result[path] = property + val base = property._get() as? PropertyBase + base?.properties()?.let { children -> visit(path, children) } + base?.fields()?.let { fields -> visit(path, fields) } + } + } + visit(null, properties) + return result +} + +private fun Property.isIntegralNumber(): Boolean = isByte || isShort || isInteger || isLong || isUnsignedLong + +private fun LogicalFieldType.operandType(): LogicalFieldType = + if (this is LogicalFieldType.Array) elementType else this + +private fun requirePhysicalPath(path: String) { + require(path.isNotBlank()) { "Elasticsearch physical path must not be blank." } + require(path.none(Char::isISOControl)) { "Elasticsearch physical path must not contain control characters." } + require(path.split('.').none(String::isBlank)) { "Elasticsearch physical path segments must not be blank." } +} + +private fun notReady(message: String): Nothing = throw ElasticsearchQueryBackendNotReadyException(message) + +private data class SystemPhysicalFields(val source: String, val exact: String) + +private const val ES_ID_FIELD = "_id" +internal const val ELASTICSEARCH_QUERY_MAPPING_VERSION_META = "wow_query_mapping_version" +internal const val ELASTICSEARCH_QUERY_DOCUMENT_KIND_META = "wow_query_document_kind" +internal const val ELASTICSEARCH_QUERY_SCHEMA_CONTRACT_META = "wow_query_schema_contract_id" +internal const val ELASTICSEARCH_QUERY_CAPABILITY_DIGEST_META = "wow_query_capability_digest" + +private val SNAPSHOT_SYSTEM_FIELDS = mapOf( + SystemFieldKind.IDENTITY to SystemPhysicalFields(MessageRecords.AGGREGATE_ID, ES_ID_FIELD), + SystemFieldKind.AGGREGATE_ID to SystemPhysicalFields(MessageRecords.AGGREGATE_ID, ES_ID_FIELD), + SystemFieldKind.TENANT_ID to SystemPhysicalFields(MessageRecords.TENANT_ID, MessageRecords.TENANT_ID), + SystemFieldKind.OWNER_ID to SystemPhysicalFields(MessageRecords.OWNER_ID, MessageRecords.OWNER_ID), + SystemFieldKind.SPACE_ID to SystemPhysicalFields(MessageRecords.SPACE_ID, MessageRecords.SPACE_ID), + SystemFieldKind.DELETED to SystemPhysicalFields(StateAggregateRecords.DELETED, StateAggregateRecords.DELETED), +) diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchRecordQueryCompiler.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchRecordQueryCompiler.kt new file mode 100644 index 00000000000..8e10e25431f --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchRecordQueryCompiler.kt @@ -0,0 +1,378 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.planned + +import co.elastic.clients.elasticsearch._types.FieldValue +import co.elastic.clients.elasticsearch._types.SortOptions +import co.elastic.clients.elasticsearch._types.SortOrder +import co.elastic.clients.elasticsearch._types.query_dsl.Query +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.bool +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.exists +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.ids +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.match +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.matchAll +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.matchNone +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.nested +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.range +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.term +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.terms +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.termsSet +import co.elastic.clients.elasticsearch._types.query_dsl.QueryBuilders.wildcard +import co.elastic.clients.elasticsearch.core.search.SourceFilter +import co.elastic.clients.json.JsonData +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendPageQueryPlan +import me.ahoo.wow.query.backend.BackendPageWindow +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.BackendRecordQueryPlan +import me.ahoo.wow.query.backend.BackendRecordResultPlan +import me.ahoo.wow.query.backend.BackendSingleQueryPlan +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.CaseSensitivity +import me.ahoo.wow.query.backend.JunctionOperator +import me.ahoo.wow.query.backend.NormalizedSortDirection +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.SystemFieldKind + +internal data class ElasticsearchCompiledRecordQuery( + val query: Query, + val sourceFilter: SourceFilter?, + val sort: List, + val limit: Int?, + val page: BackendPageWindow?, +) + +internal class ElasticsearchRecordQueryCompiler( + private val binding: ElasticsearchPreparedQueryBinding, +) { + constructor(binding: ElasticsearchSnapshotQueryBinding) : this(binding.prepared) + + fun compile(plan: BackendRecordQueryPlan): ElasticsearchCompiledRecordQuery { + if (plan.target != binding.schema.target || plan.schemaContractId != binding.schema.contractId) { + unsupported() + } + val result = plan as? BackendRecordResultPlan + return ElasticsearchCompiledRecordQuery( + compileCondition(plan.filter.condition), + result?.projection?.let(::compileProjection), + result?.sort?.map { sort -> + SortOptions.of { option -> + option.field { field -> + field.field(requireField(sort.field).sortField ?: unsupported()) + .order( + when (sort.direction) { + NormalizedSortDirection.ASC -> SortOrder.Asc + NormalizedSortDirection.DESC -> SortOrder.Desc + }, + ) + } + } + }.orEmpty(), + when (plan) { + is BackendSingleQueryPlan -> 1 + is BackendStreamQueryPlan -> plan.limit + is BackendPageQueryPlan, + is BackendCountQueryPlan, + -> null + }, + (plan as? BackendPageQueryPlan)?.page, + ) + } + + internal fun compileCondition(condition: BackendPlannedCondition): Query = + when (condition) { + BackendPlannedCondition.All -> matchAll { it } + BackendPlannedCondition.None -> matchNone { it } + is BackendPlannedCondition.Junction -> compileJunction(condition) + is BackendPlannedCondition.Predicate -> compilePredicate(condition) + is BackendPlannedCondition.ElementMatch -> compileElementMatch(condition) + is BackendPlannedCondition.Search -> compileSearch(condition) + is BackendPlannedCondition.Native -> unsupported() + } + + private fun compileJunction(condition: BackendPlannedCondition.Junction): Query { + val children = condition.children.map(::compileCondition) + return when (condition.operator) { + JunctionOperator.AND -> bool { builder -> builder.filter(children) } + JunctionOperator.OR -> bool { builder -> builder.should(children).minimumShouldMatch("1") } + JunctionOperator.NOR -> bool { builder -> builder.mustNot(children) } + } + } + + @Suppress("CyclomaticComplexMethod") + private fun compilePredicate(predicate: BackendPlannedCondition.Predicate): Query { + if (predicate.options.caseSensitivity != CaseSensitivity.SENSITIVE) { + unsupported() + } + return when (predicate.operator) { + PredicateOperator.EQ -> compileEqual(predicate) + PredicateOperator.NE -> negate(compileEqual(predicate)) + PredicateOperator.GT -> compileRange(predicate, RangeBound.GT) + PredicateOperator.LT -> compileRange(predicate, RangeBound.LT) + PredicateOperator.GTE -> compileRange(predicate, RangeBound.GTE) + PredicateOperator.LTE -> compileRange(predicate, RangeBound.LTE) + PredicateOperator.CONTAINS -> compileLiteral(predicate, LiteralKind.CONTAINS) + PredicateOperator.IN -> compileIn(predicate, negated = false) + PredicateOperator.NOT_IN -> compileIn(predicate, negated = true) + PredicateOperator.BETWEEN -> compileBetween(predicate) + PredicateOperator.ALL_IN -> compileAllIn(predicate) + PredicateOperator.STARTS_WITH -> compileLiteral(predicate, LiteralKind.STARTS_WITH) + PredicateOperator.ENDS_WITH -> compileLiteral(predicate, LiteralKind.ENDS_WITH) + PredicateOperator.IS_NULL -> compileNull(predicate) + PredicateOperator.NOT_NULL -> negate(compileNull(predicate)) + PredicateOperator.IS_TRUE -> compileTerm(predicate, NormalizedValue.BooleanValue(true)) + PredicateOperator.IS_FALSE -> compileTerm(predicate, NormalizedValue.BooleanValue(false)) + PredicateOperator.EXISTS -> { + val exists = predicate.value as? NormalizedValue.BooleanValue ?: unsupported() + if (exists.value) present(predicate) else missing(predicate) + } + } + } + + private fun compileEqual(predicate: BackendPlannedCondition.Predicate): Query { + val value = requireNotNull(predicate.value) + if (value == NormalizedValue.Null) { + return compileNull(predicate) + } + return compileTerm(predicate, value) + } + + private fun compileTerm(predicate: BackendPlannedCondition.Predicate, value: NormalizedValue): Query { + val physical = requireField(predicate.field).exactField ?: unsupported() + if (physical == ES_ID_FIELD) { + val text = value as? NormalizedValue.Text ?: unsupported() + return ids { builder -> builder.values(text.value) } + } + return term { builder -> builder.field(physical).value(value.toFieldValue(predicate.field)) } + } + + private fun compileIn(predicate: BackendPlannedCondition.Predicate, negated: Boolean): Query { + val values = predicate.value.requireList() + val nonNull = values.filterNot { value -> value == NormalizedValue.Null } + val containsNull = values.any { value -> value == NormalizedValue.Null } + val terms = if (nonNull.isEmpty()) { + matchNone { it } + } else { + compileTerms(predicate.field, nonNull) + } + val combined = if (!negated && values.any { value -> value == NormalizedValue.Null }) { + bool { builder -> builder.should(terms, compileNull(predicate)).minimumShouldMatch("1") } + } else { + terms + } + if (!negated) return combined + if (!containsNull) return negate(combined) + val nonNullValues = if (nonNull.isEmpty()) matchAll { it } else negate(terms) + val explicitNonNull = bool { builder -> builder.must(present(predicate), nonNullValues) } + return bool { builder -> builder.should(missing(predicate), explicitNonNull).minimumShouldMatch("1") } + } + + private fun compileTerms(field: QueryFieldId, values: List): Query { + val physical = requireField(field).exactField ?: unsupported() + if (physical == ES_ID_FIELD) { + return ids { builder -> + builder.values(values.map { value -> (value as? NormalizedValue.Text)?.value ?: unsupported() }) + } + } + return terms { builder -> + builder.field(physical).terms { terms -> + terms.value(values.map { value -> value.toFieldValue(field) }) + } + } + } + + private fun compileAllIn(predicate: BackendPlannedCondition.Predicate): Query { + val values = predicate.value.requireList() + if (values.any { value -> value == NormalizedValue.Null }) { + unsupported() + } + val physical = requireField(predicate.field).exactField ?: unsupported() + return termsSet { builder -> + builder.field(physical) + .terms(values.map { value -> value.toFieldValue(predicate.field) }) + .minimumShouldMatch(values.size.toString()) + } + } + + private fun compileBetween(predicate: BackendPlannedCondition.Predicate): Query { + val values = predicate.value.requireList() + if (values.size != 2 || values.any { value -> value == NormalizedValue.Null }) { + unsupported() + } + val physical = requireField(predicate.field).rangeField ?: unsupported() + return range { builder -> + builder.untyped { range -> + range.field(physical) + .gte(values[0].toJsonData(predicate.field)) + .lte(values[1].toJsonData(predicate.field)) + } + } + } + + private fun compileRange(predicate: BackendPlannedCondition.Predicate, bound: RangeBound): Query { + val value = requireNotNull(predicate.value) + if (value == NormalizedValue.Null) unsupported() + val physical = requireField(predicate.field).rangeField ?: unsupported() + return range { builder -> + builder.untyped { range -> + range.field(physical).also { + when (bound) { + RangeBound.GT -> range.gt(value.toJsonData(predicate.field)) + RangeBound.LT -> range.lt(value.toJsonData(predicate.field)) + RangeBound.GTE -> range.gte(value.toJsonData(predicate.field)) + RangeBound.LTE -> range.lte(value.toJsonData(predicate.field)) + } + } + } + } + } + + private fun compileLiteral( + predicate: BackendPlannedCondition.Predicate, + kind: LiteralKind, + ): Query { + val physical = requireField(predicate.field).literalField ?: unsupported() + val literal = (predicate.value as? NormalizedValue.Text)?.value ?: unsupported() + val escaped = literal.escapeWildcard() + val pattern = when (kind) { + LiteralKind.CONTAINS -> "*$escaped*" + LiteralKind.STARTS_WITH -> "$escaped*" + LiteralKind.ENDS_WITH -> "*$escaped" + } + return wildcard { builder -> builder.field(physical).value(pattern).caseInsensitive(false) } + } + + private fun present(predicate: BackendPlannedCondition.Predicate): Query { + val physical = requireField(predicate.field).presenceField ?: unsupported() + return term { builder -> builder.field(physical).value(true) } + } + + private fun missing(predicate: BackendPlannedCondition.Predicate): Query = negate(present(predicate)) + + private fun compileNull(predicate: BackendPlannedCondition.Predicate): Query { + val field = requireField(predicate.field) + val exact = field.exactField ?: unsupported() + return negate(exists { builder -> builder.field(exact) }) + } + + private fun compileElementMatch(condition: BackendPlannedCondition.ElementMatch): Query { + val field = requireField(condition.field) + val nestedPath = field.nestedPath ?: unsupported() + return nested { builder -> builder.path(nestedPath).query(compileCondition(condition.condition)) } + } + + private fun compileSearch(condition: BackendPlannedCondition.Search): Query { + val scope = binding.searchScopes[condition.scope] ?: unsupported() + val searches = scope.fields.values.map { physical -> + match { builder -> builder.field(physical).query(condition.text) } + } + return if (searches.size == 1) { + searches.single() + } else { + bool { builder -> builder.should(searches).minimumShouldMatch("1") } + } + } + + private fun compileProjection(projection: BackendProjection): SourceFilter? = + when (projection) { + BackendProjection.All -> null + is BackendProjection.Include -> SourceFilter.of { filter -> + filter.includes( + canonicalSourcePaths( + projection.fields + QueryFieldId.System(SystemFieldKind.IDENTITY), + ), + ) + } + + is BackendProjection.Exclude -> SourceFilter.of { filter -> + val identitySource = requireField(QueryFieldId.System(SystemFieldKind.IDENTITY)).sourceField + filter.excludes( + canonicalSourcePaths(projection.fields).filterNot { path -> path == identitySource }, + ) + } + } + + private fun canonicalSourcePaths(fields: List): List { + val paths = fields.map { field -> requireField(field).sourceField } + return paths.distinct().filter { candidate -> + paths.none { other -> other != candidate && candidate.startsWith("$other.") } + }.sorted() + } + + private fun requireField(field: QueryFieldId): ElasticsearchFieldBinding = + binding.fields[field] ?: unsupported() + + private fun NormalizedValue.toFieldValue(field: QueryFieldId): FieldValue = + when (this) { + is NormalizedValue.Text -> FieldValue.of(value) + is NormalizedValue.BooleanValue -> FieldValue.of(value) + is NormalizedValue.Int64 -> FieldValue.of(value) + is NormalizedValue.InstantValue -> { + if (requireField(field).valueEncoding != ElasticsearchValueEncoding.EPOCH_MILLIS) unsupported() + FieldValue.of(value.toEpochMilli()) + } + + NormalizedValue.Null, + is NormalizedValue.Decimal, + is NormalizedValue.Bytes, + is NormalizedValue.ListValue, + is NormalizedValue.ObjectValue, + -> unsupported() + } + + private fun NormalizedValue.toJsonData(field: QueryFieldId): JsonData = + JsonData.of(toFieldValue(field)._get()) + + private fun NormalizedValue?.requireList(): List = + (this as? NormalizedValue.ListValue)?.values ?: unsupported() + + private fun negate(query: Query): Query = bool { builder -> builder.mustNot(query) } + + private fun String.escapeWildcard(): String = buildString(length + 4) { + this@escapeWildcard.forEach { char -> + if (char in WILDCARD_META) append('\\') + append(char) + } + } + + private fun unsupported(): Nothing = throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED) + + private enum class RangeBound { + GT, + LT, + GTE, + LTE, + } + + private enum class LiteralKind { + CONTAINS, + STARTS_WITH, + ENDS_WITH, + } + + private companion object { + const val ES_ID_FIELD = "_id" + val WILDCARD_META = setOf('\\', '*', '?') + } +} diff --git a/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotRecordQueryBackend.kt b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotRecordQueryBackend.kt new file mode 100644 index 00000000000..348c3657f2f --- /dev/null +++ b/wow-elasticsearch/src/main/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotRecordQueryBackend.kt @@ -0,0 +1,517 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + +package me.ahoo.wow.elasticsearch.query.planned + +import co.elastic.clients.elasticsearch._types.ShardStatistics +import co.elastic.clients.elasticsearch.core.CountRequest +import co.elastic.clients.elasticsearch.core.SearchRequest +import co.elastic.clients.elasticsearch.core.search.Hit +import co.elastic.clients.elasticsearch.core.search.ResponseBody +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendPage +import me.ahoo.wow.query.backend.BackendPageQueryPlan +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.BackendRecord +import me.ahoo.wow.query.backend.BackendRecordCompleteness +import me.ahoo.wow.query.backend.BackendRecordQueryPlan +import me.ahoo.wow.query.backend.BackendSingleQueryPlan +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.RecordQueryBackend +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.serialization.state.StateAggregateRecords +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.math.BigDecimal +import java.math.BigInteger +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.util.IdentityHashMap +import java.util.LinkedHashMap + +internal class ElasticsearchSnapshotRecordQueryBackend( + private val client: ReactiveElasticsearchClient, + private val binding: ElasticsearchPreparedQueryBinding, + private val clock: Clock = Clock.systemUTC(), +) : RecordQueryBackend { + private val compiler = ElasticsearchRecordQueryCompiler(binding) + private val mapper = ElasticsearchSnapshotRecordMapper(binding) + private val pitPageExecutor = ElasticsearchPitPageExecutor(client, binding, compiler, mapper, clock) + + override fun single( + plan: BackendSingleQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = Mono.defer { + validateOptions(plan, options) + val compiled = compiler.compile(plan) + client.search(searchRequest(compiled, options, 0, 1, trackTotalHits = false), Map::class.java) + .flatMap { response -> + validateResponse(response) + if (response.hits().hits().size > 1) incomplete() + val hit = response.hits().hits().singleOrNull() + if (hit == null) Mono.empty() else Mono.just(hit.toRecord(plan.projection)) + } + }.mapBackendErrors() + + override fun stream( + plan: BackendStreamQueryPlan, + options: QueryBackendExecutionOptions, + ): Flux = Flux.defer { + validateOptions(plan, options) + val compiled = compiler.compile(plan) + client.search( + searchRequest(compiled, options, 0, plan.limit, trackTotalHits = false), + Map::class.java, + ).flatMapMany { response -> + validateResponse(response) + if (response.hits().hits().size > plan.limit) incomplete() + Flux.fromIterable(response.hits().hits()).map { hit -> hit.toRecord(plan.projection) } + } + }.mapBackendErrors() + + override fun page( + plan: BackendPageQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = Mono.defer { + validateOptions(plan, options) + pitPageExecutor.execute(plan, options) + }.mapBackendErrors() + + override fun count( + plan: BackendCountQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = Mono.defer { + validateOptions(plan, options) + val compiled = compiler.compile(plan) + val request = CountRequest.of { builder -> + builder.index(binding.indexName).query(compiled.query) + } + client.count(request).map { response -> + requireCompleteShards(response.shards()) + response.count() + } + }.mapBackendErrors() + + private fun searchRequest( + compiled: ElasticsearchCompiledRecordQuery, + options: QueryBackendExecutionOptions, + from: Int, + size: Int, + trackTotalHits: Boolean, + ): SearchRequest = SearchRequest.of { builder -> + builder.index(binding.indexName) + .query(compiled.query) + .from(from) + .size(size) + .allowPartialSearchResults(false) + .trackTotalHits { hits -> hits.enabled(trackTotalHits) } + .also { request -> + if (compiled.sort.isNotEmpty()) request.sort(compiled.sort) + compiled.sourceFilter?.let { sourceFilter -> + request.source { source -> source.filter(sourceFilter) } + } + options.remainingMillis()?.let { remaining -> request.timeout("${remaining}ms") } + } + } + + private fun validateOptions(plan: BackendRecordQueryPlan, options: QueryBackendExecutionOptions) { + if (plan is BackendStreamQueryPlan && plan.limit > MAX_DIRECT_STREAM_RESULT_WINDOW) { + unsupported() + } + requireSupportedOptions(options) + requireReturnedRecordsBudget(plan, options) + requirePageWindowBudget(plan, options) + options.remainingMillis() + } + + private fun requireSupportedOptions(options: QueryBackendExecutionOptions) { + val unsupportedBudgets = listOf( + options.maxScannedRecords, + options.maxCandidateBuckets, + options.maxReturnedBuckets, + ) + if (unsupportedBudgets.any { budget -> budget != null }) { + unsupported() + } + } + + private fun requireReturnedRecordsBudget( + plan: BackendRecordQueryPlan, + options: QueryBackendExecutionOptions, + ) { + options.maxReturnedRecords?.let { maximum -> + val requested = when (plan) { + is BackendSingleQueryPlan -> 1L + is BackendStreamQueryPlan -> plan.limit.toLong() + is BackendPageQueryPlan -> plan.page.size.toLong() + is BackendCountQueryPlan -> 0L + } + if (requested > maximum) budgetExceeded() + } + } + + private fun requirePageWindowBudget( + plan: BackendRecordQueryPlan, + options: QueryBackendExecutionOptions, + ) { + if (plan !is BackendPageQueryPlan) { + return + } + val endExclusive = try { + Math.addExact(plan.page.offset, plan.page.size.toLong()) + } catch (error: ArithmeticException) { + throw QueryBackendException(QueryBackendFailureKind.BUDGET_EXCEEDED, error) + } + options.maxPageWindow?.let { maximum -> + if (endExclusive > maximum) { + budgetExceeded() + } + } + } + + private fun QueryBackendExecutionOptions.remainingMillis(): Long? { + val currentDeadline = deadline ?: return null + val now = clock.instant() + val remaining = try { + Duration.between(now, currentDeadline).toMillis() + } catch (error: ArithmeticException) { + if (currentDeadline.isAfter(now)) { + Long.MAX_VALUE + } else { + throw QueryBackendException(QueryBackendFailureKind.TIMEOUT, error) + } + } + if (remaining <= 0) throw QueryBackendException(QueryBackendFailureKind.TIMEOUT) + return remaining + } + + private fun validateResponse(response: ResponseBody<*>) { + if (response.timedOut()) throw QueryBackendException(QueryBackendFailureKind.TIMEOUT) + requireCompleteShards(response.shards()) + } + + private fun requireCompleteShards(shards: ShardStatistics) { + if (shards.failed().toLong() != 0L) incomplete() + } + + @Suppress("UNCHECKED_CAST") + private fun Hit<*>.toRecord(projection: BackendProjection): BackendRecord { + if (ignored().isNotEmpty()) incomplete() + val source = source() as? Map ?: incomplete() + val identity = id() ?: incomplete() + return mapper.map(identity, source, projection) + } + + private fun Mono.mapBackendErrors(): Mono = onErrorMap(::mapBackendError) + + private fun Flux.mapBackendErrors(): Flux = onErrorMap(::mapBackendError) + + private fun mapBackendError(error: Throwable): Throwable = + if (error is QueryBackendException) error else QueryBackendException(QueryBackendFailureKind.UNAVAILABLE, error) + + private fun unsupported(): Nothing = throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED) + + private fun budgetExceeded(): Nothing = throw QueryBackendException(QueryBackendFailureKind.BUDGET_EXCEEDED) + + private companion object { + const val MAX_DIRECT_STREAM_RESULT_WINDOW = 10_000 + } +} + +internal class ElasticsearchSnapshotRecordMapper( + private val binding: ElasticsearchPreparedQueryBinding, + private val identityOutputField: String = MessageRecords.AGGREGATE_ID, + private val limits: MappingLimits = MappingLimits(), +) { + @Suppress("TooGenericExceptionCaught") + fun map( + hitIdentity: String, + source: Map, + projection: BackendProjection = BackendProjection.All, + ): BackendRecord = try { + val frozen = MappingSession(limits).objectValue(source, 0) + val sourceIdentity = (frozen.values[identityOutputField] as? NormalizedValue.Text)?.value ?: mappingFailure() + if (hitIdentity.isBlank() || sourceIdentity != hitIdentity) mappingFailure() + val logical = frozen.toLogicalDocument() + BackendRecord( + hitIdentity, + logical.apply(projection, identityOutputField), + BackendRecordCompleteness.COMPLETE, + ) + } catch (error: QueryBackendException) { + throw error + } catch (error: RuntimeException) { + throw QueryBackendException(QueryBackendFailureKind.MAPPING_FAILURE, error) + } + + private fun NormalizedValue.ObjectValue.toLogicalDocument(): NormalizedValue.ObjectValue { + val paths = binding.schema.fields.keys.filterIsInstance().map(QueryFieldId.Path::segments) + var logical = include(paths) + binding.schema.fields.filterKeys { field -> field is QueryFieldId.Path }.forEach { (field, schemaField) -> + field as QueryFieldId.Path + logical = logical.transformAt(field.segments) { value -> + decodeValue(value, schemaField.type, binding.fields.getValue(field).valueEncoding) + } + } + val values = LinkedHashMap(logical.values) + binding.schema.fields.filterKeys { field -> field is QueryFieldId.System }.forEach { (field, schemaField) -> + field as QueryFieldId.System + val physical = binding.fields.getValue(field) + val sourcePath = physical.sourceField.split('.') + valueAt(sourcePath)?.let { value -> + values[field.outputPath(identityOutputField).single()] = + decodeValue(value, schemaField.type, physical.valueEncoding) + } + } + return NormalizedValue.ObjectValue(values) + } + + private fun NormalizedValue.ObjectValue.transformAt( + path: List, + transform: (NormalizedValue) -> NormalizedValue, + ): NormalizedValue.ObjectValue { + val head = path.firstOrNull() ?: mappingFailure() + val current = values[head] ?: return this + val copy = LinkedHashMap(values) + copy[head] = current.transformAt(path.drop(1), transform) + return NormalizedValue.ObjectValue(copy) + } + + private fun NormalizedValue.transformAt( + path: List, + transform: (NormalizedValue) -> NormalizedValue, + ): NormalizedValue = when { + path.isEmpty() -> transform(this) + this == NormalizedValue.Null -> this + this is NormalizedValue.ObjectValue -> transformAt(path, transform) + this is NormalizedValue.ListValue -> NormalizedValue.ListValue( + values.map { value -> value.transformAt(path, transform) }, + ) + else -> mappingFailure() + } + + private fun decodeValue( + value: NormalizedValue, + type: LogicalFieldType, + encoding: ElasticsearchValueEncoding, + ): NormalizedValue { + if (value == NormalizedValue.Null) return value + return when (type) { + LogicalFieldType.Instant -> { + if (encoding != ElasticsearchValueEncoding.EPOCH_MILLIS || value !is NormalizedValue.Int64) { + mappingFailure() + } + NormalizedValue.InstantValue(Instant.ofEpochMilli(value.value)) + } + + is LogicalFieldType.Array -> { + val list = value as? NormalizedValue.ListValue ?: mappingFailure() + NormalizedValue.ListValue( + list.values.map { element -> + decodeValue(element, type.elementType, encoding) + }, + ) + } + + else -> value + } + } + + private fun NormalizedValue.ObjectValue.valueAt(path: List): NormalizedValue? { + var current: NormalizedValue = this + path.forEach { segment -> + current = (current as? NormalizedValue.ObjectValue)?.values?.get(segment) ?: return null + } + return current + } + + internal data class MappingLimits( + val maxDepth: Int = 32, + val maxNodes: Int = 100_000, + val maxCollectionSize: Int = 100_000, + ) { + init { + require(maxDepth > 0 && maxNodes > 0 && maxCollectionSize > 0) + } + } + + private class MappingSession(private val limits: MappingLimits) { + private val active = IdentityHashMap() + private var nodes = 0 + + fun objectValue(source: Map<*, *>, depth: Int): NormalizedValue.ObjectValue = + withContainer(source, depth) { + val values = LinkedHashMap() + var count = 0 + source.entries.forEach { entry -> + if (++count > limits.maxCollectionSize) mappingFailure() + val key = entry.key as? String ?: mappingFailure() + if (values.containsKey(key)) mappingFailure() + values[key] = value(entry.value, depth + 1) + } + NormalizedValue.ObjectValue(values) + } + + private fun value(source: Any?, depth: Int): NormalizedValue = + when (source) { + is Map<*, *> -> objectValue(source, depth) + is Iterable<*> -> listValue(source, depth) + is Array<*> -> listValue(source.asIterable(), depth) + else -> scalarValue(source, depth) + } + + private fun scalarValue(source: Any?, depth: Int): NormalizedValue { + enterNode(depth) + return when (source) { + null -> NormalizedValue.Null + is Boolean -> NormalizedValue.BooleanValue(source) + is String -> NormalizedValue.Text(source) + is Number -> numberValue(source) + is Instant -> NormalizedValue.InstantValue(source) + is ByteArray -> NormalizedValue.Bytes(source) + else -> mappingFailure() + } + } + + private fun numberValue(source: Number): NormalizedValue = when (source) { + is Byte, is Short, is Int, is Long -> NormalizedValue.Int64(source.toLong()) + is BigInteger -> try { + NormalizedValue.Int64(source.longValueExact()) + } catch (_: ArithmeticException) { + NormalizedValue.Decimal(BigDecimal(source)) + } + + is BigDecimal -> NormalizedValue.Decimal(source) + is Float, is Double -> { + val value = source.toDouble() + if (!value.isFinite()) mappingFailure() + NormalizedValue.Decimal(BigDecimal.valueOf(value)) + } + + else -> mappingFailure() + } + + private fun listValue(source: Iterable<*>, depth: Int): NormalizedValue.ListValue = + withContainer(source, depth) { + val values = ArrayList() + val iterator = source.iterator() + while (iterator.hasNext()) { + if (values.size >= limits.maxCollectionSize) mappingFailure() + values += value(iterator.next(), depth + 1) + } + NormalizedValue.ListValue(values) + } + + private fun enterNode(depth: Int) { + if (depth > limits.maxDepth || ++nodes > limits.maxNodes) mappingFailure() + } + + private fun withContainer(source: Any, depth: Int, block: () -> T): T { + enterNode(depth) + if (active.put(source, Unit) != null) mappingFailure() + return try { + block() + } finally { + active.remove(source) + } + } + } +} + +private fun NormalizedValue.ObjectValue.apply( + projection: BackendProjection, + identityOutputField: String, +): NormalizedValue.ObjectValue = + when (projection) { + BackendProjection.All -> this + is BackendProjection.Include -> include( + projection.fields.map { field -> + field.outputPath(identityOutputField) + }, + ) + is BackendProjection.Exclude -> exclude( + projection.fields.map { field -> + field.outputPath(identityOutputField) + }, + ) + } + +private fun QueryFieldId.outputPath(identityOutputField: String): List = + when (this) { + is QueryFieldId.Path -> segments + is QueryFieldId.System -> listOf( + when (kind) { + SystemFieldKind.IDENTITY -> identityOutputField + SystemFieldKind.AGGREGATE_ID -> MessageRecords.AGGREGATE_ID + SystemFieldKind.TENANT_ID -> MessageRecords.TENANT_ID + SystemFieldKind.OWNER_ID -> MessageRecords.OWNER_ID + SystemFieldKind.SPACE_ID -> MessageRecords.SPACE_ID + SystemFieldKind.DELETED -> StateAggregateRecords.DELETED + }, + ) + } + +private fun NormalizedValue.ObjectValue.include(paths: List>): NormalizedValue.ObjectValue { + val result = LinkedHashMap() + values.forEach { (key, value) -> + val matching = paths.filter { path -> path.firstOrNull() == key } + if (matching.any { path -> path.size == 1 }) { + result[key] = value + } else if (matching.isNotEmpty()) { + result[key] = value.includeNested(matching.map { path -> path.drop(1) }) + } + } + return NormalizedValue.ObjectValue(result) +} + +private fun NormalizedValue.includeNested(paths: List>): NormalizedValue = + when (this) { + NormalizedValue.Null -> NormalizedValue.Null + is NormalizedValue.ObjectValue -> include(paths) + is NormalizedValue.ListValue -> NormalizedValue.ListValue(values.map { value -> value.includeNested(paths) }) + else -> mappingFailure() + } + +private fun NormalizedValue.ObjectValue.exclude(paths: List>): NormalizedValue.ObjectValue { + val result = LinkedHashMap() + values.forEach { (key, value) -> + val matching = paths.filter { path -> path.firstOrNull() == key } + if (matching.none { path -> path.size == 1 }) { + val nested = matching.filter { path -> path.size > 1 }.map { path -> path.drop(1) } + result[key] = if (nested.isEmpty()) value else value.excludeNested(nested) + } + } + return NormalizedValue.ObjectValue(result) +} + +private fun NormalizedValue.excludeNested(paths: List>): NormalizedValue = + when (this) { + is NormalizedValue.ObjectValue -> exclude(paths) + is NormalizedValue.ListValue -> NormalizedValue.ListValue(values.map { value -> value.excludeNested(paths) }) + else -> this + } + +private fun incomplete(): Nothing = throw QueryBackendException(QueryBackendFailureKind.INCOMPLETE_RESULT) + +private fun mappingFailure(): Nothing = throw QueryBackendException(QueryBackendFailureKind.MAPPING_FAILURE) diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/BoundedElasticsearchIndexMigrationProbeVerifierTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/BoundedElasticsearchIndexMigrationProbeVerifierTest.kt new file mode 100644 index 00000000000..972f1a3123c --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/BoundedElasticsearchIndexMigrationProbeVerifierTest.kt @@ -0,0 +1,226 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import org.junit.jupiter.api.Test +import reactor.core.publisher.Mono +import java.time.Duration +import java.util.function.Consumer + +class BoundedElasticsearchIndexMigrationProbeVerifierTest { + @Test + fun `should compare every registered record and analytics probe against the exact generation`() { + val recordEqual = probe("record-equal", ElasticsearchIndexProbeKind.RECORD) + val recordDifferent = probe("record-different", ElasticsearchIndexProbeKind.RECORD) + val analyticsDifferent = probe("analytics-different", ElasticsearchIndexProbeKind.ANALYTICS) + val suite = suite(listOf(recordEqual, recordDifferent, analyticsDifferent)) + val executor = ProbeExecutor( + authority = mapOf( + recordEqual.id to evidence("1"), + recordDifferent.id to evidence("2"), + analyticsDifferent.id to evidence("3"), + ), + physical = mapOf( + recordEqual.id to evidence("1"), + recordDifferent.id to evidence("4"), + analyticsDifferent.id to evidence("5"), + ), + ) + + val result = BoundedElasticsearchIndexMigrationProbeVerifier(suite, executor) + .compare(COMMAND, MANIFEST, MANIFEST.names.physical) + .block()!! + + result.assert().isEqualTo(ElasticsearchIndexProbeVerification(1, 1)) + executor.authorityCalls.assert().containsExactly(*suite.probes.toTypedArray()) + executor.physicalCalls.map { it.first }.assert().containsExactly(*suite.probes.toTypedArray()) + executor.physicalCalls.map { it.second }.assert().containsExactly( + MANIFEST.names.physical, + MANIFEST.names.physical, + MANIFEST.names.physical, + ) + } + + @Test + fun `suite should be canonical immutable and reject duplicate or excessive definitions`() { + val source = mutableListOf( + probe("z-record", ElasticsearchIndexProbeKind.RECORD), + probe("a-analytics", ElasticsearchIndexProbeKind.ANALYTICS), + ) + val suite = suite(source) + source.clear() + + suite.probes.map { it.id.value }.assert().containsExactly("a-analytics", "z-record") + assertThrownBy { + @Suppress("UNCHECKED_CAST") + (suite.probes as MutableList).clear() + } + assertThrownBy { + suite( + listOf( + probe("duplicate", ElasticsearchIndexProbeKind.RECORD), + probe("duplicate", ElasticsearchIndexProbeKind.ANALYTICS), + ), + ) + } + assertThrownBy { + suite( + (0..MAX_ELASTICSEARCH_INDEX_MIGRATION_PROBES).map { index -> + probe("probe-$index", ElasticsearchIndexProbeKind.RECORD) + }, + ) + } + } + + @Test + fun `missing incomplete or failed evidence should fail verification without running later probes`() { + val first = probe("first", ElasticsearchIndexProbeKind.RECORD) + val later = probe("later", ElasticsearchIndexProbeKind.ANALYTICS) + val suite = suite(listOf(first, later)) + listOf( + ProbeExecutor(authority = emptyMap(), physical = mapOf(first.id to evidence("1"))), + ProbeExecutor( + authority = mapOf(first.id to evidence("1", complete = false)), + physical = mapOf(first.id to evidence("1")), + ), + ProbeExecutor( + authority = mapOf(first.id to evidence("1")), + physical = mapOf(first.id to evidence("1")), + failure = IllegalStateException("probe failed"), + ), + ProbeExecutor( + authority = mapOf(first.id to evidence("1")), + physical = mapOf(first.id to evidence("1")), + failure = ElasticsearchIndexLifecycleException( + ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED, + MANIFEST.id, + "forged typed error", + ), + ), + ).forEach { executor -> + assertThrownBy { + BoundedElasticsearchIndexMigrationProbeVerifier(suite, executor) + .compare(COMMAND, MANIFEST, MANIFEST.names.physical) + .block() + }.satisfies( + Consumer { error -> + error.code.assert().isEqualTo(ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED) + }, + ) + executor.authorityCalls.assert().containsExactly(first) + executor.authorityCalls.assert().doesNotContain(later) + } + } + + @Test + fun `suite target schema or id mismatch should fail before any probe executes`() { + val probe = probe("record", ElasticsearchIndexProbeKind.RECORD) + listOf( + suite(listOf(probe), id = ElasticsearchIndexProbeSuiteId("another-suite-v1")), + suite( + listOf(probe), + target = QueryTarget(MaterializedNamedAggregate("sales", "cart"), QueryDocumentKind.SNAPSHOT), + ), + suite(listOf(probe), schemaContractId = SchemaContractId("9".repeat(64))), + ).forEach { mismatchedSuite -> + val executor = ProbeExecutor(mapOf(probe.id to evidence("1")), mapOf(probe.id to evidence("1"))) + val verifier = BoundedElasticsearchIndexMigrationProbeVerifier(mismatchedSuite, executor) + + assertThrownBy { + verifier.compare(COMMAND, MANIFEST, MANIFEST.names.physical).block() + }.satisfies( + Consumer { error -> + error.code.assert().isEqualTo(ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED) + }, + ) + executor.authorityCalls.assert().isEmpty() + executor.physicalCalls.assert().isEmpty() + } + } + + private fun suite( + probes: Collection, + id: ElasticsearchIndexProbeSuiteId = MANIFEST.verificationContract.probeSuiteId, + target: QueryTarget = MANIFEST.target, + schemaContractId: SchemaContractId = MANIFEST.schemaContractId, + ) = ElasticsearchIndexMigrationProbeSuite(id, target, schemaContractId, probes) + + private class ProbeExecutor( + private val authority: Map, + private val physical: Map, + private val failure: RuntimeException? = null, + ) : ElasticsearchIndexMigrationProbeExecutor { + val authorityCalls = mutableListOf() + val physicalCalls = mutableListOf>() + + override fun evaluateAuthority( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + probe: ElasticsearchIndexMigrationProbe, + ): Mono = Mono.defer { + authorityCalls += probe + Mono.justOrEmpty(authority[probe.id]) + } + + override fun evaluatePhysical( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + probe: ElasticsearchIndexMigrationProbe, + ): Mono = Mono.defer { + physicalCalls += probe to physicalIndex + failure?.let { throw it } + Mono.justOrEmpty(physical[probe.id]) + } + } + + private companion object { + val MANIFEST = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("probe-verification"), + QueryTarget(MaterializedNamedAggregate("sales", "order"), QueryDocumentKind.SNAPSHOT), + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(3), + SchemaContractId("1".repeat(64)), + ElasticsearchIndexCapabilityDigest("2".repeat(64)), + ElasticsearchPhysicalIndex("wow.sales.order.snapshot-v0001-000001"), + ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM, + testVerificationContract(), + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + val COMMAND = ElasticsearchIndexLifecycleCommandId("probe-command") + } +} + +private fun probe(id: String, kind: ElasticsearchIndexProbeKind) = ElasticsearchIndexMigrationProbe( + ElasticsearchIndexProbeId(id), + kind, +) + +private fun evidence(value: String, complete: Boolean = true) = ElasticsearchIndexProbeEvidence( + resultCount = 1, + resultChecksum = ElasticsearchIndexChecksum(value.repeat(64)), + complete = complete, +) diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/DefaultElasticsearchIndexMigrationVerifierTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/DefaultElasticsearchIndexMigrationVerifierTest.kt new file mode 100644 index 00000000000..6ec4c819216 --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/DefaultElasticsearchIndexMigrationVerifierTest.kt @@ -0,0 +1,175 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import org.junit.jupiter.api.Test +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset +import java.util.function.Consumer + +class DefaultElasticsearchIndexMigrationVerifierTest { + @Test + fun `should combine authority index and query probe evidence without changing its provenance`() { + val verifier = verifier( + ElasticsearchAuthoritativeVerificationSnapshot(ALGORITHM, 2, IDENTITY, CONTENT, 7), + ElasticsearchPhysicalIndexVerificationSnapshot(ALGORITHM, DESTINATION, 2, IDENTITY, CONTENT, true, 7), + ElasticsearchIndexProbeVerification(0, 0), + ) + + val verification = verifier.verify(COMMAND, MANIFEST, DESTINATION).block()!! + + verification.assert().isEqualTo( + ElasticsearchIndexVerification( + DESTINATION, + 2, + 2, + IDENTITY, + IDENTITY, + CONTENT, + CONTENT, + true, + 7, + 7, + 0, + 0, + NOW, + ), + ) + verification.requireSatisfied(MANIFEST, DESTINATION) + } + + @Test + fun `should preserve a missing index watermark as verification evidence and fail closed`() { + val verification = verifier( + ElasticsearchAuthoritativeVerificationSnapshot(ALGORITHM, 2, IDENTITY, CONTENT, 7), + ElasticsearchPhysicalIndexVerificationSnapshot(ALGORITHM, DESTINATION, 2, IDENTITY, CONTENT, true, null), + ElasticsearchIndexProbeVerification(0, 0), + ).verify(COMMAND, MANIFEST, DESTINATION).block()!! + + verification.authoritativeWatermark.assert().isEqualTo(7) + verification.indexedWatermark.assert().isNull() + assertThrownBy { + verification.requireSatisfied(MANIFEST, DESTINATION) + }.satisfies( + Consumer { error -> + error.code.assert().isEqualTo(ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED) + }, + ) + } + + @Test + fun `should reject a probe suite that is not pinned by the migration manifest`() { + val mismatched = MANIFEST.copy( + verificationContract = MANIFEST.verificationContract.copy( + probeSuiteId = ElasticsearchIndexProbeSuiteId("another-probe-suite-v1"), + ), + ) + + assertThrownBy { + verifier( + ElasticsearchAuthoritativeVerificationSnapshot(ALGORITHM, 0, IDENTITY, CONTENT, null), + ElasticsearchPhysicalIndexVerificationSnapshot( + ALGORITHM, + DESTINATION, + 0, + IDENTITY, + CONTENT, + true, + null, + ), + ElasticsearchIndexProbeVerification(0, 0), + ).verify(COMMAND, mismatched, DESTINATION).block() + }.satisfies( + Consumer { error -> + error.code.assert().isEqualTo(ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED) + error.migrationId.assert().isEqualTo(MANIFEST.id) + }, + ) + } + + private fun verifier( + expected: ElasticsearchAuthoritativeVerificationSnapshot, + actual: ElasticsearchPhysicalIndexVerificationSnapshot, + probes: ElasticsearchIndexProbeVerification, + ) = DefaultElasticsearchIndexMigrationVerifier( + ElasticsearchAuthoritativeVerificationSource { command, manifest -> + command.assert().isEqualTo(COMMAND) + manifest.assert().isEqualTo(MANIFEST) + Mono.just(expected) + }, + ElasticsearchPhysicalIndexVerificationSource { command, manifest, physical -> + command.assert().isEqualTo(COMMAND) + manifest.assert().isEqualTo(MANIFEST) + physical.assert().isEqualTo(DESTINATION) + Mono.just(actual) + }, + object : ElasticsearchIndexMigrationProbeVerifier { + override val suiteId = ElasticsearchIndexProbeSuiteId("test-probes-v1") + + override fun compare( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono { + command.assert().isEqualTo(COMMAND) + manifest.assert().isEqualTo(MANIFEST) + physicalIndex.assert().isEqualTo(DESTINATION) + return Mono.just(probes) + } + }, + CLOCK, + ) + + private companion object { + val NOW: Instant = Instant.parse("2026-08-08T01:00:00Z") + val CLOCK: Clock = Clock.fixed(NOW, ZoneOffset.UTC) + val TARGET = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + val SOURCE = ElasticsearchPhysicalIndex("wow.sales.order.snapshot-v0001-000001") + val DESTINATION = ElasticsearchPhysicalIndex("wow.sales.order.snapshot-v0002-000002") + val IDENTITY = ElasticsearchIndexChecksum("1".repeat(64)) + val CONTENT = ElasticsearchIndexChecksum("2".repeat(64)) + val ALGORITHM = ElasticsearchIndexChecksumAlgorithm.CANONICAL_DOCUMENT_SHA256_V1 + val COMMAND = ElasticsearchIndexLifecycleCommandId("verify-command") + val MANIFEST = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("snapshot-verification"), + TARGET, + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(2), + SchemaContractId("3".repeat(64)), + ElasticsearchIndexCapabilityDigest("4".repeat(64)), + SOURCE, + ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM, + testVerificationContract(), + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleClientTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleClientTest.kt new file mode 100644 index 00000000000..17758f18caa --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleClientTest.kt @@ -0,0 +1,246 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import co.elastic.clients.elasticsearch._types.mapping.TypeMapping +import co.elastic.clients.elasticsearch.cluster.PutComponentTemplateRequest +import co.elastic.clients.elasticsearch.cluster.PutComponentTemplateResponse +import co.elastic.clients.elasticsearch.indices.CreateIndexRequest +import co.elastic.clients.elasticsearch.indices.CreateIndexResponse +import co.elastic.clients.elasticsearch.indices.GetAliasRequest +import co.elastic.clients.elasticsearch.indices.GetAliasResponse +import co.elastic.clients.elasticsearch.indices.GetMappingRequest +import co.elastic.clients.elasticsearch.indices.GetMappingResponse +import co.elastic.clients.elasticsearch.indices.PutIndexTemplateRequest +import co.elastic.clients.elasticsearch.indices.PutIndexTemplateResponse +import co.elastic.clients.elasticsearch.indices.UpdateAliasesRequest +import co.elastic.clients.elasticsearch.indices.UpdateAliasesResponse +import co.elastic.clients.elasticsearch.indices.get_alias.IndexAliases +import co.elastic.clients.elasticsearch.indices.get_mapping.IndexMappingRecord +import co.elastic.clients.json.JsonData +import io.mockk.every +import io.mockk.mockk +import io.mockk.slot +import io.mockk.verify +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import org.junit.jupiter.api.Test +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClusterClient +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchIndicesClient +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset +import java.util.function.Consumer + +class ElasticsearchIndexLifecycleClientTest { + @Test + fun `versioned templates should bind metadata component composition and exact generation pattern`() { + val template = ElasticsearchVersionedIndexTemplate(MANIFEST, mapping()) + val component = template.componentRequest() + val index = template.indexTemplateRequest() + + component.name().assert().isEqualTo("wow.sales.order.snapshot-query-mapping-v0002") + component.version().assert().isEqualTo(2L) + component.template().mappings()!!.meta().string(MAPPING_VERSION).assert().isEqualTo("v0002") + index.name().assert().isEqualTo("wow.sales.order.snapshot-query-template-v0002") + index.indexPatterns().assert().containsExactly("wow.sales.order.snapshot-v0002-*") + index.composedOf().assert().containsExactly(component.name()) + index.allowAutoCreate().assert().isFalse() + index.meta().string(CAPABILITY_DIGEST).assert().isEqualTo(MANIFEST.capabilityDigest.value) + } + + @Test + fun `template should reject metadata that does not attest the manifest`() { + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.ATTESTATION_MISMATCH) { + ElasticsearchVersionedIndexTemplate( + MANIFEST, + mapping(capabilityDigest = "b".repeat(64)), + ) + } + } + + @Test + fun `create should ensure both templates and reuse an exactly attested destination`() { + val client = mockk() + val cluster = mockk() + val indices = mockk() + every { client.cluster() } returns cluster + every { client.indices() } returns indices + every { cluster.putComponentTemplate(any()) } returns Mono.just( + PutComponentTemplateResponse.of { response -> response.acknowledged(true) }, + ) + every { indices.putIndexTemplate(any()) } returns Mono.just( + PutIndexTemplateResponse.of { response -> response.acknowledged(true) }, + ) + every { indices.getMapping(any()) } returns Mono.just(mappingResponse()) + val admin = ReactiveElasticsearchIndexAdminClient(client, CLOCK) + + val attestation = admin.create(MANIFEST, ElasticsearchVersionedIndexTemplate(MANIFEST, mapping())).block()!! + + attestation.assert().isEqualTo(MANIFEST.destinationAttestation) + verify(exactly = 1) { cluster.putComponentTemplate(any()) } + verify(exactly = 1) { indices.putIndexTemplate(any()) } + verify(exactly = 0) { indices.create(any()) } + } + + @Test + fun `create should explicitly create a missing generation and attest the applied template`() { + val client = mockk() + val cluster = mockk() + val indices = mockk() + every { client.cluster() } returns cluster + every { client.indices() } returns indices + every { cluster.putComponentTemplate(any()) } returns Mono.just( + PutComponentTemplateResponse.of { response -> response.acknowledged(true) }, + ) + every { indices.putIndexTemplate(any()) } returns Mono.just( + PutIndexTemplateResponse.of { response -> response.acknowledged(true) }, + ) + every { indices.getMapping(any()) } returnsMany listOf( + Mono.just(GetMappingResponse.of { response -> response.mappings(emptyMap()) }), + Mono.just(mappingResponse()), + ) + every { indices.create(any()) } returns Mono.just( + CreateIndexResponse.of { response -> + response.index(DESTINATION.value).acknowledged(true).shardsAcknowledged(true) + }, + ) + val admin = ReactiveElasticsearchIndexAdminClient(client, CLOCK) + + admin.create(MANIFEST, ElasticsearchVersionedIndexTemplate(MANIFEST, mapping())).block()!! + + verify(exactly = 1) { + indices.create(match { request -> request.index() == DESTINATION.value }) + } + } + + @Test + fun `cutover should remove the exact source with must-exist and add one write alias atomically`() { + val client = mockk() + val indices = mockk() + every { client.indices() } returns indices + every { indices.getAlias(any()) } returnsMany listOf( + Mono.just(aliasResponse(SOURCE)), + Mono.just(aliasResponse(DESTINATION)), + ) + val captured = slot() + every { indices.updateAliases(capture(captured)) } returns Mono.just( + UpdateAliasesResponse.of { response -> response.acknowledged(true) }, + ) + val admin = ReactiveElasticsearchIndexAdminClient(client, CLOCK) + + val transition = admin.compareAndSetAlias(MANIFEST, SOURCE, DESTINATION).block()!! + + transition.previous.assert().isEqualTo(SOURCE) + transition.current.assert().isEqualTo(DESTINATION) + captured.captured.actions().assert().hasSize(2) + captured.captured.actions()[0].remove().mustExist().assert().isTrue() + captured.captured.actions()[0].remove().index().assert().isEqualTo(SOURCE.value) + captured.captured.actions()[1].add().isWriteIndex().assert().isTrue() + captured.captured.actions()[1].add().index().assert().isEqualTo(DESTINATION.value) + } + + @Test + fun `cutover should fail closed when the expected source alias is absent`() { + val client = mockk() + val indices = mockk() + every { client.indices() } returns indices + every { indices.getAlias(any()) } returns Mono.just( + GetAliasResponse.of { response -> response.aliases(emptyMap()) }, + ) + val admin = ReactiveElasticsearchIndexAdminClient(client, CLOCK) + + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.ALIAS_CONFLICT) { + admin.compareAndSetAlias(MANIFEST, SOURCE, DESTINATION).block() + } + verify(exactly = 0) { indices.updateAliases(any()) } + } + + private fun mapping( + capabilityDigest: String = MANIFEST.capabilityDigest.value, + ): TypeMapping = TypeMapping.of { mapping -> + mapping.meta(MAPPING_VERSION, JsonData.of(MANIFEST.mappingVersion.tag)) + .meta(DOCUMENT_KIND, JsonData.of(MANIFEST.target.documentKind.name)) + .meta(SCHEMA_CONTRACT, JsonData.of(MANIFEST.schemaContractId.value)) + .meta(CAPABILITY_DIGEST, JsonData.of(capabilityDigest)) + } + + private fun mappingResponse(): GetMappingResponse = GetMappingResponse.of { response -> + response.mappings( + DESTINATION.value, + IndexMappingRecord.of { record -> record.mappings(mapping()) }, + ) + } + + private fun aliasResponse(physical: ElasticsearchPhysicalIndex): GetAliasResponse = + GetAliasResponse.of { response -> + response.aliases( + physical.value, + IndexAliases.of { index -> + index.aliases(MANIFEST.names.alias.value) { alias -> alias.isWriteIndex(true) } + }, + ) + } + + private fun assertLifecycle( + code: ElasticsearchIndexLifecycleErrorCode, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> error.code.assert().isEqualTo(code) }, + ) + } + + private fun Map.string(key: String): String = getValue(key).to(String::class.java) + + private companion object { + const val MAPPING_VERSION = "wow_query_mapping_version" + const val DOCUMENT_KIND = "wow_query_document_kind" + const val SCHEMA_CONTRACT = "wow_query_schema_contract_id" + const val CAPABILITY_DIGEST = "wow_query_capability_digest" + val NOW: Instant = Instant.parse("2026-08-08T03:00:00Z") + val CLOCK: Clock = Clock.fixed(NOW, ZoneOffset.UTC) + val TARGET = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + val SOURCE = ElasticsearchPhysicalIndex("wow.sales.order.snapshot-v0001-000001") + val DESTINATION = ElasticsearchPhysicalIndex("wow.sales.order.snapshot-v0002-000007") + val MANIFEST = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("sales-order-snapshot-v2-g7"), + TARGET, + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(7), + SchemaContractId("1".repeat(64)), + ElasticsearchIndexCapabilityDigest("a".repeat(64)), + SOURCE, + ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM, + testVerificationContract(), + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleExecutorTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleExecutorTest.kt new file mode 100644 index 00000000000..89ebd034810 --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleExecutorTest.kt @@ -0,0 +1,316 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import org.junit.jupiter.api.Test +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset +import java.util.concurrent.atomic.AtomicBoolean +import java.util.concurrent.atomic.AtomicInteger +import java.util.function.Consumer + +class ElasticsearchIndexLifecycleExecutorTest { + @Test + fun `commands should follow the explicit verified cutover and fresh rollback verification sequence`() { + val fixture = fixture() + var state = fixture.executor.register(MANIFEST).block()!! + state.phase.assert().isEqualTo(ElasticsearchIndexLifecyclePhase.NEW) + + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.VALIDATE) + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.CREATE) + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.REBUILD) + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.VERIFY) + state.phase.assert().isEqualTo(ElasticsearchIndexLifecyclePhase.VERIFIED) + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.CUTOVER) + state.phase.assert().isEqualTo(ElasticsearchIndexLifecyclePhase.CUTOVER) + state.retainedSourceUntil.assert().isEqualTo(NOW.plus(MANIFEST.minimumRetention)) + + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.VERIFY) + state.rollbackVerification!!.physicalIndex.assert().isEqualTo(SOURCE) + state.phase.assert().isEqualTo(ElasticsearchIndexLifecyclePhase.ROLLBACK_VERIFIED) + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.ROLLBACK) + + state.phase.assert().isEqualTo(ElasticsearchIndexLifecyclePhase.ROLLED_BACK) + fixture.operations.calls.assert().containsExactly( + "validate", + "create", + "rebuild", + "verify:${DESTINATION.value}", + "cutover", + "verify:${SOURCE.value}", + "rollback", + ) + } + + @Test + fun `invalid order and failed verification should fail before alias mutation`() { + val fixture = fixture() + val initial = fixture.executor.register(MANIFEST).block()!! + + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.INVALID_TRANSITION) { + fixture.execute(initial, ElasticsearchIndexLifecycleCommandType.CUTOVER) + } + fixture.operations.calls.assert().isEmpty() + + var state = fixture.execute(initial, ElasticsearchIndexLifecycleCommandType.VALIDATE) + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.CREATE) + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.REBUILD) + fixture.operations.verificationMatches.set(false) + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED) { + fixture.execute(state, ElasticsearchIndexLifecycleCommandType.VERIFY) + } + fixture.operations.calls.assert().doesNotContain("cutover") + } + + @Test + fun `snapshot cutover without a trusted write fence should fail before alias mutation`() { + val fixture = fixture(ElasticsearchSnapshotCutoverGuard.DENY) + var state = fixture.executor.register(MANIFEST).block()!! + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.VALIDATE) + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.CREATE) + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.REBUILD) + state = fixture.execute(state, ElasticsearchIndexLifecycleCommandType.VERIFY) + + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.CUTOVER_FENCE_REQUIRED) { + fixture.execute(state, ElasticsearchIndexLifecycleCommandType.CUTOVER) + } + fixture.operations.calls.assert().doesNotContain("cutover") + } + + @Test + fun `same command should resume idempotently while a different command is rejected`() { + val fixture = fixture() + val initial = fixture.executor.register(MANIFEST).block()!! + fixture.operations.failValidationOnce.set(true) + val command = command(initial, ElasticsearchIndexLifecycleCommandType.VALIDATE, "validate-1") + + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.OPERATION_FAILED) { + fixture.executor.execute(command).block() + } + val claimed = fixture.repository.get(MANIFEST.id)!! + claimed.activeCommand!!.id.assert().isEqualTo(command.id) + + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.COMMAND_CONFLICT) { + fixture.executor.execute( + command(initial, ElasticsearchIndexLifecycleCommandType.VALIDATE, "validate-2"), + ).block() + } + + val resumed = fixture.executor.execute(command).block()!! + resumed.phase.assert().isEqualTo(ElasticsearchIndexLifecyclePhase.VALIDATED) + resumed.lastCompletedCommand!!.id.assert().isEqualTo(command.id) + fixture.operations.validateCalls.get().assert().isEqualTo(2) + + fixture.executor.execute(command).block()!!.assert().isEqualTo(resumed) + fixture.operations.validateCalls.get().assert().isEqualTo(2) + } + + @Test + fun `dry run should be read only and describe the transition`() { + val fixture = fixture() + val initial = fixture.executor.register(MANIFEST).block()!! + val plan = fixture.executor.plan( + command(initial, ElasticsearchIndexLifecycleCommandType.VALIDATE, "validate-plan"), + ).block()!! + + plan.from.assert().isEqualTo(ElasticsearchIndexLifecyclePhase.NEW) + plan.to.assert().isEqualTo(ElasticsearchIndexLifecyclePhase.VALIDATED) + fixture.repository.get(MANIFEST.id).assert().isEqualTo(initial) + fixture.operations.calls.assert().isEmpty() + } + + private fun fixture( + cutoverGuard: ElasticsearchSnapshotCutoverGuard = ElasticsearchSnapshotCutoverGuard { _, _ -> Mono.empty() }, + ): Fixture { + val repository = InMemoryElasticsearchIndexLifecycleRepository() + val operations = ProbeOperations() + return Fixture( + repository, + operations, + ElasticsearchIndexLifecycleExecutor( + repository, + operations, + Clock.fixed(NOW, ZoneOffset.UTC), + cutoverGuard, + ), + ) + } + + private fun assertLifecycle( + code: ElasticsearchIndexLifecycleErrorCode, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> error.code.assert().isEqualTo(code) }, + ) + } + + private fun Fixture.execute( + state: ElasticsearchIndexMigrationState, + type: ElasticsearchIndexLifecycleCommandType, + ): ElasticsearchIndexMigrationState = executor.execute( + command(state, type, "${type.name.lowercase()}-${state.revision}"), + ).block()!! + + private fun command( + state: ElasticsearchIndexMigrationState, + type: ElasticsearchIndexLifecycleCommandType, + id: String, + ) = ElasticsearchIndexLifecycleCommand( + ElasticsearchIndexLifecycleCommandId(id), + state.manifest.id, + type, + state.revision, + ) + + private class ProbeOperations : ElasticsearchIndexLifecycleOperations { + val calls = mutableListOf() + val validateCalls = AtomicInteger() + val failValidationOnce = AtomicBoolean() + val verificationMatches = AtomicBoolean(true) + + override fun validate( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = Mono.fromCallable { + calls += "validate" + validateCalls.incrementAndGet() + if (failValidationOnce.compareAndSet(true, false)) error("transient validation failure") + ElasticsearchIndexInventory( + manifest.names.alias, + SOURCE, + mapOf(SOURCE to sourceAttestation()), + NOW, + ) + } + + override fun create( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = Mono.fromCallable { + calls += "create" + manifest.destinationAttestation + } + + override fun rebuild( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = Mono.fromCallable { + calls += "rebuild" + ElasticsearchIndexRebuildReceipt( + manifest.names.physical, + manifest.rebuildStrategy, + authoritativeWatermark = 42, + indexedWatermark = 42, + completedAt = NOW, + ) + } + + override fun verify( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + physicalIndex: ElasticsearchPhysicalIndex, + ): Mono = Mono.fromCallable { + calls += "verify:${physicalIndex.value}" + val actual = if (verificationMatches.get()) 10L else 9L + verification(physicalIndex, actual) + } + + override fun cutover( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = Mono.fromCallable { + calls += "cutover" + ElasticsearchAliasTransition(manifest.names.alias, SOURCE, DESTINATION, NOW) + } + + override fun rollback( + command: ElasticsearchIndexLifecycleCommandId, + manifest: ElasticsearchIndexMigrationManifest, + ): Mono = Mono.fromCallable { + calls += "rollback" + ElasticsearchAliasTransition(manifest.names.alias, DESTINATION, SOURCE, NOW) + } + } + + private data class Fixture( + val repository: InMemoryElasticsearchIndexLifecycleRepository, + val operations: ProbeOperations, + val executor: ElasticsearchIndexLifecycleExecutor, + ) + + private companion object { + val NOW: Instant = Instant.parse("2026-08-08T02:00:00Z") + val TARGET = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + val SOURCE = ElasticsearchPhysicalIndex("wow.sales.order.snapshot-v0001-000001") + val DESTINATION = ElasticsearchPhysicalIndex("wow.sales.order.snapshot-v0002-000007") + val MANIFEST = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("sales-order-snapshot-v2-g7"), + TARGET, + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(7), + SchemaContractId("1".repeat(64)), + ElasticsearchIndexCapabilityDigest("a".repeat(64)), + SOURCE, + ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM, + testVerificationContract(), + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + + fun sourceAttestation() = ElasticsearchIndexAttestation( + SOURCE, + ElasticsearchIndexMappingVersion(1), + QueryDocumentKind.SNAPSHOT, + SchemaContractId("0".repeat(64)), + ElasticsearchIndexCapabilityDigest("c".repeat(64)), + ) + + fun verification( + physical: ElasticsearchPhysicalIndex, + actualCount: Long, + ) = ElasticsearchIndexVerification( + physical, + expectedCount = 10, + actualCount = actualCount, + expectedIdentityChecksum = ElasticsearchIndexChecksum("2".repeat(64)), + actualIdentityChecksum = ElasticsearchIndexChecksum("2".repeat(64)), + expectedContentChecksum = ElasticsearchIndexChecksum("3".repeat(64)), + actualContentChecksum = ElasticsearchIndexChecksum("3".repeat(64)), + versionContinuity = true, + authoritativeWatermark = 42, + indexedWatermark = 42, + recordProbeMismatchCount = 0, + analyticsProbeMismatchCount = 0, + verifiedAt = NOW, + ) + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleModelTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleModelTest.kt new file mode 100644 index 00000000000..8a1fcc80c39 --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleModelTest.kt @@ -0,0 +1,267 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import org.junit.jupiter.api.Test +import java.time.Duration +import java.time.Instant +import java.util.function.Consumer + +class ElasticsearchIndexLifecycleModelTest { + @Test + fun `names should retain the stable alias and encode mapping version plus generation`() { + val snapshot = ElasticsearchIndexNames.of(SNAPSHOT_TARGET, VERSION, GENERATION) + val eventStream = ElasticsearchIndexNames.of(EVENT_TARGET, VERSION, GENERATION) + + snapshot.alias.value.assert().isEqualTo("wow.sales.order.snapshot") + snapshot.physical.value.assert().isEqualTo("wow.sales.order.snapshot-v0002-000007") + eventStream.alias.value.assert().isEqualTo("wow.sales.order.es") + eventStream.physical.value.assert().isEqualTo("wow.sales.order.es-v0002-000007") + } + + @Test + fun `verification checksum algorithm must match the document kind`() { + assertThrownBy { + eventManifest().copy(verificationContract = testVerificationContract()) + } + assertThrownBy { + manifest().copy(verificationContract = testEventStreamVerificationContract()) + } + } + + @Test + fun `validation should bind the exact alias source and destination attestation`() { + val manifest = manifest() + val expectedDestination = manifest.destinationAttestation + manifest.validate( + ElasticsearchIndexInventory( + manifest.names.alias, + SOURCE, + mapOf(SOURCE to sourceAttestation(), manifest.names.physical to expectedDestination), + OBSERVED_AT, + ), + ) + + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.ALIAS_CONFLICT) { + val drifted = ElasticsearchPhysicalIndex("wow.sales.order.snapshot-v0001-000002") + manifest.validate( + ElasticsearchIndexInventory( + manifest.names.alias, + drifted, + mapOf( + SOURCE to sourceAttestation(), + drifted to sourceAttestation().copy(physicalIndex = drifted), + ), + OBSERVED_AT, + ), + ) + } + + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.ATTESTATION_MISMATCH) { + manifest.validate( + ElasticsearchIndexInventory( + manifest.names.alias, + SOURCE, + mapOf( + SOURCE to sourceAttestation(), + manifest.names.physical to expectedDestination.copy( + capabilityDigest = ElasticsearchIndexCapabilityDigest("b".repeat(64)), + ), + ), + OBSERVED_AT, + ), + ) + } + } + + @Test + fun `inventory and evidence should be immutable value objects`() { + val mutable = linkedMapOf(SOURCE to sourceAttestation()) + val inventory = ElasticsearchIndexInventory(manifest().names.alias, SOURCE, mutable, OBSERVED_AT) + mutable.clear() + + inventory.indices.keys.assert().containsExactly(SOURCE) + assertThrownBy { + @Suppress("UNCHECKED_CAST") + (inventory.indices as MutableMap).clear() + } + inventory.assert().isEqualTo( + ElasticsearchIndexInventory( + manifest().names.alias, + SOURCE, + mapOf(SOURCE to sourceAttestation()), + OBSERVED_AT, + ), + ) + } + + @Test + fun `verification must prove exact data and probe equivalence`() { + verification().requireSatisfied(manifest(), manifest().names.physical) + + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED) { + verification(actualCount = 9).requireSatisfied(manifest(), manifest().names.physical) + } + + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED) { + verification(indexedWatermark = 41).requireSatisfied(manifest(), manifest().names.physical) + } + } + + @Test + fun `event stream rebuild and verification must prove a converged non-null watermark`() { + val manifest = eventManifest() + ElasticsearchIndexRebuildReceipt( + manifest.names.physical, + manifest.rebuildStrategy, + authoritativeWatermark = 42, + indexedWatermark = 42, + completedAt = OBSERVED_AT, + ).requireMatches(manifest) + eventVerification(manifest, 42, 42).requireSatisfied(manifest, manifest.names.physical) + + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED) { + ElasticsearchIndexRebuildReceipt( + manifest.names.physical, + manifest.rebuildStrategy, + authoritativeWatermark = null, + indexedWatermark = null, + completedAt = OBSERVED_AT, + ).requireMatches(manifest) + } + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED) { + ElasticsearchIndexRebuildReceipt( + manifest.names.physical, + manifest.rebuildStrategy, + authoritativeWatermark = 42, + indexedWatermark = 41, + completedAt = OBSERVED_AT, + ).requireMatches(manifest) + } + assertLifecycle(ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED) { + eventVerification(manifest, null, null).requireSatisfied(manifest, manifest.names.physical) + } + } + + private fun assertLifecycle( + code: ElasticsearchIndexLifecycleErrorCode, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> error.code.assert().isEqualTo(code) }, + ) + } + + private fun manifest() = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("sales-order-snapshot-v2-g7"), + SNAPSHOT_TARGET, + VERSION, + GENERATION, + SchemaContractId("1".repeat(64)), + ElasticsearchIndexCapabilityDigest("a".repeat(64)), + SOURCE, + ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM, + testVerificationContract(), + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + + private fun eventManifest() = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("sales-order-event-stream-v2-g7"), + EVENT_TARGET, + VERSION, + GENERATION, + SchemaContractId("1".repeat(64)), + ElasticsearchIndexCapabilityDigest("a".repeat(64)), + EVENT_SOURCE, + ElasticsearchIndexRebuildStrategy.EVENT_STREAM_PAUSE_AND_DRAIN, + testEventStreamVerificationContract(), + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + + private fun sourceAttestation() = ElasticsearchIndexAttestation( + SOURCE, + ElasticsearchIndexMappingVersion(1), + QueryDocumentKind.SNAPSHOT, + SchemaContractId("0".repeat(64)), + ElasticsearchIndexCapabilityDigest("c".repeat(64)), + ) + + private fun verification( + actualCount: Long = 10, + indexedWatermark: Long = 42, + ) = ElasticsearchIndexVerification( + manifest().names.physical, + expectedCount = 10, + actualCount = actualCount, + expectedIdentityChecksum = ElasticsearchIndexChecksum("2".repeat(64)), + actualIdentityChecksum = ElasticsearchIndexChecksum("2".repeat(64)), + expectedContentChecksum = ElasticsearchIndexChecksum("3".repeat(64)), + actualContentChecksum = ElasticsearchIndexChecksum("3".repeat(64)), + versionContinuity = true, + authoritativeWatermark = 42, + indexedWatermark = indexedWatermark, + recordProbeMismatchCount = 0, + analyticsProbeMismatchCount = 0, + verifiedAt = OBSERVED_AT, + ) + + private fun eventVerification( + manifest: ElasticsearchIndexMigrationManifest, + authoritativeWatermark: Long?, + indexedWatermark: Long?, + ) = ElasticsearchIndexVerification( + manifest.names.physical, + expectedCount = 10, + actualCount = 10, + expectedIdentityChecksum = ElasticsearchIndexChecksum("2".repeat(64)), + actualIdentityChecksum = ElasticsearchIndexChecksum("2".repeat(64)), + expectedContentChecksum = ElasticsearchIndexChecksum("3".repeat(64)), + actualContentChecksum = ElasticsearchIndexChecksum("3".repeat(64)), + versionContinuity = true, + authoritativeWatermark = authoritativeWatermark, + indexedWatermark = indexedWatermark, + recordProbeMismatchCount = 0, + analyticsProbeMismatchCount = 0, + verifiedAt = OBSERVED_AT, + ) + + private companion object { + val SNAPSHOT_TARGET = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + val EVENT_TARGET = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.EVENT_STREAM, + ) + val VERSION = ElasticsearchIndexMappingVersion(2) + val GENERATION = ElasticsearchIndexGeneration(7) + val SOURCE = ElasticsearchPhysicalIndex("wow.sales.order.snapshot-v0001-000001") + val EVENT_SOURCE = ElasticsearchPhysicalIndex("wow.sales.order.es-v0001-000001") + val OBSERVED_AT: Instant = Instant.parse("2026-08-08T01:00:00Z") + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleStateCodecTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleStateCodecTest.kt new file mode 100644 index 00000000000..b246bd4e092 --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleStateCodecTest.kt @@ -0,0 +1,158 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import org.junit.jupiter.api.Test +import java.nio.ByteBuffer +import java.time.Duration +import java.time.Instant + +class ElasticsearchIndexLifecycleStateCodecTest { + private val codec = ElasticsearchIndexLifecycleStateCodec() + + @Test + fun `should round trip every persisted lifecycle field`() { + val state = completeState() + + val decoded = codec.decode(MIGRATION_ID, codec.encode(state)) + + decoded.assert().isEqualTo(state) + } + + @Test + fun `should reject corrupt mismatched trailing and oversized payloads`() { + val encoded = codec.encode(ElasticsearchIndexMigrationState.initial(MANIFEST)) + val corrupt = encoded.copyOf().also { it[0] = (it[0].toInt() xor 0x7f).toByte() } + val versionOne = encoded.copyOf().also { ByteBuffer.wrap(it).putInt(Int.SIZE_BYTES, 1) } + + assertCorrupt { codec.decode(MIGRATION_ID, corrupt) } + assertCorrupt { codec.decode(MIGRATION_ID, versionOne) } + assertCorrupt { codec.decode(ElasticsearchIndexMigrationId("another-migration"), encoded) } + assertCorrupt { codec.decode(MIGRATION_ID, encoded + 0) } + assertCorrupt { + codec.decode( + MIGRATION_ID, + ByteArray(ElasticsearchIndexLifecycleStateCodec.MAX_PAYLOAD_BYTES + 1), + ) + } + } + + private fun completeState(): ElasticsearchIndexMigrationState { + val sourceAttestation = ElasticsearchIndexAttestation( + SOURCE, + ElasticsearchIndexMappingVersion(1), + QueryDocumentKind.SNAPSHOT, + SchemaContractId("0".repeat(64)), + ElasticsearchIndexCapabilityDigest("c".repeat(64)), + ) + val destinationAttestation = MANIFEST.destinationAttestation + val inventory = ElasticsearchIndexInventory( + MANIFEST.names.alias, + SOURCE, + linkedMapOf(SOURCE to sourceAttestation, DESTINATION to destinationAttestation), + NOW.minusSeconds(60), + ) + val rebuild = ElasticsearchIndexRebuildReceipt( + DESTINATION, + ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM, + authoritativeWatermark = 123, + indexedWatermark = 123, + completedAt = NOW.minusSeconds(40), + ) + val destinationVerification = verification(DESTINATION, NOW.minusSeconds(30)) + val rollbackVerification = verification(SOURCE, NOW.minusSeconds(10)) + return ElasticsearchIndexMigrationState( + manifest = MANIFEST, + phase = ElasticsearchIndexLifecyclePhase.ROLLBACK_VERIFIED, + revision = 12, + activeCommand = ElasticsearchIndexActiveCommand( + ElasticsearchIndexLifecycleCommandId("rollback-command"), + ElasticsearchIndexLifecycleCommandType.ROLLBACK, + ElasticsearchIndexLifecyclePhase.ROLLBACK_VERIFIED, + ElasticsearchIndexLifecyclePhase.ROLLED_BACK, + NOW, + ), + lastCompletedCommand = ElasticsearchIndexCompletedCommand( + ElasticsearchIndexLifecycleCommandId("verify-rollback-command"), + ElasticsearchIndexLifecycleCommandType.VERIFY, + NOW.minusSeconds(5), + ), + inventory = inventory, + destinationAttestation = destinationAttestation, + rebuildReceipt = rebuild, + destinationVerification = destinationVerification, + cutover = ElasticsearchAliasTransition(MANIFEST.names.alias, SOURCE, DESTINATION, NOW.minusSeconds(20)), + rollbackVerification = rollbackVerification, + rollback = ElasticsearchAliasTransition(MANIFEST.names.alias, DESTINATION, SOURCE, NOW.plusSeconds(10)), + retainedSourceUntil = NOW.plus(MANIFEST.minimumRetention), + ) + } + + private fun verification(index: ElasticsearchPhysicalIndex, verifiedAt: Instant) = + ElasticsearchIndexVerification( + index, + expectedCount = 10, + actualCount = 10, + expectedIdentityChecksum = ElasticsearchIndexChecksum("1".repeat(64)), + actualIdentityChecksum = ElasticsearchIndexChecksum("1".repeat(64)), + expectedContentChecksum = ElasticsearchIndexChecksum("2".repeat(64)), + actualContentChecksum = ElasticsearchIndexChecksum("2".repeat(64)), + versionContinuity = true, + authoritativeWatermark = 123, + indexedWatermark = 123, + recordProbeMismatchCount = 0, + analyticsProbeMismatchCount = 0, + verifiedAt = verifiedAt, + ) + + private fun assertCorrupt(action: () -> Unit) { + assertThrownBy(action).satisfies( + java.util.function.Consumer { error -> + error.code.assert().isEqualTo(ElasticsearchIndexLifecycleErrorCode.REPOSITORY_CORRUPTED) + }, + ) + } + + private companion object { + val NOW: Instant = Instant.parse("2026-08-08T04:00:00Z") + val MIGRATION_ID = ElasticsearchIndexMigrationId("lifecycle-order-snapshot-v2-g7") + val TARGET = QueryTarget(MaterializedNamedAggregate("lifecycle", "order"), QueryDocumentKind.SNAPSHOT) + val SOURCE = ElasticsearchPhysicalIndex("wow.lifecycle.order.snapshot-v0001-000001") + val DESTINATION = ElasticsearchPhysicalIndex("wow.lifecycle.order.snapshot-v0002-000007") + val MANIFEST = ElasticsearchIndexMigrationManifest( + MIGRATION_ID, + TARGET, + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(7), + SchemaContractId("1".repeat(64)), + ElasticsearchIndexCapabilityDigest("a".repeat(64)), + SOURCE, + ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM, + testVerificationContract(), + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleTestFixtures.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleTestFixtures.kt new file mode 100644 index 00000000000..78cd4b61b88 --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/ElasticsearchIndexLifecycleTestFixtures.kt @@ -0,0 +1,24 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.elasticsearch.query.lifecycle + +internal fun testVerificationContract() = ElasticsearchIndexVerificationContract( + ElasticsearchIndexChecksumAlgorithm.CANONICAL_DOCUMENT_SHA256_V1, + ElasticsearchIndexProbeSuiteId("test-probes-v1"), +) + +internal fun testEventStreamVerificationContract() = ElasticsearchIndexVerificationContract( + ElasticsearchIndexChecksumAlgorithm.CANONICAL_EVENT_STREAM_SHA256_V1, + ElasticsearchIndexProbeSuiteId("test-event-stream-probes-v1"), +) diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStoreElasticsearchEventStreamIndexRebuilderTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStoreElasticsearchEventStreamIndexRebuilderTest.kt new file mode 100644 index 00000000000..1d7a2aa0139 --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStoreElasticsearchEventStreamIndexRebuilderTest.kt @@ -0,0 +1,209 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.event.DomainEventStream +import me.ahoo.wow.eventsourcing.AggregateIdScanner +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.modeling.aggregateId +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.tck.event.MockDomainEventStreams +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset +import java.util.concurrent.atomic.AtomicInteger +import java.util.function.Consumer + +class EventStoreElasticsearchEventStreamIndexRebuilderTest { + @Test + fun `should rebuild every ordered event stream while the write fence remains drained`() { + val first = eventStream("001") + val second = eventStream("002") + val third = eventStream("003") + val scans = mutableListOf>() + val writes = mutableListOf>() + val checkpoints = ArrayDeque(listOf(42L, 42L)) + val source = ElasticsearchAuthoritativeEventStreamSource { target, afterId, limit -> + target.assert().isEqualTo(TARGET) + scans += afterId to limit + Flux.fromIterable( + when (afterId) { + AggregateIdScanner.FIRST_ID -> listOf(first, second) + "002" -> listOf(third) + else -> emptyList() + }, + ) + } + val writer = ElasticsearchPhysicalEventStreamWriter { index, stream -> + writes += index to "${stream.aggregateId.id}:${stream.version}" + Mono.empty() + } + val barrier = ElasticsearchEventStreamMigrationBarrier { command, manifest -> + command.assert().isEqualTo(COMMAND) + manifest.assert().isEqualTo(MANIFEST) + Mono.just(checkpoints.removeFirst()) + } + + val receipt = rebuilder(source, writer, barrier).rebuild(COMMAND, MANIFEST).block()!! + + scans.assert().containsExactly(AggregateIdScanner.FIRST_ID to 2, "002" to 2) + writes.assert().containsExactly( + DESTINATION to "001:1", + DESTINATION to "002:1", + DESTINATION to "003:1", + ) + receipt.assert().isEqualTo( + ElasticsearchIndexRebuildReceipt( + DESTINATION, + ElasticsearchIndexRebuildStrategy.EVENT_STREAM_PAUSE_AND_DRAIN, + 42, + 42, + NOW, + ), + ) + } + + @Test + fun `should fail closed when the write fence advances during the rebuild`() { + val writerCalls = AtomicInteger() + val checkpoints = ArrayDeque(listOf(42L, 43L)) + val source = ElasticsearchAuthoritativeEventStreamSource { _, _, _ -> Flux.just(eventStream("001")) } + val writer = ElasticsearchPhysicalEventStreamWriter { _, _ -> + writerCalls.incrementAndGet() + Mono.empty() + } + + assertThrownBy { + rebuilder( + source, + writer, + ElasticsearchEventStreamMigrationBarrier { _, _ -> Mono.just(checkpoints.removeFirst()) }, + ).rebuild(COMMAND, MANIFEST).block() + }.satisfies( + Consumer { error -> + error.code.assert().isEqualTo(ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED) + }, + ) + writerCalls.get().assert().isEqualTo(1) + } + + @Test + fun `should reject broken aggregate ordering version continuity and page bounds`() { + val writerCalls = AtomicInteger() + val writer = ElasticsearchPhysicalEventStreamWriter { _, _ -> + writerCalls.incrementAndGet() + Mono.empty() + } + val invalidSources = listOf( + listOf(eventStream("002"), eventStream("001")), + listOf(eventStream("001", aggregateVersion = 1)), + listOf(eventStream("001"), eventStream("002"), eventStream("003")), + ) + + invalidSources.forEach { streams -> + assertThrownBy { + rebuilder( + ElasticsearchAuthoritativeEventStreamSource { _, _, _ -> Flux.fromIterable(streams) }, + writer, + ElasticsearchEventStreamMigrationBarrier { _, _ -> Mono.just(42) }, + ).rebuild(COMMAND, MANIFEST).block() + } + } + // Validation is streaming; already validated immutable documents may be written before a later violation. + // Exact-generation writes are idempotent, while no rebuild receipt is emitted for any invalid source. + writerCalls.get().assert().isEqualTo(3) + } + + @Test + fun `should reject controlled mirror until its convergence controller is implemented`() { + val scans = AtomicInteger() + val source = ElasticsearchAuthoritativeEventStreamSource { _, _, _ -> + scans.incrementAndGet() + Flux.empty() + } + val mirrorManifest = MANIFEST.copy( + id = ElasticsearchIndexMigrationId("event-stream-mirror"), + rebuildStrategy = ElasticsearchIndexRebuildStrategy.EVENT_STREAM_CONTROLLED_MIRROR, + ) + + assertThrownBy { + rebuilder( + source, + ElasticsearchPhysicalEventStreamWriter { _, _ -> Mono.empty() }, + ElasticsearchEventStreamMigrationBarrier { _, _ -> Mono.just(42) }, + ).rebuild(COMMAND, mirrorManifest).block() + }.satisfies( + Consumer { error -> + error.code.assert().isEqualTo(ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED) + }, + ) + scans.get().assert().isZero() + } + + private fun rebuilder( + source: ElasticsearchAuthoritativeEventStreamSource, + writer: ElasticsearchPhysicalEventStreamWriter, + barrier: ElasticsearchEventStreamMigrationBarrier, + ) = EventStoreElasticsearchEventStreamIndexRebuilder( + source, + writer, + barrier, + CLOCK, + ElasticsearchEventStreamRebuildOptions(aggregateScanPageSize = 2, writeConcurrency = 1), + ) + + private fun eventStream(id: String, aggregateVersion: Int = 0): DomainEventStream = + MockDomainEventStreams.generateEventStream( + TARGET.namedAggregate.aggregateId(id, "tenant-1"), + aggregateVersion = aggregateVersion, + eventCount = 1, + ) + + private companion object { + val NOW: Instant = Instant.parse("2026-08-08T06:00:00Z") + val CLOCK: Clock = Clock.fixed(NOW, ZoneOffset.UTC) + val TARGET = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.EVENT_STREAM, + ) + val MANIFEST = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("event-stream-rebuild"), + TARGET, + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(3), + SchemaContractId("1".repeat(64)), + ElasticsearchIndexCapabilityDigest("2".repeat(64)), + ElasticsearchPhysicalIndex("wow.sales.order.es-v0001-000001"), + ElasticsearchIndexRebuildStrategy.EVENT_STREAM_PAUSE_AND_DRAIN, + testEventStreamVerificationContract(), + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + val DESTINATION: ElasticsearchPhysicalIndex = MANIFEST.names.physical + val COMMAND = ElasticsearchIndexLifecycleCommandId("event-stream-rebuild-command") + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStoreElasticsearchSnapshotIndexRebuilderTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStoreElasticsearchSnapshotIndexRebuilderTest.kt new file mode 100644 index 00000000000..edc87ec2b0f --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStoreElasticsearchSnapshotIndexRebuilderTest.kt @@ -0,0 +1,175 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import io.mockk.every +import io.mockk.mockk +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.eventsourcing.AggregateIdScanner +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.modeling.aggregateId +import me.ahoo.wow.modeling.state.ReadOnlyStateAggregate +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset +import java.util.concurrent.atomic.AtomicInteger +import java.util.function.Consumer + +class EventStoreElasticsearchSnapshotIndexRebuilderTest { + @Test + fun `should rebuild ordered authoritative pages into the exact physical index`() { + val first = aggregate("001", 1) + val second = aggregate("002", 2) + val third = aggregate("003", 3) + val requests = mutableListOf>() + val writes = mutableListOf>() + val source = ElasticsearchAuthoritativeSnapshotSource { target, afterId, limit -> + target.assert().isEqualTo(TARGET) + requests += afterId to limit + Flux.fromIterable( + when (afterId) { + AggregateIdScanner.FIRST_ID -> listOf(first, second) + "002" -> listOf(third) + else -> emptyList() + }, + ) + } + val writer = ElasticsearchPhysicalSnapshotWriter { index, snapshot -> + writes += index to snapshot.aggregateId.id + Mono.empty() + } + + val receipt = rebuilder(source, writer).rebuild(COMMAND, MANIFEST).block()!! + + requests.assert().containsExactly(AggregateIdScanner.FIRST_ID to 2, "002" to 2) + writes.assert().containsExactly( + DESTINATION to "001", + DESTINATION to "002", + DESTINATION to "003", + ) + receipt.assert().isEqualTo( + ElasticsearchIndexRebuildReceipt( + DESTINATION, + ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM, + null, + null, + NOW, + ), + ) + } + + @Test + fun `should reject unordered or over-sized source pages before writing`() { + val writes = AtomicInteger() + val writer = ElasticsearchPhysicalSnapshotWriter { _, _ -> + writes.incrementAndGet() + Mono.empty() + } + listOf( + listOf(aggregate("002", 1), aggregate("001", 1)), + listOf(aggregate("001", 1), aggregate("002", 1), aggregate("003", 1)), + ).forEach { page -> + val source = ElasticsearchAuthoritativeSnapshotSource { _, _, _ -> Flux.fromIterable(page) } + assertThrownBy { + rebuilder(source, writer).rebuild(COMMAND, MANIFEST).block() + } + } + writes.get().assert().isZero() + } + + @Test + fun `should reject a non-snapshot rebuild strategy before source access`() { + val scans = AtomicInteger() + val source = ElasticsearchAuthoritativeSnapshotSource { _, _, _ -> + scans.incrementAndGet() + Flux.empty() + } + val eventManifest = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("event-rebuild"), + QueryTarget(TARGET.namedAggregate, QueryDocumentKind.EVENT_STREAM), + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(3), + SchemaContractId("3".repeat(64)), + ElasticsearchIndexCapabilityDigest("4".repeat(64)), + ElasticsearchPhysicalIndex("wow.sales.order.event_stream-v0001-000001"), + ElasticsearchIndexRebuildStrategy.EVENT_STREAM_PAUSE_AND_DRAIN, + testEventStreamVerificationContract(), + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + + assertThrownBy { + rebuilder(source, ElasticsearchPhysicalSnapshotWriter { _, _ -> Mono.empty() }) + .rebuild(COMMAND, eventManifest) + .block() + }.satisfies( + Consumer { error -> + error.code.assert().isEqualTo(ElasticsearchIndexLifecycleErrorCode.VALIDATION_FAILED) + }, + ) + scans.get().assert().isZero() + } + + private fun rebuilder( + source: ElasticsearchAuthoritativeSnapshotSource, + writer: ElasticsearchPhysicalSnapshotWriter, + ) = EventStoreElasticsearchSnapshotIndexRebuilder( + source, + writer, + CLOCK, + ElasticsearchSnapshotRebuildOptions(scanPageSize = 2, writeConcurrency = 1), + ) + + private fun aggregate(id: String, aggregateVersion: Int): ReadOnlyStateAggregate = mockk { + every { aggregateId } returns TARGET.namedAggregate.aggregateId(id, "tenant-1") + every { version } returns aggregateVersion + } + + private companion object { + val NOW: Instant = Instant.parse("2026-08-08T00:00:00Z") + val CLOCK: Clock = Clock.fixed(NOW, ZoneOffset.UTC) + val TARGET = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + val MANIFEST = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("snapshot-rebuild"), + TARGET, + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(3), + SchemaContractId("1".repeat(64)), + ElasticsearchIndexCapabilityDigest("2".repeat(64)), + ElasticsearchPhysicalIndex("wow.sales.order.snapshot-v0001-000001"), + ElasticsearchIndexRebuildStrategy.SNAPSHOT_FROM_EVENT_STREAM, + testVerificationContract(), + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + val DESTINATION: ElasticsearchPhysicalIndex = MANIFEST.names.physical + val COMMAND = ElasticsearchIndexLifecycleCommandId("rebuild-command") + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStreamCanonicalChecksumTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStreamCanonicalChecksumTest.kt new file mode 100644 index 00000000000..70eeaddc2a1 --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStreamCanonicalChecksumTest.kt @@ -0,0 +1,107 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.serialization.MessageRecords +import org.junit.jupiter.api.Test +import java.math.BigDecimal + +class EventStreamCanonicalChecksumTest { + @Test + fun `map order and mathematical number representation should not change evidence`() { + val first = linkedMapOf( + MessageRecords.AGGREGATE_ID to "001", + MessageRecords.VERSION to 1, + MessageRecords.BODY to listOf(linkedMapOf("value" to BigDecimal("1.00"), "name" to "created")), + ) + val reordered = linkedMapOf( + MessageRecords.BODY to listOf(linkedMapOf("name" to "created", "value" to 1L)), + MessageRecords.VERSION to BigDecimal("1.0"), + MessageRecords.AGGREGATE_ID to "001", + ) + + evidence("001-1" to first).assert().isEqualTo(evidence("001-1" to reordered)) + } + + @Test + fun `content changes should preserve identity evidence and change content evidence`() { + val original = document("001", 1, listOf(mapOf("name" to "created"))) + val changed = document("001", 1, listOf(mapOf("name" to "changed"))) + + val first = evidence("001-1" to original) + val second = evidence("001-1" to changed) + + first.identityChecksum.assert().isEqualTo(second.identityChecksum) + first.contentChecksum.assert().isNotEqualTo(second.contentChecksum) + } + + @Test + fun `aggregate and version order must be complete and continuous`() { + evidence( + "001-1" to document("001", 1, listOf(1, 2)), + "001-3" to document("001", 3, listOf(3)), + "002-1" to document("002", 1, listOf(1)), + ).count.assert().isEqualTo(3) + + listOf( + arrayOf("001-2" to document("001", 2, listOf(1))), + arrayOf( + "001-1" to document("001", 1, listOf(1)), + "001-3" to document("001", 3, listOf(2)), + ), + arrayOf( + "002-1" to document("002", 1, listOf(1)), + "001-1" to document("001", 1, listOf(1)), + ), + arrayOf("other-1" to document("001", 1, listOf(1))), + arrayOf("001-1" to document("001", 1, emptyList())), + ).forEach { invalid -> + assertThrownBy { evidence(*invalid) } + } + } + + @Test + fun `version must be an exact positive integer within the supported range`() { + listOf( + BigDecimal("1.5"), + Long.MAX_VALUE, + 0, + Double.NaN, + Double.POSITIVE_INFINITY, + ).forEach { invalidVersion -> + assertThrownBy { + evidence("001-$invalidVersion" to document("001", invalidVersion, listOf(1))) + } + } + } + + private fun evidence( + vararg documents: Pair>, + ): EventStreamCanonicalChecksumEvidence = EventStreamCanonicalChecksumAccumulator().let { accumulator -> + documents.forEach { (identity, document) -> accumulator.accept(identity, document) } + accumulator.finish() + } + + private fun document( + aggregateId: String, + version: Number, + body: List, + ): Map = linkedMapOf( + MessageRecords.AGGREGATE_ID to aggregateId, + MessageRecords.VERSION to version, + MessageRecords.BODY to body, + ) +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStreamElasticsearchIndexVerificationSourcesTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStreamElasticsearchIndexVerificationSourcesTest.kt new file mode 100644 index 00000000000..c224f65a599 --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/EventStreamElasticsearchIndexVerificationSourcesTest.kt @@ -0,0 +1,112 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.event.DomainEventStream +import me.ahoo.wow.eventsourcing.AggregateIdScanner +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.modeling.aggregateId +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.toLinkedHashMap +import me.ahoo.wow.tck.event.MockDomainEventStreams +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Duration +import java.util.function.Consumer + +class EventStreamElasticsearchIndexVerificationSourcesTest { + @Test + fun `authority evidence should bind a stable drained watermark to the canonical stream checksum`() { + val first = eventStream("001") + val second = eventStream("002") + val checkpoints = ArrayDeque(listOf(42L, 42L)) + val source = ElasticsearchAuthoritativeEventStreamSource { _, afterId, _ -> + when (afterId) { + AggregateIdScanner.FIRST_ID -> Flux.just(first) + "001" -> Flux.just(second) + else -> Flux.empty() + } + } + val verification = EventStoreEventStreamVerificationSource( + source, + ElasticsearchEventStreamMigrationBarrier { _, _ -> Mono.just(checkpoints.removeFirst()) }, + EventStreamElasticsearchIndexVerificationOptions(aggregateScanPageSize = 1), + ).capture(COMMAND, MANIFEST).block()!! + + val expected = EventStreamCanonicalChecksumAccumulator().also { accumulator -> + listOf(first, second).forEach { stream -> + accumulator.accept(stream.eventStreamDocumentId(), stream.toLinkedHashMap()) + } + }.finish() + verification.count.assert().isEqualTo(2) + verification.identityChecksum.assert().isEqualTo(expected.identityChecksum) + verification.contentChecksum.assert().isEqualTo(expected.contentChecksum) + verification.watermark.assert().isEqualTo(42) + verification.checksumAlgorithm.assert() + .isEqualTo(ElasticsearchIndexChecksumAlgorithm.CANONICAL_EVENT_STREAM_SHA256_V1) + } + + @Test + fun `authority evidence should fail when the write fence advances`() { + val checkpoints = ArrayDeque(listOf(42L, 43L)) + val source = ElasticsearchAuthoritativeEventStreamSource { _, _, _ -> Flux.just(eventStream("001")) } + + assertThrownBy { + EventStoreEventStreamVerificationSource( + source, + ElasticsearchEventStreamMigrationBarrier { _, _ -> Mono.just(checkpoints.removeFirst()) }, + ).capture(COMMAND, MANIFEST).block() + }.satisfies( + Consumer { error -> + error.code.assert().isEqualTo(ElasticsearchIndexLifecycleErrorCode.VERIFICATION_FAILED) + }, + ) + } + + private fun eventStream(id: String): DomainEventStream = MockDomainEventStreams.generateEventStream( + TARGET.namedAggregate.aggregateId(id, "tenant-1"), + eventCount = 1, + ) + + private companion object { + val TARGET = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.EVENT_STREAM, + ) + val MANIFEST = ElasticsearchIndexMigrationManifest( + ElasticsearchIndexMigrationId("event-stream-verification"), + TARGET, + ElasticsearchIndexMappingVersion(2), + ElasticsearchIndexGeneration(4), + SchemaContractId("1".repeat(64)), + ElasticsearchIndexCapabilityDigest("2".repeat(64)), + ElasticsearchPhysicalIndex("wow.sales.order.es-v0001-000001"), + ElasticsearchIndexRebuildStrategy.EVENT_STREAM_PAUSE_AND_DRAIN, + testEventStreamVerificationContract(), + Duration.ofMinutes(5), + Duration.ofHours(1), + ) + val COMMAND = ElasticsearchIndexLifecycleCommandId("event-stream-verify-command") + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/SnapshotCanonicalChecksumTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/SnapshotCanonicalChecksumTest.kt new file mode 100644 index 00000000000..ff839cbad1b --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/lifecycle/SnapshotCanonicalChecksumTest.kt @@ -0,0 +1,94 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.elasticsearch.query.lifecycle + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.serialization.state.SnapshotRecords +import org.junit.jupiter.api.Test +import java.math.BigDecimal + +class SnapshotCanonicalChecksumTest { + @Test + fun `should ignore snapshot time and canonicalize object order and numeric representation`() { + val first = checksum( + linkedMapOf( + MessageRecords.AGGREGATE_ID to "order-1", + MessageRecords.VERSION to 1, + SnapshotRecords.SNAPSHOT_TIME to 1L, + "state" to linkedMapOf("quantity" to 1, "name" to "cart"), + ), + ) + val equivalent = checksum( + linkedMapOf( + "state" to linkedMapOf("name" to "cart", "quantity" to BigDecimal("1.00")), + SnapshotRecords.SNAPSHOT_TIME to 9_999L, + MessageRecords.VERSION to 1L, + MessageRecords.AGGREGATE_ID to "order-1", + ), + ) + + equivalent.assert().isEqualTo(first) + } + + @Test + fun `should change only content checksum when document content changes`() { + val original = checksum(document("order-1", "created")) + val changed = checksum(document("order-1", "paid")) + + changed.identityChecksum.assert().isEqualTo(original.identityChecksum) + changed.contentChecksum.assert().isNotEqualTo(original.contentChecksum) + } + + @Test + fun `should reject unordered duplicate or mismatched identities`() { + val accumulator = SnapshotCanonicalChecksumAccumulator() + accumulator.accept("order-2", document("order-2", "created")) + + assertThrownBy { + accumulator.accept("order-1", document("order-1", "created")) + }.hasMessageContaining("strictly ascending") + assertThrownBy { + SnapshotCanonicalChecksumAccumulator().accept("order-1", document("order-2", "created")) + }.hasMessageContaining("serialized aggregate id") + } + + @Test + fun `should reject cyclic or over-budget documents before producing evidence`() { + val cyclic = linkedMapOf() + cyclic[MessageRecords.AGGREGATE_ID] = "order-1" + cyclic[MessageRecords.VERSION] = 1 + cyclic["state"] = cyclic + + assertThrownBy { + SnapshotCanonicalChecksumAccumulator().accept("order-1", cyclic) + }.hasMessageContaining("cycle") + assertThrownBy { + SnapshotCanonicalChecksumAccumulator( + SnapshotCanonicalChecksumLimits(maxPayloadBytesPerDocument = 4), + ).accept("order-1", document("order-1", "created")) + }.hasMessageContaining("payload limit") + } + + private fun checksum(document: Map): SnapshotCanonicalChecksumEvidence = + SnapshotCanonicalChecksumAccumulator().apply { accept("order-1", document) }.finish() + + private fun document(identity: String, status: String): Map = linkedMapOf( + MessageRecords.AGGREGATE_ID to identity, + MessageRecords.VERSION to 1, + SnapshotRecords.SNAPSHOT_TIME to 1L, + "state" to linkedMapOf("status" to status), + ) +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchAnalyticsQueryBackendTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchAnalyticsQueryBackendTest.kt new file mode 100644 index 00000000000..a8665c9fa43 --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchAnalyticsQueryBackendTest.kt @@ -0,0 +1,448 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.planned + +import co.elastic.clients.elasticsearch._types.FieldValue +import co.elastic.clients.elasticsearch._types.aggregations.Aggregate +import co.elastic.clients.elasticsearch._types.aggregations.CompositeBucket +import co.elastic.clients.elasticsearch.core.ClosePointInTimeRequest +import co.elastic.clients.elasticsearch.core.ClosePointInTimeResponse +import co.elastic.clients.elasticsearch.core.OpenPointInTimeRequest +import co.elastic.clients.elasticsearch.core.OpenPointInTimeResponse +import co.elastic.clients.elasticsearch.core.SearchRequest +import co.elastic.clients.elasticsearch.core.SearchResponse +import co.elastic.clients.elasticsearch.core.search.TotalHitsRelation +import io.mockk.confirmVerified +import io.mockk.every +import io.mockk.mockk +import io.mockk.slot +import io.mockk.verify +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.BackendAnalyticsBucketOrder +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsCondition +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsDimension +import me.ahoo.wow.query.backend.BackendAnalyticsGrouping +import me.ahoo.wow.query.backend.BackendAnalyticsMetric +import me.ahoo.wow.query.backend.BackendAnalyticsMissingPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNullPlacement +import me.ahoo.wow.query.backend.BackendAnalyticsPageWindow +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.BackendAnalyticsTextCollation +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendRequiredCapabilities +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import org.junit.jupiter.api.Test +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Mono +import reactor.test.StepVerifier +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset +import java.util.function.Consumer + +class ElasticsearchAnalyticsQueryBackendTest { + @Test + fun `grouped response should use Elasticsearch response after key instead of deriving one`() { + val client = mockk() + val request = slot() + every { client.search(capture(request), Map::class.java) } returns Mono.just( + response( + aggregation = composite( + listOf(bucket("PAID", 2)), + mapOf("status" to FieldValue.of("SHIPPED")), + ), + ), + ) + + val page = backend(client).analyze(groupedPlan(), OPTIONS).block()!! + + page.buckets.single().keys[STATUS_ALIAS].assert().isEqualTo(NormalizedValue.Text("PAID")) + page.buckets.single().metrics[COUNT_ALIAS].assert().isEqualTo(NormalizedValue.Int64(2)) + page.afterKey!!.assert().containsExactly(NormalizedValue.Text("SHIPPED")) + page.consistency.assert().isEqualTo(BackendAnalyticsConsistency.EVENTUAL) + page.completeness.assert().isEqualTo(BackendAnalyticsCompleteness.EXACT) + request.captured.size().assert().isZero() + request.captured.aggregations().keys.assert().containsExactly(ANALYTICS_AGGREGATION) + } + + @Test + fun `global document count should require exact total and map zero safely`() { + val client = mockk() + every { client.search(any(), Map::class.java) } returnsMany listOf( + Mono.just(response(total = 0, totalRelation = TotalHitsRelation.Eq)), + Mono.just(response(total = 1, totalRelation = TotalHitsRelation.Gte)), + ) + + val page = backend(client).analyze(globalPlan(), OPTIONS).block()!! + page.buckets.single().metrics[COUNT_ALIAS].assert().isEqualTo(NormalizedValue.Int64(0)) + page.afterKey.assert().isNull() + assertFailure(QueryBackendFailureKind.INCOMPLETE_RESULT) { + backend(client).analyze(globalPlan(), OPTIONS).block() + } + } + + @Test + fun `timeout failed shards and malformed after key should fail closed`() { + val client = mockk() + every { client.search(any(), Map::class.java) } returnsMany listOf( + Mono.just(response(timedOut = true)), + Mono.just(response(failedShards = 1)), + Mono.just( + response( + aggregation = composite( + listOf(bucket("PAID", 1)), + mapOf("unexpected" to FieldValue.of("PAID")), + ), + ), + ), + ) + + assertFailure(QueryBackendFailureKind.TIMEOUT) { + backend(client).analyze(groupedPlan(), OPTIONS).block() + } + assertFailure(QueryBackendFailureKind.INCOMPLETE_RESULT) { + backend(client).analyze(groupedPlan(), OPTIONS).block() + } + assertFailure(QueryBackendFailureKind.MAPPING_FAILURE) { + backend(client).analyze(groupedPlan(), OPTIONS).block() + } + } + + @Test + fun `null composite bucket and response cursor should remain canonical null values`() { + val client = mockk() + every { client.search(any(), Map::class.java) } returns Mono.just( + response( + aggregation = composite( + listOf(bucket(FieldValue.NULL, 2)), + mapOf("status" to FieldValue.NULL), + ), + ), + ) + + val page = backend(client).analyze( + groupedPlan(BackendAnalyticsMissingPolicy.AS_NULL_BUCKET), + OPTIONS, + ).block()!! + + page.buckets.single().keys[STATUS_ALIAS].assert().isEqualTo(NormalizedValue.Null) + page.buckets.single().metrics[COUNT_ALIAS].assert().isEqualTo(NormalizedValue.Int64(2)) + page.afterKey!!.assert().containsExactly(NormalizedValue.Null) + } + + @Test + fun `invalid contracts and unsupported budgets should fail before Elasticsearch IO`() { + val client = mockk() + val backend = backend(client) + + assertFailure(QueryBackendFailureKind.UNSUPPORTED) { + backend.analyze(groupedPlan(), OPTIONS.copy(maxCandidateBuckets = 1)).block() + } + assertFailure(QueryBackendFailureKind.TIMEOUT) { + backend.analyze(groupedPlan(), OPTIONS.copy(deadline = NOW.minusMillis(1))).block() + } + + confirmVerified(client) + } + + @Test + fun `snapshot analytics should transfer the latest PIT state and close it through the lifecycle`() { + val client = mockk() + val firstRequest = slot() + val continuedRequest = slot() + val closeRequest = slot() + every { client.openPointInTime(any()) } returns Mono.just(openPit("pit-1")) + every { client.search(capture(firstRequest), Map::class.java) } returns Mono.just( + response( + aggregation = composite( + listOf(bucket("PAID", 2)), + mapOf("status" to FieldValue.of("PAID")), + ), + pitId = "pit-2", + ), + ) + val backend = backend(client) + + val first = backend.analyze(snapshotGroupedPlan(), OPTIONS, null).block()!! + first.consistency.assert().isEqualTo(BackendAnalyticsConsistency.SNAPSHOT) + first.cursorState!!.payload().decodeToString().assert().isEqualTo("pit-2") + firstRequest.captured.index().assert().isEmpty() + firstRequest.captured.pit()!!.id().assert().isEqualTo("pit-1") + verify(exactly = 0) { client.closePointInTime(any()) } + + every { client.search(capture(continuedRequest), Map::class.java) } returns Mono.just( + response( + aggregation = composite(listOf(bucket("SHIPPED", 1)), emptyMap()), + pitId = "pit-3", + ), + ) + val continued = backend.analyze( + snapshotGroupedPlan(listOf(NormalizedValue.Text("PAID"))), + OPTIONS, + first.cursorState, + ).block()!! + continued.afterKey.assert().isNull() + continued.cursorState!!.payload().decodeToString().assert().isEqualTo("pit-3") + continuedRequest.captured.pit()!!.id().assert().isEqualTo("pit-2") + verify(exactly = 1) { client.openPointInTime(any()) } + + every { client.closePointInTime(capture(closeRequest)) } returns Mono.just( + ClosePointInTimeResponse.of { response -> response.succeeded(true).numFreed(1) }, + ) + backend.close(continued.cursorState!!).block() + closeRequest.captured.id().assert().isEqualTo("pit-3") + } + + @Test + fun `snapshot analytics cancellation should close a PIT before ownership transfer`() { + val client = mockk() + val closeRequest = slot() + every { client.openPointInTime(any()) } returns Mono.just(openPit("pit-cancel")) + every { client.search(any(), Map::class.java) } returns Mono.never() + every { client.closePointInTime(capture(closeRequest)) } returns Mono.just( + ClosePointInTimeResponse.of { response -> response.succeeded(true).numFreed(1) }, + ) + + StepVerifier.create(backend(client).analyze(snapshotGroupedPlan(), OPTIONS, null)) + .thenAwait(Duration.ofMillis(10)) + .thenCancel() + .verify() + + closeRequest.captured.id().assert().isEqualTo("pit-cancel") + } + + @Test + fun `unsuccessful analytics PIT close should fail incomplete`() { + val client = mockk() + every { client.closePointInTime(any()) } returns Mono.just( + ClosePointInTimeResponse.of { response -> response.succeeded(false).numFreed(0) }, + ) + + assertFailure(QueryBackendFailureKind.INCOMPLETE_RESULT) { + backend(client).close( + me.ahoo.wow.query.backend.BackendAnalyticsCursorState("pit-open".encodeToByteArray()), + ).block() + } + } + + @Test + fun `expired snapshot PIT should fail incomplete and consume the leased state`() { + val client = mockk() + val closeRequest = slot() + val pitExpired = mockk { + every { status() } returns 404 + } + every { client.search(any(), Map::class.java) } returns Mono.error(pitExpired) + every { client.closePointInTime(capture(closeRequest)) } returns Mono.just( + ClosePointInTimeResponse.of { response -> response.succeeded(true).numFreed(0) }, + ) + + val error = runCatching { + backend(client).analyze( + snapshotGroupedPlan(listOf(NormalizedValue.Text("PAID"))), + OPTIONS, + me.ahoo.wow.query.backend.BackendAnalyticsCursorState("pit-expired".encodeToByteArray()), + ).block() + }.exceptionOrNull() as QueryBackendException + + error.kind.assert().isEqualTo(QueryBackendFailureKind.INCOMPLETE_RESULT) + error.cause.assert().isSameAs(pitExpired) + closeRequest.captured.id().assert().isEqualTo("pit-expired") + } + + private fun backend(client: ReactiveElasticsearchClient) = ElasticsearchAnalyticsQueryBackend( + client, + binding.prepared, + Clock.fixed(NOW, ZoneOffset.UTC), + ) + + private fun response( + aggregation: Aggregate? = null, + total: Long = 0, + totalRelation: TotalHitsRelation? = null, + timedOut: Boolean = false, + failedShards: Int = 0, + pitId: String? = null, + ): SearchResponse> = SearchResponse.of> { response -> + response.took(1) + .timedOut(timedOut) + .shards { shards -> shards.failed(failedShards).successful(1).total(1 + failedShards) } + .hits { hits -> + hits.hits(emptyList()) + totalRelation?.let { relation -> hits.total { value -> value.value(total).relation(relation) } } + hits + } + .also { builder -> aggregation?.let { builder.aggregations(ANALYTICS_AGGREGATION, it) } } + .also { builder -> pitId?.let(builder::pitId) } + } + + private fun composite( + buckets: List, + afterKey: Map, + ): Aggregate = Aggregate.of { aggregate -> + aggregate.composite { composite -> composite.buckets { values -> values.array(buckets) }.afterKey(afterKey) } + } + + private fun bucket(status: String, count: Long): CompositeBucket = CompositeBucket.of { bucket -> + bucket.key("status", status).docCount(count) + } + + private fun bucket(status: FieldValue, count: Long): CompositeBucket = CompositeBucket.of { bucket -> + bucket.key("status", status).docCount(count) + } + + private fun globalPlan() = plan(BackendAnalyticsGrouping.Global, BackendAnalyticsPageWindow(1)) + + private fun groupedPlan( + missingPolicy: BackendAnalyticsMissingPolicy = BackendAnalyticsMissingPolicy.EXCLUDE, + ) = plan( + BackendAnalyticsGrouping.By( + listOf(BackendAnalyticsDimension(STATUS_ALIAS, status, missingPolicy)), + ), + BackendAnalyticsPageWindow(10), + ) + + private fun snapshotGroupedPlan(afterKey: List? = null) = plan( + BackendAnalyticsGrouping.By( + listOf(BackendAnalyticsDimension(STATUS_ALIAS, status, BackendAnalyticsMissingPolicy.EXCLUDE)), + ), + BackendAnalyticsPageWindow(10, afterKey), + BackendAnalyticsConsistency.SNAPSHOT, + ) + + private fun plan( + grouping: BackendAnalyticsGrouping, + window: BackendAnalyticsPageWindow, + consistency: BackendAnalyticsConsistency = BackendAnalyticsConsistency.EVENTUAL, + ) = BackendAnalyticsQueryPlan( + target, + schema.contractId, + BackendEnforcedFilter(BackendPlannedCondition.All, BackendPlannedCondition.All), + grouping, + listOf(BackendAnalyticsMetric.DocumentCount(COUNT_ALIAS)), + BackendAnalyticsCondition.All, + when (grouping) { + BackendAnalyticsGrouping.Global -> BackendAnalyticsBucketOrder.Global + is BackendAnalyticsGrouping.By -> BackendAnalyticsBucketOrder.DimensionKeyAscending( + BackendAnalyticsNullPlacement.FIRST, + BackendAnalyticsTextCollation.BINARY, + ) + }, + window, + null, + consistency, + BackendAnalyticsCompleteness.EXACT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("a".repeat(64)), + ) + + private fun openPit(id: String): OpenPointInTimeResponse = OpenPointInTimeResponse.of { response -> + response.id(id).shards { shards -> shards.total(1).successful(1).failed(0) } + } + + private fun assertFailure(kind: QueryBackendFailureKind, action: () -> Unit) { + assertThrownBy(action).satisfies( + Consumer { error -> error.kind.assert().isEqualTo(kind) }, + ) + } + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + private val state = QueryFieldId.Path(listOf("state")) + private val status = QueryFieldId.Path(listOf("state", "status")) + private val schema = QueryDocumentSchema( + target, + listOf( + field(identity, LogicalFieldType.Text, setOf(FieldCapability.EXACT)), + field(state, LogicalFieldType.Object, emptySet()), + field(status, LogicalFieldType.Text, setOf(FieldCapability.AGGREGATABLE)), + ), + emptyList(), + ) + private val binding = ElasticsearchSnapshotQueryBinding( + schema, + "wow.sales.order.snapshot", + "order-query-v1", + mapOf( + identity to ElasticsearchFieldBinding( + MessageRecords.AGGREGATE_ID, + setOf(FieldCapability.EXACT), + exactField = "_id", + ), + state to ElasticsearchFieldBinding("state", emptySet()), + status to ElasticsearchFieldBinding( + "state.status", + setOf(FieldCapability.AGGREGATABLE), + groupField = "state.status.exact", + groupReadiness = GROUP_READINESS, + keywordReadiness = ElasticsearchKeywordReadiness(128, 512, true, true), + ), + ), + ) + + private fun field( + id: QueryFieldId, + type: LogicalFieldType, + capabilities: Set, + ) = QueryFieldSchema( + id, + type, + Presence.OPTIONAL, + Nullability.NULLABLE, + if (FieldCapability.EXACT in capabilities) setOf(PredicateOperator.EQ) else emptySet(), + capabilities, + ) + + private companion object { + const val ANALYTICS_AGGREGATION = "wow_analytics" + val NOW: Instant = Instant.parse("2026-08-08T00:00:00Z") + val OPTIONS = QueryBackendExecutionOptions(NOW.plusSeconds(10), null, maxReturnedBuckets = 10) + val STATUS_ALIAS = AnalyticsAlias("status") + val COUNT_ALIAS = AnalyticsAlias("count") + val GROUP_READINESS = ElasticsearchGroupReadiness(historicalValuesAudited = true) + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchAnalyticsQueryCompilerTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchAnalyticsQueryCompilerTest.kt new file mode 100644 index 00000000000..24bac7d0f03 --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchAnalyticsQueryCompilerTest.kt @@ -0,0 +1,236 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.planned + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.BackendAnalyticsBucketOrder +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsCondition +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsDimension +import me.ahoo.wow.query.backend.BackendAnalyticsGrouping +import me.ahoo.wow.query.backend.BackendAnalyticsMetric +import me.ahoo.wow.query.backend.BackendAnalyticsMissingPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNullPlacement +import me.ahoo.wow.query.backend.BackendAnalyticsPageWindow +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.BackendAnalyticsTextCollation +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendRequiredCapabilities +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import org.junit.jupiter.api.Test +import java.util.function.Consumer + +class ElasticsearchAnalyticsQueryCompilerTest { + private val compiler: ElasticsearchAnalyticsQueryCompiler + get() = ElasticsearchAnalyticsQueryCompiler(binding.prepared) + + @Test + fun `grouped exact document count should compile composite sources and response after key`() { + val compiled = compiler.compile( + plan( + BackendAnalyticsGrouping.By( + listOf( + BackendAnalyticsDimension( + AnalyticsAlias("status"), + status, + BackendAnalyticsMissingPolicy.EXCLUDE, + ), + ), + ), + BackendAnalyticsPageWindow(10, listOf(NormalizedValue.Text("PAID"))), + ), + ) + + compiled.aggregation!!.composite().size().assert().isEqualTo(10) + compiled.aggregation.composite().sources().single().name().assert().isEqualTo("status") + val terms = compiled.aggregation.composite().sources().single().value().terms() + terms.field().assert().isEqualTo("state.status.exact") + terms.missingBucket().assert().isFalse() + terms.missingOrder().assert().isNull() + compiled.aggregation.composite().after()["status"]!!.stringValue().assert().isEqualTo("PAID") + } + + @Test + fun `unsupported precision contracts should fail before IO`() { + assertUnsupported { + compiler.compile( + plan( + BackendAnalyticsGrouping.Global, + BackendAnalyticsPageWindow(1), + listOf(BackendAnalyticsMetric.Sum(AnalyticsAlias("sum"), amount)), + ), + ) + } + } + + @Test + fun `snapshot consistency should reuse the same logical aggregation compilation`() { + val compiled = compiler.compile( + plan( + BackendAnalyticsGrouping.Global, + BackendAnalyticsPageWindow(1), + consistency = BackendAnalyticsConsistency.SNAPSHOT, + ), + ) + + compiled.aggregation.assert().isNull() + } + + @Test + fun `missing and explicit null should share one composite bucket and cursor value`() { + val compiled = compiler.compile( + plan( + BackendAnalyticsGrouping.By( + listOf( + BackendAnalyticsDimension( + AnalyticsAlias("status"), + status, + BackendAnalyticsMissingPolicy.AS_NULL_BUCKET, + ), + ), + ), + BackendAnalyticsPageWindow(10, listOf(NormalizedValue.Null)), + ), + ) + + val terms = compiled.aggregation!!.composite().sources().single().value().terms() + terms.missingBucket().assert().isTrue() + terms.missingOrder()!!.jsonValue().assert().isEqualTo("first") + compiled.aggregation.composite().after()["status"]!!.isNull.assert().isTrue() + } + + private fun plan( + grouping: BackendAnalyticsGrouping, + window: BackendAnalyticsPageWindow, + metrics: List = listOf( + BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count")), + ), + consistency: BackendAnalyticsConsistency = BackendAnalyticsConsistency.EVENTUAL, + ) = BackendAnalyticsQueryPlan( + target, + schema.contractId, + BackendEnforcedFilter(BackendPlannedCondition.All, BackendPlannedCondition.All), + grouping, + metrics, + BackendAnalyticsCondition.All, + when (grouping) { + BackendAnalyticsGrouping.Global -> BackendAnalyticsBucketOrder.Global + is BackendAnalyticsGrouping.By -> BackendAnalyticsBucketOrder.DimensionKeyAscending( + BackendAnalyticsNullPlacement.FIRST, + BackendAnalyticsTextCollation.BINARY, + ) + }, + window, + null, + consistency, + BackendAnalyticsCompleteness.EXACT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("9".repeat(64)), + ) + + private fun assertUnsupported(action: () -> Unit) { + assertThrownBy(action).satisfies( + Consumer { error -> error.kind.assert().isEqualTo(QueryBackendFailureKind.UNSUPPORTED) }, + ) + } + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + private val state = QueryFieldId.Path(listOf("state")) + private val status = QueryFieldId.Path(listOf("state", "status")) + private val amount = QueryFieldId.Path(listOf("state", "amount")) + private val schema = QueryDocumentSchema( + target, + listOf( + field(identity, LogicalFieldType.Text, setOf(FieldCapability.EXACT)), + field(state, LogicalFieldType.Object, emptySet()), + field(status, LogicalFieldType.Text, setOf(FieldCapability.AGGREGATABLE)), + field(amount, LogicalFieldType.Int64, setOf(FieldCapability.AGGREGATABLE)), + ), + emptyList(), + ) + private val binding = ElasticsearchSnapshotQueryBinding( + schema, + "wow.sales.order.snapshot", + "order-query-v1", + mapOf( + identity to ElasticsearchFieldBinding( + MessageRecords.AGGREGATE_ID, + setOf(FieldCapability.EXACT), + exactField = "_id", + ), + state to ElasticsearchFieldBinding("state", emptySet()), + status to ElasticsearchFieldBinding( + "state.status", + setOf(FieldCapability.AGGREGATABLE), + groupField = "state.status.exact", + groupReadiness = GROUP_READINESS, + keywordReadiness = ElasticsearchKeywordReadiness(128, 512, true, true), + ), + amount to ElasticsearchFieldBinding( + "state.amount", + setOf(FieldCapability.AGGREGATABLE), + groupField = "state.amount", + groupReadiness = GROUP_READINESS, + ), + ), + ) + + private fun field( + id: QueryFieldId, + type: LogicalFieldType, + capabilities: Set, + ) = QueryFieldSchema( + id, + type, + Presence.OPTIONAL, + Nullability.NULLABLE, + if (FieldCapability.EXACT in capabilities) setOf(PredicateOperator.EQ) else emptySet(), + capabilities, + ) + + private companion object { + val GROUP_READINESS = ElasticsearchGroupReadiness(historicalValuesAudited = true) + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchRecordQueryCompilerTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchRecordQueryCompilerTest.kt new file mode 100644 index 00000000000..4673e227bb1 --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchRecordQueryCompilerTest.kt @@ -0,0 +1,306 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.planned + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.BackendRequiredCapabilities +import me.ahoo.wow.query.backend.BackendSort +import me.ahoo.wow.query.backend.BackendSortOrigin +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.CaseSensitivity +import me.ahoo.wow.query.backend.EmptyArraySemantics +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedPredicateOptions +import me.ahoo.wow.query.backend.NormalizedSortDirection +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.QuerySearchScopeDefinition +import me.ahoo.wow.query.backend.RecordResultShape +import me.ahoo.wow.query.backend.SearchScopeId +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import org.junit.jupiter.api.Test +import java.util.function.Consumer + +class ElasticsearchRecordQueryCompilerTest { + private val compiler: ElasticsearchRecordQueryCompiler + get() = ElasticsearchRecordQueryCompiler(binding) + + @Test + fun `compiler should bind enforced exact filter projection and stable sort roles`() { + val user = predicate(name, PredicateOperator.EQ, NormalizedValue.Text("alice")) + val mandatory = predicate(tenant, PredicateOperator.EQ, NormalizedValue.Text("tenant-1")) + val compiled = compiler.compile( + streamPlan( + BackendEnforcedFilter(user, mandatory), + BackendProjection.Include(listOf(state, name)), + listOf(BackendSort(identity, NormalizedSortDirection.ASC, BackendSortOrigin.STABILITY_TIE_BREAKER)), + ), + ) + + val json = compiled.query.toString() + json.assert().contains("state.name.exact", MessageRecords.TENANT_ID, "alice", "tenant-1") + compiled.sourceFilter!!.includes().assert().containsExactly(MessageRecords.AGGREGATE_ID, "state") + compiled.sort.assert().hasSize(1) + compiled.sort.single().field().field().assert().isEqualTo(MessageRecords.AGGREGATE_ID) + compiled.limit.assert().isEqualTo(10) + } + + @Test + fun `compiler should preserve literal wildcard text and use explicit search scope`() { + val literal = compiler.compileCondition( + predicate(name, PredicateOperator.CONTAINS, NormalizedValue.Text("a*?\\b")), + ) + literal.wildcard().field().assert().isEqualTo("state.name.exact") + literal.wildcard().value().assert().isEqualTo("*a\\*\\?\\\\b*") + literal.wildcard().caseInsensitive().assert().isFalse() + + val search = compiler.compileCondition(BackendPlannedCondition.Search(scope, "search text")) + search.match().field().assert().isEqualTo("state.name") + search.match().query().stringValue().assert().isEqualTo("search text") + } + + @Test + fun `compiler should preserve Mongo null missing baseline where Elasticsearch can prove equivalence`() { + val isNull = compiler.compileCondition(predicate(name, PredicateOperator.EQ, NormalizedValue.Null)) + isNull.bool().mustNot().single().exists().field().assert().isEqualTo("state.name.exact") + + val isNotNull = compiler.compileCondition(predicate(name, PredicateOperator.NE, NormalizedValue.Null)) + isNotNull.bool().mustNot().single().bool().mustNot().single().exists().field().assert() + .isEqualTo("state.name.exact") + + val includesNull = compiler.compileCondition( + predicate( + name, + PredicateOperator.IN, + NormalizedValue.ListValue(listOf(NormalizedValue.Text("alice"), NormalizedValue.Null)), + ), + ) + includesNull.bool().should().assert().hasSize(2) + + val exists = compiler.compileCondition( + predicate(name, PredicateOperator.EXISTS, NormalizedValue.BooleanValue(true)), + ) + exists.term().field().assert().isEqualTo("state.name.present") + exists.term().value().booleanValue().assert().isTrue() + val missing = compiler.compileCondition( + predicate(name, PredicateOperator.EXISTS, NormalizedValue.BooleanValue(false)), + ) + missing.bool().mustNot().single().term().field().assert().isEqualTo("state.name.present") + + val excludesNull = compiler.compileCondition( + predicate( + name, + PredicateOperator.NOT_IN, + NormalizedValue.ListValue(listOf(NormalizedValue.Null)), + ), + ) + excludesNull.bool().should().assert().hasSize(2) + } + + @Test + fun `compiler should require nested mapping role and reject insensitive or native input`() { + val nested = compiler.compileCondition( + BackendPlannedCondition.ElementMatch( + items, + predicate(itemName, PredicateOperator.EQ, NormalizedValue.Text("item")), + ), + ) + nested.nested().path().assert().isEqualTo("state.items") + nested.nested().query().term().field().assert().isEqualTo("state.items.name") + + assertUnsupported { + compiler.compileCondition( + BackendPlannedCondition.Predicate( + name, + PredicateOperator.EQ, + NormalizedValue.Text("ALICE"), + NormalizedPredicateOptions(CaseSensitivity.INSENSITIVE), + ), + ) + } + } + + private fun streamPlan( + filter: BackendEnforcedFilter, + projection: BackendProjection, + sort: List, + ) = BackendStreamQueryPlan( + target, + schema.contractId, + filter, + RecordResultShape.DYNAMIC, + projection, + sort, + 10, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("8".repeat(64)), + ) + + private fun predicate( + field: QueryFieldId, + operator: PredicateOperator, + value: NormalizedValue? = null, + ) = BackendPlannedCondition.Predicate(field, operator, value) + + private fun assertUnsupported(action: () -> Unit) { + assertThrownBy(action).satisfies( + Consumer { error -> error.kind.assert().isEqualTo(QueryBackendFailureKind.UNSUPPORTED) }, + ) + } + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + private val tenant = QueryFieldId.System(SystemFieldKind.TENANT_ID) + private val state = QueryFieldId.Path(listOf("state")) + private val name = QueryFieldId.Path(listOf("state", "name")) + private val items = QueryFieldId.Path(listOf("state", "items")) + private val itemName = QueryFieldId.Path(listOf("state", "items", "name")) + private val scope = SearchScopeId("state-name") + private val schema = QueryDocumentSchema( + target, + listOf( + field(identity, LogicalFieldType.Text, setOf(PredicateOperator.EQ), EXACT_SORT_PROJECT), + field(tenant, LogicalFieldType.Text, setOf(PredicateOperator.EQ), setOf(FieldCapability.EXACT)), + field(state, LogicalFieldType.Object), + field( + name, + LogicalFieldType.Text, + setOf( + PredicateOperator.EQ, + PredicateOperator.NE, + PredicateOperator.IN, + PredicateOperator.NOT_IN, + PredicateOperator.CONTAINS, + ), + setOf( + FieldCapability.EXACT, + FieldCapability.PRESENCE, + FieldCapability.FULL_TEXT, + FieldCapability.LITERAL_PATTERN, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + FieldCapability.AGGREGATABLE, + ), + ), + field( + items, + LogicalFieldType.Array( + LogicalFieldType.Object, + Nullability.NON_NULL, + EmptyArraySemantics.DISTINCT, + ), + capabilities = setOf(FieldCapability.ELEMENT_MATCH), + ), + field(itemName, LogicalFieldType.Text, setOf(PredicateOperator.EQ), setOf(FieldCapability.EXACT)), + ), + listOf(QuerySearchScopeDefinition(scope, null, listOf(name), listOf(name))), + ) + private val binding = ElasticsearchSnapshotQueryBinding( + schema, + "wow.sales.order.snapshot", + "order-query-v1", + linkedMapOf( + identity to ElasticsearchFieldBinding( + MessageRecords.AGGREGATE_ID, + EXACT_SORT_PROJECT, + exactField = "_id", + sortField = MessageRecords.AGGREGATE_ID, + keywordReadiness = KEYWORD_READINESS, + ), + tenant to ElasticsearchFieldBinding( + MessageRecords.TENANT_ID, + setOf(FieldCapability.EXACT), + exactField = MessageRecords.TENANT_ID, + keywordReadiness = KEYWORD_READINESS, + ), + state to ElasticsearchFieldBinding("state", emptySet()), + name to ElasticsearchFieldBinding( + "state.name", + setOf( + FieldCapability.EXACT, + FieldCapability.FULL_TEXT, + FieldCapability.LITERAL_PATTERN, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + FieldCapability.AGGREGATABLE, + ), + exactField = "state.name.exact", + presenceField = "state.name.present", + searchField = "state.name", + searchAnalyzer = "standard", + literalField = "state.name.exact", + sortField = "state.name.exact", + groupField = "state.name.exact", + groupReadiness = GROUP_READINESS, + keywordReadiness = KEYWORD_READINESS, + ), + items to ElasticsearchFieldBinding( + "state.items", + setOf(FieldCapability.ELEMENT_MATCH), + nestedPath = "state.items", + ), + itemName to ElasticsearchFieldBinding( + "state.items.name", + setOf(FieldCapability.EXACT), + exactField = "state.items.name", + keywordReadiness = KEYWORD_READINESS, + ), + ), + listOf(ElasticsearchSearchScopeBinding(scope, mapOf(name to "state.name"))), + ) + + private fun field( + id: QueryFieldId, + type: LogicalFieldType, + operators: Set = emptySet(), + capabilities: Set = emptySet(), + ) = QueryFieldSchema(id, type, Presence.OPTIONAL, Nullability.NULLABLE, operators, capabilities) + + private companion object { + val KEYWORD_READINESS = ElasticsearchKeywordReadiness(128, 512, true, true) + val GROUP_READINESS = ElasticsearchGroupReadiness(historicalValuesAudited = true) + val EXACT_SORT_PROJECT = setOf( + FieldCapability.EXACT, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + ) + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotQueryBindingTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotQueryBindingTest.kt new file mode 100644 index 00000000000..f96d5dfab7b --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotQueryBindingTest.kt @@ -0,0 +1,359 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.planned + +import co.elastic.clients.elasticsearch._types.mapping.Property +import co.elastic.clients.elasticsearch._types.mapping.TypeMapping +import co.elastic.clients.json.JsonData +import io.mockk.mockk +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.EmptyArraySemantics +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.QuerySearchScopeDefinition +import me.ahoo.wow.query.backend.SearchScopeId +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import org.junit.jupiter.api.Test +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient + +class ElasticsearchSnapshotQueryBindingTest { + @Test + fun `readiness should attest every concrete mapping and explicit physical role`() { + binding.prepared.attestReadiness( + mapOf( + "sales.order.snapshot-000001" to mapping(), + "sales.order.snapshot-000002" to mapping(), + ), + ) + + binding.fields[name]!!.exactField.assert().isEqualTo("state.name.exact") + binding.searchScopes[scope]!!.fields[name].assert().isEqualTo("state.name") + + val contribution = binding.prepared.createContribution(mockk()) + contribution.supportedOperations.assert().contains(QueryOperation.ANALYZE) + contribution.analyticsBackend.assert().isNotNull() + } + + @Test + fun `readiness should reject mapping version and field role drift`() { + assertThrownBy { + binding.prepared.attestReadiness(mapOf("generation" to mapping(version = "stale"))) + } + assertThrownBy { + binding.prepared.attestReadiness( + mapOf("generation" to mapping(nameExact = keyword(ignoreAbove = 63))), + ) + } + assertThrownBy { + binding.prepared.attestReadiness( + mapOf("generation" to mapping(nameExact = keyword(ignoreAbove = 128, docValues = false))), + ) + } + assertThrownBy { + binding.prepared.attestReadiness( + mapOf("generation" to mapping(searchAnalyzer = "whitespace")), + ) + } + assertThrownBy { + binding.prepared.attestReadiness( + mapOf("generation" to mapping(nameExact = keyword(128, normalizer = "lowercase"))), + ) + } + assertThrownBy { + binding.prepared.attestReadiness( + mapOf("generation" to mapping(nameExact = keyword(128, nullValue = "__null__"))), + ) + } + } + + @Test + fun `readiness should reject object arrays masquerading as nested mappings`() { + assertThrownBy { + binding.prepared.attestReadiness(mapOf("generation" to mapping(nestedItems = false))) + } + } + + @Test + fun `readiness should reject sort and group roles whose mapping type differs from the logical field`() { + val mismatchedFields = binding.fields.toMutableMap() + mismatchedFields[name] = ElasticsearchFieldBinding( + "state.name", + binding.fields.getValue(name).capabilities, + exactField = "state.name.exact", + searchField = "state.name", + searchAnalyzer = "standard", + literalField = "state.name.exact", + sortField = "state.rank", + groupField = "state.rank", + groupReadiness = GROUP_READINESS, + keywordReadiness = KEYWORD_READINESS, + ) + val mismatchedBinding = ElasticsearchSnapshotQueryBinding( + schema, + INDEX, + VERSION, + mismatchedFields, + listOf(searchBinding), + ) + + assertThrownBy { + mismatchedBinding.prepared.attestReadiness(mapOf("generation" to mapping(includeRank = true))) + } + } + + @Test + fun `binding should reject guessed physical roles and incomplete keyword attestation`() { + assertThrownBy { + ElasticsearchFieldBinding( + "state.name", + setOf(FieldCapability.EXACT), + ) + } + assertThrownBy { + ElasticsearchKeywordReadiness(128, 512, historicalValuesAudited = false, writeConstraintEnforced = true) + } + assertThrownBy { + ElasticsearchFieldBinding( + "state.note", + setOf(FieldCapability.PRESENCE), + ) + } + assertThrownBy { + ElasticsearchSnapshotQueryBinding( + schema, + INDEX, + VERSION, + binding.fields + ( + QueryFieldId.System(SystemFieldKind.TENANT_ID) to ElasticsearchFieldBinding( + MessageRecords.OWNER_ID, + setOf(FieldCapability.EXACT), + exactField = MessageRecords.OWNER_ID, + keywordReadiness = KEYWORD_READINESS, + ) + ), + listOf(searchBinding), + ) + } + } + + private fun mapping( + version: String = VERSION, + nameExact: Property = keyword(ignoreAbove = 128), + nestedItems: Boolean = true, + includeRank: Boolean = false, + analyzer: String = "standard", + searchAnalyzer: String = "standard", + ): TypeMapping = TypeMapping.of { mapping -> + mapping.meta(MAPPING_VERSION_META, JsonData.of(version)) + .meta(DOCUMENT_KIND_META, JsonData.of(QueryDocumentKind.SNAPSHOT.name)) + .meta(SCHEMA_CONTRACT_META, JsonData.of(schema.contractId.value)) + .meta(CAPABILITY_DIGEST_META, JsonData.of(binding.prepared.capabilityDigest)) + .properties(MessageRecords.AGGREGATE_ID, keyword(ignoreAbove = 128)) + .properties(MessageRecords.TENANT_ID, keyword(ignoreAbove = 128)) + .properties( + "state", + Property.of { state -> + state.`object` { objectField -> + objectField + .properties( + "name", + Property.of { text -> + text.text { definition -> + definition.analyzer(analyzer).searchAnalyzer(searchAnalyzer) + .fields("exact", nameExact) + } + }, + ) + .properties( + "items", + if (nestedItems) { + Property.of { items -> + items.nested { nested -> nested.properties("name", keyword(ignoreAbove = 128)) } + } + } else { + Property.of { items -> + items.`object` { objectField -> + objectField.properties("name", keyword(ignoreAbove = 128)) + } + } + }, + ) + .also { state -> + if (includeRank) { + state.properties( + "rank", + Property.of { property -> property.boolean_ { boolean -> boolean } }, + ) + } + } + } + }, + ) + } + + private fun keyword( + ignoreAbove: Int, + docValues: Boolean = true, + normalizer: String? = null, + nullValue: String? = null, + ): Property = + Property.of { property -> + property.keyword { keyword -> + keyword.ignoreAbove(ignoreAbove).docValues(docValues).also { definition -> + normalizer?.let(definition::normalizer) + nullValue?.let(definition::nullValue) + } + } + } + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + private val tenant = QueryFieldId.System(SystemFieldKind.TENANT_ID) + private val state = QueryFieldId.Path(listOf("state")) + private val name = QueryFieldId.Path(listOf("state", "name")) + private val items = QueryFieldId.Path(listOf("state", "items")) + private val itemName = QueryFieldId.Path(listOf("state", "items", "name")) + private val scope = SearchScopeId("state-name") + private val schema = QueryDocumentSchema( + target, + listOf( + field(identity, LogicalFieldType.Text, setOf(PredicateOperator.EQ), EXACT_SORT_PROJECT), + field(tenant, LogicalFieldType.Text, setOf(PredicateOperator.EQ), setOf(FieldCapability.EXACT)), + field(state, LogicalFieldType.Object), + field( + name, + LogicalFieldType.Text, + setOf( + PredicateOperator.EQ, + PredicateOperator.CONTAINS, + ), + setOf( + FieldCapability.EXACT, + FieldCapability.FULL_TEXT, + FieldCapability.LITERAL_PATTERN, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + FieldCapability.AGGREGATABLE, + ), + ), + field( + items, + LogicalFieldType.Array( + LogicalFieldType.Object, + Nullability.NON_NULL, + EmptyArraySemantics.DISTINCT, + ), + capabilities = setOf(FieldCapability.ELEMENT_MATCH), + ), + field(itemName, LogicalFieldType.Text, setOf(PredicateOperator.EQ), setOf(FieldCapability.EXACT)), + ), + listOf(QuerySearchScopeDefinition(scope, null, listOf(name), listOf(name))), + ) + private val searchBinding = ElasticsearchSearchScopeBinding(scope, mapOf(name to "state.name")) + private val binding = ElasticsearchSnapshotQueryBinding( + schema, + INDEX, + VERSION, + linkedMapOf( + identity to ElasticsearchFieldBinding( + MessageRecords.AGGREGATE_ID, + EXACT_SORT_PROJECT, + exactField = "_id", + sortField = MessageRecords.AGGREGATE_ID, + keywordReadiness = KEYWORD_READINESS, + ), + tenant to ElasticsearchFieldBinding( + MessageRecords.TENANT_ID, + setOf(FieldCapability.EXACT), + exactField = MessageRecords.TENANT_ID, + keywordReadiness = KEYWORD_READINESS, + ), + state to ElasticsearchFieldBinding("state", emptySet()), + name to ElasticsearchFieldBinding( + "state.name", + setOf( + FieldCapability.EXACT, + FieldCapability.FULL_TEXT, + FieldCapability.LITERAL_PATTERN, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + FieldCapability.AGGREGATABLE, + ), + exactField = "state.name.exact", + searchField = "state.name", + searchAnalyzer = "standard", + literalField = "state.name.exact", + sortField = "state.name.exact", + groupField = "state.name.exact", + groupReadiness = GROUP_READINESS, + keywordReadiness = KEYWORD_READINESS, + ), + items to ElasticsearchFieldBinding( + "state.items", + setOf(FieldCapability.ELEMENT_MATCH), + nestedPath = "state.items", + ), + itemName to ElasticsearchFieldBinding( + "state.items.name", + setOf(FieldCapability.EXACT), + exactField = "state.items.name", + keywordReadiness = KEYWORD_READINESS, + ), + ), + listOf(searchBinding), + ) + + private fun field( + id: QueryFieldId, + type: LogicalFieldType, + operators: Set = emptySet(), + capabilities: Set = emptySet(), + ) = QueryFieldSchema(id, type, Presence.OPTIONAL, Nullability.NULLABLE, operators, capabilities) + + private companion object { + const val INDEX = "wow.sales.order.snapshot" + const val VERSION = "order-query-v1" + const val MAPPING_VERSION_META = "wow_query_mapping_version" + const val DOCUMENT_KIND_META = "wow_query_document_kind" + const val SCHEMA_CONTRACT_META = "wow_query_schema_contract_id" + const val CAPABILITY_DIGEST_META = "wow_query_capability_digest" + val KEYWORD_READINESS = ElasticsearchKeywordReadiness(128, 512, true, true) + val GROUP_READINESS = ElasticsearchGroupReadiness(historicalValuesAudited = true) + val EXACT_SORT_PROJECT = setOf( + FieldCapability.EXACT, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + ) + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotRecordMapperTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotRecordMapperTest.kt new file mode 100644 index 00000000000..1cee6fc3dfc --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotRecordMapperTest.kt @@ -0,0 +1,172 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + +package me.ahoo.wow.elasticsearch.query.planned + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import org.junit.jupiter.api.Test +import java.time.Instant +import java.util.function.Consumer + +class ElasticsearchSnapshotRecordMapperTest { + @Test + fun `mapper should freeze source and apply logical projection without leaking identity`() { + val bytes = byteArrayOf(1, 2) + val state = linkedMapOf("name" to "alice", "secret" to "hidden", "bytes" to bytes) + val source = linkedMapOf( + MessageRecords.AGGREGATE_ID to "order-1", + "state" to state, + "backendOnly" to "hidden", + ) + + val record = mapper.map( + "order-1", + source, + BackendProjection.Include(listOf(QueryFieldId.Path(listOf("state", "name")))), + ) + state["name"] = "changed" + bytes[0] = 9 + source["backendOnly"] = "changed" + + record.identity.assert().isEqualTo("order-1") + record.document.values.keys.assert().containsExactly("state") + val projectedState = record.document.values["state"] as NormalizedValue.ObjectValue + projectedState.values.assert().containsEntry("name", NormalizedValue.Text("alice")) + projectedState.values.assert().doesNotContainKey("secret") + } + + @Test + fun `mapper should reject missing or inconsistent source identity and cyclic values`() { + assertMappingFailure { + mapper.map("order-1", emptyMap()) + } + assertMappingFailure { + mapper.map( + "order-1", + mapOf(MessageRecords.AGGREGATE_ID to "order-2"), + ) + } + val cycle = linkedMapOf() + cycle[MessageRecords.AGGREGATE_ID] = "order-1" + cycle["cycle"] = cycle + assertMappingFailure { + mapper.map("order-1", cycle) + } + } + + @Test + fun `mapper should decode epoch millis using the logical field schema`() { + val record = mapper.map( + "order-1", + mapOf( + MessageRecords.AGGREGATE_ID to "order-1", + "state" to mapOf("createdAt" to 1_234L), + ), + ) + + val state = record.document.values["state"] as NormalizedValue.ObjectValue + state.values["createdAt"].assert().isEqualTo( + NormalizedValue.InstantValue(Instant.ofEpochMilli(1_234L)), + ) + } + + private fun assertMappingFailure(action: () -> Unit) { + assertThrownBy(action).satisfies( + Consumer { error -> error.kind.assert().isEqualTo(QueryBackendFailureKind.MAPPING_FAILURE) }, + ) + } + + private val mapper: ElasticsearchSnapshotRecordMapper + get() { + val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + val state = QueryFieldId.Path(listOf("state")) + val name = QueryFieldId.Path(listOf("state", "name")) + val createdAt = QueryFieldId.Path(listOf("state", "createdAt")) + val schema = QueryDocumentSchema( + QueryTarget(MaterializedNamedAggregate("sales", "order"), QueryDocumentKind.SNAPSHOT), + listOf( + QueryFieldSchema( + identity, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + emptyList(), + emptyList() + ), + QueryFieldSchema( + state, + LogicalFieldType.Object, + Presence.REQUIRED, + Nullability.NON_NULL, + emptyList(), + emptyList() + ), + QueryFieldSchema( + name, + LogicalFieldType.Text, + Presence.OPTIONAL, + Nullability.NULLABLE, + emptyList(), + emptyList() + ), + QueryFieldSchema( + createdAt, + LogicalFieldType.Instant, + Presence.OPTIONAL, + Nullability.NULLABLE, + emptyList(), + emptyList(), + ), + ), + emptyList(), + ) + return ElasticsearchSnapshotRecordMapper( + ElasticsearchPreparedQueryBinding( + schema, + "wow.sales.order.snapshot", + "v1", + mapOf( + identity to ElasticsearchFieldBinding(MessageRecords.AGGREGATE_ID, emptySet()), + state to ElasticsearchFieldBinding("state", emptySet()), + name to ElasticsearchFieldBinding("state.name", emptySet()), + createdAt to ElasticsearchFieldBinding( + "state.createdAt", + emptySet(), + valueEncoding = ElasticsearchValueEncoding.EPOCH_MILLIS, + ), + ), + emptyMap(), + BackendId("elasticsearch"), + ), + ) + } +} diff --git a/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotRecordQueryBackendTest.kt b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotRecordQueryBackendTest.kt new file mode 100644 index 00000000000..dd0a24b3e7d --- /dev/null +++ b/wow-elasticsearch/src/test/kotlin/me/ahoo/wow/elasticsearch/query/planned/ElasticsearchSnapshotRecordQueryBackendTest.kt @@ -0,0 +1,404 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.elasticsearch.query.planned + +import co.elastic.clients.elasticsearch._types.ElasticsearchException +import co.elastic.clients.elasticsearch._types.ErrorCause +import co.elastic.clients.elasticsearch.core.ClosePointInTimeRequest +import co.elastic.clients.elasticsearch.core.ClosePointInTimeResponse +import co.elastic.clients.elasticsearch.core.OpenPointInTimeRequest +import co.elastic.clients.elasticsearch.core.OpenPointInTimeResponse +import co.elastic.clients.elasticsearch.core.SearchRequest +import co.elastic.clients.elasticsearch.core.SearchResponse +import co.elastic.clients.elasticsearch.core.search.Hit +import co.elastic.clients.elasticsearch.core.search.TotalHitsRelation +import io.mockk.confirmVerified +import io.mockk.every +import io.mockk.mockk +import io.mockk.slot +import io.mockk.verify +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPageQueryPlan +import me.ahoo.wow.query.backend.BackendPageWindow +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.BackendRequiredCapabilities +import me.ahoo.wow.query.backend.BackendRequiredConsistency +import me.ahoo.wow.query.backend.BackendSingleQueryPlan +import me.ahoo.wow.query.backend.BackendSort +import me.ahoo.wow.query.backend.BackendSortOrigin +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.BackendTotalMode +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedSortDirection +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.RecordResultShape +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import org.junit.jupiter.api.Test +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Mono +import reactor.test.StepVerifier +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset +import java.util.function.Consumer + +class ElasticsearchSnapshotRecordQueryBackendTest { + @Test + fun `search response failures should preserve timeout incomplete and exact total categories`() { + val client = mockk() + val backend = backend(client) + every { client.openPointInTime(any()) } returns Mono.just(openPit("pit-1")) + every { client.closePointInTime(any()) } returns Mono.just(closePit()) + every { client.search(any(), Map::class.java) } returnsMany listOf( + Mono.just(response(timedOut = true)), + Mono.just(response(failedShards = 1)), + Mono.just(response(totalRelation = TotalHitsRelation.Gte)), + ) + + assertFailure(QueryBackendFailureKind.TIMEOUT) { backend.single(singlePlan, OPTIONS).block() } + assertFailure(QueryBackendFailureKind.INCOMPLETE_RESULT) { backend.single(singlePlan, OPTIONS).block() } + assertFailure(QueryBackendFailureKind.INCOMPLETE_RESULT) { backend.page(pagePlan, OPTIONS).block() } + + verify(exactly = 3) { client.search(any(), Map::class.java) } + verify(exactly = 1) { client.openPointInTime(any()) } + verify(exactly = 1) { client.closePointInTime(any()) } + } + + @Test + fun `deep page should traverse search after and close the latest PIT`() { + val client = mockk() + val searches = mutableListOf() + val closeRequest = slot() + every { client.openPointInTime(any()) } returns Mono.just(openPit("pit-1")) + every { client.search(capture(searches), Map::class.java) } returnsMany listOf( + Mono.just( + response( + totalRelation = TotalHitsRelation.Eq, + total = 1_001, + ids = (0 until 1_000).map(Int::toString), + pitId = "pit-2", + ), + ), + Mono.just( + response( + totalRelation = TotalHitsRelation.Eq, + total = 1_001, + ids = listOf("1000"), + pitId = "pit-3", + ), + ), + ) + every { client.closePointInTime(capture(closeRequest)) } returns Mono.just(closePit()) + + val result = backend(client).page(pagePlan(1_000, 1), OPTIONS.copy(maxCursorPages = 2)).block()!! + + result.records.single().identity.assert().isEqualTo("1000") + result.total.assert().isEqualTo(1_001) + searches.assert().hasSize(2) + searches.first().searchAfter().assert().isEmpty() + searches.last().searchAfter().single().stringValue().assert().isEqualTo("999") + closeRequest.captured.id().assert().isEqualTo("pit-3") + } + + @Test + fun `deep page cancellation should close the latest PIT`() { + val client = mockk() + val closeRequest = slot() + every { client.openPointInTime(any()) } returns Mono.just(openPit("pit-1")) + every { client.search(any(), Map::class.java) } returnsMany listOf( + Mono.just( + response( + totalRelation = TotalHitsRelation.Eq, + total = 1_001, + ids = (0 until 1_000).map(Int::toString), + pitId = "pit-2", + ), + ), + Mono.never(), + ) + every { client.closePointInTime(capture(closeRequest)) } returns Mono.just(closePit()) + + StepVerifier.create(backend(client).page(pagePlan(1_000, 1), OPTIONS.copy(maxCursorPages = 2))) + .thenAwait(Duration.ofMillis(10)) + .thenCancel() + .verify() + + verify(timeout = 1_000, exactly = 1) { client.closePointInTime(any()) } + closeRequest.captured.id().assert().isEqualTo("pit-2") + } + + @Test + fun `expired PIT should fail incomplete and close the latest lease without replacing the original error`() { + val client = mockk() + val closeRequest = slot() + val pitExpired = mockk { + every { status() } returns NOT_FOUND + } + val closeFailure = IllegalStateException("close failed") + every { client.openPointInTime(any()) } returns Mono.just(openPit("pit-1")) + every { client.search(any(), Map::class.java) } returnsMany listOf( + Mono.just( + response( + totalRelation = TotalHitsRelation.Eq, + total = 1_001, + ids = (0 until 1_000).map(Int::toString), + pitId = "pit-2", + ), + ), + Mono.error(pitExpired), + ) + every { client.closePointInTime(capture(closeRequest)) } returns Mono.error(closeFailure) + + val error = runCatching { + backend(client).page(pagePlan(1_000, 1), OPTIONS.copy(maxCursorPages = 2)).block() + }.exceptionOrNull() as QueryBackendException + + error.kind.assert().isEqualTo(QueryBackendFailureKind.INCOMPLETE_RESULT) + error.cause.assert().isSameAs(pitExpired) + error.suppressed.any { suppressed -> suppressed === closeFailure }.assert().isTrue() + closeRequest.captured.id().assert().isEqualTo("pit-2") + } + + @Test + fun `expired PIT wrapped as search phase failure should fail incomplete`() { + val client = mockk() + val missingContext = mockk { + every { type() } returns SEARCH_CONTEXT_MISSING_TYPE + every { causedBy() } returns null + every { rootCause() } returns emptyList() + every { suppressed() } returns emptyList() + } + val searchPhaseFailure = mockk { + every { type() } returns "search_phase_execution_exception" + every { causedBy() } returns null + every { rootCause() } returns listOf(missingContext) + every { suppressed() } returns emptyList() + } + val pitExpired = mockk { + every { status() } returns BAD_REQUEST + every { error() } returns searchPhaseFailure + } + every { client.openPointInTime(any()) } returns Mono.just(openPit("pit-1")) + every { client.search(any(), Map::class.java) } returns Mono.error(pitExpired) + every { client.closePointInTime(any()) } returns Mono.just(closePit()) + + val error = runCatching { + backend(client).page(pagePlan(0, 1), OPTIONS).block() + }.exceptionOrNull() as QueryBackendException + + error.kind.assert().isEqualTo(QueryBackendFailureKind.INCOMPLETE_RESULT) + error.cause.assert().isSameAs(pitExpired) + verify(exactly = 1) { client.closePointInTime(any()) } + } + + @Test + fun `expired PIT during successful cleanup should fail incomplete rather than unavailable`() { + val client = mockk() + every { client.openPointInTime(any()) } returns Mono.just(openPit("pit-1")) + every { client.search(any(), Map::class.java) } returns Mono.just( + response(totalRelation = TotalHitsRelation.Eq, total = 0), + ) + every { client.closePointInTime(any()) } returns Mono.empty() + + assertFailure(QueryBackendFailureKind.INCOMPLETE_RESULT) { + backend(client).page(pagePlan(0, 1), OPTIONS).block() + } + } + + @Test + fun `unsupported budgets and expired deadline should fail before Elasticsearch IO`() { + val client = mockk() + val backend = backend(client) + + assertFailure(QueryBackendFailureKind.UNSUPPORTED) { + backend.single(singlePlan, OPTIONS.copy(maxScannedRecords = 1)).block() + } + assertFailure(QueryBackendFailureKind.TIMEOUT) { + backend.single(singlePlan, OPTIONS.copy(deadline = NOW.minusMillis(1))).block() + } + + confirmVerified(client) + } + + @Test + fun `bounded stream beyond the supported result window should fail before Elasticsearch IO`() { + val client = mockk() + + assertFailure(QueryBackendFailureKind.UNSUPPORTED) { + backend(client).stream( + streamPlan(10_001), + OPTIONS.copy(maxReturnedRecords = 20_000), + ).blockLast() + } + + confirmVerified(client) + } + + private fun backend(client: ReactiveElasticsearchClient) = ElasticsearchSnapshotRecordQueryBackend( + client, + binding.prepared, + Clock.fixed(NOW, ZoneOffset.UTC), + ) + + private fun response( + timedOut: Boolean = false, + failedShards: Int = 0, + totalRelation: TotalHitsRelation? = null, + total: Long = 0, + ids: List = emptyList(), + pitId: String? = null, + ): SearchResponse> = SearchResponse.of> { response -> + response.took(1) + .timedOut(timedOut) + .shards { shards -> shards.failed(failedShards).successful(1).total(1 + failedShards) } + .hits { hits -> + hits.hits( + ids.map { id -> + Hit.of> { hit -> + hit.index("wow.sales.order.snapshot") + .id(id) + .source(mapOf(MessageRecords.AGGREGATE_ID to id)) + .sort(id) + } + }, + ) + totalRelation?.let { relation -> + hits.total { totalHits -> totalHits.relation(relation).value(total) } + } + hits + } + .also { builder -> pitId?.let(builder::pitId) } + } + + private fun openPit(id: String): OpenPointInTimeResponse = + OpenPointInTimeResponse.of { response -> + response.id(id).shards { shards -> shards.failed(0).successful(1).total(1) } + } + + private fun closePit(): ClosePointInTimeResponse = ClosePointInTimeResponse.of { response -> + response.succeeded(true).numFreed(1) + } + + private fun assertFailure(kind: QueryBackendFailureKind, action: () -> Unit) { + assertThrownBy(action).satisfies( + Consumer { error -> error.kind.assert().isEqualTo(kind) }, + ) + } + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + private val schema = QueryDocumentSchema( + target, + listOf( + QueryFieldSchema( + identity, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT, FieldCapability.SORTABLE), + ), + ), + emptyList(), + ) + private val binding = ElasticsearchSnapshotQueryBinding( + schema, + "wow.sales.order.snapshot", + "order-query-v1", + mapOf( + identity to ElasticsearchFieldBinding( + MessageRecords.AGGREGATE_ID, + setOf(FieldCapability.EXACT, FieldCapability.SORTABLE), + exactField = "_id", + sortField = MessageRecords.AGGREGATE_ID, + ), + ), + ) + private val filter = BackendEnforcedFilter(BackendPlannedCondition.All, BackendPlannedCondition.All) + private val singlePlan = BackendSingleQueryPlan( + target, + schema.contractId, + filter, + RecordResultShape.DYNAMIC, + BackendProjection.All, + emptyList(), + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("1".repeat(64)), + ) + private val pagePlan = pagePlan(0, 10) + + private fun streamPlan(limit: Int) = BackendStreamQueryPlan( + target, + schema.contractId, + filter, + RecordResultShape.DYNAMIC, + BackendProjection.All, + emptyList(), + limit, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("3".repeat(64)), + ) + + private fun pagePlan(offset: Long, size: Int) = BackendPageQueryPlan( + target, + schema.contractId, + filter, + RecordResultShape.DYNAMIC, + BackendProjection.All, + listOf(BackendSort(identity, NormalizedSortDirection.ASC, BackendSortOrigin.STABILITY_TIE_BREAKER)), + BackendPageWindow(offset, size), + BackendTotalMode.EXACT, + BackendRequiredConsistency.SAME_INPUT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("2".repeat(64)), + ) + + private companion object { + const val BAD_REQUEST = 400 + const val NOT_FOUND = 404 + const val SEARCH_CONTEXT_MISSING_TYPE = "search_context_missing_exception" + val NOW: Instant = Instant.parse("2026-08-08T00:00:00Z") + val OPTIONS = QueryBackendExecutionOptions(NOW.plusSeconds(10), 100) + } +} diff --git a/wow-mongo/src/integrationTest/kotlin/me/ahoo/wow/mongo/query/cursor/MongoAnalyticsCursorGatewayIntegrationTest.kt b/wow-mongo/src/integrationTest/kotlin/me/ahoo/wow/mongo/query/cursor/MongoAnalyticsCursorGatewayIntegrationTest.kt new file mode 100644 index 00000000000..aea2a441469 --- /dev/null +++ b/wow-mongo/src/integrationTest/kotlin/me/ahoo/wow/mongo/query/cursor/MongoAnalyticsCursorGatewayIntegrationTest.kt @@ -0,0 +1,294 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.cursor + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.query.analytics.AnalyticsBucketWindow +import me.ahoo.wow.api.query.analytics.AnalyticsDimension +import me.ahoo.wow.api.query.analytics.AnalyticsGrouping +import me.ahoo.wow.api.query.analytics.AnalyticsMetric +import me.ahoo.wow.api.query.analytics.AnalyticsMetricKind +import me.ahoo.wow.api.query.analytics.AnalyticsQuery +import me.ahoo.wow.api.query.analytics.AnalyticsValue +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.AnalyticsQueryBackend +import me.ahoo.wow.query.backend.BackendAnalyticsBucket +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsPage +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.BackendRecord +import me.ahoo.wow.query.backend.BackendSingleQueryPlan +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.BackendStreamSupport +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendComposition +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.RecordQueryBackend +import me.ahoo.wow.query.backend.RecordQueryBackendContribution +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.cursor.QueryCursorHmacKey +import me.ahoo.wow.query.cursor.QueryCursorLeaseConfiguration +import me.ahoo.wow.query.cursor.QueryCursorSigningKeys +import me.ahoo.wow.query.event.NoOpEventStreamQueryServiceFactory +import me.ahoo.wow.query.gateway.QueryAuthority +import me.ahoo.wow.query.gateway.QueryAuthorityResolver +import me.ahoo.wow.query.gateway.QueryCall +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryElementPathMode +import me.ahoo.wow.query.gateway.QueryErrorCategory +import me.ahoo.wow.query.gateway.QueryExecutionException +import me.ahoo.wow.query.gateway.QueryExecutionMode +import me.ahoo.wow.query.gateway.QueryExecutionProfile +import me.ahoo.wow.query.gateway.QueryExecutionProfiles +import me.ahoo.wow.query.gateway.QueryGatewayRuntime +import me.ahoo.wow.query.gateway.QueryLegacyDialect +import me.ahoo.wow.query.gateway.QueryLegacyDialectResolver +import me.ahoo.wow.query.gateway.QueryMatchScopeMode +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryOperationProfileKey +import me.ahoo.wow.query.gateway.QueryPurpose +import me.ahoo.wow.query.gateway.QueryRawServiceSource +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.gateway.QueryValidationMode +import me.ahoo.wow.query.snapshot.NoOpSnapshotQueryServiceFactory +import me.ahoo.wow.tck.container.MongoTestFixture +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.extension.RegisterExtension +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.Instant +import java.time.ZoneOffset +import java.util.concurrent.atomic.AtomicInteger +import java.util.function.Consumer + +class MongoAnalyticsCursorGatewayIntegrationTest { + @JvmField + @RegisterExtension + val mongo = MongoTestFixture("analytics_cursor_gateway") + + @Test + fun `should replay an analytics cursor across nodes and rotate signing keys exactly once`() { + val options = MongoQueryCursorLeaseStoreOptions(collectionName = "analytics_cursor_lease", maxEntries = 8) + val firstStore = MongoQueryCursorLeaseStore(mongo.database(), options, CLOCK) + val secondStore = MongoQueryCursorLeaseStore( + mongo.newClient().getDatabase(mongo.databaseName), + options, + CLOCK, + ) + firstStore.ensureIndexes().block() + val backend = ThreePageAnalyticsBackend() + val firstNode = runtime(firstStore, backend, QueryCursorSigningKeys(KEY_ONE)).analyticsGateway + val secondNode = runtime( + secondStore, + backend, + QueryCursorSigningKeys(KEY_TWO, listOf(KEY_ONE)), + ).analyticsGateway + + val first = firstNode.analyze(CALL, query()).block()!! + first.buckets.single().keys.assert().containsEntry("status", AnalyticsValue.of("A")) + val firstToken = checkNotNull(first.nextCursor) + + val second = secondNode.analyze(CALL, query(firstToken)).block()!! + second.buckets.single().keys.assert().containsEntry("status", AnalyticsValue.of("B")) + val rotatedToken = checkNotNull(second.nextCursor) + + assertInvalidCursor { + firstNode.analyze(CALL, query(rotatedToken)).block() + } + val third = secondNode.analyze(CALL, query(rotatedToken)).block()!! + third.buckets.single().keys.assert().containsEntry("status", AnalyticsValue.of("C")) + third.nextCursor.assert().isNull() + + assertInvalidCursor { + secondNode.analyze(CALL, query(firstToken)).block() + } + backend.calls.get().assert().isEqualTo(3) + } + + private fun runtime( + store: MongoQueryCursorLeaseStore, + backend: AnalyticsQueryBackend, + keys: QueryCursorSigningKeys, + ): QueryGatewayRuntime = QueryGatewayRuntime.create( + namedAggregates = listOf(AGGREGATE), + backendComposition = composition(backend), + cursorLeaseConfiguration = QueryCursorLeaseConfiguration(store, keys), + rawServiceSource = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate) = + NoOpSnapshotQueryServiceFactory.create(namedAggregate) + + override fun eventStream(namedAggregate: NamedAggregate) = + NoOpEventStreamQueryServiceFactory.create(namedAggregate) + }, + dialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.DOCUMENT) + }, + authorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.System("mongo-cursor-test", "cross-node cursor")) + }, + executionProfiles = PROFILES, + clock = CLOCK, + ) + + private fun composition(backend: AnalyticsQueryBackend): QueryBackendComposition = QueryBackendComposition( + listOf( + RecordQueryBackendContribution( + schema = SCHEMA, + backendId = BACKEND_ID, + supportedOperations = setOf(QueryOperation.ANALYZE), + streamSupport = BackendStreamSupport.NONE, + semanticTiers = setOf(SemanticTier.PORTABLE), + fieldCapabilities = mapOf( + DELETED to setOf(FieldCapability.EXACT), + STATUS to setOf(FieldCapability.AGGREGATABLE), + ), + backend = NoOpRecordBackend, + analyticsBackend = backend, + mappingGenerationDigest = "a".repeat(64), + ), + ), + mapOf(TARGET to BACKEND_ID), + ) + + private fun query(cursor: me.ahoo.wow.api.query.analytics.AnalyticsCursor? = null): AnalyticsQuery = AnalyticsQuery( + grouping = AnalyticsGrouping.by(listOf(AnalyticsDimension("status", "state.status"))), + metrics = listOf(AnalyticsMetric("count", AnalyticsMetricKind.DOCUMENT_COUNT)), + window = AnalyticsBucketWindow(1, cursor), + ) + + private fun assertInvalidCursor(action: () -> Unit) { + assertThrownBy(action).satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.INVALID_CURSOR) + error.path.assert().isEqualTo("$.cursor") + error.code.assert().isEqualTo("INVALID_CURSOR_TOKEN") + }, + ) + } + + private class ThreePageAnalyticsBackend : AnalyticsQueryBackend { + val calls = AtomicInteger() + + override fun analyze( + plan: me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono { + calls.incrementAndGet() + val previous = (plan.bucketWindow.afterKey?.singleOrNull() as? NormalizedValue.Text)?.value + val current = when (previous) { + null -> "A" + "A" -> "B" + "B" -> "C" + else -> error("Unexpected analytics cursor position: $previous") + } + return Mono.just( + BackendAnalyticsPage( + listOf( + BackendAnalyticsBucket( + mapOf(AnalyticsAlias("status") to NormalizedValue.Text(current)), + mapOf(AnalyticsAlias("count") to NormalizedValue.Int64(1)), + ), + ), + if (current == "C") null else listOf(NormalizedValue.Text(current)), + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + ), + ) + } + } + + private object NoOpRecordBackend : RecordQueryBackend { + override fun single( + plan: BackendSingleQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = Mono.empty() + + override fun stream( + plan: BackendStreamQueryPlan, + options: QueryBackendExecutionOptions, + ): Flux = Flux.empty() + + override fun count(plan: BackendCountQueryPlan, options: QueryBackendExecutionOptions): Mono = Mono.just(0) + } + + private companion object { + val CLOCK: Clock = Clock.fixed(Instant.parse("2026-08-09T00:00:00Z"), ZoneOffset.UTC) + val KEY_ONE = QueryCursorHmacKey(1, ByteArray(32) { 1 }) + val KEY_TWO = QueryCursorHmacKey(2, ByteArray(32) { 2 }) + val AGGREGATE = MaterializedNamedAggregate("sales", "order") + val TARGET = QueryTarget(AGGREGATE, QueryDocumentKind.SNAPSHOT) + val CALL = QueryCall(TARGET, QueryPurpose("mongo-cursor-integration")) + val BACKEND_ID = BackendId("mongo-cursor-test") + val DELETED = QueryFieldId.System(SystemFieldKind.DELETED) + val STATE = QueryFieldId.Path(listOf("state")) + val STATUS = QueryFieldId.Path(listOf("state", "status")) + val SCHEMA = QueryDocumentSchema( + TARGET, + listOf( + QueryFieldSchema( + DELETED, + LogicalFieldType.Boolean, + Presence.REQUIRED, + Nullability.NON_NULL, + listOf(PredicateOperator.IS_TRUE, PredicateOperator.IS_FALSE), + listOf(FieldCapability.EXACT), + ), + QueryFieldSchema( + STATE, + LogicalFieldType.Object, + Presence.REQUIRED, + Nullability.NON_NULL, + emptyList(), + emptyList(), + ), + QueryFieldSchema( + STATUS, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + emptyList(), + listOf(FieldCapability.AGGREGATABLE), + ), + ), + emptyList(), + ) + val PROFILES = QueryExecutionProfiles( + operationProfiles = mapOf( + QueryOperationProfileKey(TARGET, QueryOperation.ANALYZE) to + QueryExecutionProfile(QueryExecutionMode.PLANNED, QueryValidationMode.STRICT), + ), + ) + } +} diff --git a/wow-mongo/src/integrationTest/kotlin/me/ahoo/wow/mongo/query/cursor/MongoQueryCursorLeaseStoreIntegrationTest.kt b/wow-mongo/src/integrationTest/kotlin/me/ahoo/wow/mongo/query/cursor/MongoQueryCursorLeaseStoreIntegrationTest.kt new file mode 100644 index 00000000000..091c9fcb60f --- /dev/null +++ b/wow-mongo/src/integrationTest/kotlin/me/ahoo/wow/mongo/query/cursor/MongoQueryCursorLeaseStoreIntegrationTest.kt @@ -0,0 +1,169 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class) + +package me.ahoo.wow.mongo.query.cursor + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.cursor.QueryCursorLeaseCreateResult +import me.ahoo.wow.query.cursor.QueryCursorLeaseEntry +import me.ahoo.wow.query.cursor.QueryCursorLeaseId +import me.ahoo.wow.query.cursor.QueryCursorPayloadFormat +import me.ahoo.wow.query.cursor.QueryCursorStoreRevision +import me.ahoo.wow.query.cursor.StoredQueryCursorLease +import me.ahoo.wow.tck.container.MongoTestFixture +import org.junit.jupiter.api.BeforeEach +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.extension.RegisterExtension +import reactor.core.publisher.Mono +import reactor.kotlin.core.publisher.toFlux +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset + +class MongoQueryCursorLeaseStoreIntegrationTest { + @JvmField + @RegisterExtension + val mongo = MongoTestFixture("query_cursor") + + private val now = Instant.parse("2026-08-09T00:00:00Z") + private val clock = Clock.fixed(now, ZoneOffset.UTC) + private lateinit var store: MongoQueryCursorLeaseStore + + @BeforeEach + fun setup() { + store = MongoQueryCursorLeaseStore( + mongo.database(), + MongoQueryCursorLeaseStoreOptions( + collectionName = "wow_query_cursor_lease_test", + maxEntries = 8, + retentionGrace = Duration.ofMinutes(5), + maxPayloadBytes = 128, + maxScanSize = 10, + ), + clock, + ) + store.ensureIndexes().block() + } + + @Test + fun `should initialize compatible unique and grace TTL indexes idempotently`() { + store.ensureIndexes().block() + + val indexes = mongo.database().getCollection(store.options.collectionName) + .listIndexes().toFlux().collectList().block()!!.associateBy { index -> index.getString("name") } + + indexes.getValue("lease_id_unique").getBoolean("unique").assert().isTrue() + (indexes.getValue("purge_at_ttl")["expireAfterSeconds"] as Number).toLong().assert().isZero() + } + + @Test + fun `should persist and load an immutable lease across store instances`() { + val payload = byteArrayOf(1, 2, 3) + val entry = entry("lease-a", now.plusSeconds(60), payload) + + store.create(entry).block().assert().isEqualTo(QueryCursorLeaseCreateResult.CREATED) + payload[0] = 9 + + val anotherStore = MongoQueryCursorLeaseStore( + mongo.newClient().getDatabase(mongo.databaseName), + store.options, + clock, + ) + val loaded = anotherStore.load(entry.id).block()!! + + loaded.entry.assert().isEqualTo(entry) + loaded.entry.payload().toList().assert().containsExactly(1.toByte(), 2.toByte(), 3.toByte()) + loaded.revision.value.assert().isNotBlank() + } + + @Test + fun `should distinguish lease id collision from bounded capacity`() { + val capacityStore = MongoQueryCursorLeaseStore( + mongo.database(), + store.options.copy(collectionName = "wow_query_cursor_capacity_test", maxEntries = 2), + clock, + ) + capacityStore.ensureIndexes().block() + val first = entry("lease-1", now.plusSeconds(60)) + + capacityStore.create(first).block().assert().isEqualTo(QueryCursorLeaseCreateResult.CREATED) + capacityStore.create(first).block().assert().isEqualTo(QueryCursorLeaseCreateResult.COLLISION) + capacityStore.create(entry("lease-2", now.plusSeconds(60))).block() + .assert().isEqualTo(QueryCursorLeaseCreateResult.CREATED) + capacityStore.create(entry("lease-3", now.plusSeconds(60))).block() + .assert().isEqualTo(QueryCursorLeaseCreateResult.CAPACITY_EXCEEDED) + } + + @Test + fun `should transfer one-time ownership with revision compare and delete across nodes`() { + val entry = entry("lease-cas", now.plusSeconds(60)) + store.create(entry).block().assert().isEqualTo(QueryCursorLeaseCreateResult.CREATED) + val loaded = store.load(entry.id).block()!! + val wrongRevision = StoredQueryCursorLease( + loaded.entry, + QueryCursorStoreRevision("wrong-revision"), + ) + + store.compareAndDelete(wrongRevision).block().assert().isFalse() + + val anotherStore = MongoQueryCursorLeaseStore( + mongo.newClient().getDatabase(mongo.databaseName), + store.options, + clock, + ) + val winners = Mono.zip( + store.compareAndDelete(loaded), + anotherStore.compareAndDelete(loaded), + ).block()!!.let { result -> listOf(result.t1, result.t2) } + + winners.count { won -> won }.assert().isEqualTo(1) + store.load(entry.id).block().assert().isNull() + } + + @Test + fun `should scan expired leases by stable lease id keyset`() { + listOf( + entry("lease-b", now.plusSeconds(20)), + entry("lease-a", now.plusSeconds(10)), + entry("lease-c", now.plusSeconds(40)), + ).forEach { entry -> + store.create(entry).block().assert().isEqualTo(QueryCursorLeaseCreateResult.CREATED) + } + + val firstPage = store.scanExpired(now.plusSeconds(30), null, 1).collectList().block()!! + firstPage.map { lease -> lease.entry.id.value }.assert().containsExactly("lease-a") + + val secondPage = store.scanExpired(now.plusSeconds(30), firstPage.single().entry.id, 10) + .collectList().block()!! + secondPage.map { lease -> lease.entry.id.value }.assert().containsExactly("lease-b") + + assertThrownBy { + store.scanExpired(now.plusSeconds(30), null, 11).collectList().block() + } + } + + private fun entry( + id: String, + expiresAt: Instant, + payload: ByteArray = byteArrayOf(1), + ): QueryCursorLeaseEntry = QueryCursorLeaseEntry( + QueryCursorLeaseId(id), + expiresAt, + QueryCursorPayloadFormat.WOW_QUERY_CURSOR_V1, + payload, + ) +} diff --git a/wow-mongo/src/integrationTest/kotlin/me/ahoo/wow/mongo/query/planned/MongoEventStreamRecordQueryBackendIntegrationTest.kt b/wow-mongo/src/integrationTest/kotlin/me/ahoo/wow/mongo/query/planned/MongoEventStreamRecordQueryBackendIntegrationTest.kt new file mode 100644 index 00000000000..9397dca6dd1 --- /dev/null +++ b/wow-mongo/src/integrationTest/kotlin/me/ahoo/wow/mongo/query/planned/MongoEventStreamRecordQueryBackendIntegrationTest.kt @@ -0,0 +1,171 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.MongoNamespace +import me.ahoo.test.asserts.assert +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.PagedQuery +import me.ahoo.wow.api.query.Pagination +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.mongo.Documents +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendComposition +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.event.NoOpEventStreamQueryServiceFactory +import me.ahoo.wow.query.gateway.QueryAuthority +import me.ahoo.wow.query.gateway.QueryAuthorityResolver +import me.ahoo.wow.query.gateway.QueryCall +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryElementPathMode +import me.ahoo.wow.query.gateway.QueryExecutionMode +import me.ahoo.wow.query.gateway.QueryExecutionProfile +import me.ahoo.wow.query.gateway.QueryExecutionProfiles +import me.ahoo.wow.query.gateway.QueryGatewayRuntime +import me.ahoo.wow.query.gateway.QueryLegacyDialect +import me.ahoo.wow.query.gateway.QueryLegacyDialectResolver +import me.ahoo.wow.query.gateway.QueryMatchScopeMode +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryOperationProfileKey +import me.ahoo.wow.query.gateway.QueryPurpose +import me.ahoo.wow.query.gateway.QueryRawServiceSource +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.gateway.QueryValidationMode +import me.ahoo.wow.query.snapshot.NoOpSnapshotQueryServiceFactory +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.tck.container.MongoTestFixture +import org.bson.Document +import org.junit.jupiter.api.BeforeEach +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.extension.RegisterExtension +import reactor.core.publisher.Mono +import reactor.kotlin.core.publisher.toMono + +class MongoEventStreamRecordQueryBackendIntegrationTest { + @JvmField + @RegisterExtension + val mongo = MongoTestFixture("planned_event_query") + + private lateinit var collectionNamespace: MongoNamespace + + @BeforeEach + fun setup() { + collectionNamespace = MongoNamespace(mongo.databaseName, "order_event_stream") + collection().insertMany( + listOf( + event("stream-1", "order-1", "tenant-1"), + event("stream-2", "order-2", "tenant-1"), + event("stream-3", "order-3", "tenant-2"), + ), + ).toMono().block() + } + + @Test + fun `gateway planned EventStream should enforce tenant without snapshot deletion`() { + val contribution = MongoEventStreamQueryBinding.frameworkFields(schema(), collectionNamespace) + .toContribution(collection()) + contribution.analyticsBackend.assert().isNull() + contribution.supportedOperations.assert().doesNotContain(QueryOperation.ANALYZE) + val gateway = QueryGatewayRuntime.create( + namedAggregates = listOf(target.namedAggregate), + backendComposition = QueryBackendComposition( + listOf(contribution), + mapOf(target to BackendId("mongo")), + ), + rawServiceSource = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate) = + NoOpSnapshotQueryServiceFactory.create(namedAggregate) + + override fun eventStream(namedAggregate: NamedAggregate) = + NoOpEventStreamQueryServiceFactory.create(namedAggregate) + }, + dialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.FIELD) + }, + authorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.Subject("subject-1", "tenant-1")) + }, + executionProfiles = QueryExecutionProfiles( + operationProfiles = mapOf( + QueryOperationProfileKey(target, QueryOperation.COUNT) to planned, + QueryOperationProfileKey(target, QueryOperation.PAGE) to planned, + ), + ), + ).gateway + val call = QueryCall(target, QueryPurpose("event-audit")) + + gateway.count(call, Condition.ALL).block().assert().isEqualTo(2) + val page = gateway.page(call, PagedQuery(Condition.ALL, pagination = Pagination(1, 1))).block()!! + page.total.assert().isEqualTo(2) + page.list.assert().hasSize(1) + page.list.single()[MessageRecords.ID].assert().isEqualTo("stream-1") + page.list.single()[MessageRecords.AGGREGATE_ID].assert().isEqualTo("order-1") + } + + private fun schema(): QueryDocumentSchema = QueryDocumentSchema( + target, + listOf( + textField(QueryFieldId.System(SystemFieldKind.IDENTITY), sortable = true), + textField(QueryFieldId.System(SystemFieldKind.AGGREGATE_ID)), + textField(QueryFieldId.System(SystemFieldKind.TENANT_ID)), + ), + emptyList(), + ) + + private fun textField(id: QueryFieldId, sortable: Boolean = false) = QueryFieldSchema( + id, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ, PredicateOperator.IN), + buildSet { + add(FieldCapability.EXACT) + add(FieldCapability.PROJECTABLE) + if (sortable) add(FieldCapability.SORTABLE) + }, + ) + + private fun collection() = mongo.database().getCollection(collectionNamespace.collectionName) + + private fun event(id: String, aggregateId: String, tenantId: String) = Document( + linkedMapOf( + Documents.ID_FIELD to id, + MessageRecords.AGGREGATE_ID to aggregateId, + MessageRecords.TENANT_ID to tenantId, + MessageRecords.VERSION to 1, + ), + ) + + private companion object { + val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.EVENT_STREAM, + ) + val planned = QueryExecutionProfile(QueryExecutionMode.PLANNED, QueryValidationMode.STRICT) + } +} diff --git a/wow-mongo/src/integrationTest/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotRecordQueryBackendIntegrationTest.kt b/wow-mongo/src/integrationTest/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotRecordQueryBackendIntegrationTest.kt new file mode 100644 index 00000000000..16dff2d5e6b --- /dev/null +++ b/wow-mongo/src/integrationTest/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotRecordQueryBackendIntegrationTest.kt @@ -0,0 +1,1240 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.ExplainVerbosity +import com.mongodb.MongoNamespace +import com.mongodb.client.model.Collation +import com.mongodb.client.model.IndexOptions +import com.mongodb.client.model.Indexes +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.mongo.Documents +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.AnalyticsQueryBackend +import me.ahoo.wow.query.backend.BackendAnalyticsBucketOrder +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsCondition +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsDimension +import me.ahoo.wow.query.backend.BackendAnalyticsGrouping +import me.ahoo.wow.query.backend.BackendAnalyticsMetric +import me.ahoo.wow.query.backend.BackendAnalyticsMissingPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNullPlacement +import me.ahoo.wow.query.backend.BackendAnalyticsNumericPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNumericPromotion +import me.ahoo.wow.query.backend.BackendAnalyticsOverflowPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsPageWindow +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.BackendAnalyticsTextCollation +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPageConsistency +import me.ahoo.wow.query.backend.BackendPageQueryPlan +import me.ahoo.wow.query.backend.BackendPageWindow +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.BackendRecordCompleteness +import me.ahoo.wow.query.backend.BackendRequiredCapabilities +import me.ahoo.wow.query.backend.BackendRequiredConsistency +import me.ahoo.wow.query.backend.BackendSort +import me.ahoo.wow.query.backend.BackendSortOrigin +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.BackendTotalMode +import me.ahoo.wow.query.backend.BackendTotalRelation +import me.ahoo.wow.query.backend.EmptyArraySemantics +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.JunctionOperator +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedSortDirection +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.RecordQueryBackend +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.QuerySearchScopeDefinition +import me.ahoo.wow.query.backend.RecordResultShape +import me.ahoo.wow.query.backend.QueryBackendComposition +import me.ahoo.wow.query.backend.SearchScopeId +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DynamicDocument +import me.ahoo.wow.api.query.IListQuery +import me.ahoo.wow.api.query.IPagedQuery +import me.ahoo.wow.api.query.ISingleQuery +import me.ahoo.wow.api.query.PagedQuery +import me.ahoo.wow.api.query.PagedList +import me.ahoo.wow.api.query.Pagination +import me.ahoo.wow.query.event.NoOpEventStreamQueryServiceFactory +import me.ahoo.wow.query.gateway.QueryAuthority +import me.ahoo.wow.query.gateway.QueryAuthorityResolver +import me.ahoo.wow.query.gateway.QueryCall +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryElementPathMode +import me.ahoo.wow.query.gateway.QueryExecutionMode +import me.ahoo.wow.query.gateway.QueryExecutionProfile +import me.ahoo.wow.query.gateway.QueryExecutionProfiles +import me.ahoo.wow.query.gateway.QueryGatewayRuntime +import me.ahoo.wow.query.gateway.QueryLegacyDialect +import me.ahoo.wow.query.gateway.QueryLegacyDialectResolver +import me.ahoo.wow.query.gateway.QueryMatchScopeMode +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryOperationProfileKey +import me.ahoo.wow.query.gateway.QueryPurpose +import me.ahoo.wow.query.gateway.QueryRawServiceSource +import me.ahoo.wow.query.gateway.QueryRuntimeHealthObserver +import me.ahoo.wow.query.gateway.QueryShadowObservation +import me.ahoo.wow.query.gateway.QueryShadowObserver +import me.ahoo.wow.query.gateway.QueryShadowOutcome +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.gateway.QueryValidationMode +import me.ahoo.wow.query.snapshot.SnapshotQueryService +import me.ahoo.wow.mongo.query.snapshot.SnapshotConditionConverter +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.serialization.state.StateAggregateRecords +import me.ahoo.wow.tck.container.MongoTestFixture +import me.ahoo.wow.tck.query.ExactNumericAnalyticsExpectation +import me.ahoo.wow.tck.query.PlannedAnalyticsQueryBackendSpec +import me.ahoo.wow.tck.query.PlannedRecordQueryBackendSpec +import org.bson.Document +import org.bson.types.Decimal128 +import org.junit.jupiter.api.BeforeEach +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.extension.RegisterExtension +import reactor.kotlin.core.publisher.toMono +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit +import java.math.BigDecimal +import java.math.RoundingMode +import java.time.Instant +import java.util.function.Consumer +import java.util.concurrent.atomic.AtomicInteger + +class MongoSnapshotRecordQueryBackendIntegrationTest : + PlannedAnalyticsQueryBackendSpec, + PlannedRecordQueryBackendSpec { + @JvmField + @RegisterExtension + val mongo = MongoTestFixture("planned_query") + + private lateinit var fixture: Fixture + + override val analyticsBackend: AnalyticsQueryBackend + get() = fixture.binding.prepareContribution(fixture.collection).block()!!.analyticsBackend!! + override val analyticsOptions: QueryBackendExecutionOptions + get() = fixture.analyticsOptions().copy(maxReturnedBuckets = 10) + override val expectedGlobalCount: Long = 2 + override val expectedUnrestrictedGlobalCount: Long = 4 + override val expectedFirstKey: NormalizedValue = NormalizedValue.Text("CREATED") + override val expectedSecondKey: NormalizedValue = NormalizedValue.Text("PAID") + override val expectedNullBucketCount: Long = 2 + override val dimensionAlias: AnalyticsAlias = AnalyticsAlias("status") + override val countAlias: AnalyticsAlias = AnalyticsAlias("count") + override val exactNumericAnalyticsExpectation: ExactNumericAnalyticsExpectation = + ExactNumericAnalyticsExpectation.Supported( + mapOf( + AnalyticsAlias("minimum") to NormalizedValue.Decimal(BigDecimal.ONE), + AnalyticsAlias("maximum") to NormalizedValue.Decimal(BigDecimal("2")), + AnalyticsAlias("total") to NormalizedValue.Decimal(BigDecimal("3")), + AnalyticsAlias("average") to NormalizedValue.Decimal(BigDecimal("1.5")), + ), + ) + override val recordBackend: RecordQueryBackend + get() = fixture.binding.prepareContribution(fixture.collection).block()!!.backend + override val recordOptions: QueryBackendExecutionOptions + get() = OPTIONS + override val expectedRecordIdentities: List = listOf("order-1", "order-4") + + override fun globalCountPlan(): BackendAnalyticsQueryPlan = fixture.analyticsPlan( + BackendAnalyticsGrouping.Global, + listOf(BackendAnalyticsMetric.DocumentCount(countAlias)), + BackendEnforcedFilter(BackendPlannedCondition.All, analyticsMandatory()), + ) + + override fun unrestrictedGlobalCountPlan(): BackendAnalyticsQueryPlan = fixture.analyticsPlan( + BackendAnalyticsGrouping.Global, + listOf(BackendAnalyticsMetric.DocumentCount(countAlias)), + BackendEnforcedFilter(BackendPlannedCondition.All, BackendPlannedCondition.All), + ) + + override fun groupedCountPlan(afterKey: List?, limit: Int): BackendAnalyticsQueryPlan = + fixture.analyticsPlan( + BackendAnalyticsGrouping.By( + listOf( + BackendAnalyticsDimension( + dimensionAlias, + fixture.status, + BackendAnalyticsMissingPolicy.EXCLUDE, + ), + ), + ), + listOf(BackendAnalyticsMetric.DocumentCount(countAlias)), + BackendEnforcedFilter(BackendPlannedCondition.All, analyticsMandatory()), + BackendAnalyticsPageWindow(limit, afterKey), + ) + + override fun nullBucketCountPlan(): BackendAnalyticsQueryPlan = fixture.analyticsPlan( + BackendAnalyticsGrouping.By( + listOf( + BackendAnalyticsDimension( + AnalyticsAlias("note"), + fixture.note, + BackendAnalyticsMissingPolicy.AS_NULL_BUCKET, + ), + ), + ), + listOf(BackendAnalyticsMetric.DocumentCount(countAlias)), + BackendEnforcedFilter(BackendPlannedCondition.All, analyticsMandatory()), + BackendAnalyticsPageWindow(10), + ) + + override fun exactNumericMetricPlan(): BackendAnalyticsQueryPlan = fixture.analyticsPlan( + BackendAnalyticsGrouping.Global, + listOf( + BackendAnalyticsMetric.Min(AnalyticsAlias("minimum"), fixture.amount), + BackendAnalyticsMetric.Max(AnalyticsAlias("maximum"), fixture.amount), + BackendAnalyticsMetric.Sum(AnalyticsAlias("total"), fixture.amount), + BackendAnalyticsMetric.Average(AnalyticsAlias("average"), fixture.amount), + ), + BackendEnforcedFilter(BackendPlannedCondition.All, analyticsMandatory()), + numericPolicy = fixture.numericPolicy, + ) + + private fun analyticsMandatory() = BackendPlannedCondition.Junction( + JunctionOperator.AND, + listOf( + fixture.predicate(fixture.tenant, PredicateOperator.EQ, NormalizedValue.Text("tenant-1")), + fixture.predicate(fixture.deleted, PredicateOperator.IS_FALSE), + ), + ) + + override fun portableCountPlan(): BackendCountQueryPlan = fixture.countPlan( + BackendEnforcedFilter(BackendPlannedCondition.All, analyticsMandatory()), + ) + + override fun portableStreamPlan(): BackendStreamQueryPlan = fixture.streamPlan( + BackendEnforcedFilter(BackendPlannedCondition.All, analyticsMandatory()), + ) + + override fun portableSecondPagePlan(): BackendPageQueryPlan = fixture.pagePlan( + BackendEnforcedFilter(BackendPlannedCondition.All, analyticsMandatory()), + 1, + 1, + ) + + @BeforeEach + fun setup() { + fixture = Fixture(MongoNamespace(mongo.databaseName, "order_snapshot")) + fixture.collection.createIndex( + Indexes.text("description"), + IndexOptions().name(TEXT_INDEX), + ).toMono().block() + val documents = listOf( + fixture.document("order-1", "tenant-1", false, "PAID", listOf("priority", "vip"), "paid order", 1), + fixture.document("order-2", "tenant-2", false, "PAID", listOf("priority"), "paid order", 1L), + fixture.document( + "order-3", + "tenant-1", + true, + "PAID", + listOf("priority"), + "paid order", + Decimal128(BigDecimal("1.0")), + ), + fixture.document("order-4", "tenant-1", false, "CREATED", emptyList(), "new order", 2), + ) + documents[0]["note"] = null + documents[2]["note"] = "value" + fixture.collection.insertMany(documents).toMono().block() + } + + @Test + fun `planned backend should enforce mandatory conditions and preserve identity outside projection`() { + val backend = fixture.binding.prepareContribution(fixture.collection).block()!!.backend + val mandatory = BackendPlannedCondition.Junction( + JunctionOperator.AND, + listOf( + fixture.predicate(fixture.tenant, PredicateOperator.EQ, NormalizedValue.Text("tenant-1")), + fixture.predicate(fixture.deleted, PredicateOperator.IS_FALSE), + ), + ) + val user = BackendPlannedCondition.Junction( + JunctionOperator.AND, + listOf( + fixture.predicate(fixture.status, PredicateOperator.EQ, NormalizedValue.Text("PAID")), + fixture.predicate( + fixture.tags, + PredicateOperator.ALL_IN, + NormalizedValue.ListValue(listOf(NormalizedValue.Text("priority"))), + ), + ), + ) + val filter = BackendEnforcedFilter(user, mandatory) + + backend.count(fixture.countPlan(filter), OPTIONS).block().assert().isEqualTo(1) + val records = backend.stream(fixture.streamPlan(filter), OPTIONS).collectList().block()!! + + records.assert().hasSize(1) + records.single().identity.assert().isEqualTo("order-1") + records.single().completeness.assert().isEqualTo(BackendRecordCompleteness.COMPLETE) + records.single().document.values.keys.assert().containsExactly("state") + (records.single().document.values.getValue("state") as NormalizedValue.ObjectValue) + .values.keys.assert().containsExactly("status") + } + + @Test + fun `attested text scope should execute against the real Mongo text index`() { + val contribution = fixture.binding.prepareContribution(fixture.collection).block()!! + val search = BackendPlannedCondition.Search(fixture.searchScope, "paid") + val mandatory = BackendPlannedCondition.Junction( + JunctionOperator.AND, + listOf( + fixture.predicate(fixture.tenant, PredicateOperator.EQ, NormalizedValue.Text("tenant-1")), + fixture.predicate(fixture.deleted, PredicateOperator.IS_FALSE), + ), + ) + + contribution.searchScopes.assert().containsExactly(fixture.searchScope) + contribution.semanticTiers.assert().containsExactly(SemanticTier.PORTABLE, SemanticTier.SEARCH) + contribution.backend.count( + fixture.countPlan(BackendEnforcedFilter(search, mandatory), SemanticTier.SEARCH), + OPTIONS, + ).block().assert().isEqualTo(1) + } + + @Test + fun `planned backend should preserve Mongo null missing and numeric equality baseline`() { + val backend = fixture.binding.prepareContribution(fixture.collection).block()!!.backend + + backend.count( + fixture.countPlan( + BackendEnforcedFilter( + fixture.predicate(fixture.note, PredicateOperator.IS_NULL), + BackendPlannedCondition.All, + ), + ), + OPTIONS, + ).block().assert().isEqualTo(3) + backend.count( + fixture.countPlan( + BackendEnforcedFilter( + fixture.predicate(fixture.note, PredicateOperator.NOT_NULL), + BackendPlannedCondition.All, + ), + ), + OPTIONS, + ).block().assert().isEqualTo(1) + backend.count( + fixture.countPlan( + BackendEnforcedFilter( + fixture.predicate( + fixture.amount, + PredicateOperator.EQ, + NormalizedValue.Decimal(BigDecimal("1.00")), + ), + BackendPlannedCondition.All, + ), + ), + OPTIONS, + ).block().assert().isEqualTo(3) + } + + @Test + fun `planned page should return exact records and total from one input stream`() { + val backend = fixture.binding.prepareContribution(fixture.collection).block()!!.backend + val mandatory = BackendPlannedCondition.Junction( + JunctionOperator.AND, + listOf( + fixture.predicate(fixture.tenant, PredicateOperator.EQ, NormalizedValue.Text("tenant-1")), + fixture.predicate(fixture.deleted, PredicateOperator.IS_FALSE), + ), + ) + + val page = backend.page( + fixture.pagePlan(BackendEnforcedFilter(BackendPlannedCondition.All, mandatory), offset = 1, size = 1), + QueryBackendExecutionOptions( + deadline = Instant.now().plusSeconds(300), + maxReturnedRecords = 1, + maxPageWindow = 2, + allowDiskUse = true, + ), + ).block()!! + + page.total.assert().isEqualTo(2) + page.totalRelation.assert().isEqualTo(BackendTotalRelation.EXACT) + page.consistency.assert().isEqualTo(BackendPageConsistency.SAME_INPUT) + page.records.assert().hasSize(1) + page.records.single().identity.assert().isEqualTo("order-4") + } + + @Test + fun `planned page should emit an exact total for empty and out of range pages`() { + val backend = fixture.binding.prepareContribution(fixture.collection).block()!!.backend + val noMatch = fixture.predicate( + fixture.tenant, + PredicateOperator.EQ, + NormalizedValue.Text("tenant-missing"), + ) + val empty = backend.page( + fixture.pagePlan( + BackendEnforcedFilter(noMatch, BackendPlannedCondition.All), + offset = 0, + size = 1, + ), + OPTIONS, + ).block()!! + val outOfRange = backend.page( + fixture.pagePlan( + BackendEnforcedFilter(BackendPlannedCondition.All, analyticsMandatory()), + offset = 20, + size = 1, + ), + OPTIONS, + ).block()!! + + empty.total.assert().isEqualTo(0) + empty.records.assert().isEmpty() + outOfRange.total.assert().isEqualTo(2) + outOfRange.records.assert().isEmpty() + } + + @Test + fun `planned page should not pack large records into one BSON document`() { + fixture.collection.deleteMany(Document()).toMono().block() + val largeStatus = "x".repeat(8 * 1024 * 1024) + fixture.collection.insertMany( + listOf( + fixture.document("large-1", "tenant-1", false, largeStatus, emptyList(), "large", 1), + fixture.document("large-2", "tenant-1", false, largeStatus, emptyList(), "large", 2), + ), + ).toMono().block() + val backend = fixture.binding.prepareContribution(fixture.collection).block()!!.backend + + val page = backend.page( + fixture.pagePlan( + BackendEnforcedFilter(BackendPlannedCondition.All, analyticsMandatory()), + offset = 0, + size = 2, + ), + QueryBackendExecutionOptions( + deadline = Instant.now().plusSeconds(300), + maxReturnedRecords = 2, + maxPageWindow = 2, + allowDiskUse = true, + ), + ).block()!! + + page.total.assert().isEqualTo(2) + page.records.map { record -> record.identity }.assert().containsExactly("large-1", "large-2") + } + + @Test + fun `planned analytics should enforce mandatory filter and compute exact global metrics`() { + val contribution = fixture.binding.prepareContribution(fixture.collection).block()!! + val backend = requireNotNull(contribution.analyticsBackend) + contribution.supportedOperations.assert().contains(QueryOperation.ANALYZE) + val mandatory = BackendPlannedCondition.Junction( + JunctionOperator.AND, + listOf( + fixture.predicate(fixture.tenant, PredicateOperator.EQ, NormalizedValue.Text("tenant-1")), + fixture.predicate(fixture.deleted, PredicateOperator.IS_FALSE), + ), + ) + + val page = backend.analyze( + fixture.analyticsPlan( + BackendAnalyticsGrouping.Global, + listOf( + BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count")), + BackendAnalyticsMetric.Min(AnalyticsAlias("minimum"), fixture.amount), + BackendAnalyticsMetric.Max(AnalyticsAlias("maximum"), fixture.amount), + BackendAnalyticsMetric.Sum(AnalyticsAlias("total"), fixture.amount), + BackendAnalyticsMetric.Average(AnalyticsAlias("average"), fixture.amount), + ), + BackendEnforcedFilter(BackendPlannedCondition.All, mandatory), + numericPolicy = fixture.numericPolicy, + ), + fixture.analyticsOptions(), + ).block()!! + + page.consistency.assert().isEqualTo(BackendAnalyticsConsistency.EVENTUAL) + page.completeness.assert().isEqualTo(BackendAnalyticsCompleteness.EXACT) + page.afterKey.assert().isNull() + page.buckets.assert().hasSize(1) + page.buckets.single().keys.assert().isEmpty() + page.buckets.single().metrics.assert().containsEntry(AnalyticsAlias("count"), NormalizedValue.Int64(2)) + page.buckets.single().metrics.assert() + .containsEntry(AnalyticsAlias("minimum"), NormalizedValue.Decimal(BigDecimal.ONE)) + page.buckets.single().metrics.assert() + .containsEntry(AnalyticsAlias("maximum"), NormalizedValue.Decimal(BigDecimal("2"))) + page.buckets.single().metrics.assert() + .containsEntry(AnalyticsAlias("total"), NormalizedValue.Decimal(BigDecimal("3"))) + page.buckets.single().metrics.assert() + .containsEntry(AnalyticsAlias("average"), NormalizedValue.Decimal(BigDecimal("1.5"))) + + val empty = backend.analyze( + fixture.analyticsPlan( + BackendAnalyticsGrouping.Global, + listOf( + BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count")), + BackendAnalyticsMetric.Sum(AnalyticsAlias("total"), fixture.amount), + BackendAnalyticsMetric.Average(AnalyticsAlias("average"), fixture.amount), + ), + BackendEnforcedFilter( + fixture.predicate( + fixture.tenant, + PredicateOperator.EQ, + NormalizedValue.Text("tenant-without-documents"), + ), + BackendPlannedCondition.All, + ), + numericPolicy = fixture.numericPolicy, + ), + fixture.analyticsOptions(), + ).block()!! + empty.buckets.assert().hasSize(1) + empty.buckets.single().metrics.assert().containsEntry(AnalyticsAlias("count"), NormalizedValue.Int64(0)) + empty.buckets.single().metrics.assert() + .containsEntry(AnalyticsAlias("total"), NormalizedValue.Decimal(BigDecimal.ZERO)) + empty.buckets.single().metrics.assert() + .containsEntry(AnalyticsAlias("average"), NormalizedValue.Null) + } + + @Test + fun `planned analytics should page stable dimension keys and preserve missing null baseline`() { + val backend = requireNotNull(fixture.binding.prepareContribution(fixture.collection).block()!!.analyticsBackend) + val mandatory = BackendPlannedCondition.Junction( + JunctionOperator.AND, + listOf( + fixture.predicate(fixture.tenant, PredicateOperator.EQ, NormalizedValue.Text("tenant-1")), + fixture.predicate(fixture.deleted, PredicateOperator.IS_FALSE), + ), + ) + val filter = BackendEnforcedFilter(BackendPlannedCondition.All, mandatory) + val grouping = BackendAnalyticsGrouping.By( + listOf( + BackendAnalyticsDimension( + AnalyticsAlias("status"), + fixture.status, + BackendAnalyticsMissingPolicy.EXCLUDE, + ), + ), + ) + val metrics = listOf(BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count"))) + + val first = backend.analyze( + fixture.analyticsPlan(grouping, metrics, filter, BackendAnalyticsPageWindow(1)), + fixture.analyticsOptions(), + ).block()!! + first.buckets.single().keys.assert() + .containsEntry(AnalyticsAlias("status"), NormalizedValue.Text("CREATED")) + first.afterKey!!.assert().containsExactly(NormalizedValue.Text("CREATED")) + + val second = backend.analyze( + fixture.analyticsPlan( + grouping, + metrics, + filter, + BackendAnalyticsPageWindow(1, first.afterKey), + ), + fixture.analyticsOptions(), + ).block()!! + second.buckets.single().keys.assert() + .containsEntry(AnalyticsAlias("status"), NormalizedValue.Text("PAID")) + second.afterKey.assert().isNull() + + val nullBucket = backend.analyze( + fixture.analyticsPlan( + BackendAnalyticsGrouping.By( + listOf( + BackendAnalyticsDimension( + AnalyticsAlias("note"), + fixture.note, + BackendAnalyticsMissingPolicy.AS_NULL_BUCKET, + ), + ), + ), + metrics, + filter, + BackendAnalyticsPageWindow(10), + ), + fixture.analyticsOptions(), + ).block()!! + nullBucket.buckets.assert().hasSize(1) + nullBucket.buckets.single().keys.assert().containsEntry(AnalyticsAlias("note"), NormalizedValue.Null) + nullBucket.buckets.single().metrics.assert() + .containsEntry(AnalyticsAlias("count"), NormalizedValue.Int64(2)) + + val excludedMissingAndNull = backend.analyze( + fixture.analyticsPlan( + BackendAnalyticsGrouping.By( + listOf( + BackendAnalyticsDimension( + AnalyticsAlias("note"), + fixture.note, + BackendAnalyticsMissingPolicy.EXCLUDE, + ), + ), + ), + metrics, + filter, + BackendAnalyticsPageWindow(10), + ), + fixture.analyticsOptions(), + ).block()!! + excludedMissingAndNull.buckets.assert().isEmpty() + excludedMissingAndNull.afterKey.assert().isNull() + } + + @Test + fun `planned analytics should replay every high cardinality bucket without gaps or duplicates`() { + val tenantId = "tenant-high-cardinality" + val statuses = (0 until 257).map { index -> "HC-${index.toString().padStart(4, '0')}" } + fixture.collection.insertMany( + statuses.mapIndexed { index, status -> + fixture.document( + id = "high-cardinality-$index", + tenantId = tenantId, + deleted = false, + status = status, + tags = emptyList(), + description = "high cardinality", + amount = index, + ) + }, + ).toMono().block() + val backend = requireNotNull(fixture.binding.prepareContribution(fixture.collection).block()!!.analyticsBackend) + val grouping = BackendAnalyticsGrouping.By( + listOf( + BackendAnalyticsDimension( + dimensionAlias, + fixture.status, + BackendAnalyticsMissingPolicy.EXCLUDE, + ), + ), + ) + val filter = BackendEnforcedFilter( + fixture.predicate(fixture.tenant, PredicateOperator.EQ, NormalizedValue.Text(tenantId)), + fixture.predicate(fixture.deleted, PredicateOperator.IS_FALSE), + ) + val observed = mutableListOf() + var afterKey: List? = null + var pages = 0 + do { + val page = backend.analyze( + fixture.analyticsPlan( + grouping, + listOf(BackendAnalyticsMetric.DocumentCount(countAlias)), + filter, + BackendAnalyticsPageWindow(31, afterKey), + ), + fixture.analyticsOptions().copy(maxReturnedBuckets = 31), + ).block()!! + observed += page.buckets.map { bucket -> + (bucket.keys.getValue(dimensionAlias) as NormalizedValue.Text).value + } + afterKey = page.afterKey + pages++ + check(pages <= 10) { "High-cardinality cursor did not terminate within the expected page bound." } + } while (afterKey != null) + + observed.assert().containsExactly(*statuses.toTypedArray()) + observed.toSet().assert().hasSize(statuses.size) + } + + @Test + fun `planned analytics eventual cursor should observe a later concurrent bucket without claiming snapshot`() { + val tenantId = "tenant-eventual" + fixture.collection.insertMany( + listOf( + fixture.document("eventual-a", tenantId, false, "A", emptyList(), "eventual", 1), + fixture.document("eventual-c", tenantId, false, "C", emptyList(), "eventual", 1), + ), + ).toMono().block() + val backend = requireNotNull(fixture.binding.prepareContribution(fixture.collection).block()!!.analyticsBackend) + val grouping = BackendAnalyticsGrouping.By( + listOf(BackendAnalyticsDimension(dimensionAlias, fixture.status, BackendAnalyticsMissingPolicy.EXCLUDE)), + ) + val filter = BackendEnforcedFilter( + fixture.predicate(fixture.tenant, PredicateOperator.EQ, NormalizedValue.Text(tenantId)), + fixture.predicate(fixture.deleted, PredicateOperator.IS_FALSE), + ) + val metrics = listOf(BackendAnalyticsMetric.DocumentCount(countAlias)) + + val first = backend.analyze( + fixture.analyticsPlan(grouping, metrics, filter, BackendAnalyticsPageWindow(1)), + fixture.analyticsOptions(), + ).block()!! + first.consistency.assert().isEqualTo(BackendAnalyticsConsistency.EVENTUAL) + first.buckets.single().keys.getValue(dimensionAlias).assert().isEqualTo(NormalizedValue.Text("A")) + + fixture.collection.insertOne( + fixture.document("eventual-b", tenantId, false, "B", emptyList(), "eventual", 1), + ).toMono().block() + + val second = backend.analyze( + fixture.analyticsPlan(grouping, metrics, filter, BackendAnalyticsPageWindow(1, first.afterKey)), + fixture.analyticsOptions(), + ).block()!! + second.consistency.assert().isEqualTo(BackendAnalyticsConsistency.EVENTUAL) + second.buckets.single().keys.getValue(dimensionAlias).assert().isEqualTo(NormalizedValue.Text("B")) + } + + @Test + fun `planned analytics should reject real Decimal128 aggregation overflow`() { + val tenantId = "tenant-overflow" + val maximum = Decimal128.parse("9.999999999999999999999999999999999E+6144") + fixture.collection.insertMany( + listOf( + fixture.document("overflow-1", tenantId, false, "OVERFLOW", emptyList(), "overflow", maximum), + fixture.document("overflow-2", tenantId, false, "OVERFLOW", emptyList(), "overflow", maximum), + ), + ).toMono().block() + val backend = requireNotNull(fixture.binding.prepareContribution(fixture.collection).block()!!.analyticsBackend) + val filter = BackendEnforcedFilter( + fixture.predicate(fixture.tenant, PredicateOperator.EQ, NormalizedValue.Text(tenantId)), + fixture.predicate(fixture.deleted, PredicateOperator.IS_FALSE), + ) + val policy = fixture.numericPolicy.copy(scale = 0) + + assertThrownBy { + backend.analyze( + fixture.analyticsPlan( + BackendAnalyticsGrouping.Global, + listOf(BackendAnalyticsMetric.Sum(AnalyticsAlias("total"), fixture.amount)), + filter, + numericPolicy = policy, + ), + fixture.analyticsOptions(), + ).block() + }.satisfies( + Consumer { error -> error.kind.assert().isEqualTo(QueryBackendFailureKind.MAPPING_FAILURE) }, + ) + } + + @Test + fun `planned page explain should use the declared tenant deletion identity index`() { + val explainFixture = Fixture(MongoNamespace(mongo.databaseName, "order_snapshot_explain")) + explainFixture.collection.createIndex( + Indexes.compoundIndex( + Indexes.ascending(MessageRecords.TENANT_ID), + Indexes.ascending(StateAggregateRecords.DELETED), + Indexes.ascending(Documents.ID_FIELD), + ), + IndexOptions().name(PAGE_INDEX), + ).toMono().block() + explainFixture.collection.insertMany( + (0 until 2_000).map { index -> + explainFixture.document( + id = "order-$index", + tenantId = if (index % 10 == 0) "tenant-1" else "tenant-other", + deleted = false, + status = "PAID", + tags = emptyList(), + description = "order", + amount = index, + ) + }, + ).toMono().block() + val mandatory = BackendPlannedCondition.Junction( + JunctionOperator.AND, + listOf( + explainFixture.predicate( + explainFixture.tenant, + PredicateOperator.EQ, + NormalizedValue.Text("tenant-1"), + ), + explainFixture.predicate(explainFixture.deleted, PredicateOperator.IS_FALSE), + ), + ) + val plan = explainFixture.pagePlan( + BackendEnforcedFilter(BackendPlannedCondition.All, mandatory), + offset = 20, + size = 10, + ) + val query = MongoRecordQueryCompiler(explainFixture.binding).compile(plan) + + val explain = Mono.from( + explainFixture.collection.aggregate(query.pagePipeline()) + .collation(Collation.builder().locale("simple").build()) + .allowDiskUse(false) + .explain(ExplainVerbosity.EXECUTION_STATS), + ).block()!! + val rendered = explain.toJson() + + rendered.assert().contains("IXSCAN", "executionStats") + rendered.contains("COLLSCAN").assert().isFalse() + } + + @Test + fun `planned high cardinality analytics explain should use the declared tenant deletion group index`() { + val explainFixture = Fixture(MongoNamespace(mongo.databaseName, "order_snapshot_analytics_explain")) + explainFixture.collection.createIndex( + Indexes.compoundIndex( + Indexes.ascending(MessageRecords.TENANT_ID), + Indexes.ascending(StateAggregateRecords.DELETED), + Indexes.ascending("state.status"), + ), + IndexOptions().name(ANALYTICS_INDEX), + ).toMono().block() + explainFixture.collection.insertMany( + (0 until 2_000).map { index -> + explainFixture.document( + id = "analytics-order-$index", + tenantId = if (index % 10 == 0) "tenant-analytics" else "tenant-other", + deleted = false, + status = "STATUS-${index.toString().padStart(4, '0')}", + tags = emptyList(), + description = "analytics explain", + amount = index, + ) + }, + ).toMono().block() + val filter = BackendEnforcedFilter( + explainFixture.predicate( + explainFixture.tenant, + PredicateOperator.EQ, + NormalizedValue.Text("tenant-analytics"), + ), + explainFixture.predicate(explainFixture.deleted, PredicateOperator.IS_FALSE), + ) + val plan = explainFixture.analyticsPlan( + BackendAnalyticsGrouping.By( + listOf( + BackendAnalyticsDimension( + AnalyticsAlias("status"), + explainFixture.status, + BackendAnalyticsMissingPolicy.EXCLUDE, + ), + ), + ), + listOf(BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + filter, + BackendAnalyticsPageWindow(100), + ) + val query = MongoAnalyticsQueryCompiler(explainFixture.binding.prepared).compile(plan) + + val explain = Mono.from( + explainFixture.collection.aggregate(query.pipeline) + .collation(Collation.builder().locale("simple").build()) + .allowDiskUse(true) + .explain(ExplainVerbosity.EXECUTION_STATS), + ).block()!! + val rendered = explain.toJson() + + rendered.assert().contains("IXSCAN", ANALYTICS_INDEX, "executionStats") + rendered.contains("COLLSCAN").assert().isFalse() + } + + @Test + fun `gateway shadow should compare legacy and planned Mongo against the same collection`() { + val contribution = fixture.binding.prepareContribution(fixture.collection).block()!! + val observation = arrayOfNulls(1) + val observed = CountDownLatch(1) + val raw = LegacyMongoCountQueryService(fixture.target.namedAggregate, fixture.collection) + val gateway = QueryGatewayRuntime.create( + namedAggregates = listOf(fixture.target.namedAggregate), + backendComposition = QueryBackendComposition( + listOf(contribution), + mapOf(fixture.target to contribution.backendId), + ), + rawServiceSource = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate): SnapshotQueryService<*> = raw + + override fun eventStream(namedAggregate: NamedAggregate) = + NoOpEventStreamQueryServiceFactory.create(namedAggregate) + }, + dialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.DOCUMENT) + }, + authorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.System("integration-test", "mongo-shadow")) + }, + executionProfiles = QueryExecutionProfiles( + operationProfiles = mapOf( + QueryOperationProfileKey(fixture.target, QueryOperation.COUNT) to + QueryExecutionProfile(QueryExecutionMode.SHADOW, QueryValidationMode.STRICT), + ), + ), + shadowObserver = QueryShadowObserver { current -> + observation[0] = current + observed.countDown() + }, + runtimeHealthObserver = QueryRuntimeHealthObserver { }, + ).gateway + + gateway.count(QueryCall(fixture.target, QueryPurpose("mongo-shadow-test")), Condition.ALL) + .block().assert().isEqualTo(3) + + observed.await(5, TimeUnit.SECONDS).assert().isTrue() + observation.single()!!.outcome.assert().isEqualTo(QueryShadowOutcome.MATCH) + } + + @Test + fun `gateway planned page should execute one exact Mongo input stream`() { + val contribution = fixture.binding.prepareContribution(fixture.collection).block()!! + val raw = LegacyMongoCountQueryService(fixture.target.namedAggregate, fixture.collection) + val gateway = QueryGatewayRuntime.create( + namedAggregates = listOf(fixture.target.namedAggregate), + backendComposition = QueryBackendComposition( + listOf(contribution), + mapOf(fixture.target to contribution.backendId), + ), + rawServiceSource = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate): SnapshotQueryService<*> = raw + + override fun eventStream(namedAggregate: NamedAggregate) = + NoOpEventStreamQueryServiceFactory.create(namedAggregate) + }, + dialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.DOCUMENT) + }, + authorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.System("integration-test", "mongo-planned-page")) + }, + executionProfiles = QueryExecutionProfiles( + operationProfiles = mapOf( + QueryOperationProfileKey(fixture.target, QueryOperation.PAGE) to + QueryExecutionProfile(QueryExecutionMode.PLANNED, QueryValidationMode.STRICT), + ), + ), + ).gateway + + val page = gateway.page( + QueryCall(fixture.target, QueryPurpose("mongo-planned-page-test")), + PagedQuery(Condition.ALL, pagination = Pagination(2, 1)), + ).block()!! + + page.total.assert().isEqualTo(3) + page.list.assert().hasSize(1) + page.list.single()[MessageRecords.AGGREGATE_ID].assert().isEqualTo("order-2") + } + + @Test + fun `gateway Mongo rehearsal should shadow cut over and roll back against one collection`() { + val contribution = fixture.binding.prepareContribution(fixture.collection).block()!! + val raw = LegacyMongoCountQueryService(fixture.target.namedAggregate, fixture.collection) + val observation = arrayOfNulls(1) + val observed = CountDownLatch(1) + val call = QueryCall(fixture.target, QueryPurpose("mongo-rollout-rehearsal")) + + fun gateway( + mode: QueryExecutionMode, + shadowObserver: QueryShadowObserver = QueryShadowObserver.NONE, + ) = QueryGatewayRuntime.create( + namedAggregates = listOf(fixture.target.namedAggregate), + backendComposition = QueryBackendComposition( + listOf(contribution), + mapOf(fixture.target to contribution.backendId), + ), + rawServiceSource = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate): SnapshotQueryService<*> = raw + + override fun eventStream(namedAggregate: NamedAggregate) = + NoOpEventStreamQueryServiceFactory.create(namedAggregate) + }, + dialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.DOCUMENT) + }, + authorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.System("integration-test", "mongo-rollout-rehearsal")) + }, + executionProfiles = QueryExecutionProfiles( + operationProfiles = mapOf( + QueryOperationProfileKey(fixture.target, QueryOperation.COUNT) to + QueryExecutionProfile(mode, QueryValidationMode.STRICT), + ), + ), + shadowObserver = shadowObserver, + runtimeHealthObserver = QueryRuntimeHealthObserver { }, + ).gateway + + gateway( + QueryExecutionMode.SHADOW, + QueryShadowObserver { current -> + observation[0] = current + observed.countDown() + }, + ).count(call, Condition.ALL).block().assert().isEqualTo(3) + observed.await(5, TimeUnit.SECONDS).assert().isTrue() + observation.single()!!.outcome.assert().isEqualTo(QueryShadowOutcome.MATCH) + raw.countInvocations.get().assert().isEqualTo(1) + + gateway(QueryExecutionMode.PLANNED) + .count(call, Condition.ALL).block().assert().isEqualTo(3) + raw.countInvocations.get().assert().isEqualTo(1) + + gateway(QueryExecutionMode.LEGACY) + .count(call, Condition.ALL).block().assert().isEqualTo(3) + raw.countInvocations.get().assert().isEqualTo(2) + } + + private inner class Fixture(val namespace: MongoNamespace) { + val collection = mongo.database().getCollection(namespace.collectionName) + val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + val tenant = QueryFieldId.System(SystemFieldKind.TENANT_ID) + val deleted = QueryFieldId.System(SystemFieldKind.DELETED) + val state = QueryFieldId.Path(listOf("state")) + val status = QueryFieldId.Path(listOf("state", "status")) + val tags = QueryFieldId.Path(listOf("state", "tags")) + val note = QueryFieldId.Path(listOf("note")) + val amount = QueryFieldId.Path(listOf("amount")) + private val description = QueryFieldId.Path(listOf("description")) + val searchScope = SearchScopeId("document-text") + val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val schema = QueryDocumentSchema( + target, + listOf( + field(identity, LogicalFieldType.Text, setOf(PredicateOperator.EQ), EXACT_SORT_PROJECT), + field(tenant, LogicalFieldType.Text, setOf(PredicateOperator.EQ), setOf(FieldCapability.EXACT)), + field(deleted, LogicalFieldType.Boolean, setOf(PredicateOperator.IS_FALSE), setOf(FieldCapability.EXACT)), + field(state, LogicalFieldType.Object), + field( + status, + LogicalFieldType.Text, + setOf(PredicateOperator.EQ), + setOf( + FieldCapability.EXACT, + FieldCapability.PROJECTABLE, + FieldCapability.AGGREGATABLE, + ), + ), + field( + tags, + LogicalFieldType.Array( + LogicalFieldType.Text, + Nullability.NON_NULL, + EmptyArraySemantics.DISTINCT, + ), + setOf(PredicateOperator.ALL_IN), + setOf(FieldCapability.EXACT), + ), + field(description, LogicalFieldType.Text, capabilities = setOf(FieldCapability.FULL_TEXT)), + field( + note, + LogicalFieldType.Text, + setOf(PredicateOperator.IS_NULL, PredicateOperator.NOT_NULL), + setOf(FieldCapability.PRESENCE, FieldCapability.AGGREGATABLE), + ), + field( + amount, + LogicalFieldType.Decimal, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT, FieldCapability.AGGREGATABLE), + ), + ), + listOf(QuerySearchScopeDefinition(searchScope, null, listOf(description), listOf(description))), + ) + val binding = MongoSnapshotQueryBinding( + schema, + namespace, + linkedMapOf( + identity to MongoFieldBinding(Documents.ID_FIELD, EXACT_SORT_PROJECT), + tenant to MongoFieldBinding(MessageRecords.TENANT_ID, setOf(FieldCapability.EXACT)), + deleted to MongoFieldBinding(StateAggregateRecords.DELETED, setOf(FieldCapability.EXACT)), + state to MongoFieldBinding("state", emptySet()), + status to MongoFieldBinding( + "state.status", + setOf( + FieldCapability.EXACT, + FieldCapability.PROJECTABLE, + FieldCapability.AGGREGATABLE, + ), + ), + tags to MongoFieldBinding("state.tags", setOf(FieldCapability.EXACT)), + description to MongoFieldBinding("description", setOf(FieldCapability.FULL_TEXT)), + note to MongoFieldBinding( + "note", + setOf(FieldCapability.PRESENCE, FieldCapability.AGGREGATABLE), + ), + amount to MongoFieldBinding( + "amount", + setOf(FieldCapability.EXACT, FieldCapability.AGGREGATABLE), + MongoValueEncoding.DECIMAL128, + ), + ), + textSearch = MongoTextSearchBinding(searchScope, TEXT_INDEX), + ) + + fun countPlan( + filter: BackendEnforcedFilter, + tier: SemanticTier = SemanticTier.PORTABLE, + ) = BackendCountQueryPlan( + target, + schema.contractId, + filter, + BackendRequiredCapabilities(), + tier, + PlanFingerprint("1".repeat(64)), + ) + + fun streamPlan(filter: BackendEnforcedFilter) = BackendStreamQueryPlan( + target, + schema.contractId, + filter, + RecordResultShape.DYNAMIC, + BackendProjection.Include(listOf(status)), + listOf(BackendSort(identity, NormalizedSortDirection.ASC, BackendSortOrigin.STABILITY_TIE_BREAKER)), + 10, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("2".repeat(64)), + ) + + fun pagePlan(filter: BackendEnforcedFilter, offset: Long, size: Int) = BackendPageQueryPlan( + target, + schema.contractId, + filter, + RecordResultShape.DYNAMIC, + BackendProjection.Include(listOf(status)), + listOf(BackendSort(identity, NormalizedSortDirection.ASC, BackendSortOrigin.STABILITY_TIE_BREAKER)), + BackendPageWindow(offset, size), + BackendTotalMode.EXACT, + BackendRequiredConsistency.SAME_INPUT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("4".repeat(64)), + ) + + val numericPolicy = BackendAnalyticsNumericPolicy( + BackendAnalyticsNumericPromotion.DECIMAL128, + 34, + 4, + RoundingMode.HALF_UP, + BackendAnalyticsOverflowPolicy.REJECT, + ) + + fun analyticsPlan( + grouping: BackendAnalyticsGrouping, + metrics: List, + filter: BackendEnforcedFilter, + window: BackendAnalyticsPageWindow = BackendAnalyticsPageWindow(1), + numericPolicy: BackendAnalyticsNumericPolicy? = null, + ) = BackendAnalyticsQueryPlan( + target, + schema.contractId, + filter, + grouping, + metrics, + BackendAnalyticsCondition.All, + when (grouping) { + BackendAnalyticsGrouping.Global -> BackendAnalyticsBucketOrder.Global + is BackendAnalyticsGrouping.By -> BackendAnalyticsBucketOrder.DimensionKeyAscending( + BackendAnalyticsNullPlacement.FIRST, + BackendAnalyticsTextCollation.BINARY, + ) + }, + window, + numericPolicy, + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("7".repeat(64)), + ) + + fun analyticsOptions() = QueryBackendExecutionOptions( + deadline = Instant.now().plusSeconds(300), + maxReturnedRecords = null, + maxReturnedBuckets = 10, + allowDiskUse = true, + ) + + fun predicate( + field: QueryFieldId, + operator: PredicateOperator, + value: NormalizedValue? = null, + ) = BackendPlannedCondition.Predicate(field, operator, value) + + fun document( + id: String, + tenantId: String, + deleted: Boolean, + status: String, + tags: List, + description: String, + amount: Any, + ): Document = Document(Documents.ID_FIELD, id) + .append(MessageRecords.TENANT_ID, tenantId) + .append(StateAggregateRecords.DELETED, deleted) + .append("state", Document("status", status).append("tags", tags)) + .append("description", description) + .append("amount", amount) + + private fun field( + id: QueryFieldId, + type: LogicalFieldType, + operators: Set = emptySet(), + capabilities: Set = emptySet(), + ) = QueryFieldSchema(id, type, Presence.OPTIONAL, Nullability.NULLABLE, operators, capabilities) + } + + private class LegacyMongoCountQueryService( + override val namedAggregate: NamedAggregate, + private val collection: com.mongodb.reactivestreams.client.MongoCollection, + ) : SnapshotQueryService { + override val name: String = "legacy-mongo-integration-test" + val countInvocations = AtomicInteger() + + override fun single(singleQuery: ISingleQuery) = Mono.empty>() + + override fun dynamicSingle(singleQuery: ISingleQuery): Mono = Mono.empty() + + override fun list(listQuery: IListQuery) = Flux.empty>() + + override fun dynamicList(listQuery: IListQuery): Flux = Flux.empty() + + override fun paged(pagedQuery: IPagedQuery) = + Mono.just(PagedList.empty>()) + + override fun dynamicPaged(pagedQuery: IPagedQuery) = Mono.just(PagedList.empty()) + + override fun count(condition: Condition): Mono = Mono.defer { + countInvocations.incrementAndGet() + Mono.from(collection.countDocuments(SnapshotConditionConverter.convert(condition))) + } + } + + private companion object { + const val ANALYTICS_INDEX = "tenant_deleted_status" + const val PAGE_INDEX = "tenant_deleted_identity" + const val TEXT_INDEX = "description_text" + val OPTIONS = QueryBackendExecutionOptions(null, null) + val EXACT_SORT_PROJECT = setOf( + FieldCapability.EXACT, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + ) + } +} diff --git a/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/cursor/MongoQueryCursorLeaseStore.kt b/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/cursor/MongoQueryCursorLeaseStore.kt new file mode 100644 index 00000000000..b0fd8c53548 --- /dev/null +++ b/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/cursor/MongoQueryCursorLeaseStore.kt @@ -0,0 +1,299 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class) + +package me.ahoo.wow.mongo.query.cursor + +import com.mongodb.ErrorCategory +import com.mongodb.MongoWriteException +import com.mongodb.client.model.Filters +import com.mongodb.client.model.IndexOptions +import com.mongodb.client.model.Indexes +import com.mongodb.client.model.Projections +import com.mongodb.client.model.Sorts +import com.mongodb.reactivestreams.client.MongoCollection +import com.mongodb.reactivestreams.client.MongoDatabase +import me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi +import me.ahoo.wow.query.cursor.QueryCursorLeaseCreateResult +import me.ahoo.wow.query.cursor.QueryCursorLeaseEntry +import me.ahoo.wow.query.cursor.QueryCursorLeaseId +import me.ahoo.wow.query.cursor.QueryCursorLeaseStore +import me.ahoo.wow.query.cursor.QueryCursorPayloadFormat +import me.ahoo.wow.query.cursor.QueryCursorStoreRevision +import me.ahoo.wow.query.cursor.StoredQueryCursorLease +import org.bson.Document +import org.bson.types.Binary +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.kotlin.core.publisher.toFlux +import reactor.kotlin.core.publisher.toMono +import java.nio.ByteBuffer +import java.nio.charset.StandardCharsets +import java.security.MessageDigest +import java.time.Clock +import java.time.DateTimeException +import java.time.Duration +import java.time.Instant +import java.util.Date +import java.util.UUID +import java.util.concurrent.TimeUnit + +/** Configuration for the bounded MongoDB cursor-lease namespace. */ +@ExperimentalQueryCursorApi +data class MongoQueryCursorLeaseStoreOptions( + val collectionName: String = DEFAULT_COLLECTION_NAME, + val maxEntries: Int = DEFAULT_MAX_ENTRIES, + val retentionGrace: Duration = DEFAULT_RETENTION_GRACE, + val maxPayloadBytes: Int = DEFAULT_MAX_PAYLOAD_BYTES, + val maxScanSize: Int = DEFAULT_MAX_SCAN_SIZE, +) { + init { + require(COLLECTION_NAME.matches(collectionName)) { "Mongo cursor lease collection name is invalid." } + require(maxEntries in 1..MAX_ENTRIES) { "Mongo cursor lease capacity is outside its supported range." } + require(!retentionGrace.isZero && !retentionGrace.isNegative) { + "Mongo cursor lease retention grace must be positive." + } + require(retentionGrace <= MAX_RETENTION_GRACE) { + "Mongo cursor lease retention grace exceeds its supported maximum." + } + require(maxPayloadBytes in 1..MAX_PAYLOAD_BYTES) { + "Mongo cursor lease payload limit is outside its supported range." + } + require(maxScanSize in 1..MAX_SCAN_SIZE) { + "Mongo cursor lease scan limit is outside its supported range." + } + } + + companion object { + const val DEFAULT_COLLECTION_NAME = "wow_query_cursor_lease_v1" + const val DEFAULT_MAX_ENTRIES = 65_536 + const val DEFAULT_MAX_PAYLOAD_BYTES = 64 * 1024 + const val DEFAULT_MAX_SCAN_SIZE = 512 + const val MAX_ENTRIES = 1_000_000 + const val MAX_PAYLOAD_BYTES = 1024 * 1024 + const val MAX_SCAN_SIZE = 4096 + val DEFAULT_RETENTION_GRACE: Duration = Duration.ofMinutes(5) + val MAX_RETENTION_GRACE: Duration = Duration.ofDays(1) + private val COLLECTION_NAME = Regex("[A-Za-z0-9][A-Za-z0-9_-]{0,119}") + } +} + +/** + * Cross-node cursor lease store backed by a bounded set of MongoDB slots. + * + * [ensureIndexes] is deliberately explicit. Applications must run it as a controlled schema/readiness operation + * before supplying this store to Query Gateway. The TTL index uses a grace deadline so the framework reaper has an + * opportunity to atomically acquire an expired lease and close its Backend state before MongoDB removes abandoned + * documents as a final safety net. + */ +@ExperimentalQueryCursorApi +class MongoQueryCursorLeaseStore( + database: MongoDatabase, + val options: MongoQueryCursorLeaseStoreOptions = MongoQueryCursorLeaseStoreOptions(), + private val clock: Clock = Clock.systemUTC(), +) : QueryCursorLeaseStore { + private val collection: MongoCollection = database.getCollection(options.collectionName) + + fun ensureIndexes(): Mono = Flux.concat( + collection.createIndex( + Indexes.ascending(LEASE_ID_FIELD), + IndexOptions().name(LEASE_ID_INDEX).unique(true), + ).toMono(), + collection.createIndex( + Indexes.ascending(PURGE_AT_FIELD), + IndexOptions().name(PURGE_AT_INDEX).expireAfter(0, TimeUnit.SECONDS), + ).toMono(), + ).then() + + override fun create(entry: QueryCursorLeaseEntry): Mono = Mono.defer { + require(entry.expiresAt.isAfter(clock.instant())) { "Mongo cursor lease expiry must be in the future." } + require(entry.payload().size <= options.maxPayloadBytes) { + "Mongo cursor lease payload exceeds its configured limit." + } + insert(entry, initialSlot(entry.id), attempt = 0) + } + + override fun load(id: QueryCursorLeaseId): Mono = Mono.defer { + collection.find(Filters.eq(LEASE_ID_FIELD, id.value)) + .first() + .toMono() + .map { document -> document.toStoredLease(id) } + } + + override fun compareAndDelete(expected: StoredQueryCursorLease): Mono = Mono.defer { + collection.deleteOne( + Filters.and( + Filters.eq(LEASE_ID_FIELD, expected.entry.id.value), + Filters.eq(REVISION_FIELD, expected.revision.value), + ), + ).toMono().map { result -> result.deletedCount == 1L } + } + + override fun scanExpired( + before: Instant, + afterId: QueryCursorLeaseId?, + limit: Int, + ): Flux { + require(limit in 1..options.maxScanSize) { "Mongo cursor lease scan exceeds its configured bound." } + val filter = afterId?.let { cursor -> + Filters.and( + Filters.lte(EXPIRES_AT_FIELD, Date.from(before)), + Filters.gt(LEASE_ID_FIELD, cursor.value), + ) + } ?: Filters.lte(EXPIRES_AT_FIELD, Date.from(before)) + return Flux.defer { + collection.find(filter) + .sort(Sorts.ascending(LEASE_ID_FIELD)) + .limit(limit) + .toFlux() + .map { document -> document.toStoredLease() } + } + } + + private fun insert( + entry: QueryCursorLeaseEntry, + initialSlot: Int, + attempt: Int, + ): Mono { + if (attempt == options.maxEntries) { + return Mono.just(QueryCursorLeaseCreateResult.CAPACITY_EXCEEDED) + } + val slot = ((initialSlot.toLong() + attempt) % options.maxEntries).toInt() + return collection.insertOne(entry.toDocument(slot)).toMono() + .map { QueryCursorLeaseCreateResult.CREATED } + .onErrorResume(MongoWriteException::class.java) { error -> + if (ErrorCategory.fromErrorCode(error.code) != ErrorCategory.DUPLICATE_KEY) { + Mono.error(error) + } else { + contains(entry.id).flatMap { duplicateId -> + if (duplicateId) { + Mono.just(QueryCursorLeaseCreateResult.COLLISION) + } else { + insert(entry, initialSlot, attempt + 1) + } + } + } + } + } + + private fun contains(id: QueryCursorLeaseId): Mono = collection + .find(Filters.eq(LEASE_ID_FIELD, id.value)) + .projection(Projections.include(LEASE_ID_FIELD)) + .first() + .toMono() + .hasElement() + + private fun QueryCursorLeaseEntry.toDocument(slot: Int): Document { + val purgeAt = try { + expiresAt.plus(options.retentionGrace) + } catch (error: DateTimeException) { + throw IllegalArgumentException("Mongo cursor lease purge deadline cannot be represented.", error) + } catch (error: ArithmeticException) { + throw IllegalArgumentException("Mongo cursor lease purge deadline cannot be represented.", error) + } + return Document(ID_FIELD, slot) + .append(FORMAT_VERSION_FIELD, FORMAT_VERSION) + .append(LEASE_ID_FIELD, id.value) + .append(EXPIRES_AT_FIELD, Date.from(expiresAt)) + .append(PURGE_AT_FIELD, Date.from(purgeAt)) + .append(PAYLOAD_FORMAT_FIELD, payloadFormat.name) + .append(PAYLOAD_FIELD, Binary(payload())) + .append(REVISION_FIELD, UUID.randomUUID().toString()) + } + + private fun Document.toStoredLease(expectedId: QueryCursorLeaseId? = null): StoredQueryCursorLease { + if (keys != DOCUMENT_FIELDS) corrupt("Mongo cursor lease document has an invalid shape.") + exactInt(ID_FIELD).let { slot -> + if (slot !in 0 until options.maxEntries) corrupt("Mongo cursor lease slot is outside configured capacity.") + } + if (exactInt(FORMAT_VERSION_FIELD) != FORMAT_VERSION) { + corrupt("Mongo cursor lease document has an unsupported format version.") + } + val id = QueryCursorLeaseId(requiredString(LEASE_ID_FIELD)) + if (expectedId != null && id != expectedId) corrupt("Mongo cursor lease response identity does not match.") + val expiresAt = requiredDate(EXPIRES_AT_FIELD).toInstant() + val purgeAt = requiredDate(PURGE_AT_FIELD).toInstant() + if (purgeAt.isBefore(expiresAt)) corrupt("Mongo cursor lease purge deadline precedes expiry.") + val payloadFormat = try { + QueryCursorPayloadFormat.valueOf(requiredString(PAYLOAD_FORMAT_FIELD)) + } catch (error: IllegalArgumentException) { + throw corrupted("Mongo cursor lease payload format is invalid.", error) + } + val payload = when (val value = this[PAYLOAD_FIELD]) { + is Binary -> value.data + is ByteArray -> value + else -> corrupt("Mongo cursor lease payload is invalid.") + } + if (payload.isEmpty() || payload.size > options.maxPayloadBytes) { + corrupt("Mongo cursor lease payload is outside configured bounds.") + } + return StoredQueryCursorLease( + QueryCursorLeaseEntry(id, expiresAt, payloadFormat, payload), + QueryCursorStoreRevision(requiredString(REVISION_FIELD)), + ) + } + + private fun Document.requiredString(field: String): String = (this[field] as? String) + ?.takeIf(String::isNotBlank) + ?: corrupt("Mongo cursor lease field [$field] is invalid.") + + private fun Document.requiredDate(field: String): Date = this[field] as? Date + ?: corrupt("Mongo cursor lease field [$field] is invalid.") + + private fun Document.exactInt(field: String): Int { + val value = this[field] as? Number ?: corrupt("Mongo cursor lease field [$field] is not an integer.") + val long = value.toLong() + if (long !in Int.MIN_VALUE.toLong()..Int.MAX_VALUE.toLong() || value.toDouble() != long.toDouble()) { + corrupt("Mongo cursor lease field [$field] is not an exact integer.") + } + return long.toInt() + } + + private fun initialSlot(id: QueryCursorLeaseId): Int { + val digest = MessageDigest.getInstance("SHA-256") + .digest(id.value.toByteArray(StandardCharsets.UTF_8)) + val unsigned = ByteBuffer.wrap(digest).int.toLong() and UNSIGNED_INT_MASK + return (unsigned % options.maxEntries).toInt() + } + + private fun corrupted(message: String, cause: Throwable? = null): IllegalStateException = + IllegalStateException(message, cause) + + private fun corrupt(message: String): Nothing = throw corrupted(message) + + private companion object { + const val FORMAT_VERSION = 1 + const val ID_FIELD = "_id" + const val FORMAT_VERSION_FIELD = "formatVersion" + const val LEASE_ID_FIELD = "leaseId" + const val EXPIRES_AT_FIELD = "expiresAt" + const val PURGE_AT_FIELD = "purgeAt" + const val PAYLOAD_FORMAT_FIELD = "payloadFormat" + const val PAYLOAD_FIELD = "payload" + const val REVISION_FIELD = "revision" + const val LEASE_ID_INDEX = "lease_id_unique" + const val PURGE_AT_INDEX = "purge_at_ttl" + const val UNSIGNED_INT_MASK = 0xffffffffL + val DOCUMENT_FIELDS = setOf( + ID_FIELD, + FORMAT_VERSION_FIELD, + LEASE_ID_FIELD, + EXPIRES_AT_FIELD, + PURGE_AT_FIELD, + PAYLOAD_FORMAT_FIELD, + PAYLOAD_FIELD, + REVISION_FIELD, + ) + } +} diff --git a/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoAnalyticsQueryBackend.kt b/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoAnalyticsQueryBackend.kt new file mode 100644 index 00000000000..2b3ccba4456 --- /dev/null +++ b/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoAnalyticsQueryBackend.kt @@ -0,0 +1,315 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.client.model.Collation +import com.mongodb.reactivestreams.client.MongoCollection +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.AnalyticsQueryBackend +import me.ahoo.wow.query.backend.BackendAnalyticsBucket +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsGrouping +import me.ahoo.wow.query.backend.BackendAnalyticsMetric +import me.ahoo.wow.query.backend.BackendAnalyticsNumericPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsPage +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.gateway.QueryDocumentKind +import org.bson.Document +import org.bson.types.Decimal128 +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.math.BigDecimal +import java.math.BigInteger +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.util.LinkedHashMap +import java.util.concurrent.TimeUnit + +internal class MongoAnalyticsQueryBackend( + private val collection: MongoCollection, + private val binding: MongoPreparedQueryBinding, + private val clock: Clock = Clock.systemUTC(), +) : AnalyticsQueryBackend { + private val compiler = MongoAnalyticsQueryCompiler(binding) + private val mapper = MongoAnalyticsResultMapper(binding) + private val collation = when (binding.collationMode) { + MongoCollationMode.SIMPLE_BINARY -> Collation.builder().locale("simple").build() + } + + init { + require(binding.documentKind == QueryDocumentKind.SNAPSHOT) { + "Mongo analytics supports Snapshot documents only." + } + } + + override fun analyze( + plan: BackendAnalyticsQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = Mono.defer { + validateOptions(plan, options) + val compiled = compiler.compile(plan) + var publisher = collection.aggregate(compiled.pipeline) + .collation(collation) + .allowDiskUse(options.allowDiskUse) + options.remainingMillis()?.let { remaining -> + publisher = publisher.maxTime(remaining, TimeUnit.MILLISECONDS) + } + Flux.from(publisher).collectList().map { documents -> + mapPage(plan, compiled, documents) + } + }.onErrorMap(::mapBackendError) + + private fun validateOptions(plan: BackendAnalyticsQueryPlan, options: QueryBackendExecutionOptions) { + if (options.maxScannedRecords != null || + options.maxCandidateBuckets != null || + options.maxCursorPages != null + ) { + unsupportedBudget() + } + options.maxReturnedBuckets?.let { maximum -> + if (plan.bucketWindow.limit > maximum) { + exceededBudget() + } + } + options.remainingMillis() + } + + private fun mapPage( + plan: BackendAnalyticsQueryPlan, + compiled: MongoCompiledAnalyticsQuery, + documents: List, + ): BackendAnalyticsPage { + if (documents.size > compiled.resultLimit) { + mappingFailure() + } + val mapped = documents.map { document -> mapper.map(document, plan) }.toMutableList() + if (mapped.isEmpty() && plan.grouping == BackendAnalyticsGrouping.Global) { + mapped += mapper.emptyGlobal(plan) + } + val hasMore = mapped.size > plan.bucketWindow.limit + val buckets = mapped.take(plan.bucketWindow.limit) + val afterKey = if (hasMore) { + val dimensions = (plan.grouping as? BackendAnalyticsGrouping.By)?.dimensions ?: mappingFailure() + val last = buckets.lastOrNull() ?: mappingFailure() + dimensions.map { dimension -> last.keys[dimension.alias] ?: mappingFailure() } + } else { + null + } + return BackendAnalyticsPage( + buckets, + afterKey, + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + ) + } + + private fun QueryBackendExecutionOptions.remainingMillis(): Long? { + val currentDeadline = deadline ?: return null + val now = clock.instant() + val remaining = try { + Duration.between(now, currentDeadline).toMillis() + } catch (error: ArithmeticException) { + if (!currentDeadline.isAfter(now)) { + throw QueryBackendException( + QueryBackendFailureKind.TIMEOUT, + error, + ) + } + Long.MAX_VALUE + } + if (remaining <= 0) { + throw QueryBackendException(QueryBackendFailureKind.TIMEOUT) + } + return remaining + } + + private fun mapBackendError(error: Throwable): Throwable = + if (error is QueryBackendException) error else QueryBackendException(QueryBackendFailureKind.UNAVAILABLE, error) + + private fun unsupportedBudget(): Nothing = throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED) + + private fun exceededBudget(): Nothing = throw QueryBackendException(QueryBackendFailureKind.BUDGET_EXCEEDED) +} + +internal class MongoAnalyticsResultMapper( + private val binding: MongoPreparedQueryBinding, +) { + fun map(source: Document, plan: BackendAnalyticsQueryPlan): BackendAnalyticsBucket = mapFailures { + val keys = when (val grouping = plan.grouping) { + BackendAnalyticsGrouping.Global -> { + if (source[GROUP_ID] != null) mappingFailure() + emptyMap() + } + + is BackendAnalyticsGrouping.By -> { + val id = source[GROUP_ID] as? Map<*, *> ?: mappingFailure() + LinkedHashMap(grouping.dimensions.size).also { values -> + grouping.dimensions.forEach { dimension -> + if (!id.containsKey(dimension.alias.value)) mappingFailure() + values[dimension.alias] = mapFieldValue(dimension.field, id[dimension.alias.value]) + } + } + } + } + val metrics = LinkedHashMap(plan.metrics.size) + plan.metrics.forEach { metric -> + metrics[metric.alias] = mapMetric(metric, source[metric.alias.value], plan.numericPolicy) + } + BackendAnalyticsBucket(keys, metrics) + } + + fun emptyGlobal(plan: BackendAnalyticsQueryPlan): BackendAnalyticsBucket = mapFailures { + if (plan.grouping != BackendAnalyticsGrouping.Global) mappingFailure() + val metrics = LinkedHashMap(plan.metrics.size) + plan.metrics.forEach { metric -> + metrics[metric.alias] = when (metric) { + is BackendAnalyticsMetric.DocumentCount -> NormalizedValue.Int64(0) + is BackendAnalyticsMetric.Sum -> normalizeNumeric(BigDecimal.ZERO, requireNotNull(plan.numericPolicy)) + is BackendAnalyticsMetric.Min, + is BackendAnalyticsMetric.Max, + is BackendAnalyticsMetric.Average, + -> NormalizedValue.Null + } + } + BackendAnalyticsBucket(emptyMap(), metrics) + } + + private fun mapMetric( + metric: BackendAnalyticsMetric, + raw: Any?, + numericPolicy: BackendAnalyticsNumericPolicy?, + ): NormalizedValue = when (metric) { + is BackendAnalyticsMetric.DocumentCount -> { + val value = raw.toExactLong() + if (value < 0) mappingFailure() + NormalizedValue.Int64(value) + } + + is BackendAnalyticsMetric.Min -> mapMetricField(metric.field, raw, numericPolicy) + is BackendAnalyticsMetric.Max -> mapMetricField(metric.field, raw, numericPolicy) + is BackendAnalyticsMetric.Sum -> normalizeNumeric(raw.toBigDecimalExact(), requireNotNull(numericPolicy)) + is BackendAnalyticsMetric.Average -> if (raw == null) { + NormalizedValue.Null + } else { + normalizeNumeric(raw.toBigDecimalExact(), requireNotNull(numericPolicy)) + } + } + + private fun mapMetricField( + field: QueryFieldId, + raw: Any?, + numericPolicy: BackendAnalyticsNumericPolicy?, + ): NormalizedValue { + if (raw == null) { + return NormalizedValue.Null + } + val type = requireNotNull(binding.schema.fields[field]).type + return when (type) { + LogicalFieldType.Int64, + LogicalFieldType.Decimal, + -> normalizeNumeric(raw.toBigDecimalExact(), requireNotNull(numericPolicy)) + + LogicalFieldType.Instant -> mapFieldValue(field, raw) + else -> mappingFailure() + } + } + + private fun mapFieldValue(field: QueryFieldId, raw: Any?): NormalizedValue { + if (raw == null) { + return NormalizedValue.Null + } + return when (requireNotNull(binding.schema.fields[field]).type) { + LogicalFieldType.Text -> NormalizedValue.Text(raw as? String ?: mappingFailure()) + LogicalFieldType.Boolean -> NormalizedValue.BooleanValue(raw as? Boolean ?: mappingFailure()) + LogicalFieldType.Int64 -> NormalizedValue.Int64(raw.toExactLong()) + LogicalFieldType.Decimal -> NormalizedValue.Decimal(raw.toBigDecimalExact()) + LogicalFieldType.Instant -> NormalizedValue.InstantValue( + Instant.ofEpochMilli(raw.toExactLong()), + ) + + LogicalFieldType.Bytes, + LogicalFieldType.Object, + is LogicalFieldType.Array, + -> mappingFailure() + } + } + + private fun normalizeNumeric( + value: BigDecimal, + policy: BackendAnalyticsNumericPolicy, + ): NormalizedValue.Decimal { + val normalized = value.setScale(policy.scale, policy.roundingMode) + if (normalized.precision() > policy.precision) { + mappingFailure() + } + try { + Decimal128(normalized) + } catch (error: NumberFormatException) { + throw QueryBackendException(QueryBackendFailureKind.MAPPING_FAILURE, error) + } + return NormalizedValue.Decimal(normalized) + } + + private fun Any?.toExactLong(): Long = when (this) { + is Byte -> toLong() + is Short -> toLong() + is Int -> toLong() + is Long -> this + is BigInteger -> longValueExact() + is BigDecimal -> longValueExact() + is Decimal128 -> bigDecimalValue().longValueExact() + else -> mappingFailure() + } + + private fun Any?.toBigDecimalExact(): BigDecimal = when (this) { + is Decimal128 -> bigDecimalValue() + is BigDecimal -> this + is BigInteger -> toBigDecimal() + is Byte -> BigDecimal.valueOf(toLong()) + is Short -> BigDecimal.valueOf(toLong()) + is Int -> toBigDecimal() + is Long -> toBigDecimal() + is Float -> if (isFinite()) BigDecimal.valueOf(toDouble()) else mappingFailure() + is Double -> if (isFinite()) BigDecimal.valueOf(this) else mappingFailure() + else -> mappingFailure() + } + + @Suppress("TooGenericExceptionCaught") + private inline fun mapFailures(block: () -> T): T = try { + block() + } catch (error: QueryBackendException) { + throw error + } catch (error: RuntimeException) { + throw QueryBackendException(QueryBackendFailureKind.MAPPING_FAILURE, error) + } + + private companion object { + const val GROUP_ID = "_id" + } +} + +private fun mappingFailure(): Nothing = throw QueryBackendException(QueryBackendFailureKind.MAPPING_FAILURE) diff --git a/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoAnalyticsQueryCompiler.kt b/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoAnalyticsQueryCompiler.kt new file mode 100644 index 00000000000..c3e2337c1f0 --- /dev/null +++ b/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoAnalyticsQueryCompiler.kt @@ -0,0 +1,294 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.client.model.Aggregates +import com.mongodb.client.model.Filters +import me.ahoo.wow.query.backend.BackendAnalyticsBucketOrder +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsCondition +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsDimension +import me.ahoo.wow.query.backend.BackendAnalyticsGrouping +import me.ahoo.wow.query.backend.BackendAnalyticsMetric +import me.ahoo.wow.query.backend.BackendAnalyticsMissingPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNullPlacement +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.BackendAnalyticsTextCollation +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.SemanticTier +import org.bson.Document +import org.bson.conversions.Bson + +internal data class MongoCompiledAnalyticsQuery( + val pipeline: List, + val dimensions: List, + val metrics: List, + val resultLimit: Int, +) + +internal class MongoAnalyticsQueryCompiler( + private val binding: MongoPreparedQueryBinding, +) { + private val recordCompiler = MongoRecordQueryCompiler(binding) + + fun compile(plan: BackendAnalyticsQueryPlan): MongoCompiledAnalyticsQuery { + validatePlan(plan) + val dimensions = (plan.grouping as? BackendAnalyticsGrouping.By)?.dimensions.orEmpty() + val resultLimit = try { + Math.addExact(plan.bucketWindow.limit, 1) + } catch (error: ArithmeticException) { + throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED, error) + } + val pipeline = buildList { + add(Aggregates.match(recordCompiler.compileFilter(plan.filter))) + compileMissingFilter(dimensions)?.let { missing -> add(Aggregates.match(missing)) } + add(compileGroup(plan.grouping, plan.metrics)) + compileCursor(dimensions, plan.bucketWindow.afterKey)?.let { cursor -> add(Aggregates.match(cursor)) } + compileSort(plan.bucketOrder, dimensions)?.let { sort -> add(Aggregates.sort(sort)) } + add(Aggregates.limit(resultLimit)) + } + return MongoCompiledAnalyticsQuery(pipeline, dimensions, plan.metrics, resultLimit) + } + + private fun validatePlan(plan: BackendAnalyticsQueryPlan) { + validateContract(plan) + validateGrouping(plan) + validateMetrics(plan) + } + + private fun validateContract(plan: BackendAnalyticsQueryPlan) { + if (plan.target != binding.schema.target || plan.schemaContractId != binding.schema.contractId) { + unsupported() + } + if (plan.having != BackendAnalyticsCondition.All || plan.semanticTier != SemanticTier.PORTABLE) { + unsupported() + } + if (plan.requiredConsistency != BackendAnalyticsConsistency.EVENTUAL || + plan.requiredCompleteness != BackendAnalyticsCompleteness.EXACT + ) { + unsupported() + } + } + + private fun validateGrouping(plan: BackendAnalyticsQueryPlan) { + when (val grouping = plan.grouping) { + BackendAnalyticsGrouping.Global -> validateGlobalGrouping(plan) + + is BackendAnalyticsGrouping.By -> validateGroupedGrouping(plan, grouping) + } + } + + private fun validateGlobalGrouping(plan: BackendAnalyticsQueryPlan) { + if (plan.bucketOrder != BackendAnalyticsBucketOrder.Global || + plan.bucketWindow.afterKey != null || + plan.bucketWindow.limit != 1 + ) { + unsupported() + } + } + + private fun validateGroupedGrouping( + plan: BackendAnalyticsQueryPlan, + grouping: BackendAnalyticsGrouping.By, + ) { + val order = plan.bucketOrder as? BackendAnalyticsBucketOrder.DimensionKeyAscending ?: unsupported() + if (order.nullPlacement != BackendAnalyticsNullPlacement.FIRST || + order.textCollation != BackendAnalyticsTextCollation.BINARY || + plan.bucketWindow.afterKey?.size?.let { size -> size != grouping.dimensions.size } == true + ) { + unsupported() + } + grouping.dimensions.forEach { dimension -> requireAggregatable(dimension.field) } + plan.bucketWindow.afterKey?.forEachIndexed { index, value -> + if (!acceptsCursorValue(grouping.dimensions[index], value)) { + unsupported() + } + } + } + + private fun validateMetrics(plan: BackendAnalyticsQueryPlan) { + var needsNumericPolicy = false + plan.metrics.forEach { metric -> + when (metric) { + is BackendAnalyticsMetric.DocumentCount -> Unit + is BackendAnalyticsMetric.Min -> { + needsNumericPolicy = validateMetricField(metric.field, allowInstant = true) || needsNumericPolicy + } + + is BackendAnalyticsMetric.Max -> { + needsNumericPolicy = validateMetricField(metric.field, allowInstant = true) || needsNumericPolicy + } + + is BackendAnalyticsMetric.Sum -> { + validateMetricField(metric.field, allowInstant = false) + needsNumericPolicy = true + } + + is BackendAnalyticsMetric.Average -> { + validateMetricField(metric.field, allowInstant = false) + needsNumericPolicy = true + } + } + } + if ((plan.numericPolicy != null) != needsNumericPolicy) { + unsupported() + } + } + + private fun validateMetricField(field: QueryFieldId, allowInstant: Boolean): Boolean { + requireAggregatable(field) + return when (binding.schema.fields.getValue(field).type) { + LogicalFieldType.Int64, + LogicalFieldType.Decimal, + -> true + + LogicalFieldType.Instant -> if (allowInstant) false else unsupported() + else -> unsupported() + } + } + + private fun requireAggregatable(field: QueryFieldId): MongoFieldBinding = + recordCompiler.requireFieldBinding(field).also { fieldBinding -> + if (FieldCapability.AGGREGATABLE !in fieldBinding.capabilities || + field is QueryFieldId.Path && binding.schema.elementOwner(field) != null + ) { + unsupported() + } + } + + private fun acceptsCursorValue( + dimension: BackendAnalyticsDimension, + value: NormalizedValue, + ): Boolean { + val schemaField = binding.schema.fields.getValue(dimension.field) + if (value == NormalizedValue.Null) { + return dimension.missingPolicy == BackendAnalyticsMissingPolicy.AS_NULL_BUCKET && + (schemaField.presence == Presence.OPTIONAL || schemaField.nullability == Nullability.NULLABLE) + } + return when (schemaField.type) { + LogicalFieldType.Text -> value is NormalizedValue.Text + LogicalFieldType.Boolean -> value is NormalizedValue.BooleanValue + LogicalFieldType.Int64 -> value is NormalizedValue.Int64 + LogicalFieldType.Decimal -> value is NormalizedValue.Decimal + LogicalFieldType.Instant -> value is NormalizedValue.InstantValue + LogicalFieldType.Bytes, + LogicalFieldType.Object, + is LogicalFieldType.Array, + -> false + } + } + + private fun compileMissingFilter(dimensions: List): Bson? { + val filters = dimensions.filter { dimension -> + dimension.missingPolicy == BackendAnalyticsMissingPolicy.EXCLUDE + }.map { dimension -> + val path = requireAggregatable(dimension.field).path + Filters.and(Filters.exists(path, true), Filters.ne(path, null)) + } + return filters.takeIf(List<*>::isNotEmpty)?.let(Filters::and) + } + + private fun compileGroup( + grouping: BackendAnalyticsGrouping, + metrics: List, + ): Bson { + val id = when (grouping) { + BackendAnalyticsGrouping.Global -> null + is BackendAnalyticsGrouping.By -> Document().also { document -> + grouping.dimensions.forEach { dimension -> + val field = requireAggregatable(dimension.field) + document[dimension.alias.value] = when (dimension.missingPolicy) { + BackendAnalyticsMissingPolicy.EXCLUDE -> "\$${field.path}" + BackendAnalyticsMissingPolicy.AS_NULL_BUCKET -> Document( + "\$ifNull", + listOf("\$${field.path}", null), + ) + } + } + } + } + val group = Document("_id", id) + metrics.forEach { metric -> group[metric.alias.value] = compileMetric(metric) } + return Document("\$group", group) + } + + private fun compileMetric(metric: BackendAnalyticsMetric): Any = + when (metric) { + is BackendAnalyticsMetric.DocumentCount -> Document("\$sum", 1) + is BackendAnalyticsMetric.Min -> Document("\$min", fieldExpression(metric.field)) + is BackendAnalyticsMetric.Max -> Document("\$max", fieldExpression(metric.field)) + is BackendAnalyticsMetric.Sum -> Document("\$sum", decimalExpression(metric.field)) + is BackendAnalyticsMetric.Average -> Document("\$avg", decimalExpression(metric.field)) + } + + private fun fieldExpression(field: QueryFieldId): String = + "\$${requireAggregatable(field).path}" + + private fun decimalExpression(field: QueryFieldId): Document { + val fieldBinding = requireAggregatable(field) + if (fieldBinding.valueEncoding != MongoValueEncoding.DECIMAL128) { + unsupported() + } + return Document("\$toDecimal", "\$${fieldBinding.path}") + } + + private fun compileCursor( + dimensions: List, + afterKey: List?, + ): Bson? { + if (afterKey == null) { + return null + } + if (dimensions.size != afterKey.size) { + unsupported() + } + val encoded = dimensions.mapIndexed { index, dimension -> + recordCompiler.encodeFieldValue(dimension.field, afterKey[index]) + } + val branches = dimensions.indices.map { index -> + Document().also { branch -> + repeat(index) { prefix -> branch[dimensionKey(dimensions[prefix])] = encoded[prefix] } + branch[dimensionKey(dimensions[index])] = Document("\$gt", encoded[index]) + } + } + return Document("\$or", branches) + } + + private fun compileSort( + order: BackendAnalyticsBucketOrder, + dimensions: List, + ): Bson? = when (order) { + BackendAnalyticsBucketOrder.Global -> null + is BackendAnalyticsBucketOrder.DimensionKeyAscending -> Document().also { sort -> + dimensions.forEach { dimension -> sort[dimensionKey(dimension)] = 1 } + } + } + + private fun dimensionKey(dimension: BackendAnalyticsDimension): String = "_id.${dimension.alias.value}" + + private fun unsupported(): Nothing = throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED) +} diff --git a/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoRecordQueryCompiler.kt b/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoRecordQueryCompiler.kt new file mode 100644 index 00000000000..2121d77c3f8 --- /dev/null +++ b/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoRecordQueryCompiler.kt @@ -0,0 +1,505 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.client.model.Aggregates +import com.mongodb.client.model.Filters +import com.mongodb.client.model.Projections +import com.mongodb.client.model.Sorts +import me.ahoo.wow.mongo.Documents +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPageQueryPlan +import me.ahoo.wow.query.backend.BackendPageWindow +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.BackendRecordQueryPlan +import me.ahoo.wow.query.backend.BackendRecordResultPlan +import me.ahoo.wow.query.backend.BackendSingleQueryPlan +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.CaseSensitivity +import me.ahoo.wow.query.backend.JunctionOperator +import me.ahoo.wow.query.backend.NormalizedSortDirection +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryFieldId +import org.bson.Document +import org.bson.conversions.Bson +import org.bson.types.Binary +import org.bson.types.Decimal128 + +internal object MongoPagePipelineFields { + const val PREFIX = "__wowQueryPage" + const val KIND = "${PREFIX}Kind" + const val POSITION = "${PREFIX}Position" + const val TOTAL = "${PREFIX}Total" + const val RECORD_KIND = 0 + const val SENTINEL_KIND = 1 +} + +internal data class MongoCompiledRecordQuery( + val filter: Bson, + val projection: Bson?, + val pageProjection: Bson?, + val sort: Bson?, + val limit: Int?, + val page: BackendPageWindow?, +) { + fun pagePipeline(): List { + val window = requireNotNull(page) { "Mongo page pipeline requires a page window." } + return buildList { + add(Aggregates.match(filter)) + add(markRecords()) + add(appendSentinel()) + add(setPageWindow()) + add(matchPageRows(window)) + pageProjection?.let { currentProjection -> add(Aggregates.project(currentProjection)) } + add(replaceSentinel()) + add(clearPageFields()) + } + } + + private fun markRecords(): Bson = + Document("\$set", Document(MongoPagePipelineFields.KIND, MongoPagePipelineFields.RECORD_KIND)) + + private fun appendSentinel(): Bson = Document( + "\$unionWith", + Document( + "pipeline", + listOf( + Document( + "\$documents", + listOf(Document(MongoPagePipelineFields.KIND, MongoPagePipelineFields.SENTINEL_KIND)), + ), + ), + ), + ) + + private fun setPageWindow(): Bson = Document( + "\$setWindowFields", + Document("sortBy", pageSort().toBsonDocument()).append("output", pageWindowOutput()), + ) + + private fun pageSort(): Bson = sort?.let { currentSort -> + Sorts.orderBy(Sorts.ascending(MongoPagePipelineFields.KIND), currentSort) + } ?: Sorts.ascending(MongoPagePipelineFields.KIND) + + private fun pageWindowOutput(): Bson = Document( + MongoPagePipelineFields.POSITION, + Document("\$sum", 1).append( + "window", + Document("documents", listOf("unbounded", "current")), + ), + ).append( + MongoPagePipelineFields.TOTAL, + Document( + "\$sum", + Document( + "\$cond", + listOf( + Document( + "\$eq", + listOf("\$${MongoPagePipelineFields.KIND}", MongoPagePipelineFields.RECORD_KIND), + ), + 1, + 0, + ), + ), + ).append( + "window", + Document("documents", listOf("unbounded", "unbounded")), + ), + ) + + private fun matchPageRows(window: BackendPageWindow): Bson { + val endInclusive = pageEndInclusive(window) + return Document( + "\$match", + Document( + "\$or", + listOf( + Document(MongoPagePipelineFields.KIND, MongoPagePipelineFields.SENTINEL_KIND), + Document( + "\$and", + listOf( + Document(MongoPagePipelineFields.KIND, MongoPagePipelineFields.RECORD_KIND), + Document(MongoPagePipelineFields.POSITION, Document("\$gt", window.offset)), + Document(MongoPagePipelineFields.POSITION, Document("\$lte", endInclusive)), + ), + ), + ), + ), + ) + } + + private fun pageEndInclusive(window: BackendPageWindow): Long = try { + Math.addExact(window.offset, window.size.toLong()) + } catch (error: ArithmeticException) { + throw QueryBackendException(QueryBackendFailureKind.BUDGET_EXCEEDED, error) + } + + private fun replaceSentinel(): Bson = Document( + "\$replaceWith", + Document( + "\$cond", + listOf( + Document( + "\$eq", + listOf("\$${MongoPagePipelineFields.KIND}", MongoPagePipelineFields.SENTINEL_KIND), + ), + Document(PAGE_TOTAL_VALUE, "\$${MongoPagePipelineFields.TOTAL}"), + "\$\$ROOT", + ), + ), + ) + + private fun clearPageFields(): Bson = Document( + "\$unset", + listOf( + MongoPagePipelineFields.KIND, + MongoPagePipelineFields.POSITION, + MongoPagePipelineFields.TOTAL, + ), + ) + + companion object { + const val PAGE_TOTAL_VALUE = "value" + } +} + +internal class MongoRecordQueryCompiler( + private val binding: MongoPreparedQueryBinding, +) { + constructor(binding: MongoSnapshotQueryBinding) : this(binding.prepared) + + constructor(binding: MongoEventStreamQueryBinding) : this(binding.prepared) + + fun compile(plan: BackendRecordQueryPlan): MongoCompiledRecordQuery { + require(plan.target == binding.schema.target) { "Mongo query plan target does not match its binding." } + require(plan.schemaContractId == binding.schema.contractId) { + "Mongo query plan schema contract does not match its binding." + } + val resultPlan = plan as? BackendRecordResultPlan + return MongoCompiledRecordQuery( + filter = compileFilter(plan.filter), + projection = resultPlan?.projection?.let(::compileProjection), + pageProjection = (plan as? BackendPageQueryPlan)?.projection?.let(::compilePageProjection), + sort = resultPlan?.sort?.takeIf(List<*>::isNotEmpty)?.let { sorts -> + Sorts.orderBy( + sorts.map { sort -> + val path = requireField(sort.field).path + when (sort.direction) { + NormalizedSortDirection.ASC -> Sorts.ascending(path) + NormalizedSortDirection.DESC -> Sorts.descending(path) + } + }, + ) + }, + limit = when (plan) { + is BackendSingleQueryPlan -> 1 + is BackendStreamQueryPlan -> plan.limit + is BackendPageQueryPlan, + is BackendCountQueryPlan -> null + }, + page = (plan as? BackendPageQueryPlan)?.page, + ) + } + + internal fun compileFilter(filter: BackendEnforcedFilter): Bson { + validateSearchShape(filter.condition) + return compileCondition(filter.condition, null) + } + + internal fun requireFieldBinding(field: QueryFieldId): MongoFieldBinding = requireField(field) + + internal fun encodeFieldValue(field: QueryFieldId, value: NormalizedValue): Any? = + encodeValue(requireField(field), value) + + private fun compileCondition(condition: BackendPlannedCondition, elementOwner: MongoFieldBinding?): Bson = + when (condition) { + BackendPlannedCondition.All -> Filters.empty() + BackendPlannedCondition.None -> Document("\$expr", false) + is BackendPlannedCondition.Junction -> { + val children = condition.children.map { child -> compileCondition(child, elementOwner) } + when (condition.operator) { + JunctionOperator.AND -> Filters.and(children) + JunctionOperator.OR -> Filters.or(children) + JunctionOperator.NOR -> Filters.nor(children) + } + } + + is BackendPlannedCondition.Predicate -> compilePredicate(condition, elementOwner) + is BackendPlannedCondition.ElementMatch -> { + val owner = requireField(condition.field) + Filters.elemMatch( + relativePath(owner.path, elementOwner?.path), + compileCondition(condition.condition, owner), + ) + } + + is BackendPlannedCondition.Search -> { + if (binding.textSearch?.scope != condition.scope) { + unsupported() + } + Filters.text(condition.text) + } + + is BackendPlannedCondition.Native -> unsupported() + } + + private fun validateSearchShape(condition: BackendPlannedCondition) { + if (condition.searchCount() > 1) { + unsupported() + } + when (condition) { + is BackendPlannedCondition.ElementMatch -> { + if (condition.condition.containsSearch()) { + unsupported() + } + } + + is BackendPlannedCondition.Junction -> when (condition.operator) { + JunctionOperator.AND -> condition.children.forEach(::validateSearchShape) + JunctionOperator.OR, + JunctionOperator.NOR, + -> if (condition.containsSearch()) { + unsupported() + } + } + + else -> Unit + } + } + + private fun BackendPlannedCondition.searchCount(): Int = when (this) { + is BackendPlannedCondition.Search -> 1 + is BackendPlannedCondition.Junction -> children.sumOf { child -> child.searchCount() } + is BackendPlannedCondition.ElementMatch -> condition.searchCount() + else -> 0 + } + + private fun BackendPlannedCondition.containsSearch(): Boolean = searchCount() > 0 + + private fun compilePredicate( + predicate: BackendPlannedCondition.Predicate, + elementOwner: MongoFieldBinding?, + ): Bson { + if (predicate.options.caseSensitivity != CaseSensitivity.SENSITIVE) { + unsupported() + } + val field = requireField(predicate.field) + val path = relativePath(field.path, elementOwner?.path) + val value = predicate.value?.let { normalized -> encodeValue(field, normalized) } + return when (predicate.operator) { + PredicateOperator.EQ, + PredicateOperator.NE, + PredicateOperator.GT, + PredicateOperator.LT, + PredicateOperator.GTE, + PredicateOperator.LTE, + -> compileComparison(path, predicate.operator, value) + + PredicateOperator.IN, + PredicateOperator.NOT_IN, + PredicateOperator.BETWEEN, + PredicateOperator.ALL_IN, + -> compileCollection(path, predicate.operator, value) + + PredicateOperator.CONTAINS, + PredicateOperator.STARTS_WITH, + PredicateOperator.ENDS_WITH, + -> compileLiteral(path, predicate.operator, value) + + PredicateOperator.IS_NULL, + PredicateOperator.NOT_NULL, + PredicateOperator.IS_TRUE, + PredicateOperator.IS_FALSE, + PredicateOperator.EXISTS, + -> compileState(path, predicate.operator, value) + } + } + + private fun compileComparison(path: String, operator: PredicateOperator, value: Any?): Bson = when (operator) { + PredicateOperator.EQ -> Filters.eq(path, value) + PredicateOperator.NE -> Filters.ne(path, value) + PredicateOperator.GT -> Filters.gt(path, value ?: unsupported()) + PredicateOperator.LT -> Filters.lt(path, value ?: unsupported()) + PredicateOperator.GTE -> Filters.gte(path, value ?: unsupported()) + PredicateOperator.LTE -> Filters.lte(path, value ?: unsupported()) + else -> unsupported() + } + + private fun compileCollection(path: String, operator: PredicateOperator, value: Any?): Bson { + val values = value.requireList() + return when (operator) { + PredicateOperator.IN -> Filters.`in`(path, values) + PredicateOperator.NOT_IN -> Filters.nin(path, values) + PredicateOperator.ALL_IN -> Filters.all(path, values) + PredicateOperator.BETWEEN -> { + require(values.size == 2) { "BETWEEN requires exactly two planned values." } + Filters.and( + Filters.gte(path, values[0] ?: unsupported()), + Filters.lte(path, values[1] ?: unsupported()), + ) + } + + else -> unsupported() + } + } + + private fun compileLiteral(path: String, operator: PredicateOperator, value: Any?): Bson { + val literal = value.requireText().escapeRegex() + return when (operator) { + PredicateOperator.CONTAINS -> Filters.regex(path, literal) + PredicateOperator.STARTS_WITH -> Filters.regex(path, "^$literal") + PredicateOperator.ENDS_WITH -> Filters.regex(path, "$literal$") + else -> unsupported() + } + } + + private fun compileState(path: String, operator: PredicateOperator, value: Any?): Bson = when (operator) { + PredicateOperator.IS_NULL -> Filters.eq(path, null) + PredicateOperator.NOT_NULL -> Filters.ne(path, null) + PredicateOperator.IS_TRUE -> Filters.eq(path, true) + PredicateOperator.IS_FALSE -> Filters.eq(path, false) + PredicateOperator.EXISTS -> Filters.exists(path, value as? Boolean ?: unsupported()) + else -> unsupported() + } + + private fun compileProjection(projection: BackendProjection): Bson? = + when (projection) { + BackendProjection.All -> null + is BackendProjection.Include -> Projections.include( + canonicalPhysicalPaths( + projection.fields.map { field -> requireField(field).path } + Documents.ID_FIELD, + ), + ) + + is BackendProjection.Exclude -> { + val excluded = canonicalPhysicalPaths( + projection.fields.map { field -> requireField(field).path } + .filterNot(Documents.ID_FIELD::equals), + ) + excluded.takeIf(List<*>::isNotEmpty)?.let(Projections::exclude) + } + } + + private fun compilePageProjection(projection: BackendProjection): Bson? = + when (projection) { + BackendProjection.All -> null + is BackendProjection.Include -> Projections.include( + canonicalPhysicalPaths( + projection.fields.map { field -> requireField(field).path } + + Documents.ID_FIELD + + listOf( + MongoPagePipelineFields.KIND, + MongoPagePipelineFields.POSITION, + MongoPagePipelineFields.TOTAL, + ), + ), + ) + + is BackendProjection.Exclude -> compileProjection(projection) + } + + private fun requireField(field: QueryFieldId): MongoFieldBinding = + binding.fields[field] ?: unsupported() + + private fun encodeValue(field: MongoFieldBinding, value: NormalizedValue): Any? = + when (value) { + NormalizedValue.Null -> null + is NormalizedValue.BooleanValue -> value.value + is NormalizedValue.Text -> value.value + is NormalizedValue.Int64 -> encodeInt64(field, value) + is NormalizedValue.Decimal -> encodeDecimal(field, value) + is NormalizedValue.InstantValue -> encodeInstant(field, value) + + is NormalizedValue.Bytes -> Binary(value.toByteArray()) + is NormalizedValue.ListValue -> value.values.map { nested -> encodeValue(field, nested) } + is NormalizedValue.ObjectValue -> Document( + value.values.mapValues { (_, nested) -> encodeValue(field, nested) }, + ) + } + + private fun encodeInt64(field: MongoFieldBinding, value: NormalizedValue.Int64): Any = + if (field.valueEncoding == MongoValueEncoding.DECIMAL128) Decimal128(value.value) else value.value + + private fun encodeDecimal(field: MongoFieldBinding, value: NormalizedValue.Decimal): Decimal128 { + if (field.valueEncoding != MongoValueEncoding.DECIMAL128) { + unsupported() + } + return decimal128(value) + } + + private fun encodeInstant(field: MongoFieldBinding, value: NormalizedValue.InstantValue): Long { + if (field.valueEncoding != MongoValueEncoding.EPOCH_MILLIS) { + unsupported() + } + return value.value.toEpochMilli() + } + + private fun decimal128(value: NormalizedValue.Decimal): Decimal128 = + try { + Decimal128(value.value) + } catch (error: NumberFormatException) { + throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED, error) + } + + private fun relativePath(path: String, owner: String?): String { + if (owner == null) { + return path + } + val prefix = "$owner." + if (!path.startsWith(prefix)) { + unsupported() + } + return path.removePrefix(prefix).also { relative -> + if (relative.isBlank()) { + unsupported() + } + } + } + + private fun Any?.requireText(): String = this as? String ?: unsupported() + + @Suppress("UNCHECKED_CAST") + private fun Any?.requireList(): List = this as? List ?: unsupported() + + private fun String.escapeRegex(): String = buildString(length + 8) { + this@escapeRegex.forEach { char -> + if (char in REGEX_META) { + append('\\') + } + append(char) + } + } + + private fun canonicalPhysicalPaths(paths: List): List = + paths.distinct().filter { candidate -> + paths.none { other -> other != candidate && candidate.startsWith("$other.") } + } + + private fun unsupported(): Nothing = throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED) + + private companion object { + val REGEX_META = setOf('\\', '^', '$', '.', '|', '?', '*', '+', '(', ')', '[', ']', '{', '}') + } +} diff --git a/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotQueryBinding.kt b/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotQueryBinding.kt new file mode 100644 index 00000000000..54a58f4e973 --- /dev/null +++ b/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotQueryBinding.kt @@ -0,0 +1,451 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.MongoNamespace +import com.mongodb.reactivestreams.client.MongoCollection +import me.ahoo.wow.mongo.Documents +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.BackendStreamSupport +import me.ahoo.wow.query.backend.ExperimentalQueryBackendApi +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.RecordQueryBackendContribution +import me.ahoo.wow.query.backend.SearchScopeId +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.serialization.state.StateAggregateRecords +import org.bson.Document +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.io.ByteArrayOutputStream +import java.io.DataOutputStream +import java.security.MessageDigest +import java.util.Collections +import java.util.LinkedHashMap + +@ExperimentalQueryBackendApi +enum class MongoValueEncoding { + DEFAULT, + EPOCH_MILLIS, + DECIMAL128, +} + +@ExperimentalQueryBackendApi +enum class MongoCollationMode { + SIMPLE_BINARY, +} + +@ExperimentalQueryBackendApi +data class MongoTextSearchBinding( + val scope: SearchScopeId, + val indexName: String, +) { + init { + require(indexName.isNotBlank()) { "Mongo text index name must not be blank." } + require(indexName.none(Char::isISOControl)) { "Mongo text index name must not contain control characters." } + } +} + +@ExperimentalQueryBackendApi +class MongoQueryBackendNotReadyException( + message: String, + cause: Throwable? = null, +) : RuntimeException(message, cause) + +@ExperimentalQueryBackendApi +fun MongoSnapshotQueryBinding.toContribution( + collection: MongoCollection, +): RecordQueryBackendContribution = prepared.toContribution(collection) + +@ExperimentalQueryBackendApi +fun MongoEventStreamQueryBinding.toContribution( + collection: MongoCollection, +): RecordQueryBackendContribution = prepared.toContribution(collection) + +private fun MongoPreparedQueryBinding.toContribution( + collection: MongoCollection, +): RecordQueryBackendContribution { + requireCollection(collection) + require(textSearch == null) { + "Mongo text search requires prepareContribution() to attest the configured text index." + } + return createContribution(collection) +} + +@ExperimentalQueryBackendApi +fun MongoSnapshotQueryBinding.prepareContribution( + collection: MongoCollection, +): Mono = prepared.prepareContribution(collection) + +@ExperimentalQueryBackendApi +fun MongoEventStreamQueryBinding.prepareContribution( + collection: MongoCollection, +): Mono = prepared.prepareContribution(collection) + +private fun MongoPreparedQueryBinding.prepareContribution( + collection: MongoCollection, +): Mono { + requireCollection(collection) + if (textSearch == null) { + return Mono.fromSupplier { createContribution(collection) } + } + return Flux.from(collection.listIndexes()).collectList() + .onErrorMap { error -> + MongoQueryBackendNotReadyException("Mongo text index readiness could not be inspected.", error) + }.map { indexes -> + attestTextIndexReadiness(indexes) + createContribution(collection) + } +} + +private fun MongoPreparedQueryBinding.createContribution( + collection: MongoCollection, +): RecordQueryBackendContribution { + requireCollection(collection) + val analyticsBackend = if (documentKind == QueryDocumentKind.SNAPSHOT) { + MongoAnalyticsQueryBackend(collection, this) + } else { + null + } + return RecordQueryBackendContribution( + schema = schema, + backendId = backendId, + supportedOperations = setOf( + QueryOperation.SINGLE, + QueryOperation.STREAM, + QueryOperation.PAGE, + QueryOperation.COUNT, + if (analyticsBackend != null) QueryOperation.ANALYZE else null, + ).filterNotNull().toSet(), + streamSupport = BackendStreamSupport.BOUNDED_ONLY, + semanticTiers = buildSet { + add(SemanticTier.PORTABLE) + if (textSearch != null) { + add(SemanticTier.SEARCH) + } + }, + fieldCapabilities = fields.mapValues { (_, field) -> field.capabilities }, + searchScopes = textSearch?.let { search -> setOf(search.scope) }.orEmpty(), + backend = MongoRecordQueryBackend(collection, this), + analyticsBackend = analyticsBackend, + mappingGenerationDigest = capabilityDigest, + ) +} + +private fun MongoPreparedQueryBinding.requireCollection(collection: MongoCollection) { + require(collection.namespace == namespace) { + "Mongo query binding namespace[$namespace] does not match collection[${collection.namespace}]." + } +} + +@ExperimentalQueryBackendApi +class MongoFieldBinding( + val path: String, + capabilities: Set, + val valueEncoding: MongoValueEncoding = MongoValueEncoding.DEFAULT, +) { + val capabilities: Set = Collections.unmodifiableSet(LinkedHashSet(capabilities)) + + init { + require(path.isNotBlank()) { "Mongo field path must not be blank." } + val segments = path.split('.') + require(segments.none(String::isBlank)) { "Mongo field path segments must not be blank." } + require(segments.none { segment -> segment.startsWith('$') }) { + "Mongo field path segments must not start with '$'." + } + require(segments.none { segment -> segment.startsWith(MongoPagePipelineFields.PREFIX) }) { + "Mongo field path segments must not use the reserved planned-page prefix." + } + require(path.none(Char::isISOControl)) { "Mongo field path must not contain control characters." } + } +} + +@ExperimentalQueryBackendApi +class MongoSnapshotQueryBinding( + val schema: QueryDocumentSchema, + val namespace: MongoNamespace, + fields: Map, + val backendId: BackendId = BackendId("mongo"), + val collationMode: MongoCollationMode = MongoCollationMode.SIMPLE_BINARY, + val textSearch: MongoTextSearchBinding? = null, +) { + internal val prepared = prepareMongoBinding( + schema, + namespace, + fields, + backendId, + collationMode, + textSearch, + QueryDocumentKind.SNAPSHOT, + MessageRecords.AGGREGATE_ID, + SNAPSHOT_SYSTEM_PATHS, + ) + val fields: Map = prepared.fields + + companion object { + fun frameworkFields(schema: QueryDocumentSchema, namespace: MongoNamespace): MongoSnapshotQueryBinding { + val bindings = linkedMapOf() + fun bind(kind: SystemFieldKind, path: String) { + val id = QueryFieldId.System(kind) + schema.fields[id]?.let { field -> bindings[id] = MongoFieldBinding(path, field.capabilities) } + } + SNAPSHOT_SYSTEM_PATHS.forEach(::bind) + return MongoSnapshotQueryBinding(schema, namespace, bindings) + } + } +} + +@ExperimentalQueryBackendApi +class MongoEventStreamQueryBinding( + val schema: QueryDocumentSchema, + val namespace: MongoNamespace, + fields: Map, + val backendId: BackendId = BackendId("mongo"), + val collationMode: MongoCollationMode = MongoCollationMode.SIMPLE_BINARY, + val textSearch: MongoTextSearchBinding? = null, +) { + internal val prepared = prepareMongoBinding( + schema, + namespace, + fields, + backendId, + collationMode, + textSearch, + QueryDocumentKind.EVENT_STREAM, + MessageRecords.ID, + EVENT_STREAM_SYSTEM_PATHS, + ) + val fields: Map = prepared.fields + + companion object { + fun frameworkFields(schema: QueryDocumentSchema, namespace: MongoNamespace): MongoEventStreamQueryBinding { + val bindings = linkedMapOf() + EVENT_STREAM_SYSTEM_PATHS.forEach { (kind, path) -> + val id = QueryFieldId.System(kind) + schema.fields[id]?.let { field -> bindings[id] = MongoFieldBinding(path, field.capabilities) } + } + return MongoEventStreamQueryBinding(schema, namespace, bindings) + } + } +} + +internal class MongoPreparedQueryBinding( + val schema: QueryDocumentSchema, + val namespace: MongoNamespace, + val fields: Map, + val backendId: BackendId, + val collationMode: MongoCollationMode, + val textSearch: MongoTextSearchBinding?, + val identityOutputField: String, + val documentKind: QueryDocumentKind, +) { + val capabilityDigest: String = MongoCapabilityDigestEncoder.encode(this) +} + +private fun prepareMongoBinding( + schema: QueryDocumentSchema, + namespace: MongoNamespace, + fields: Map, + backendId: BackendId, + collationMode: MongoCollationMode, + textSearch: MongoTextSearchBinding?, + documentKind: QueryDocumentKind, + identityOutputField: String, + systemPaths: Map, +): MongoPreparedQueryBinding { + require(schema.target.documentKind == documentKind) { + "Mongo query binding requires a $documentKind target." + } + val copy = LinkedHashMap(fields.size) + fields.forEach { (field, binding) -> + val fieldSchema = requireNotNull(schema.fields[field]) { + "Mongo field $field is not declared by the logical schema." + } + require(fieldSchema.capabilities.containsAll(binding.capabilities)) { + "Mongo field $field overclaims the logical schema contract." + } + requireValidEncoding(fieldSchema.type, binding.valueEncoding) + if (field is QueryFieldId.Path) { + require(field.segments.joinToString(".") == binding.path) { + "Mongo record materialization requires logical and physical user paths to match." + } + require(binding.path !in systemPaths.values) { + "Mongo user field $field must not collide with a framework system field." + } + } + copy[field] = binding + } + val immutableFields = Collections.unmodifiableMap(copy) + require(immutableFields.keys == schema.fields.keys) { + "Mongo query binding must cover every field in the logical schema." + } + systemPaths.forEach { (kind, expectedPath) -> + val id = QueryFieldId.System(kind) + if (id in schema.fields && id in immutableFields) { + require(immutableFields[id]?.path == expectedPath) { + "Mongo $documentKind system field $kind must bind to $expectedPath." + } + } + } + require(immutableFields[QueryFieldId.System(SystemFieldKind.IDENTITY)]?.path == Documents.ID_FIELD) { + "Mongo $documentKind identity must bind to ${Documents.ID_FIELD}." + } + if (documentKind == QueryDocumentKind.EVENT_STREAM) { + require(QueryFieldId.System(SystemFieldKind.DELETED) !in schema.fields) { + "Mongo EventStream schema must not declare snapshot deletion semantics." + } + } + textSearch?.let { search -> validateTextSearch(schema, immutableFields, search) } + return MongoPreparedQueryBinding( + schema, + namespace, + immutableFields, + backendId, + collationMode, + textSearch, + identityOutputField, + documentKind, + ) +} + +private fun validateTextSearch( + schema: QueryDocumentSchema, + fields: Map, + search: MongoTextSearchBinding, +) { + val definition = requireNotNull(schema.searchScopes[search.scope]) { + "Mongo text search scope ${search.scope} is not declared by the logical schema." + } + require(definition.owner == null) { + "Mongo text search supports root document scopes only." + } + definition.fields.forEach { field -> + require(FieldCapability.FULL_TEXT in requireNotNull(fields[field]).capabilities) { + "Mongo text search field $field must bind FULL_TEXT capability." + } + } +} + +private object MongoCapabilityDigestEncoder { + fun encode(binding: MongoPreparedQueryBinding): String { + val bytes = ByteArrayOutputStream().use { buffer -> + DataOutputStream(buffer).use { output -> + output.writeUTF("wow.mongo.query.binding.v1") + output.writeUTF(binding.schema.contractId.value) + output.writeUTF(binding.namespace.databaseName) + output.writeUTF(binding.namespace.collectionName) + output.writeUTF(binding.documentKind.name) + output.writeUTF(binding.backendId.value) + output.writeUTF(binding.collationMode.name) + output.writeInt(binding.fields.size) + binding.fields.entries.sortedBy { entry -> entry.key.stableKey() }.forEach { (field, physical) -> + output.writeUTF(field.stableKey()) + output.writeUTF(physical.path) + output.writeUTF(physical.valueEncoding.name) + output.writeInt(physical.capabilities.size) + physical.capabilities.sortedBy(FieldCapability::name).forEach { capability -> + output.writeUTF(capability.name) + } + } + output.writeBoolean(binding.textSearch != null) + binding.textSearch?.let { search -> + output.writeUTF(search.scope.value) + output.writeUTF(search.indexName) + } + } + buffer.toByteArray() + } + return MessageDigest.getInstance("SHA-256").digest(bytes).toHex() + } + + private fun QueryFieldId.stableKey(): String = when (this) { + is QueryFieldId.Path -> "path:${segments.joinToString("\u0000")}" + is QueryFieldId.System -> "system:${kind.name}" + } + + private fun ByteArray.toHex(): String = joinToString(separator = "") { byte -> "%02x".format(byte) } +} + +internal fun MongoPreparedQueryBinding.attestTextIndexReadiness(indexes: List) { + val search = requireNotNull(textSearch) { "Mongo text search is not configured." } + val definition = requireNotNull(schema.searchScopes[search.scope]) + val expectedFields = definition.fields.map { field -> requireNotNull(fields[field]).path }.toSet() + val candidates = indexes.filter { index -> index.getString("name") == search.indexName } + if (candidates.size != 1 || !candidates.single().isExactTextIndex(expectedFields)) { + throw MongoQueryBackendNotReadyException( + "Mongo text index ${search.indexName} is missing or does not match scope ${search.scope}.", + ) + } +} + +internal fun MongoSnapshotQueryBinding.attestTextIndexReadiness(indexes: List) { + prepared.attestTextIndexReadiness(indexes) +} + +internal fun MongoEventStreamQueryBinding.attestTextIndexReadiness(indexes: List) { + prepared.attestTextIndexReadiness(indexes) +} + +private val SNAPSHOT_SYSTEM_PATHS = linkedMapOf( + SystemFieldKind.IDENTITY to Documents.ID_FIELD, + SystemFieldKind.AGGREGATE_ID to Documents.ID_FIELD, + SystemFieldKind.TENANT_ID to MessageRecords.TENANT_ID, + SystemFieldKind.OWNER_ID to MessageRecords.OWNER_ID, + SystemFieldKind.SPACE_ID to MessageRecords.SPACE_ID, + SystemFieldKind.DELETED to StateAggregateRecords.DELETED, +) + +private val EVENT_STREAM_SYSTEM_PATHS = linkedMapOf( + SystemFieldKind.IDENTITY to Documents.ID_FIELD, + SystemFieldKind.AGGREGATE_ID to MessageRecords.AGGREGATE_ID, + SystemFieldKind.TENANT_ID to MessageRecords.TENANT_ID, + SystemFieldKind.OWNER_ID to MessageRecords.OWNER_ID, + SystemFieldKind.SPACE_ID to MessageRecords.SPACE_ID, +) + +private fun Document.isExactTextIndex(expectedFields: Set): Boolean { + val key = this["key"] as? Document ?: return false + val weights = this["weights"] as? Document ?: return false + val collation = this["collation"] as? Document + return key.keys == setOf("_fts", "_ftsx") && + key["_fts"] == "text" && + (key["_ftsx"] as? Number)?.toInt() == 1 && + weights.keys == expectedFields && + (collation == null || collation.getString("locale") == "simple") +} + +private fun requireValidEncoding(type: LogicalFieldType, encoding: MongoValueEncoding) { + val scalarType = generateSequence(type) { current -> + (current as? LogicalFieldType.Array)?.elementType + }.last() + val expected = when (scalarType) { + LogicalFieldType.Instant -> MongoValueEncoding.EPOCH_MILLIS + LogicalFieldType.Decimal -> MongoValueEncoding.DECIMAL128 + else -> MongoValueEncoding.DEFAULT + } + require(encoding == expected) { + "Mongo value encoding $encoding does not match logical field type $type; expected $expected." + } +} diff --git a/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotRecordQueryBackend.kt b/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotRecordQueryBackend.kt new file mode 100644 index 00000000000..4fe86cdba54 --- /dev/null +++ b/wow-mongo/src/main/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotRecordQueryBackend.kt @@ -0,0 +1,472 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.client.model.Collation +import com.mongodb.client.model.CountOptions +import com.mongodb.reactivestreams.client.FindPublisher +import com.mongodb.reactivestreams.client.MongoCollection +import me.ahoo.wow.mongo.Documents +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendPage +import me.ahoo.wow.query.backend.BackendPageConsistency +import me.ahoo.wow.query.backend.BackendPageQueryPlan +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.BackendRecord +import me.ahoo.wow.query.backend.BackendRecordCompleteness +import me.ahoo.wow.query.backend.BackendRecordQueryPlan +import me.ahoo.wow.query.backend.BackendSingleQueryPlan +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.BackendTotalRelation +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.RecordQueryBackend +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.serialization.state.StateAggregateRecords +import org.bson.Document +import org.bson.types.Binary +import org.bson.types.Decimal128 +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.math.BigDecimal +import java.math.BigInteger +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.util.Date +import java.util.IdentityHashMap +import java.util.LinkedHashMap +import java.util.concurrent.TimeUnit + +internal class MongoRecordQueryBackend( + private val collection: MongoCollection, + private val binding: MongoPreparedQueryBinding, + private val clock: Clock = Clock.systemUTC(), +) : RecordQueryBackend { + private val compiler = MongoRecordQueryCompiler(binding) + private val mapper = MongoRecordMapper(binding) + private val pageMapper = MongoPageResultMapper(binding) + private val collation = when (binding.collationMode) { + MongoCollationMode.SIMPLE_BINARY -> Collation.builder().locale("simple").build() + } + + override fun single( + plan: BackendSingleQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = Mono.defer { + validateOptions(plan, options) + val query = compiler.compile(plan) + Mono.from(applyQuery(collection.find(query.filter), query, options).first()) + .map { source -> mapper.map(source, plan.projection) } + }.mapBackendErrors() + + override fun stream( + plan: BackendStreamQueryPlan, + options: QueryBackendExecutionOptions, + ): Flux = Flux.defer { + validateOptions(plan, options) + val query = compiler.compile(plan) + Flux.from(applyQuery(collection.find(query.filter), query, options)) + .map { source -> mapper.map(source, plan.projection) } + }.mapBackendErrors() + + override fun page( + plan: BackendPageQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = Mono.defer { + validateOptions(plan, options) + val query = compiler.compile(plan) + var publisher = collection.aggregate(query.pagePipeline()) + .collation(collation) + .allowDiskUse(options.allowDiskUse) + options.remainingMillis()?.let { remaining -> + publisher = publisher.maxTime(remaining, TimeUnit.MILLISECONDS) + } + Flux.from(publisher) + .collectList() + .map { results -> pageMapper.map(results, plan.projection) } + }.mapBackendErrors() + + override fun count( + plan: BackendCountQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = Mono.defer { + validateOptions(plan, options) + val countOptions = CountOptions().collation(collation) + options.remainingMillis()?.let { remaining -> countOptions.maxTime(remaining, TimeUnit.MILLISECONDS) } + Mono.from( + collection.countDocuments( + compiler.compile(plan).filter, + countOptions, + ), + ) + }.mapBackendErrors() + + private fun applyQuery( + source: FindPublisher, + query: MongoCompiledRecordQuery, + options: QueryBackendExecutionOptions, + ): FindPublisher { + var result = source.collation(collation).allowDiskUse(options.allowDiskUse) + options.remainingMillis()?.let { remaining -> + result = result.maxTime(remaining, TimeUnit.MILLISECONDS) + } + query.projection?.let { projection -> result = result.projection(projection) } + query.sort?.let { sort -> result = result.sort(sort) } + query.limit?.let { limit -> result = result.limit(limit) } + return result + } + + private fun validateOptions(plan: BackendRecordQueryPlan, options: QueryBackendExecutionOptions) { + requireSupportedRecordBudget(options) + validateReturnedBudget(plan, options) + if (plan is BackendPageQueryPlan) { + validatePageBudget(plan, options) + } + options.remainingMillis() + } + + private fun requireSupportedRecordBudget(options: QueryBackendExecutionOptions) { + val unsupported = listOfNotNull( + options.maxScannedRecords, + options.maxCandidateBuckets, + options.maxReturnedBuckets, + options.maxCursorPages, + ) + if (unsupported.isNotEmpty()) { + unsupportedBudget() + } + } + + private fun validateReturnedBudget(plan: BackendRecordQueryPlan, options: QueryBackendExecutionOptions) { + options.maxReturnedRecords?.let { maximum -> + val requested = when (plan) { + is BackendSingleQueryPlan -> 1L + is BackendStreamQueryPlan -> plan.limit.toLong() + is BackendPageQueryPlan -> plan.page.size.toLong() + is BackendCountQueryPlan -> 0L + } + if (requested > maximum) { + exceededBudget() + } + } + } + + private fun validatePageBudget(plan: BackendPageQueryPlan, options: QueryBackendExecutionOptions) { + options.maxPageWindow?.let { maximum -> + val endExclusive = try { + Math.addExact(plan.page.offset, plan.page.size.toLong()) + } catch (error: ArithmeticException) { + throw QueryBackendException(QueryBackendFailureKind.BUDGET_EXCEEDED, error) + } + if (endExclusive > maximum) { + exceededBudget() + } + } + } + + private fun QueryBackendExecutionOptions.remainingMillis(): Long? { + val currentDeadline = deadline ?: return null + val now = clock.instant() + val remaining = try { + Duration.between(now, currentDeadline).toMillis() + } catch (error: ArithmeticException) { + if (currentDeadline.isAfter(now)) { + Long.MAX_VALUE + } else { + throw QueryBackendException( + QueryBackendFailureKind.TIMEOUT, + error, + ) + } + } + if (remaining <= 0) { + throw QueryBackendException(QueryBackendFailureKind.TIMEOUT) + } + return remaining + } + + private fun unsupportedBudget(): Nothing = throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED) + + private fun exceededBudget(): Nothing = throw QueryBackendException(QueryBackendFailureKind.BUDGET_EXCEEDED) + + private fun Mono.mapBackendErrors(): Mono = onErrorMap(::mapBackendError) + + private fun Flux.mapBackendErrors(): Flux = onErrorMap(::mapBackendError) + + private fun mapBackendError(error: Throwable): Throwable = + if (error is QueryBackendException) error else QueryBackendException(QueryBackendFailureKind.UNAVAILABLE, error) +} + +internal class MongoPageResultMapper(binding: MongoPreparedQueryBinding) { + private val mapper = MongoRecordMapper(binding) + + fun map(results: List, projection: BackendProjection): BackendPage { + val records = ArrayList(results.size) + var total: Long? = null + results.forEach { result -> + if (result.containsKey(Documents.ID_FIELD)) { + records += mapper.map(result, projection) + } else { + if (result.keys != setOf(MongoCompiledRecordQuery.PAGE_TOTAL_VALUE) || total != null) mappingFailure() + total = (result[MongoCompiledRecordQuery.PAGE_TOTAL_VALUE] as? Number)?.toLong() + ?.takeIf { value -> value >= 0 } + ?: mappingFailure() + } + } + return BackendPage( + records, + total ?: 0L, + BackendTotalRelation.EXACT, + BackendPageConsistency.SAME_INPUT, + ) + } +} + +internal class MongoRecordMapper( + private val binding: MongoPreparedQueryBinding, + private val limits: MappingLimits = MappingLimits(), +) { + @Suppress("TooGenericExceptionCaught") + fun map( + source: Document, + projection: BackendProjection = BackendProjection.All, + ): BackendRecord = try { + mapResult(source, projection) + } catch (error: QueryBackendException) { + throw error + } catch (error: RuntimeException) { + throw QueryBackendException(QueryBackendFailureKind.MAPPING_FAILURE, error) + } + + private fun mapResult(source: Document, projection: BackendProjection): BackendRecord { + val session = MappingSession(limits) + val frozen = session.objectValue(source, 0) + val identity = (frozen.values[Documents.ID_FIELD] as? NormalizedValue.Text)?.value + ?: mappingFailure() + val document = frozen.toLogicalDocument().apply(projection, binding.identityOutputField) + return BackendRecord( + identity, + document, + BackendRecordCompleteness.COMPLETE, + ) + } + + private fun NormalizedValue.ObjectValue.toLogicalDocument(): NormalizedValue.ObjectValue { + val pathFields = binding.schema.fields.keys.filterIsInstance() + .map(QueryFieldId.Path::segments) + val logical = LinkedHashMap(include(pathFields).values) + binding.schema.fields.keys.filterIsInstance().forEach { field -> + val physicalPath = requireNotNull(binding.fields[field]).path.split('.') + valueAt(physicalPath)?.let { value -> + logical[field.outputPath(binding.identityOutputField).single()] = value + } + } + return NormalizedValue.ObjectValue(logical) + } + + private fun NormalizedValue.ObjectValue.valueAt(path: List): NormalizedValue? { + var current: NormalizedValue = this + path.forEach { segment -> + current = (current as? NormalizedValue.ObjectValue)?.values?.get(segment) ?: return null + } + return current + } + + internal data class MappingLimits( + val maxDepth: Int = 32, + val maxNodes: Int = 100_000, + val maxCollectionSize: Int = 100_000, + ) { + init { + require(maxDepth > 0 && maxNodes > 0 && maxCollectionSize > 0) + } + } + + private class MappingSession(private val limits: MappingLimits) { + private val active = IdentityHashMap() + private var nodes: Int = 0 + + fun objectValue(source: Map<*, *>, depth: Int): NormalizedValue.ObjectValue = + withContainer(source, depth) { + val values = LinkedHashMap() + var count = 0 + source.entries.forEach { entry -> + count++ + if (count > limits.maxCollectionSize) { + mappingFailure() + } + val key = entry.key as? String ?: mappingFailure() + if (values.containsKey(key)) { + mappingFailure() + } + values[key] = value(entry.value, depth + 1) + } + NormalizedValue.ObjectValue(values) + } + + private fun value(source: Any?, depth: Int): NormalizedValue = + when (source) { + is Map<*, *> -> objectValue(source, depth) + is Iterable<*> -> listValue(source, depth) + is Array<*> -> listValue(source.asIterable(), depth) + else -> scalarValue(source, depth) + } + + private fun scalarValue(source: Any?, depth: Int): NormalizedValue { + enterNode(depth) + return when (source) { + null -> NormalizedValue.Null + is Boolean -> NormalizedValue.BooleanValue(source) + is String -> NormalizedValue.Text(source) + is Decimal128 -> NormalizedValue.Decimal(source.bigDecimalValue()) + is Number -> numberValue(source) + is Instant -> NormalizedValue.InstantValue(source) + is Date -> NormalizedValue.InstantValue(source.toInstant()) + is ByteArray -> NormalizedValue.Bytes(source) + is Binary -> NormalizedValue.Bytes(source.data) + else -> mappingFailure() + } + } + + private fun numberValue(source: Number): NormalizedValue = when (source) { + is Byte, is Short, is Int, is Long -> NormalizedValue.Int64(source.toLong()) + is BigInteger -> source.longValueExactOrDecimal() + is BigDecimal -> NormalizedValue.Decimal(source) + is Float, is Double -> { + val value = source.toDouble() + if (!value.isFinite()) mappingFailure() + NormalizedValue.Decimal(BigDecimal.valueOf(value)) + } + + else -> mappingFailure() + } + + private fun listValue(source: Iterable<*>, depth: Int): NormalizedValue.ListValue = + withContainer(source, depth) { + val values = ArrayList() + val iterator = source.iterator() + while (iterator.hasNext()) { + if (values.size >= limits.maxCollectionSize) { + mappingFailure() + } + values += value(iterator.next(), depth + 1) + } + NormalizedValue.ListValue(values) + } + + private fun enterNode(depth: Int) { + if (depth > limits.maxDepth || ++nodes > limits.maxNodes) { + mappingFailure() + } + } + + private fun withContainer(source: Any, depth: Int, block: () -> T): T { + enterNode(depth) + if (active.put(source, Unit) != null) { + mappingFailure() + } + return try { + block() + } finally { + active.remove(source) + } + } + + private fun BigInteger.longValueExactOrDecimal(): NormalizedValue = + try { + NormalizedValue.Int64(longValueExact()) + } catch (_: ArithmeticException) { + NormalizedValue.Decimal(BigDecimal(this)) + } + } +} + +private fun NormalizedValue.ObjectValue.apply( + projection: BackendProjection, + identityOutputField: String, +): NormalizedValue.ObjectValue = + when (projection) { + BackendProjection.All -> this + is BackendProjection.Include -> include( + projection.fields.map { field -> field.outputPath(identityOutputField) } + ) + is BackendProjection.Exclude -> exclude( + projection.fields.map { field -> field.outputPath(identityOutputField) } + ) + } + +private fun QueryFieldId.outputPath(identityOutputField: String): List = + when (this) { + is QueryFieldId.Path -> segments + is QueryFieldId.System -> listOf( + when (kind) { + SystemFieldKind.IDENTITY -> identityOutputField + SystemFieldKind.AGGREGATE_ID -> MessageRecords.AGGREGATE_ID + + SystemFieldKind.TENANT_ID -> MessageRecords.TENANT_ID + SystemFieldKind.OWNER_ID -> MessageRecords.OWNER_ID + SystemFieldKind.SPACE_ID -> MessageRecords.SPACE_ID + SystemFieldKind.DELETED -> StateAggregateRecords.DELETED + }, + ) + } + +private fun NormalizedValue.ObjectValue.include(paths: List>): NormalizedValue.ObjectValue { + val result = LinkedHashMap() + values.forEach { (key, value) -> + val matching = paths.filter { path -> path.firstOrNull() == key } + if (matching.any { path -> path.size == 1 }) { + result[key] = value + } else if (matching.isNotEmpty()) { + result[key] = value.includeNested(matching.map { path -> path.drop(1) }) + } + } + return NormalizedValue.ObjectValue(result) +} + +private fun NormalizedValue.includeNested(paths: List>): NormalizedValue = + when (this) { + NormalizedValue.Null -> NormalizedValue.Null + is NormalizedValue.ObjectValue -> include(paths) + is NormalizedValue.ListValue -> NormalizedValue.ListValue(values.map { value -> value.includeNested(paths) }) + else -> mappingFailure() + } + +private fun NormalizedValue.ObjectValue.exclude(paths: List>): NormalizedValue.ObjectValue { + val result = LinkedHashMap() + values.forEach { (key, value) -> + val matching = paths.filter { path -> path.firstOrNull() == key } + if (matching.none { path -> path.size == 1 }) { + val nested = matching.filter { path -> path.size > 1 }.map { path -> path.drop(1) } + result[key] = if (nested.isEmpty()) value else value.excludeNested(nested) + } + } + return NormalizedValue.ObjectValue(result) +} + +private fun NormalizedValue.excludeNested(paths: List>): NormalizedValue = + when (this) { + is NormalizedValue.ObjectValue -> exclude(paths) + is NormalizedValue.ListValue -> NormalizedValue.ListValue(values.map { value -> value.excludeNested(paths) }) + else -> this + } + +private fun mappingFailure(): Nothing = throw QueryBackendException(QueryBackendFailureKind.MAPPING_FAILURE) diff --git a/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoAnalyticsQueryBackendTest.kt b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoAnalyticsQueryBackendTest.kt new file mode 100644 index 00000000000..0fdb98f472d --- /dev/null +++ b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoAnalyticsQueryBackendTest.kt @@ -0,0 +1,327 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.MongoNamespace +import com.mongodb.reactivestreams.client.AggregatePublisher +import com.mongodb.reactivestreams.client.MongoCollection +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.mongo.Documents +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.BackendAnalyticsBucketOrder +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsCondition +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsDimension +import me.ahoo.wow.query.backend.BackendAnalyticsGrouping +import me.ahoo.wow.query.backend.BackendAnalyticsMetric +import me.ahoo.wow.query.backend.BackendAnalyticsMissingPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNullPlacement +import me.ahoo.wow.query.backend.BackendAnalyticsNumericPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNumericPromotion +import me.ahoo.wow.query.backend.BackendAnalyticsOverflowPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsPageWindow +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.BackendAnalyticsTextCollation +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendRequiredCapabilities +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import org.bson.Document +import org.bson.types.Decimal128 +import org.junit.jupiter.api.Test +import reactor.test.StepVerifier +import reactor.test.publisher.TestPublisher +import java.lang.reflect.Proxy +import java.math.BigDecimal +import java.math.RoundingMode +import java.time.Clock +import java.time.Instant +import java.time.ZoneOffset +import java.util.concurrent.atomic.AtomicInteger +import java.util.concurrent.atomic.AtomicLong +import java.util.function.Consumer + +class MongoAnalyticsQueryBackendTest { + @Test + fun `result mapper should apply Decimal128 policy and synthesize one empty global bucket`() { + val mapper = MongoAnalyticsResultMapper(binding.prepared) + val plan = plan( + listOf( + BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count")), + BackendAnalyticsMetric.Sum(AnalyticsAlias("total"), amount), + BackendAnalyticsMetric.Average(AnalyticsAlias("average"), amount), + ), + ) + + val mapped = mapper.map( + Document("_id", null) + .append("count", 2L) + .append("total", Decimal128(BigDecimal("20.125"))) + .append("average", Decimal128(BigDecimal("10.0625"))), + plan, + ) + mapped.metrics[AnalyticsAlias("count")].assert().isEqualTo(NormalizedValue.Int64(2)) + mapped.metrics[AnalyticsAlias("total")].assert() + .isEqualTo(NormalizedValue.Decimal(BigDecimal("20.13"))) + mapped.metrics[AnalyticsAlias("average")].assert() + .isEqualTo(NormalizedValue.Decimal(BigDecimal("10.06"))) + + val empty = mapper.emptyGlobal(plan) + empty.metrics[AnalyticsAlias("count")].assert().isEqualTo(NormalizedValue.Int64(0)) + empty.metrics[AnalyticsAlias("total")].assert().isEqualTo(NormalizedValue.Decimal(BigDecimal.ZERO)) + empty.metrics[AnalyticsAlias("average")].assert().isEqualTo(NormalizedValue.Null) + } + + @Test + fun `result mapper should reject values exceeding the declared Decimal128 precision`() { + val mapper = MongoAnalyticsResultMapper(binding.prepared) + val policy = BackendAnalyticsNumericPolicy( + BackendAnalyticsNumericPromotion.DECIMAL128, + 3, + 0, + RoundingMode.UNNECESSARY, + BackendAnalyticsOverflowPolicy.REJECT, + ) + val plan = plan( + listOf(BackendAnalyticsMetric.Sum(AnalyticsAlias("total"), amount)), + policy, + ) + + assertBackendFailure(QueryBackendFailureKind.MAPPING_FAILURE) { + mapper.map( + Document("_id", null).append("total", Decimal128(BigDecimal("1234"))), + plan, + ) + } + } + + @Test + fun `unsupported exceeded and expired analytics budgets should fail before Mongo access`() { + val storageCalls = AtomicInteger() + val backend = MongoAnalyticsQueryBackend( + rejectingCollection(storageCalls), + binding.prepared, + Clock.fixed(NOW, ZoneOffset.UTC), + ) + val plan = plan(listOf(BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count")))) + + listOf( + QueryBackendFailureKind.UNSUPPORTED to QueryBackendExecutionOptions(null, null, maxScannedRecords = 1), + QueryBackendFailureKind.UNSUPPORTED to QueryBackendExecutionOptions(null, null, maxCandidateBuckets = 1), + QueryBackendFailureKind.UNSUPPORTED to QueryBackendExecutionOptions(null, null, maxCursorPages = 1), + QueryBackendFailureKind.BUDGET_EXCEEDED to QueryBackendExecutionOptions( + null, + null, + maxReturnedBuckets = 1, + ), + QueryBackendFailureKind.TIMEOUT to QueryBackendExecutionOptions(NOW, null), + ).forEach { (kind, options) -> + assertBackendFailure(kind) { backend.analyze(groupedPlan(plan, 2), options).block() } + } + storageCalls.get().assert().isZero() + } + + @Test + fun `analytics backend should propagate remaining deadline and cancellation to Mongo publisher`() { + val publisher = TestPublisher.createCold() + val maxTimeMillis = AtomicLong(-1) + val backend = MongoAnalyticsQueryBackend( + aggregateCollection(publisher, maxTimeMillis), + binding.prepared, + Clock.fixed(NOW, ZoneOffset.UTC), + ) + + StepVerifier.create( + backend.analyze( + plan(listOf(BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count")))), + QueryBackendExecutionOptions(NOW.plusSeconds(2), null), + ), + ).thenCancel().verify() + + maxTimeMillis.get().assert().isEqualTo(2_000) + publisher.assertWasSubscribed() + publisher.assertCancelled() + } + + private fun groupedPlan(source: BackendAnalyticsQueryPlan, limit: Int) = BackendAnalyticsQueryPlan( + source.target, + source.schemaContractId, + source.filter, + BackendAnalyticsGrouping.By( + listOf( + BackendAnalyticsDimension( + AnalyticsAlias("amount"), + amount, + BackendAnalyticsMissingPolicy.AS_NULL_BUCKET, + ), + ), + ), + source.metrics, + source.having, + BackendAnalyticsBucketOrder.DimensionKeyAscending( + BackendAnalyticsNullPlacement.FIRST, + BackendAnalyticsTextCollation.BINARY, + ), + BackendAnalyticsPageWindow(limit), + source.numericPolicy, + source.requiredConsistency, + source.requiredCompleteness, + source.requiredCapabilities, + source.semanticTier, + source.fingerprint, + ) + + private fun plan( + metrics: List, + numericPolicy: BackendAnalyticsNumericPolicy? = NUMERIC_POLICY.takeIf { + metrics.any { metric -> metric !is BackendAnalyticsMetric.DocumentCount } + }, + ) = BackendAnalyticsQueryPlan( + target, + schema.contractId, + BackendEnforcedFilter(BackendPlannedCondition.All, BackendPlannedCondition.All), + BackendAnalyticsGrouping.Global, + metrics, + BackendAnalyticsCondition.All, + BackendAnalyticsBucketOrder.Global, + BackendAnalyticsPageWindow(1), + numericPolicy, + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("6".repeat(64)), + ) + + private fun assertBackendFailure(kind: QueryBackendFailureKind, action: () -> Unit) { + assertThrownBy(action).satisfies( + Consumer { error -> error.kind.assert().isEqualTo(kind) }, + ) + } + + @Suppress("UNCHECKED_CAST") + private fun rejectingCollection(calls: AtomicInteger): MongoCollection = Proxy.newProxyInstance( + MongoCollection::class.java.classLoader, + arrayOf(MongoCollection::class.java), + ) { _, method, _ -> + calls.incrementAndGet() + error("Mongo collection method ${method.name} must not be called before budget validation.") + } as MongoCollection + + @Suppress("UNCHECKED_CAST") + private fun aggregateCollection( + publisher: TestPublisher, + maxTimeMillis: AtomicLong, + ): MongoCollection { + lateinit var aggregate: AggregatePublisher + aggregate = Proxy.newProxyInstance( + AggregatePublisher::class.java.classLoader, + arrayOf(AggregatePublisher::class.java), + ) { _, method, arguments -> + when (method.name) { + "allowDiskUse", "collation" -> aggregate + "maxTime" -> aggregate.also { + maxTimeMillis.set((arguments!![0] as Number).toLong()) + } + + "subscribe" -> { + publisher.subscribe(arguments!![0] as org.reactivestreams.Subscriber) + null + } + + else -> error("Unexpected AggregatePublisher method: ${method.name}") + } + } as AggregatePublisher + return Proxy.newProxyInstance( + MongoCollection::class.java.classLoader, + arrayOf(MongoCollection::class.java), + ) { _, method, _ -> + when (method.name) { + "aggregate" -> aggregate + else -> error("Unexpected MongoCollection method: ${method.name}") + } + } as MongoCollection + } + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + private val amount = QueryFieldId.Path(listOf("amount")) + private val schema = QueryDocumentSchema( + target, + listOf( + QueryFieldSchema( + identity, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + emptySet(), + setOf(FieldCapability.EXACT), + ), + QueryFieldSchema( + amount, + LogicalFieldType.Decimal, + Presence.OPTIONAL, + Nullability.NULLABLE, + emptySet(), + setOf(FieldCapability.AGGREGATABLE), + ), + ), + emptyList(), + ) + private val binding = MongoSnapshotQueryBinding( + schema, + MongoNamespace("sales", "order_snapshot"), + mapOf( + identity to MongoFieldBinding(Documents.ID_FIELD, setOf(FieldCapability.EXACT)), + amount to MongoFieldBinding("amount", setOf(FieldCapability.AGGREGATABLE), MongoValueEncoding.DECIMAL128), + ), + ) + + private companion object { + val NOW: Instant = Instant.parse("2026-08-08T00:00:00Z") + val NUMERIC_POLICY = BackendAnalyticsNumericPolicy( + BackendAnalyticsNumericPromotion.DECIMAL128, + 34, + 2, + RoundingMode.HALF_UP, + BackendAnalyticsOverflowPolicy.REJECT, + ) + } +} diff --git a/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoAnalyticsQueryCompilerTest.kt b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoAnalyticsQueryCompilerTest.kt new file mode 100644 index 00000000000..860d5653b43 --- /dev/null +++ b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoAnalyticsQueryCompilerTest.kt @@ -0,0 +1,274 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.MongoNamespace +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.mongo.Documents +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.BackendAnalyticsBucketOrder +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsCondition +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsDimension +import me.ahoo.wow.query.backend.BackendAnalyticsGrouping +import me.ahoo.wow.query.backend.BackendAnalyticsMetric +import me.ahoo.wow.query.backend.BackendAnalyticsMissingPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNullPlacement +import me.ahoo.wow.query.backend.BackendAnalyticsNumericPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNumericPromotion +import me.ahoo.wow.query.backend.BackendAnalyticsOverflowPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsPageWindow +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.BackendAnalyticsTextCollation +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendRequiredCapabilities +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.serialization.state.StateAggregateRecords +import org.bson.BsonDocument +import org.junit.jupiter.api.Test +import java.math.BigDecimal +import java.math.RoundingMode +import java.util.function.Consumer + +class MongoAnalyticsQueryCompilerTest { + private val fixture = Fixture() + private val compiler = MongoAnalyticsQueryCompiler(fixture.binding.prepared) + + @Test + fun `global pipeline should preserve enforced filter and exact numeric metrics`() { + val filter = BackendEnforcedFilter( + fixture.predicate(fixture.status, NormalizedValue.Text("PAID")), + fixture.predicate(fixture.tenant, NormalizedValue.Text("tenant-1")), + ) + val compiled = compiler.compile( + fixture.plan( + BackendAnalyticsGrouping.Global, + listOf( + BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count")), + BackendAnalyticsMetric.Sum(AnalyticsAlias("total"), fixture.amount), + ), + filter = filter, + ), + ) + val pipeline = compiled.pipeline.map { stage -> stage.toBsonDocument() } + + pipeline.map { stage -> stage.keys.single() }.assert().containsExactly("\$match", "\$group", "\$limit") + pipeline[0].getDocument("\$match").toJson().assert().contains(MessageRecords.TENANT_ID) + val group = pipeline[1].getDocument("\$group") + requireNotNull(group["_id"]).isNull.assert().isTrue() + group.getDocument("count").getInt32("\$sum").value.assert().isEqualTo(1) + group.getDocument("total").getDocument("\$sum").getString("\$toDecimal").value.assert() + .isEqualTo("\$state.amount") + pipeline[2].getInt32("\$limit").value.assert().isEqualTo(2) + } + + @Test + fun `grouped pipeline should apply missing policy compound cursor and stable order`() { + val plan = fixture.plan( + BackendAnalyticsGrouping.By( + listOf( + BackendAnalyticsDimension( + AnalyticsAlias("status"), + fixture.status, + BackendAnalyticsMissingPolicy.EXCLUDE, + ), + BackendAnalyticsDimension( + AnalyticsAlias("amount"), + fixture.amount, + BackendAnalyticsMissingPolicy.AS_NULL_BUCKET, + ), + ), + ), + listOf( + BackendAnalyticsMetric.Min(AnalyticsAlias("minimum"), fixture.amount), + BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count")), + ), + window = BackendAnalyticsPageWindow( + 2, + listOf(NormalizedValue.Text("PAID"), NormalizedValue.Decimal(BigDecimal.TEN)), + ), + ) + val pipeline = compiler.compile(plan).pipeline.map { stage -> stage.toBsonDocument() } + + pipeline.map { stage -> stage.keys.single() }.assert().containsExactly( + "\$match", + "\$match", + "\$group", + "\$match", + "\$sort", + "\$limit", + ) + pipeline[1].getDocument("\$match").toJson().assert().contains("state.status") + val groupId = pipeline[2].getDocument("\$group").getDocument("_id") + groupId.getString("status").value.assert().isEqualTo("\$state.status") + groupId.getDocument("amount").getArray("\$ifNull").values.last().isNull.assert().isTrue() + val cursor = pipeline[3].getDocument("\$match").getArray("\$or") + cursor.size.assert().isEqualTo(2) + cursor.values[1].asDocument().getString("_id.status").value.assert().isEqualTo("PAID") + cursor.values[1].asDocument().getDocument("_id.amount").getDecimal128("\$gt").value + .bigDecimalValue().compareTo(BigDecimal.TEN).assert().isZero() + pipeline[4].getDocument("\$sort").assertKeys("_id.status", "_id.amount") + pipeline[5].getInt32("\$limit").value.assert().isEqualTo(3) + } + + @Test + fun `cursor should retain the exact canonical dimension type`() { + val plan = fixture.plan( + BackendAnalyticsGrouping.By( + listOf( + BackendAnalyticsDimension( + AnalyticsAlias("amount"), + fixture.amount, + BackendAnalyticsMissingPolicy.AS_NULL_BUCKET, + ), + ), + ), + listOf(BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + window = BackendAnalyticsPageWindow(2, listOf(NormalizedValue.Text("10"))), + ) + + assertThrownBy { compiler.compile(plan) }.satisfies( + Consumer { error -> error.kind.assert().isEqualTo(QueryBackendFailureKind.UNSUPPORTED) }, + ) + } + + private fun BsonDocument.assertKeys(vararg keys: String) { + this.keys.toList().assert().containsExactly(*keys) + } + + private class Fixture { + val target = QueryTarget(MaterializedNamedAggregate("sales", "order"), QueryDocumentKind.SNAPSHOT) + val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + val tenant = QueryFieldId.System(SystemFieldKind.TENANT_ID) + val deleted = QueryFieldId.System(SystemFieldKind.DELETED) + val state = QueryFieldId.Path(listOf("state")) + val status = QueryFieldId.Path(listOf("state", "status")) + val amount = QueryFieldId.Path(listOf("state", "amount")) + val schema = QueryDocumentSchema( + target, + listOf( + field(identity, LogicalFieldType.Text, FieldCapability.EXACT), + field(tenant, LogicalFieldType.Text, FieldCapability.EXACT), + field(deleted, LogicalFieldType.Boolean, FieldCapability.EXACT), + field(state, LogicalFieldType.Object), + field(status, LogicalFieldType.Text, FieldCapability.EXACT, FieldCapability.AGGREGATABLE), + field(amount, LogicalFieldType.Decimal, FieldCapability.AGGREGATABLE), + ), + emptyList(), + ) + val binding = MongoSnapshotQueryBinding( + schema, + MongoNamespace("sales", "order_snapshot"), + linkedMapOf( + identity to MongoFieldBinding(Documents.ID_FIELD, setOf(FieldCapability.EXACT)), + tenant to MongoFieldBinding(MessageRecords.TENANT_ID, setOf(FieldCapability.EXACT)), + deleted to MongoFieldBinding(StateAggregateRecords.DELETED, setOf(FieldCapability.EXACT)), + state to MongoFieldBinding("state", emptySet()), + status to MongoFieldBinding( + "state.status", + setOf(FieldCapability.EXACT, FieldCapability.AGGREGATABLE), + ), + amount to MongoFieldBinding( + "state.amount", + setOf(FieldCapability.AGGREGATABLE), + MongoValueEncoding.DECIMAL128, + ), + ), + ) + + fun predicate(field: QueryFieldId, value: NormalizedValue) = BackendPlannedCondition.Predicate( + field, + PredicateOperator.EQ, + value, + ) + + fun plan( + grouping: BackendAnalyticsGrouping, + metrics: List, + filter: BackendEnforcedFilter = BackendEnforcedFilter( + BackendPlannedCondition.All, + BackendPlannedCondition.All, + ), + window: BackendAnalyticsPageWindow = BackendAnalyticsPageWindow(1), + ) = BackendAnalyticsQueryPlan( + target, + schema.contractId, + filter, + grouping, + metrics, + BackendAnalyticsCondition.All, + when (grouping) { + BackendAnalyticsGrouping.Global -> BackendAnalyticsBucketOrder.Global + is BackendAnalyticsGrouping.By -> BackendAnalyticsBucketOrder.DimensionKeyAscending( + BackendAnalyticsNullPlacement.FIRST, + BackendAnalyticsTextCollation.BINARY, + ) + }, + window, + BackendAnalyticsNumericPolicy( + BackendAnalyticsNumericPromotion.DECIMAL128, + 34, + 2, + RoundingMode.HALF_UP, + BackendAnalyticsOverflowPolicy.REJECT, + ), + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("5".repeat(64)), + ) + + private fun field( + id: QueryFieldId, + type: LogicalFieldType, + vararg capabilities: FieldCapability, + ): QueryFieldSchema { + val capabilitySet = capabilities.toSet() + return QueryFieldSchema( + id, + type, + Presence.OPTIONAL, + Nullability.NULLABLE, + if (FieldCapability.EXACT in capabilitySet) setOf(PredicateOperator.EQ) else emptySet(), + capabilitySet, + ) + } + } +} diff --git a/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoEventStreamQueryBindingTest.kt b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoEventStreamQueryBindingTest.kt new file mode 100644 index 00000000000..7e966d51682 --- /dev/null +++ b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoEventStreamQueryBindingTest.kt @@ -0,0 +1,164 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.MongoNamespace +import com.mongodb.client.model.Filters +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.mongo.Documents +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.BackendRequiredCapabilities +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import org.bson.Document +import org.junit.jupiter.api.Test + +class MongoEventStreamQueryBindingTest { + @Test + fun `event stream system fields should bind without snapshot deletion semantics`() { + val binding = MongoEventStreamQueryBinding.frameworkFields(schema(), namespace) + + binding.fields.getValue(identity).path.assert().isEqualTo(Documents.ID_FIELD) + binding.fields.getValue(aggregateId).path.assert().isEqualTo(MessageRecords.AGGREGATE_ID) + binding.fields.containsKey(deleted).assert().isFalse() + + val plan = BackendCountQueryPlan( + target, + binding.schema.contractId, + BackendEnforcedFilter( + BackendPlannedCondition.Predicate( + aggregateId, + PredicateOperator.EQ, + NormalizedValue.Text("order-1"), + ), + BackendPlannedCondition.Predicate( + tenant, + PredicateOperator.EQ, + NormalizedValue.Text("tenant-1"), + ), + ), + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("5".repeat(64)), + ) + + MongoRecordQueryCompiler(binding).compile(plan).filter.toBsonDocument().assert().isEqualTo( + Filters.and( + Filters.eq(MessageRecords.AGGREGATE_ID, "order-1"), + Filters.eq(MessageRecords.TENANT_ID, "tenant-1"), + ).toBsonDocument(), + ) + } + + @Test + fun `event stream mapper should restore id independently from aggregate id`() { + val binding = MongoEventStreamQueryBinding.frameworkFields(schema(), namespace) + val source = Document( + linkedMapOf( + Documents.ID_FIELD to "stream-1", + MessageRecords.AGGREGATE_ID to "order-1", + MessageRecords.TENANT_ID to "tenant-1", + ), + ) + + val record = MongoRecordMapper(binding.prepared).map(source) + record.identity.assert().isEqualTo("stream-1") + (record.document.values.getValue(MessageRecords.ID) as NormalizedValue.Text).value.assert() + .isEqualTo("stream-1") + (record.document.values.getValue(MessageRecords.AGGREGATE_ID) as NormalizedValue.Text).value.assert() + .isEqualTo("order-1") + + val excluded = MongoRecordMapper(binding.prepared).map( + source, + BackendProjection.Exclude(listOf(identity)), + ) + excluded.document.values.containsKey(MessageRecords.ID).assert().isFalse() + excluded.document.values.containsKey(MessageRecords.AGGREGATE_ID).assert().isTrue() + } + + @Test + fun `event stream schema should reject snapshot deleted field`() { + assertThrownBy { + MongoEventStreamQueryBinding.frameworkFields(schema(includeDeleted = true), namespace) + } + } + + private fun schema(includeDeleted: Boolean = false): QueryDocumentSchema = QueryDocumentSchema( + target, + buildList { + add( + textField(identity, setOf(FieldCapability.EXACT, FieldCapability.SORTABLE, FieldCapability.PROJECTABLE)) + ) + add(textField(aggregateId, setOf(FieldCapability.EXACT, FieldCapability.PROJECTABLE))) + add(textField(tenant, setOf(FieldCapability.EXACT, FieldCapability.PROJECTABLE))) + if (includeDeleted) { + add( + QueryFieldSchema( + deleted, + LogicalFieldType.Boolean, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.IS_FALSE), + setOf(FieldCapability.EXACT), + ), + ) + } + }, + emptyList(), + ) + + private fun textField(id: QueryFieldId, capabilities: Set) = QueryFieldSchema( + id, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ), + capabilities, + ) + + private companion object { + val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.EVENT_STREAM, + ) + val namespace = MongoNamespace("sales", "order_event_stream") + val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + val aggregateId = QueryFieldId.System(SystemFieldKind.AGGREGATE_ID) + val tenant = QueryFieldId.System(SystemFieldKind.TENANT_ID) + val deleted = QueryFieldId.System(SystemFieldKind.DELETED) + } +} diff --git a/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoRecordQueryBackendBudgetTest.kt b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoRecordQueryBackendBudgetTest.kt new file mode 100644 index 00000000000..b6fcf0b5f13 --- /dev/null +++ b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoRecordQueryBackendBudgetTest.kt @@ -0,0 +1,170 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.MongoNamespace +import com.mongodb.reactivestreams.client.MongoCollection +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPageQueryPlan +import me.ahoo.wow.query.backend.BackendPageWindow +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.BackendRequiredCapabilities +import me.ahoo.wow.query.backend.BackendRequiredConsistency +import me.ahoo.wow.query.backend.BackendSort +import me.ahoo.wow.query.backend.BackendSortOrigin +import me.ahoo.wow.query.backend.BackendTotalMode +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedSortDirection +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.RecordResultShape +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import org.bson.Document +import org.junit.jupiter.api.Test +import java.lang.reflect.Proxy +import java.time.Clock +import java.time.Instant +import java.time.ZoneOffset +import java.util.concurrent.atomic.AtomicInteger +import java.util.function.Consumer + +class MongoRecordQueryBackendBudgetTest { + @Test + fun `unsupported and exceeded budgets should fail before obtaining a Mongo publisher`() { + val storageCalls = AtomicInteger() + val backend = MongoRecordQueryBackend( + rejectingCollection(storageCalls), + binding.prepared, + Clock.fixed(NOW, ZoneOffset.UTC), + ) + + assertBackendFailure(QueryBackendFailureKind.UNSUPPORTED) { + backend.count(countPlan, QueryBackendExecutionOptions(null, null, maxScannedRecords = 1)).block() + } + assertBackendFailure(QueryBackendFailureKind.BUDGET_EXCEEDED) { + backend.page(pagePlan, QueryBackendExecutionOptions(null, 1)).block() + } + assertBackendFailure(QueryBackendFailureKind.BUDGET_EXCEEDED) { + backend.page( + pagePlan, + QueryBackendExecutionOptions(null, null, maxPageWindow = 1), + ).block() + } + assertBackendFailure(QueryBackendFailureKind.TIMEOUT) { + backend.count( + countPlan, + QueryBackendExecutionOptions(NOW, null), + ).block() + } + + storageCalls.get().assert().isZero() + } + + private fun assertBackendFailure(kind: QueryBackendFailureKind, action: () -> Unit) { + assertThrownBy(action).satisfies( + Consumer { error -> error.kind.assert().isEqualTo(kind) }, + ) + } + + @Suppress("UNCHECKED_CAST") + private fun rejectingCollection(calls: AtomicInteger): MongoCollection = Proxy.newProxyInstance( + MongoCollection::class.java.classLoader, + arrayOf(MongoCollection::class.java), + ) { _, method, _ -> + calls.incrementAndGet() + error("Mongo collection method ${method.name} must not be called before budget validation.") + } as MongoCollection + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + private val deleted = QueryFieldId.System(SystemFieldKind.DELETED) + private val schema = QueryDocumentSchema( + target, + listOf( + QueryFieldSchema( + identity, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + listOf(PredicateOperator.EQ), + listOf(FieldCapability.EXACT, FieldCapability.SORTABLE), + ), + QueryFieldSchema( + deleted, + LogicalFieldType.Boolean, + Presence.REQUIRED, + Nullability.NON_NULL, + listOf(PredicateOperator.IS_FALSE), + listOf(FieldCapability.EXACT), + ), + ), + emptyList(), + ) + private val binding = MongoSnapshotQueryBinding.frameworkFields( + schema, + MongoNamespace("test", "order_snapshot"), + ) + private val filter = BackendEnforcedFilter(BackendPlannedCondition.All, BackendPlannedCondition.All) + private val countPlan = BackendCountQueryPlan( + target, + schema.contractId, + filter, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("1".repeat(64)), + ) + private val pagePlan = BackendPageQueryPlan( + target, + schema.contractId, + filter, + RecordResultShape.DYNAMIC, + BackendProjection.All, + listOf(BackendSort(identity, NormalizedSortDirection.ASC, BackendSortOrigin.STABILITY_TIE_BREAKER)), + BackendPageWindow(1, 2), + BackendTotalMode.EXACT, + BackendRequiredConsistency.SAME_INPUT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("2".repeat(64)), + ) + + private companion object { + val NOW: Instant = Instant.parse("2024-01-01T00:00:00Z") + } +} diff --git a/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoRecordQueryCompilerTest.kt b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoRecordQueryCompilerTest.kt new file mode 100644 index 00000000000..df696e62e53 --- /dev/null +++ b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoRecordQueryCompilerTest.kt @@ -0,0 +1,536 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.MongoNamespace +import com.mongodb.client.model.Filters +import com.mongodb.client.model.Projections +import com.mongodb.client.model.Sorts +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.mongo.Documents +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPageQueryPlan +import me.ahoo.wow.query.backend.BackendPageWindow +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.BackendRequiredCapabilities +import me.ahoo.wow.query.backend.BackendRequiredConsistency +import me.ahoo.wow.query.backend.BackendSingleQueryPlan +import me.ahoo.wow.query.backend.BackendSort +import me.ahoo.wow.query.backend.BackendSortOrigin +import me.ahoo.wow.query.backend.BackendTotalMode +import me.ahoo.wow.query.backend.CaseSensitivity +import me.ahoo.wow.query.backend.EmptyArraySemantics +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedPredicateOptions +import me.ahoo.wow.query.backend.NormalizedSortDirection +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.QuerySearchScopeDefinition +import me.ahoo.wow.query.backend.RecordResultShape +import me.ahoo.wow.query.backend.SearchScopeId +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.serialization.state.StateAggregateRecords +import org.bson.Document +import org.bson.conversions.Bson +import org.bson.types.Decimal128 +import org.junit.jupiter.api.DynamicTest +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.TestFactory +import java.math.BigDecimal +import java.time.Instant +import java.util.function.Consumer + +class MongoRecordQueryCompilerTest { + private val fixture = Fixture() + private val compiler = MongoRecordQueryCompiler(fixture.binding) + + @Test + fun `should compile user and mandatory predicates from logical binding`() { + val plan = fixture.plan( + user = fixture.predicate(fixture.name, PredicateOperator.EQ, NormalizedValue.Text("Alice")), + mandatory = fixture.predicate(fixture.tenant, PredicateOperator.EQ, NormalizedValue.Text("tenant-1")), + ) + + compiler.compile(plan).filter.toBsonDocument().assert().isEqualTo( + Filters.and( + Filters.eq("state.name", "Alice"), + Filters.eq(MessageRecords.TENANT_ID, "tenant-1"), + ).toBsonDocument(), + ) + } + + @Test + fun `should make element match child path relative to current element`() { + val element = BackendPlannedCondition.ElementMatch( + fixture.items, + fixture.predicate(fixture.itemName, PredicateOperator.EQ, NormalizedValue.Text("book")), + ) + + compiler.compile(fixture.plan(element)).filter.toBsonDocument().assert().isEqualTo( + Filters.and( + Filters.elemMatch("state.items", Filters.eq("name", "book")), + Filters.empty(), + ).toBsonDocument(), + ) + } + + @Test + fun `should preserve literal regex and encode instant as epoch millis`() { + val condition = BackendPlannedCondition.Junction( + me.ahoo.wow.query.backend.JunctionOperator.AND, + listOf( + fixture.predicate(fixture.name, PredicateOperator.CONTAINS, NormalizedValue.Text("a.*?\\b")), + fixture.predicate( + fixture.createdAt, + PredicateOperator.GTE, + NormalizedValue.InstantValue(Instant.ofEpochMilli(1234)), + ), + ), + ) + + compiler.compile(fixture.plan(condition)).filter.toBsonDocument().assert().isEqualTo( + Filters.and( + Filters.and( + Filters.regex("state.name", "a\\.\\*\\?\\\\b"), + Filters.gte("state.createdAt", 1234L), + ), + Filters.empty(), + ).toBsonDocument(), + ) + } + + @Test + fun `should encode Decimal128 exactly and reject out of range decimal before storage`() { + compiler.compile( + fixture.plan( + user = fixture.predicate( + fixture.amount, + PredicateOperator.EQ, + NormalizedValue.Decimal(BigDecimal("1.00")), + ), + ), + ).filter.toBsonDocument().assert().isEqualTo( + Filters.and(Filters.eq("state.amount", Decimal128(BigDecimal.ONE)), Filters.empty()).toBsonDocument(), + ) + + assertThrownBy { + compiler.compile( + fixture.plan( + user = fixture.predicate( + fixture.amount, + PredicateOperator.EQ, + NormalizedValue.Decimal(BigDecimal("1E+7000")), + ), + ), + ) + }.satisfies( + Consumer { error -> error.kind.assert().isEqualTo(QueryBackendFailureKind.UNSUPPORTED) }, + ) + } + + @Test + fun `should force identity fetch while preserving logical projection and sort`() { + val plan = fixture.plan( + projection = BackendProjection.Include(listOf(fixture.name)), + sort = listOf( + BackendSort(fixture.name, NormalizedSortDirection.DESC, BackendSortOrigin.USER), + BackendSort(fixture.identity, NormalizedSortDirection.ASC, BackendSortOrigin.STABILITY_TIE_BREAKER), + ), + ) + val compiled = compiler.compile(plan) + + compiled.projection?.toBsonDocument().assert().isEqualTo( + Projections.include(listOf("state.name", Documents.ID_FIELD)).toBsonDocument(), + ) + compiled.sort?.toBsonDocument().assert().isEqualTo( + Sorts.orderBy(Sorts.descending("state.name"), Sorts.ascending(Documents.ID_FIELD)).toBsonDocument(), + ) + } + + @Test + fun `should canonicalize ancestor projection before creating Mongo projection`() { + val compiled = compiler.compile( + fixture.plan(projection = BackendProjection.Include(listOf(fixture.state, fixture.name))), + ) + + compiled.projection?.toBsonDocument().assert().isEqualTo( + Projections.include(listOf("state", Documents.ID_FIELD)).toBsonDocument(), + ) + } + + @Test + fun `should preserve one exact same-input page without a collection union target`() { + val compiled = compiler.compile(fixture.pagePlan(offset = 3, size = 2)) + + compiled.page.assert().isEqualTo(BackendPageWindow(3, 2)) + (compiled.limit == null).assert().isTrue() + val pipeline = compiled.pagePipeline().map(Bson::toBsonDocument) + pipeline.none { stage -> stage.containsKey("\$facet") }.assert().isTrue() + pipeline.none { stage -> stage.containsKey("\$skip") || stage.containsKey("\$limit") }.assert().isTrue() + pipeline.any { stage -> stage.containsKey("\$setWindowFields") }.assert().isTrue() + val union = requireNotNull( + pipeline.single { stage -> stage.containsKey("\$unionWith") }["\$unionWith"], + ).asDocument() + union.containsKey("coll").assert().isFalse() + requireNotNull(union["pipeline"]) + .asArray() + .single() + .asDocument() + .containsKey("\$documents") + .assert() + .isTrue() + pipeline.last().containsKey("\$unset").assert().isTrue() + } + + @Test + fun `should reject overflowing page window before storage`() { + assertThrownBy { + compiler.compile(fixture.pagePlan(offset = Long.MAX_VALUE, size = 1)).pagePipeline() + }.satisfies( + Consumer { error -> error.kind.assert().isEqualTo(QueryBackendFailureKind.BUDGET_EXCEEDED) }, + ) + } + + @Test + fun `page result should keep records separate from the exact total row`() { + val results = listOf( + Document(Documents.ID_FIELD, "order-1") + .append(MessageRecords.TENANT_ID, "tenant-1") + .append(StateAggregateRecords.DELETED, false) + .append("state", Document("name", "Alice")), + Document(MongoCompiledRecordQuery.PAGE_TOTAL_VALUE, 3L), + ) + + val page = MongoPageResultMapper(fixture.binding.prepared).map(results, BackendProjection.All) + + page.total.assert().isEqualTo(3) + page.records.single().identity.assert().isEqualTo("order-1") + page.records.single().document.values["state"].assert().isEqualTo( + NormalizedValue.ObjectValue(mapOf("name" to NormalizedValue.Text("Alice"))), + ) + } + + @Test + fun `should reject insensitive predicate before storage`() { + val predicate = BackendPlannedCondition.Predicate( + fixture.name, + PredicateOperator.EQ, + NormalizedValue.Text("Alice"), + NormalizedPredicateOptions(CaseSensitivity.INSENSITIVE), + ) + + assertThrownBy { + compiler.compile(fixture.plan(predicate)) + } + } + + @Test + fun `should compile one attested root text scope`() { + val search = BackendPlannedCondition.Search(fixture.searchScope, "paid order") + + compiler.compile(fixture.plan(user = search)).filter.toBsonDocument().assert().isEqualTo( + Filters.and(Filters.text("paid order"), Filters.empty()).toBsonDocument(), + ) + } + + @Test + fun `should reject text search below non-conjunctive structure`() { + val search = BackendPlannedCondition.Search(fixture.searchScope, "paid") + val disjunction = BackendPlannedCondition.Junction( + me.ahoo.wow.query.backend.JunctionOperator.OR, + listOf(search, fixture.predicate(fixture.name, PredicateOperator.EQ, NormalizedValue.Text("Alice"))), + ) + + assertThrownBy { + compiler.compile(fixture.plan(user = disjunction)) + } + } + + @TestFactory + fun `should compile every portable predicate operator`(): List = predicateCases().map { case -> + DynamicTest.dynamicTest(case.operator.name) { + val actual = compiler.compile( + fixture.plan(user = fixture.predicate(case.field, case.operator, case.value)), + ).filter + actual.toBsonDocument().assert().isEqualTo( + Filters.and(case.expected, Filters.empty()).toBsonDocument(), + ) + } + } + + private fun predicateCases(): List { + val text = NormalizedValue.Text("Alice") + val texts = NormalizedValue.ListValue(listOf(NormalizedValue.Text("Alice"), NormalizedValue.Text("Bob"))) + val instants = NormalizedValue.ListValue(listOf(instant(10), instant(20))) + return listOf( + PredicateCase(fixture.name, PredicateOperator.EQ, text, Filters.eq("state.name", "Alice")), + PredicateCase(fixture.name, PredicateOperator.NE, text, Filters.ne("state.name", "Alice")), + PredicateCase(fixture.createdAt, PredicateOperator.GT, instant(10), Filters.gt("state.createdAt", 10L)), + PredicateCase(fixture.createdAt, PredicateOperator.LT, instant(10), Filters.lt("state.createdAt", 10L)), + PredicateCase(fixture.createdAt, PredicateOperator.GTE, instant(10), Filters.gte("state.createdAt", 10L)), + PredicateCase(fixture.createdAt, PredicateOperator.LTE, instant(10), Filters.lte("state.createdAt", 10L)), + PredicateCase(fixture.name, PredicateOperator.CONTAINS, text, Filters.regex("state.name", "Alice")), + PredicateCase( + fixture.name, + PredicateOperator.IN, + texts, + Filters.`in`("state.name", listOf("Alice", "Bob")), + ), + PredicateCase( + fixture.name, + PredicateOperator.NOT_IN, + texts, + Filters.nin("state.name", listOf("Alice", "Bob")), + ), + PredicateCase( + fixture.createdAt, + PredicateOperator.BETWEEN, + instants, + Filters.and(Filters.gte("state.createdAt", 10L), Filters.lte("state.createdAt", 20L)), + ), + PredicateCase( + fixture.tags, + PredicateOperator.ALL_IN, + texts, + Filters.all("state.tags", listOf("Alice", "Bob")), + ), + PredicateCase(fixture.name, PredicateOperator.STARTS_WITH, text, Filters.regex("state.name", "^Alice")), + PredicateCase(fixture.name, PredicateOperator.ENDS_WITH, text, Filters.regex("state.name", "Alice$")), + PredicateCase(fixture.name, PredicateOperator.IS_NULL, null, Filters.eq("state.name", null)), + PredicateCase(fixture.name, PredicateOperator.NOT_NULL, null, Filters.ne("state.name", null)), + PredicateCase(fixture.active, PredicateOperator.IS_TRUE, null, Filters.eq("state.active", true)), + PredicateCase(fixture.active, PredicateOperator.IS_FALSE, null, Filters.eq("state.active", false)), + PredicateCase( + fixture.name, + PredicateOperator.EXISTS, + NormalizedValue.BooleanValue(true), + Filters.exists("state.name", true), + ), + ) + } + + private fun instant(epochMilli: Long) = NormalizedValue.InstantValue(Instant.ofEpochMilli(epochMilli)) + + private class Fixture { + val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + val tenant = QueryFieldId.System(SystemFieldKind.TENANT_ID) + private val deleted = QueryFieldId.System(SystemFieldKind.DELETED) + val state = QueryFieldId.Path(listOf("state")) + val name = QueryFieldId.Path(listOf("state", "name")) + val createdAt = QueryFieldId.Path(listOf("state", "createdAt")) + val amount = QueryFieldId.Path(listOf("state", "amount")) + val active = QueryFieldId.Path(listOf("state", "active")) + val tags = QueryFieldId.Path(listOf("state", "tags")) + val items = QueryFieldId.Path(listOf("state", "items")) + val itemName = QueryFieldId.Path(listOf("state", "items", "name")) + private val description = QueryFieldId.Path(listOf("description")) + val searchScope = SearchScopeId("document-text") + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val schema = QueryDocumentSchema( + target, + listOf( + field(identity, LogicalFieldType.Text, setOf(PredicateOperator.EQ), EXACT_SORT_PROJECT), + field(tenant, LogicalFieldType.Text, setOf(PredicateOperator.EQ), setOf(FieldCapability.EXACT)), + field( + deleted, + LogicalFieldType.Boolean, + setOf(PredicateOperator.IS_FALSE), + setOf(FieldCapability.EXACT) + ), + field(state, LogicalFieldType.Object), + field( + name, + LogicalFieldType.Text, + setOf( + PredicateOperator.EQ, + PredicateOperator.NE, + PredicateOperator.CONTAINS, + PredicateOperator.IN, + PredicateOperator.NOT_IN, + PredicateOperator.STARTS_WITH, + PredicateOperator.ENDS_WITH, + PredicateOperator.IS_NULL, + PredicateOperator.NOT_NULL, + PredicateOperator.EXISTS, + ), + setOf( + FieldCapability.EXACT, + FieldCapability.PRESENCE, + FieldCapability.LITERAL_PATTERN, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + ), + ), + field( + createdAt, + LogicalFieldType.Instant, + setOf( + PredicateOperator.GT, + PredicateOperator.LT, + PredicateOperator.GTE, + PredicateOperator.LTE, + PredicateOperator.BETWEEN, + ), + setOf(FieldCapability.RANGE), + ), + field( + amount, + LogicalFieldType.Decimal, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT), + ), + field( + active, + LogicalFieldType.Boolean, + setOf(PredicateOperator.IS_TRUE, PredicateOperator.IS_FALSE), + setOf(FieldCapability.EXACT), + ), + field( + tags, + LogicalFieldType.Array( + LogicalFieldType.Text, + Nullability.NON_NULL, + EmptyArraySemantics.DISTINCT, + ), + setOf(PredicateOperator.ALL_IN), + setOf(FieldCapability.EXACT), + ), + field( + items, + LogicalFieldType.Array(LogicalFieldType.Object, Nullability.NON_NULL, EmptyArraySemantics.DISTINCT), + capabilities = setOf(FieldCapability.ELEMENT_MATCH), + ), + field(itemName, LogicalFieldType.Text, setOf(PredicateOperator.EQ), setOf(FieldCapability.EXACT)), + field(description, LogicalFieldType.Text, capabilities = setOf(FieldCapability.FULL_TEXT)), + ), + listOf(QuerySearchScopeDefinition(searchScope, null, listOf(description), listOf(description))), + ) + val binding = MongoSnapshotQueryBinding( + schema, + MongoNamespace("sales", "order_snapshot"), + linkedMapOf( + identity to MongoFieldBinding(Documents.ID_FIELD, EXACT_SORT_PROJECT), + tenant to MongoFieldBinding(MessageRecords.TENANT_ID, setOf(FieldCapability.EXACT)), + deleted to MongoFieldBinding(StateAggregateRecords.DELETED, setOf(FieldCapability.EXACT)), + state to MongoFieldBinding("state", emptySet()), + name to MongoFieldBinding( + "state.name", + setOf( + FieldCapability.EXACT, + FieldCapability.PRESENCE, + FieldCapability.LITERAL_PATTERN, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + ), + ), + createdAt to MongoFieldBinding("state.createdAt", setOf(FieldCapability.RANGE), MongoValueEncoding.EPOCH_MILLIS), + amount to MongoFieldBinding( + "state.amount", + setOf(FieldCapability.EXACT), + MongoValueEncoding.DECIMAL128, + ), + active to MongoFieldBinding("state.active", setOf(FieldCapability.EXACT)), + tags to MongoFieldBinding("state.tags", setOf(FieldCapability.EXACT)), + items to MongoFieldBinding("state.items", setOf(FieldCapability.ELEMENT_MATCH)), + itemName to MongoFieldBinding("state.items.name", setOf(FieldCapability.EXACT)), + description to MongoFieldBinding("description", setOf(FieldCapability.FULL_TEXT)), + ), + textSearch = MongoTextSearchBinding(searchScope, "description_text"), + ) + + fun predicate( + field: QueryFieldId, + operator: PredicateOperator, + value: NormalizedValue? = null, + ) = BackendPlannedCondition.Predicate(field, operator, value) + + fun plan( + user: BackendPlannedCondition = BackendPlannedCondition.All, + mandatory: BackendPlannedCondition = BackendPlannedCondition.All, + projection: BackendProjection = BackendProjection.All, + sort: List = emptyList(), + ) = BackendSingleQueryPlan( + target, + schema.contractId, + BackendEnforcedFilter(user, mandatory), + RecordResultShape.DYNAMIC, + projection, + sort, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("0".repeat(64)), + ) + + fun pagePlan(offset: Long, size: Int) = BackendPageQueryPlan( + target, + schema.contractId, + BackendEnforcedFilter(BackendPlannedCondition.All, BackendPlannedCondition.All), + RecordResultShape.DYNAMIC, + BackendProjection.All, + listOf(BackendSort(identity, NormalizedSortDirection.ASC, BackendSortOrigin.STABILITY_TIE_BREAKER)), + BackendPageWindow(offset, size), + BackendTotalMode.EXACT, + BackendRequiredConsistency.SAME_INPUT, + BackendRequiredCapabilities(), + SemanticTier.PORTABLE, + PlanFingerprint("3".repeat(64)), + ) + + private fun field( + id: QueryFieldId, + type: LogicalFieldType, + operators: Set = emptySet(), + capabilities: Set = emptySet(), + ) = QueryFieldSchema(id, type, Presence.OPTIONAL, Nullability.NULLABLE, operators, capabilities) + + companion object { + val EXACT_SORT_PROJECT = setOf( + FieldCapability.EXACT, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + ) + } + } + + private data class PredicateCase( + val field: QueryFieldId, + val operator: PredicateOperator, + val value: NormalizedValue?, + val expected: Bson, + ) +} diff --git a/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotQueryBindingTest.kt b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotQueryBindingTest.kt new file mode 100644 index 00000000000..8d669465665 --- /dev/null +++ b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotQueryBindingTest.kt @@ -0,0 +1,234 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.MongoNamespace +import com.mongodb.reactivestreams.client.MongoCollection +import io.mockk.every +import io.mockk.mockk +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.mongo.Documents +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.QuerySearchScopeDefinition +import me.ahoo.wow.query.backend.SearchScopeId +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import org.bson.Document +import org.junit.jupiter.api.Test + +class MongoSnapshotQueryBindingTest { + @Test + fun `binding should reject unsafe physical paths and wrong system mapping`() { + assertThrownBy { + MongoFieldBinding("state.\$expr", emptySet()) + } + assertThrownBy { + MongoFieldBinding("state.__wowQueryPageKind", emptySet()) + } + assertThrownBy { + binding( + mapOf( + identity to MongoFieldBinding(Documents.ID_FIELD, setOf(FieldCapability.EXACT)), + tenant to MongoFieldBinding(MessageRecords.OWNER_ID, setOf(FieldCapability.EXACT)), + createdAt to MongoFieldBinding( + "createdAt", + setOf(FieldCapability.RANGE), + MongoValueEncoding.EPOCH_MILLIS, + ), + ), + ) + } + + val collidingPath = QueryFieldId.Path(listOf(MessageRecords.TENANT_ID)) + val collidingSchema = QueryDocumentSchema( + target, + schema.fields.values + QueryFieldSchema( + collidingPath, + LogicalFieldType.Text, + Presence.OPTIONAL, + Nullability.NON_NULL, + emptySet(), + emptySet(), + ), + emptyList(), + ) + assertThrownBy { + MongoSnapshotQueryBinding( + collidingSchema, + namespace, + linkedMapOf( + identity to MongoFieldBinding(Documents.ID_FIELD, setOf(FieldCapability.EXACT)), + tenant to MongoFieldBinding(MessageRecords.TENANT_ID, setOf(FieldCapability.EXACT)), + createdAt to MongoFieldBinding( + "createdAt", + setOf(FieldCapability.RANGE), + MongoValueEncoding.EPOCH_MILLIS, + ), + collidingPath to MongoFieldBinding(MessageRecords.TENANT_ID, emptySet()), + ), + ) + } + assertThrownBy { + binding( + mapOf( + identity to MongoFieldBinding(Documents.ID_FIELD, setOf(FieldCapability.EXACT)), + tenant to MongoFieldBinding(MessageRecords.TENANT_ID, setOf(FieldCapability.EXACT)), + ), + ) + } + } + + @Test + fun `binding should reject logical type and value encoding mismatch`() { + assertThrownBy { + binding( + mapOf( + identity to MongoFieldBinding(Documents.ID_FIELD, setOf(FieldCapability.EXACT)), + createdAt to MongoFieldBinding( + "createdAt", + setOf(FieldCapability.RANGE), + MongoValueEncoding.DEFAULT, + ), + tenant to MongoFieldBinding(MessageRecords.TENANT_ID, setOf(FieldCapability.EXACT)), + ), + ) + } + } + + @Test + fun `contribution should require the exact collection namespace`() { + val binding = binding( + mapOf( + identity to MongoFieldBinding(Documents.ID_FIELD, setOf(FieldCapability.EXACT)), + tenant to MongoFieldBinding(MessageRecords.TENANT_ID, setOf(FieldCapability.EXACT)), + createdAt to MongoFieldBinding( + "createdAt", + setOf(FieldCapability.RANGE), + MongoValueEncoding.EPOCH_MILLIS, + ), + ), + ) + val collection = mockk>() + every { collection.namespace } returns MongoNamespace("sales", "cart_snapshot") + + assertThrownBy { + binding.toContribution(collection) + } + } + + @Test + fun `text search contribution should require an exact attested text index`() { + val description = QueryFieldId.Path(listOf("description")) + val scopeId = SearchScopeId("document-text") + val searchSchema = QueryDocumentSchema( + target, + schema.fields.values + QueryFieldSchema( + description, + LogicalFieldType.Text, + Presence.OPTIONAL, + Nullability.NULLABLE, + emptySet(), + setOf(FieldCapability.FULL_TEXT), + ), + listOf(QuerySearchScopeDefinition(scopeId, null, listOf(description), listOf(description))), + ) + val binding = MongoSnapshotQueryBinding( + searchSchema, + namespace, + mapOf( + identity to MongoFieldBinding(Documents.ID_FIELD, setOf(FieldCapability.EXACT)), + tenant to MongoFieldBinding(MessageRecords.TENANT_ID, setOf(FieldCapability.EXACT)), + createdAt to MongoFieldBinding( + "createdAt", + setOf(FieldCapability.RANGE), + MongoValueEncoding.EPOCH_MILLIS, + ), + description to MongoFieldBinding("description", setOf(FieldCapability.FULL_TEXT)), + ), + textSearch = MongoTextSearchBinding(scopeId, "description_text"), + ) + val ready = Document("name", "description_text") + .append("key", Document("_fts", "text").append("_ftsx", 1)) + .append("weights", Document("description", 1)) + + binding.attestTextIndexReadiness(listOf(ready)) + assertThrownBy { + binding.attestTextIndexReadiness(listOf(Document(ready).append("name", "stale_text"))) + } + assertThrownBy { + binding.attestTextIndexReadiness( + listOf(Document(ready).append("weights", Document("other", 1))), + ) + } + binding.textSearch?.scope.assert().isEqualTo(scopeId) + } + + private fun binding(fields: Map): MongoSnapshotQueryBinding = + MongoSnapshotQueryBinding(schema, namespace, fields) + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val namespace = MongoNamespace("sales", "order_snapshot") + private val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + private val tenant = QueryFieldId.System(SystemFieldKind.TENANT_ID) + private val createdAt = QueryFieldId.Path(listOf("createdAt")) + private val schema = QueryDocumentSchema( + target, + listOf( + QueryFieldSchema( + identity, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT), + ), + QueryFieldSchema( + tenant, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT), + ), + QueryFieldSchema( + createdAt, + LogicalFieldType.Instant, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.GTE), + setOf(FieldCapability.RANGE), + ), + ), + emptyList(), + ) +} diff --git a/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotRecordMapperTest.kt b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotRecordMapperTest.kt new file mode 100644 index 00000000000..a0112a610df --- /dev/null +++ b/wow-mongo/src/test/kotlin/me/ahoo/wow/mongo/query/planned/MongoSnapshotRecordMapperTest.kt @@ -0,0 +1,205 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.mongo.query.planned + +import com.mongodb.MongoNamespace +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.mongo.Documents +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import org.bson.Document +import org.junit.jupiter.api.Test +import java.util.function.Consumer + +class MongoSnapshotRecordMapperTest { + private val mapper = MongoRecordMapper(binding.prepared) + + @Test + fun `should freeze one source observation and restore logical identity`() { + val bytes = byteArrayOf(1, 2) + val nested = Document("bytes", bytes) + val source = Document( + linkedMapOf( + Documents.ID_FIELD to "order-1", + "state" to nested, + "backendOnly" to "must-not-leak", + ), + ) + + val record = mapper.map(source) + bytes[0] = 9 + nested["bytes"] = byteArrayOf(9) + source[Documents.ID_FIELD] = "order-2" + + record.identity.assert().isEqualTo("order-1") + (record.document.values[MessageRecords.AGGREGATE_ID] as NormalizedValue.Text).value.assert() + .isEqualTo("order-1") + val frozenBytes = ( + (record.document.values["state"] as NormalizedValue.ObjectValue).values["bytes"] as NormalizedValue.Bytes + ).toByteArray() + frozenBytes.assert().isEqualTo(byteArrayOf(1, 2)) + record.document.values.containsKey(Documents.ID_FIELD).assert().isFalse() + record.document.values.containsKey("backendOnly").assert().isFalse() + } + + @Test + fun `should reject missing or non-text identity`() { + assertThrownBy { + mapper.map(Document("state", Document())) + } + assertThrownBy { + mapper.map(Document(Documents.ID_FIELD, 1)) + } + } + + @Test + fun `should reject cyclic or over-budget source before materialization`() { + val cyclic = Document(Documents.ID_FIELD, "id") + cyclic["self"] = cyclic + assertThrownBy { + mapper.map(cyclic) + } + + val oversized = Document(Documents.ID_FIELD, "id") + oversized["values"] = listOf(1, 2, 3) + assertThrownBy { + MongoRecordMapper( + binding.prepared, + limits = MongoRecordMapper.MappingLimits(maxDepth = 8, maxNodes = 3, maxCollectionSize = 8), + ).map(oversized) + } + } + + @Test + fun `should apply logical projection after extracting the physical identity`() { + val source = Document( + linkedMapOf( + Documents.ID_FIELD to "order-1", + "tenantId" to "tenant-1", + "state" to Document(linkedMapOf("name" to "Alice", "internal" to "hidden")), + "backendOnly" to "must-not-leak", + ), + ) + val stateName = QueryFieldId.Path(listOf("state", "name")) + val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + + val included = mapper.map(source, BackendProjection.Include(listOf(stateName))) + included.identity.assert().isEqualTo("order-1") + included.document.values.keys.assert().containsExactly("state") + (included.document.values["state"] as NormalizedValue.ObjectValue).values.keys.assert() + .containsExactly("name") + + val excluded = mapper.map(source, BackendProjection.Exclude(listOf(identity))) + excluded.document.values.containsKey(MessageRecords.AGGREGATE_ID).assert().isFalse() + excluded.document.values.containsKey("tenantId").assert().isTrue() + } + + @Test + fun `should classify non-finite numeric source as mapping failure`() { + listOf(Double.NaN, Double.POSITIVE_INFINITY, Double.NEGATIVE_INFINITY).forEach { value -> + assertThrownBy { + mapper.map( + Document(linkedMapOf(Documents.ID_FIELD to "order-1", "value" to value)), + ) + }.satisfies( + Consumer { error -> error.kind.assert().isEqualTo(QueryBackendFailureKind.MAPPING_FAILURE) }, + ) + } + } + + private companion object { + val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + val tenant = QueryFieldId.System(SystemFieldKind.TENANT_ID) + val state = QueryFieldId.Path(listOf("state")) + val stateName = QueryFieldId.Path(listOf("state", "name")) + val schema = QueryDocumentSchema( + QueryTarget(MaterializedNamedAggregate("sales", "order"), QueryDocumentKind.SNAPSHOT), + listOf( + QueryFieldSchema( + identity, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT, FieldCapability.PROJECTABLE), + ), + QueryFieldSchema( + tenant, + LogicalFieldType.Text, + Presence.OPTIONAL, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT, FieldCapability.PROJECTABLE), + ), + QueryFieldSchema( + state, + LogicalFieldType.Object, + Presence.OPTIONAL, + Nullability.NON_NULL, + emptySet(), + emptySet(), + ), + QueryFieldSchema( + stateName, + LogicalFieldType.Text, + Presence.OPTIONAL, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT, FieldCapability.PROJECTABLE), + ), + ), + emptyList(), + ) + val binding = MongoSnapshotQueryBinding( + schema, + MongoNamespace("sales", "order_snapshot"), + linkedMapOf( + identity to MongoFieldBinding( + Documents.ID_FIELD, + setOf(FieldCapability.EXACT, FieldCapability.PROJECTABLE), + ), + tenant to MongoFieldBinding( + MessageRecords.TENANT_ID, + setOf(FieldCapability.EXACT, FieldCapability.PROJECTABLE), + ), + state to MongoFieldBinding("state", emptySet()), + stateName to MongoFieldBinding( + "state.name", + setOf(FieldCapability.EXACT, FieldCapability.PROJECTABLE), + ), + ), + ) + } +} diff --git a/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/Https.kt b/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/Https.kt index a0a916aa6a5..c84c11fc41e 100644 --- a/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/Https.kt +++ b/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/Https.kt @@ -21,6 +21,7 @@ object Https { object Code { const val OK = "200" const val BAD_REQUEST = "400" + const val FORBIDDEN = "403" const val NOT_FOUND = "404" const val NOT_ACCEPTABLE = "406" const val REQUEST_TIMEOUT = "408" diff --git a/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/QueryComponent.kt b/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/QueryComponent.kt index 941d9789659..39c813e7b7f 100644 --- a/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/QueryComponent.kt +++ b/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/QueryComponent.kt @@ -13,14 +13,27 @@ package me.ahoo.wow.openapi +import io.swagger.v3.oas.models.media.ArraySchema +import io.swagger.v3.oas.models.media.ComposedSchema +import io.swagger.v3.oas.models.media.IntegerSchema +import io.swagger.v3.oas.models.media.ObjectSchema +import io.swagger.v3.oas.models.media.StringSchema import me.ahoo.wow.api.Wow import me.ahoo.wow.api.query.Condition import me.ahoo.wow.api.query.ListQuery import me.ahoo.wow.api.query.PagedList import me.ahoo.wow.api.query.PagedQuery +import me.ahoo.wow.api.query.analytics.AnalyticsCompleteness +import me.ahoo.wow.api.query.analytics.AnalyticsConsistency +import me.ahoo.wow.api.query.analytics.AnalyticsCursor +import me.ahoo.wow.api.query.analytics.AnalyticsDimension +import me.ahoo.wow.api.query.analytics.AnalyticsNumericPolicy +import me.ahoo.wow.api.query.analytics.AnalyticsPage import me.ahoo.wow.modeling.metadata.AggregateMetadata import me.ahoo.wow.modeling.toStringWithAlias import me.ahoo.wow.openapi.CommonComponent.Response.withErrorCodeHeader +import me.ahoo.wow.openapi.QueryComponent.Schema.analyticsPageSchema +import me.ahoo.wow.openapi.QueryComponent.Schema.analyticsQuerySchema import me.ahoo.wow.openapi.QueryComponent.Schema.conditionSchema import me.ahoo.wow.openapi.QueryComponent.Schema.listQuerySchema import me.ahoo.wow.openapi.QueryComponent.Schema.pagedQuerySchema @@ -32,6 +45,8 @@ import me.ahoo.wow.schema.typed.query.AggregatedPagedQuery import me.ahoo.wow.schema.typed.query.AggregatedSingleQuery object QueryComponent { + const val ANALYTICS_QUERY_SUFFIX = ".AnalyticsQuery" + const val ANALYTICS_PAGE_SUFFIX = ".AnalyticsPage" const val SINGLE_QUERY_SUFFIX = ".SingleQuery" const val COUNT_QUERY_SUFFIX = ".CountQuery" const val LIST_QUERY_SUFFIX = ".ListQuery" @@ -39,6 +54,8 @@ object QueryComponent { const val COUNT_QUERY_KEY = Wow.WOW + COUNT_QUERY_SUFFIX const val LIST_QUERY_KEY = Wow.WOW + LIST_QUERY_SUFFIX const val PAGED_QUERY_KEY = Wow.WOW + PAGED_QUERY_SUFFIX + const val ANALYTICS_QUERY_KEY = Wow.WOW + ANALYTICS_QUERY_SUFFIX + const val ANALYTICS_PAGE_KEY = Wow.WOW + ANALYTICS_PAGE_SUFFIX object Schema { @@ -53,6 +70,87 @@ object QueryComponent { fun OpenAPIComponentContext.pagedQuerySchema(): io.swagger.v3.oas.models.media.Schema<*> { return schema(PagedQuery::class.java) } + + fun OpenAPIComponentContext.analyticsQuerySchema(): io.swagger.v3.oas.models.media.Schema<*> { + return ComposedSchema().oneOf( + listOf( + analyticsQueryVariant(global = true), + analyticsQueryVariant(global = false), + ), + ) + } + + private fun OpenAPIComponentContext.analyticsQueryVariant(global: Boolean): ObjectSchema = + ObjectSchema().apply { + additionalProperties = false + addProperties("condition", schema(Condition::class.java)) + addProperties("grouping", analyticsGroupingSchema(global)) + addProperties("metrics", ArraySchema().items(analyticsMetricSchema()).minItems(1)) + addProperties("window", analyticsBucketWindowSchema(global)) + addProperties("numericPolicy", schema(AnalyticsNumericPolicy::class.java)) + addProperties("consistency", schema(AnalyticsConsistency::class.java)) + addProperties("completeness", schema(AnalyticsCompleteness::class.java)) + required = listOf("grouping", "metrics", "window") + } + + private fun OpenAPIComponentContext.analyticsGroupingSchema(global: Boolean): ObjectSchema = + ObjectSchema().apply { + additionalProperties = false + addProperties( + "kind", + StringSchema()._const(if (global) "GLOBAL" else "BY"), + ) + addProperties( + "dimensions", + ArraySchema().items(schema(AnalyticsDimension::class.java)).also { dimensions -> + if (global) dimensions.maxItems(0) else dimensions.minItems(1) + }, + ) + required = if (global) listOf("kind") else listOf("kind", "dimensions") + } + + private fun OpenAPIComponentContext.analyticsMetricSchema(): ComposedSchema = ComposedSchema().apply { + oneOf = listOf( + ObjectSchema().apply { + additionalProperties = false + addProperties("alias", boundedString(128)) + addProperties("kind", StringSchema()._const("DOCUMENT_COUNT")) + required = listOf("alias", "kind") + }, + ObjectSchema().apply { + additionalProperties = false + addProperties("alias", boundedString(128)) + addProperties( + "kind", + StringSchema()._enum(listOf("MIN", "MAX", "SUM", "AVERAGE")), + ) + addProperties("field", boundedString(512)) + required = listOf("alias", "kind", "field") + }, + ) + } + + private fun OpenAPIComponentContext.analyticsBucketWindowSchema(global: Boolean): ObjectSchema = + ObjectSchema().apply { + additionalProperties = false + addProperties( + "limit", + IntegerSchema().minimum(java.math.BigDecimal.ONE).also { limit -> + if (global) limit.maximum(java.math.BigDecimal.ONE) + }, + ) + if (!global) addProperties("cursor", schema(AnalyticsCursor::class.java)) + required = listOf("limit") + } + + private fun boundedString(maxLength: Int): StringSchema = StringSchema().apply { + minLength = 1 + this.maxLength = maxLength + } + + fun OpenAPIComponentContext.analyticsPageSchema(): io.swagger.v3.oas.models.media.Schema<*> { + return schema(AnalyticsPage::class.java) + } } object RequestBody { @@ -102,6 +200,12 @@ object QueryComponent { content(schema = schema(AggregatedPagedQuery::class.java, aggregateMetadata.command.aggregateType)) } } + + fun OpenAPIComponentContext.analyticsQueryRequestBody(): io.swagger.v3.oas.models.parameters.RequestBody { + return requestBody(ANALYTICS_QUERY_KEY) { + content(schema = analyticsQuerySchema()) + } + } } object Response { @@ -123,6 +227,13 @@ object QueryComponent { ).build() } + fun OpenAPIComponentContext.analyticsPageResponse(): io.swagger.v3.oas.models.responses.ApiResponse { + return response(ANALYTICS_PAGE_KEY) { + withErrorCodeHeader(this@analyticsPageResponse) + content(Https.MediaType.APPLICATION_JSON, schema = analyticsPageSchema()) + } + } + fun OpenAPIComponentContext.loadEventStreamResponse(aggregateMetadata: AggregateMetadata<*, *>): io.swagger.v3.oas.models.responses.ApiResponse { return ApiResponseBuilder().withErrorCodeHeader(this) .listContent( diff --git a/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contract/BuiltInHttpRoutes.kt b/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contract/BuiltInHttpRoutes.kt index 483a2ba9a78..0bbcb2165ba 100644 --- a/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contract/BuiltInHttpRoutes.kt +++ b/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contract/BuiltInHttpRoutes.kt @@ -53,6 +53,7 @@ object BuiltInHttpRouteHandlerKeys { } object Snapshot { + const val ANALYZE = "$AGGREGATE_SNAPSHOT.analyze" const val COUNT = "$AGGREGATE_SNAPSHOT.count" const val LIST_QUERY = "$AGGREGATE_SNAPSHOT.list-query" const val LIST_QUERY_STATE = "$AGGREGATE_SNAPSHOT.list-query-state" diff --git a/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contributor/QueryContractComponentSupport.kt b/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contributor/QueryContractComponentSupport.kt index 1fe8be74e5e..13b2d0b64f9 100644 --- a/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contributor/QueryContractComponentSupport.kt +++ b/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contributor/QueryContractComponentSupport.kt @@ -13,19 +13,24 @@ package me.ahoo.wow.openapi.contributor +import me.ahoo.wow.api.Wow import me.ahoo.wow.api.query.MaterializedSnapshot import me.ahoo.wow.api.query.PagedList import me.ahoo.wow.modeling.metadata.AggregateMetadata import me.ahoo.wow.modeling.toStringWithAlias +import me.ahoo.wow.openapi.CommonComponent.Response.withErrorCodeHeader +import me.ahoo.wow.openapi.CommonComponent.Schema.errorInfoSchema import me.ahoo.wow.openapi.Https import me.ahoo.wow.openapi.QueryComponent import me.ahoo.wow.openapi.QueryComponent.RequestBody.aggregatedCountQueryRequestBody import me.ahoo.wow.openapi.QueryComponent.RequestBody.aggregatedListQueryRequestBody import me.ahoo.wow.openapi.QueryComponent.RequestBody.aggregatedPagedQueryRequestBody import me.ahoo.wow.openapi.QueryComponent.RequestBody.aggregatedSingleQueryRequestBody +import me.ahoo.wow.openapi.QueryComponent.RequestBody.analyticsQueryRequestBody import me.ahoo.wow.openapi.QueryComponent.RequestBody.countQueryRequestBody import me.ahoo.wow.openapi.QueryComponent.RequestBody.listQueryRequestBody import me.ahoo.wow.openapi.QueryComponent.RequestBody.pagedQueryRequestBody +import me.ahoo.wow.openapi.QueryComponent.Response.analyticsPageResponse import me.ahoo.wow.openapi.QueryComponent.Response.countQueryResponse import me.ahoo.wow.openapi.context.OpenAPIComponentContext import me.ahoo.wow.openapi.contract.HttpContent @@ -41,6 +46,11 @@ internal fun OpenAPIComponentContext.countQueryRequestBodyRef(): HttpRequestBody return HttpRequestBody(componentRef = QueryComponent.COUNT_QUERY_KEY) } +internal fun OpenAPIComponentContext.analyticsQueryRequestBodyRef(): HttpRequestBody { + analyticsQueryRequestBody() + return HttpRequestBody(componentRef = QueryComponent.ANALYTICS_QUERY_KEY) +} + internal fun OpenAPIComponentContext.listQueryRequestBodyRef(): HttpRequestBody { listQueryRequestBody() return HttpRequestBody(componentRef = QueryComponent.LIST_QUERY_KEY) @@ -87,6 +97,47 @@ internal fun OpenAPIComponentContext.countQueryResponseRef(): HttpResponse { ) } +internal fun OpenAPIComponentContext.analyticsPageResponseRef(): HttpResponse { + analyticsPageResponse() + return HttpResponse( + statusCode = Https.Code.OK, + componentRef = QueryComponent.ANALYTICS_PAGE_KEY, + ) +} + +internal fun OpenAPIComponentContext.queryRouteResponses( + vararg successResponses: HttpResponse, +): List = successResponses.toList() + listOf( + badRequestResponseRef(), + queryErrorResponseRef( + Https.Code.FORBIDDEN, + QUERY_ACCESS_DENIED_RESPONSE_KEY, + "Query access was denied.", + ), + requestTimeoutResponseRef(), + tooManyRequestsResponseRef(), + queryErrorResponseRef( + Https.Code.BAD_GATEWAY, + QUERY_INCOMPLETE_RESULT_RESPONSE_KEY, + "The query result was incomplete.", + ), + queryErrorResponseRef( + Https.Code.SERVICE_UNAVAILABLE, + QUERY_BACKEND_UNAVAILABLE_RESPONSE_KEY, + "The query backend is unavailable.", + ), + queryErrorResponseRef( + Https.Code.GATEWAY_TIMEOUT, + QUERY_BACKEND_TIMEOUT_RESPONSE_KEY, + "The query backend timed out.", + ), + queryErrorResponseRef( + Https.Code.INTERNAL_SERVER_ERROR, + QUERY_INTERNAL_FAILURE_RESPONSE_KEY, + "The query result could not be produced.", + ), +) + internal fun OpenAPIComponentContext.eventStreamListResponse( aggregateMetadata: AggregateMetadata<*, *> ): HttpResponse { @@ -195,3 +246,22 @@ private fun OpenAPIComponentContext.responseWithJson(schema: HttpSchema): HttpRe content = listOf(HttpContent(Https.MediaType.APPLICATION_JSON, schema)) ) } + +private fun OpenAPIComponentContext.queryErrorResponseRef( + statusCode: String, + componentKey: String, + description: String, +): HttpResponse { + response(componentKey) { + withErrorCodeHeader(this@queryErrorResponseRef) + description(description) + content(schema = errorInfoSchema()) + } + return HttpResponse(statusCode = statusCode, componentRef = componentKey) +} + +private const val QUERY_ACCESS_DENIED_RESPONSE_KEY = "${Wow.WOW_PREFIX}QueryAccessDenied" +private const val QUERY_INCOMPLETE_RESULT_RESPONSE_KEY = "${Wow.WOW_PREFIX}QueryIncompleteResult" +private const val QUERY_BACKEND_UNAVAILABLE_RESPONSE_KEY = "${Wow.WOW_PREFIX}QueryBackendUnavailable" +private const val QUERY_BACKEND_TIMEOUT_RESPONSE_KEY = "${Wow.WOW_PREFIX}QueryBackendTimeout" +private const val QUERY_INTERNAL_FAILURE_RESPONSE_KEY = "${Wow.WOW_PREFIX}QueryInternalFailure" diff --git a/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contributor/aggregate/event/EventRouteContributor.kt b/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contributor/aggregate/event/EventRouteContributor.kt index 6a673c2f69b..2777d7a0541 100644 --- a/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contributor/aggregate/event/EventRouteContributor.kt +++ b/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contributor/aggregate/event/EventRouteContributor.kt @@ -48,6 +48,7 @@ import me.ahoo.wow.openapi.contributor.eventStreamPagedResponse import me.ahoo.wow.openapi.contributor.headVersionPathParameterRef import me.ahoo.wow.openapi.contributor.listQueryRequestBodyRef import me.ahoo.wow.openapi.contributor.pagedQueryRequestBodyRef +import me.ahoo.wow.openapi.contributor.queryRouteResponses import me.ahoo.wow.openapi.contributor.requestTimeoutResponseRef import me.ahoo.wow.openapi.contributor.tailVersionPathParameterRef import me.ahoo.wow.openapi.metadata.AggregateRouteMetadata @@ -93,7 +94,7 @@ object EventRouteContributor : RouteContributor { appendOwnerPath = variant.appendOwnerPath, appendPathSuffix = "event/count", requestBody = componentContext.countQueryRequestBodyRef(), - responses = listOf(componentContext.countQueryResponseRef()) + responses = componentContext.queryRouteResponses(componentContext.countQueryResponseRef()) ), eventRoute( currentContext = currentContext, @@ -108,7 +109,9 @@ object EventRouteContributor : RouteContributor { appendPathSuffix = "event/list", accept = STREAMING_ACCEPT, requestBody = componentContext.listQueryRequestBodyRef(), - responses = listOf(componentContext.eventStreamListResponse(aggregateMetadata)) + responses = componentContext.queryRouteResponses( + componentContext.eventStreamListResponse(aggregateMetadata) + ) ), eventRoute( currentContext = currentContext, @@ -122,7 +125,9 @@ object EventRouteContributor : RouteContributor { appendOwnerPath = variant.appendOwnerPath, appendPathSuffix = "event/paged", requestBody = componentContext.pagedQueryRequestBodyRef(), - responses = listOf(componentContext.eventStreamPagedResponse(aggregateMetadata)) + responses = componentContext.queryRouteResponses( + componentContext.eventStreamPagedResponse(aggregateMetadata) + ) ) ) } @@ -150,7 +155,9 @@ object EventRouteContributor : RouteContributor { componentContext.headVersionPathParameterRef(), componentContext.tailVersionPathParameterRef() ), - responses = listOf(componentContext.eventStreamListResponse(aggregateRouteMetadata.aggregateMetadata)) + responses = componentContext.queryRouteResponses( + componentContext.eventStreamListResponse(aggregateRouteMetadata.aggregateMetadata) + ) ) } diff --git a/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contributor/aggregate/snapshot/SnapshotRouteContributor.kt b/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contributor/aggregate/snapshot/SnapshotRouteContributor.kt index e7158928043..eaca2dca200 100644 --- a/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contributor/aggregate/snapshot/SnapshotRouteContributor.kt +++ b/wow-openapi/src/main/kotlin/me/ahoo/wow/openapi/contributor/aggregate/snapshot/SnapshotRouteContributor.kt @@ -40,6 +40,8 @@ import me.ahoo.wow.openapi.contributor.aggregatedCountQueryRequestBodyRef import me.ahoo.wow.openapi.contributor.aggregatedListQueryRequestBodyRef import me.ahoo.wow.openapi.contributor.aggregatedPagedQueryRequestBodyRef import me.ahoo.wow.openapi.contributor.aggregatedSingleQueryRequestBodyRef +import me.ahoo.wow.openapi.contributor.analyticsPageResponseRef +import me.ahoo.wow.openapi.contributor.analyticsQueryRequestBodyRef import me.ahoo.wow.openapi.contributor.batchAfterIdPathParameterRef import me.ahoo.wow.openapi.contributor.batchLimitPathParameterRef import me.ahoo.wow.openapi.contributor.batchResultResponseRef @@ -49,11 +51,11 @@ import me.ahoo.wow.openapi.contributor.materializedSnapshotListResponse import me.ahoo.wow.openapi.contributor.materializedSnapshotPagedResponse import me.ahoo.wow.openapi.contributor.materializedSnapshotSingleResponse import me.ahoo.wow.openapi.contributor.notFoundResponseRef +import me.ahoo.wow.openapi.contributor.queryRouteResponses import me.ahoo.wow.openapi.contributor.requestTimeoutResponseRef import me.ahoo.wow.openapi.contributor.stateListResponse import me.ahoo.wow.openapi.contributor.statePagedResponse import me.ahoo.wow.openapi.contributor.stateSingleResponse -import me.ahoo.wow.openapi.contributor.tooManyRequestsResponseRef import me.ahoo.wow.openapi.metadata.AggregateRouteMetadata object SnapshotRouteContributor : RouteContributor { @@ -84,6 +86,7 @@ object SnapshotRouteContributor : RouteContributor { ): List { val aggregateMetadata = aggregateRouteMetadata.aggregateMetadata return listOf( + analyticsRoute(currentContext, aggregateRouteMetadata, componentContext, variant), snapshotRoute( currentContext = currentContext, aggregateRouteMetadata = aggregateRouteMetadata, @@ -96,11 +99,7 @@ object SnapshotRouteContributor : RouteContributor { appendOwnerPath = variant.appendOwnerPath, appendPathSuffix = "snapshot/count", requestBody = componentContext.aggregatedCountQueryRequestBodyRef(aggregateMetadata), - responses = listOf( - componentContext.countQueryResponseRef(), - componentContext.requestTimeoutResponseRef(), - componentContext.tooManyRequestsResponseRef() - ) + responses = componentContext.queryRouteResponses(componentContext.countQueryResponseRef()) ), listQuerySnapshotRoute(currentContext, aggregateRouteMetadata, componentContext, variant), listQuerySnapshotStateRoute(currentContext, aggregateRouteMetadata, componentContext, variant), @@ -111,6 +110,28 @@ object SnapshotRouteContributor : RouteContributor { ) } + private fun analyticsRoute( + currentContext: NamedBoundedContext, + aggregateRouteMetadata: AggregateRouteMetadata<*>, + componentContext: OpenAPIComponentContext, + variant: TenantOwnerVariant, + ): HttpRouteContract = snapshotRoute( + currentContext = currentContext, + aggregateRouteMetadata = aggregateRouteMetadata, + componentContext = componentContext, + handlerKey = BuiltInHttpRouteHandlerKeys.Snapshot.ANALYZE, + resourceName = SNAPSHOT, + operation = "analyze", + operationSummary = "Analyze Snapshot", + appendTenantPath = variant.appendTenantPath, + appendOwnerPath = variant.appendOwnerPath, + appendPathSuffix = "snapshot/analyze", + requestBody = componentContext.analyticsQueryRequestBodyRef(), + responses = componentContext.queryRouteResponses( + componentContext.analyticsPageResponseRef(), + ), + ) + private fun listQuerySnapshotRoute( currentContext: NamedBoundedContext, aggregateRouteMetadata: AggregateRouteMetadata<*>, @@ -130,7 +151,7 @@ object SnapshotRouteContributor : RouteContributor { appendPathSuffix = "snapshot/list", accept = STREAMING_ACCEPT, requestBody = componentContext.aggregatedListQueryRequestBodyRef(aggregateRouteMetadata.aggregateMetadata), - responses = listOf( + responses = componentContext.queryRouteResponses( componentContext.materializedSnapshotListResponse(aggregateRouteMetadata.aggregateMetadata) ) ) @@ -155,7 +176,9 @@ object SnapshotRouteContributor : RouteContributor { appendPathSuffix = "snapshot/list/state", accept = STREAMING_ACCEPT, requestBody = componentContext.aggregatedListQueryRequestBodyRef(aggregateRouteMetadata.aggregateMetadata), - responses = listOf(componentContext.stateListResponse(aggregateRouteMetadata.aggregateMetadata)) + responses = componentContext.queryRouteResponses( + componentContext.stateListResponse(aggregateRouteMetadata.aggregateMetadata) + ) ) } @@ -177,7 +200,7 @@ object SnapshotRouteContributor : RouteContributor { appendOwnerPath = variant.appendOwnerPath, appendPathSuffix = "snapshot/paged", requestBody = componentContext.aggregatedPagedQueryRequestBodyRef(aggregateRouteMetadata.aggregateMetadata), - responses = listOf( + responses = componentContext.queryRouteResponses( componentContext.materializedSnapshotPagedResponse(aggregateRouteMetadata.aggregateMetadata) ) ) @@ -201,7 +224,9 @@ object SnapshotRouteContributor : RouteContributor { appendOwnerPath = variant.appendOwnerPath, appendPathSuffix = "snapshot/paged/state", requestBody = componentContext.aggregatedPagedQueryRequestBodyRef(aggregateRouteMetadata.aggregateMetadata), - responses = listOf(componentContext.statePagedResponse(aggregateRouteMetadata.aggregateMetadata)) + responses = componentContext.queryRouteResponses( + componentContext.statePagedResponse(aggregateRouteMetadata.aggregateMetadata) + ) ) } @@ -225,7 +250,7 @@ object SnapshotRouteContributor : RouteContributor { requestBody = componentContext.aggregatedSingleQueryRequestBodyRef( aggregateRouteMetadata.aggregateMetadata ), - responses = listOf( + responses = componentContext.queryRouteResponses( componentContext.materializedSnapshotSingleResponse(aggregateRouteMetadata.aggregateMetadata), componentContext.notFoundResponseRef() ) @@ -252,7 +277,7 @@ object SnapshotRouteContributor : RouteContributor { requestBody = componentContext.aggregatedSingleQueryRequestBodyRef( aggregateRouteMetadata.aggregateMetadata ), - responses = listOf( + responses = componentContext.queryRouteResponses( componentContext.stateSingleResponse(aggregateRouteMetadata.aggregateMetadata), componentContext.notFoundResponseRef() ) @@ -277,7 +302,9 @@ object SnapshotRouteContributor : RouteContributor { appendOwnerPath = aggregateRouteMetadata.defaultAppendOwnerPath(), appendIdPath = aggregateRouteMetadata.owner != AggregateRoute.Owner.AGGREGATE_ID, appendPathSuffix = "snapshot", - responses = loadSnapshotResponses(aggregateRouteMetadata, componentContext) + responses = componentContext.queryRouteResponses( + *loadSnapshotResponses(aggregateRouteMetadata, componentContext).toTypedArray() + ) ) } diff --git a/wow-openapi/src/test/kotlin/me/ahoo/wow/openapi/snapshot/OpenApiCompatibilitySnapshotTest.kt b/wow-openapi/src/test/kotlin/me/ahoo/wow/openapi/snapshot/OpenApiCompatibilitySnapshotTest.kt index 01eb17da37d..fecc16dc0bb 100644 --- a/wow-openapi/src/test/kotlin/me/ahoo/wow/openapi/snapshot/OpenApiCompatibilitySnapshotTest.kt +++ b/wow-openapi/src/test/kotlin/me/ahoo/wow/openapi/snapshot/OpenApiCompatibilitySnapshotTest.kt @@ -19,6 +19,7 @@ import io.swagger.v3.oas.models.parameters.Parameter import me.ahoo.test.asserts.assert import me.ahoo.wow.naming.MaterializedNamedBoundedContext import me.ahoo.wow.openapi.RouterSpecs +import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys import me.ahoo.wow.openapi.contract.HttpParameter import me.ahoo.wow.openapi.snapshot.OpenApiSnapshotSupport.assertContractSnapshot import me.ahoo.wow.openapi.snapshot.OpenApiSnapshotSupport.assertOpenApiSnapshot @@ -94,6 +95,95 @@ internal class OpenApiCompatibilitySnapshotTest { .containsExactly("FAIL", "RAW_JSON") } + @Test + fun `generated analytics contract should expose an opaque scalar cursor and complete route`() { + val openAPI = OpenAPI() + RouterSpecs(currentContext).build().mergeOpenAPIFromCatalog(openAPI) + + val document = mapper.valueToTree(openAPI) + val schemas = document.path("components").path("schemas") + val cursor = schemas.path("wow.api.query.AnalyticsCursor") + cursor.path("type").asText().assert().isEqualTo("string") + cursor.path("maxLength").asInt().assert().isEqualTo(256) + cursor.path("pattern").asText().assert().isEqualTo("^[A-Za-z0-9._-]+$") + + val request = document.path("components").path("requestBodies").path("wow.AnalyticsQuery") + .path("content").path("application/json").path("schema") + val variants = request.path("oneOf") + variants.assert().hasSize(2) + variants.forEach { variant -> + variant.path("additionalProperties").asBoolean().assert().isFalse() + variant.path("required").map { it.asText() }.assert().contains("grouping", "metrics", "window") + variant.path("properties").path("metrics").path("minItems").asInt().assert().isEqualTo(1) + variant.path("properties").path("metrics").path("items").path("oneOf").assert().hasSize(2) + } + val global = variants.first() + global.path("properties").path("grouping").path("properties").path("kind").path("const") + .asText().assert().isEqualTo("GLOBAL") + global.path("properties").path("grouping").path("properties").path("dimensions").path("maxItems") + .asInt().assert().isZero() + global.path("properties").path("window").path("properties").path("limit").path("maximum") + .asInt().assert().isEqualTo(1) + global.path("properties").path("window").path("properties").has("cursor").assert().isFalse() + val globalMetricVariants = global.path("properties").path("metrics").path("items").path("oneOf") + globalMetricVariants.first().path("properties").has("field").assert().isFalse() + globalMetricVariants.last().path("required").map { it.asText() }.assert().contains("field") + val grouped = variants.last() + grouped.path("properties").path("grouping").path("properties").path("kind").path("const") + .asText().assert().isEqualTo("BY") + grouped.path("properties").path("grouping").path("properties").path("dimensions").path("minItems") + .asInt().assert().isEqualTo(1) + grouped.path("properties").path("window").path("properties").path("cursor").path("\$ref") + .asText().assert().isEqualTo("#/components/schemas/wow.api.query.AnalyticsCursor") + + val route = document.path("paths").path("/cart/snapshot/analyze").path("post") + route.path("requestBody").path("\$ref").asText().assert() + .isEqualTo("#/components/requestBodies/wow.AnalyticsQuery") + route.path("responses").path("200").path("\$ref").asText().assert() + .isEqualTo("#/components/responses/wow.AnalyticsPage") + route.path("responses").fieldNames().asSequence().toList().sorted().assert() + .containsExactly("200", "400", "403", "408", "429", "500", "502", "503", "504") + listOf("400", "403", "408", "429", "500", "502", "503", "504").forEach { status -> + val responseRef = route.path("responses").path(status).path("\$ref").asText() + responseRef.assert().startsWith("#/components/responses/") + val response = document.path("components").path("responses").path(responseRef.substringAfterLast('/')) + response.path("headers").path("Wow-Error-Code").path("\$ref").asText() + .assert().isEqualTo("#/components/headers/wow.Wow-Error-Code") + response.path("content").path("application/json").path("schema").path("\$ref").asText() + .assert().isEqualTo("#/components/schemas/wow.api.DefaultErrorInfo") + } + } + + @Test + fun `generated query routes should declare the runtime failure status contract`() { + val queryHandlerKeys = setOf( + BuiltInHttpRouteHandlerKeys.Snapshot.ANALYZE, + BuiltInHttpRouteHandlerKeys.Snapshot.COUNT, + BuiltInHttpRouteHandlerKeys.Snapshot.LIST_QUERY, + BuiltInHttpRouteHandlerKeys.Snapshot.LIST_QUERY_STATE, + BuiltInHttpRouteHandlerKeys.Snapshot.PAGED_QUERY, + BuiltInHttpRouteHandlerKeys.Snapshot.PAGED_QUERY_STATE, + BuiltInHttpRouteHandlerKeys.Snapshot.SINGLE, + BuiltInHttpRouteHandlerKeys.Snapshot.SINGLE_STATE, + BuiltInHttpRouteHandlerKeys.Snapshot.LOAD, + BuiltInHttpRouteHandlerKeys.Event.COUNT, + BuiltInHttpRouteHandlerKeys.Event.LIST_QUERY, + BuiltInHttpRouteHandlerKeys.Event.PAGED_QUERY, + BuiltInHttpRouteHandlerKeys.Event.LOAD, + ) + val requiredFailureStatuses = setOf("400", "403", "408", "429", "500", "502", "503", "504") + val queryRoutes = RouterSpecs(currentContext).build().toRouteCatalog().routes + .filter { it.handlerKey in queryHandlerKeys } + + queryRoutes.assert().isNotEmpty() + queryRoutes.forEach { route -> + val responseCodes = route.responses.map { it.statusCode }.toSet() + requiredFailureStatuses.forEach { status -> + responseCodes.assert().contains(status) + } + } + } + @Test fun `generated route contracts should match example domain compatibility snapshot`() { val routerSpecs = RouterSpecs(currentContext).build() diff --git a/wow-openapi/src/test/resources/openapi/example-domain-contract.snapshot.json b/wow-openapi/src/test/resources/openapi/example-domain-contract.snapshot.json index 8ed1bf7a7cd..e776a79831f 100644 --- a/wow-openapi/src/test/resources/openapi/example-domain-contract.snapshot.json +++ b/wow-openapi/src/test/resources/openapi/example-domain-contract.snapshot.json @@ -5,7 +5,7 @@ "parameterNames" : [ ], "path" : "/cart/event/count", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -14,7 +14,7 @@ "parameterNames" : [ ], "path" : "/cart/event/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -23,7 +23,16 @@ "parameterNames" : [ ], "path" : "/cart/event/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], + "tagNames" : [ "customer", "example.cart" ] +}, { + "accept" : [ "application/json" ], + "id" : "example.cart.snapshot.analyze", + "method" : "POST", + "parameterNames" : [ ], + "path" : "/cart/snapshot/analyze", + "requestBody" : true, + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -32,7 +41,7 @@ "parameterNames" : [ ], "path" : "/cart/snapshot/count", "requestBody" : true, - "responseCodes" : [ "200", "408", "429" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -41,7 +50,7 @@ "parameterNames" : [ ], "path" : "/cart/snapshot/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -50,7 +59,7 @@ "parameterNames" : [ ], "path" : "/cart/snapshot/list/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -59,7 +68,7 @@ "parameterNames" : [ ], "path" : "/cart/snapshot/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -68,7 +77,7 @@ "parameterNames" : [ ], "path" : "/cart/snapshot/paged/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -77,7 +86,7 @@ "parameterNames" : [ ], "path" : "/cart/snapshot/single", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -86,7 +95,7 @@ "parameterNames" : [ ], "path" : "/cart/snapshot/single/state", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -113,7 +122,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.id", "ref:#/components/parameters/wow.headVersion", "ref:#/components/parameters/wow.tailVersion" ], "path" : "/cart/{id}/event/{headVersion}/{tailVersion}", "requestBody" : false, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -176,7 +185,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId" ], "path" : "/owner/{ownerId}/cart/event/count", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -185,7 +194,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId" ], "path" : "/owner/{ownerId}/cart/event/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -194,7 +203,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId" ], "path" : "/owner/{ownerId}/cart/event/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -230,7 +239,16 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId" ], "path" : "/owner/{ownerId}/cart/snapshot", "requestBody" : false, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], + "tagNames" : [ "customer", "example.cart" ] +}, { + "accept" : [ "application/json" ], + "id" : "example.cart.owner.snapshot.analyze", + "method" : "POST", + "parameterNames" : [ "ref:#/components/parameters/wow.ownerId" ], + "path" : "/owner/{ownerId}/cart/snapshot/analyze", + "requestBody" : true, + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -239,7 +257,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId" ], "path" : "/owner/{ownerId}/cart/snapshot/count", "requestBody" : true, - "responseCodes" : [ "200", "408", "429" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -248,7 +266,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId" ], "path" : "/owner/{ownerId}/cart/snapshot/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -257,7 +275,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId" ], "path" : "/owner/{ownerId}/cart/snapshot/list/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -266,7 +284,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId" ], "path" : "/owner/{ownerId}/cart/snapshot/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -275,7 +293,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId" ], "path" : "/owner/{ownerId}/cart/snapshot/paged/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -284,7 +302,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId" ], "path" : "/owner/{ownerId}/cart/snapshot/single", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -293,7 +311,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId" ], "path" : "/owner/{ownerId}/cart/snapshot/single/state", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "customer", "example.cart" ] }, { "accept" : [ "application/json" ], @@ -347,7 +365,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/owner/{ownerId}/sales-order/event/count", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -356,7 +374,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/owner/{ownerId}/sales-order/event/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -365,7 +383,16 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/owner/{ownerId}/sales-order/event/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], + "tagNames" : [ "example.order" ] +}, { + "accept" : [ "application/json" ], + "id" : "example.order.owner.snapshot.analyze", + "method" : "POST", + "parameterNames" : [ "ref:#/components/parameters/wow.ownerId", "ref:#/components/parameters/wow.Wow-Space-Id" ], + "path" : "/owner/{ownerId}/sales-order/snapshot/analyze", + "requestBody" : true, + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -374,7 +401,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/owner/{ownerId}/sales-order/snapshot/count", "requestBody" : true, - "responseCodes" : [ "200", "408", "429" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -383,7 +410,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/owner/{ownerId}/sales-order/snapshot/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -392,7 +419,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/owner/{ownerId}/sales-order/snapshot/list/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -401,7 +428,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/owner/{ownerId}/sales-order/snapshot/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -410,7 +437,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/owner/{ownerId}/sales-order/snapshot/paged/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -419,7 +446,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/owner/{ownerId}/sales-order/snapshot/single", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -428,7 +455,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.ownerId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/owner/{ownerId}/sales-order/snapshot/single/state", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -437,7 +464,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/sales-order/event/count", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -446,7 +473,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/sales-order/event/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -455,7 +482,16 @@ "parameterNames" : [ "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/sales-order/event/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], + "tagNames" : [ "example.order" ] +}, { + "accept" : [ "application/json" ], + "id" : "example.order.snapshot.analyze", + "method" : "POST", + "parameterNames" : [ "ref:#/components/parameters/wow.Wow-Space-Id" ], + "path" : "/sales-order/snapshot/analyze", + "requestBody" : true, + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -464,7 +500,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/sales-order/snapshot/count", "requestBody" : true, - "responseCodes" : [ "200", "408", "429" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -473,7 +509,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/sales-order/snapshot/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -482,7 +518,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/sales-order/snapshot/list/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -491,7 +527,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/sales-order/snapshot/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -500,7 +536,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/sales-order/snapshot/paged/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -509,7 +545,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/sales-order/snapshot/single", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -518,7 +554,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/sales-order/snapshot/single/state", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -545,7 +581,7 @@ "parameterNames" : [ ], "path" : "/tck/mock_aggregate/event/count", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -554,7 +590,7 @@ "parameterNames" : [ ], "path" : "/tck/mock_aggregate/event/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -563,7 +599,16 @@ "parameterNames" : [ ], "path" : "/tck/mock_aggregate/event/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], + "tagNames" : [ "tck.mock_aggregate" ] +}, { + "accept" : [ "application/json" ], + "id" : "tck.mock_aggregate.snapshot.analyze", + "method" : "POST", + "parameterNames" : [ ], + "path" : "/tck/mock_aggregate/snapshot/analyze", + "requestBody" : true, + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -572,7 +617,7 @@ "parameterNames" : [ ], "path" : "/tck/mock_aggregate/snapshot/count", "requestBody" : true, - "responseCodes" : [ "200", "408", "429" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -581,7 +626,7 @@ "parameterNames" : [ ], "path" : "/tck/mock_aggregate/snapshot/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -590,7 +635,7 @@ "parameterNames" : [ ], "path" : "/tck/mock_aggregate/snapshot/list/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -599,7 +644,7 @@ "parameterNames" : [ ], "path" : "/tck/mock_aggregate/snapshot/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -608,7 +653,7 @@ "parameterNames" : [ ], "path" : "/tck/mock_aggregate/snapshot/paged/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -617,7 +662,7 @@ "parameterNames" : [ ], "path" : "/tck/mock_aggregate/snapshot/single", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -626,7 +671,7 @@ "parameterNames" : [ ], "path" : "/tck/mock_aggregate/snapshot/single/state", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -653,7 +698,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_with_tenant_id/event/count", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -662,7 +707,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_with_tenant_id/event/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -671,7 +716,16 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_with_tenant_id/event/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], + "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] +}, { + "accept" : [ "application/json" ], + "id" : "tck.modeling_command_aggregate_with_tenant_id.snapshot.analyze", + "method" : "POST", + "parameterNames" : [ ], + "path" : "/tck/modeling_command_aggregate_with_tenant_id/snapshot/analyze", + "requestBody" : true, + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -680,7 +734,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_with_tenant_id/snapshot/count", "requestBody" : true, - "responseCodes" : [ "200", "408", "429" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -689,7 +743,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_with_tenant_id/snapshot/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -698,7 +752,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_with_tenant_id/snapshot/list/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -707,7 +761,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_with_tenant_id/snapshot/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -716,7 +770,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_with_tenant_id/snapshot/paged/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -725,7 +779,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_with_tenant_id/snapshot/single", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -734,7 +788,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_with_tenant_id/snapshot/single/state", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -761,7 +815,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_without_ctor_parameters/event/count", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -770,7 +824,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_without_ctor_parameters/event/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -779,7 +833,16 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_without_ctor_parameters/event/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], + "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] +}, { + "accept" : [ "application/json" ], + "id" : "tck.modeling_command_aggregate_without_ctor_parameters.snapshot.analyze", + "method" : "POST", + "parameterNames" : [ ], + "path" : "/tck/modeling_command_aggregate_without_ctor_parameters/snapshot/analyze", + "requestBody" : true, + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -788,7 +851,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_without_ctor_parameters/snapshot/count", "requestBody" : true, - "responseCodes" : [ "200", "408", "429" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -797,7 +860,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_without_ctor_parameters/snapshot/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -806,7 +869,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_without_ctor_parameters/snapshot/list/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -815,7 +878,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_without_ctor_parameters/snapshot/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -824,7 +887,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_without_ctor_parameters/snapshot/paged/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -833,7 +896,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_without_ctor_parameters/snapshot/single", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -842,7 +905,7 @@ "parameterNames" : [ ], "path" : "/tck/modeling_command_aggregate_without_ctor_parameters/snapshot/single/state", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -869,7 +932,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/mock_aggregate/event/count", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -878,7 +941,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/mock_aggregate/event/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -887,7 +950,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/mock_aggregate/event/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -907,6 +970,15 @@ "requestBody" : true, "responseCodes" : [ "200", "400", "404", "408", "409", "410", "429" ], "tagNames" : [ "tck.mock_aggregate" ] +}, { + "accept" : [ "application/json" ], + "id" : "tck.mock_aggregate.tenant.snapshot.analyze", + "method" : "POST", + "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], + "path" : "/tck/tenant/{tenantId}/mock_aggregate/snapshot/analyze", + "requestBody" : true, + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], + "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], "id" : "tck.mock_aggregate.tenant.snapshot.count", @@ -914,7 +986,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/mock_aggregate/snapshot/count", "requestBody" : true, - "responseCodes" : [ "200", "408", "429" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -923,7 +995,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/mock_aggregate/snapshot/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -932,7 +1004,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/mock_aggregate/snapshot/list/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -941,7 +1013,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/mock_aggregate/snapshot/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -950,7 +1022,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/mock_aggregate/snapshot/paged/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -959,7 +1031,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/mock_aggregate/snapshot/single", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -968,7 +1040,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/mock_aggregate/snapshot/single/state", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -986,7 +1058,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.id", "ref:#/components/parameters/wow.headVersion", "ref:#/components/parameters/wow.tailVersion" ], "path" : "/tck/tenant/{tenantId}/mock_aggregate/{id}/event/{headVersion}/{tailVersion}", "requestBody" : false, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1013,7 +1085,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.id" ], "path" : "/tck/tenant/{tenantId}/mock_aggregate/{id}/snapshot", "requestBody" : false, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.mock_aggregate" ] }, { "accept" : [ "application/json" ], @@ -1085,7 +1157,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/event/count", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1094,7 +1166,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/event/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -1103,7 +1175,16 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/event/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], + "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] +}, { + "accept" : [ "application/json" ], + "id" : "tck.modeling_command_aggregate_with_tenant_id.tenant.snapshot.analyze", + "method" : "POST", + "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], + "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/snapshot/analyze", + "requestBody" : true, + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -1112,7 +1193,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/snapshot/count", "requestBody" : true, - "responseCodes" : [ "200", "408", "429" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1121,7 +1202,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/snapshot/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1130,7 +1211,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/snapshot/list/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -1139,7 +1220,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/snapshot/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -1148,7 +1229,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/snapshot/paged/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -1157,7 +1238,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/snapshot/single", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -1166,7 +1247,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/snapshot/single/state", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1184,7 +1265,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.id", "ref:#/components/parameters/wow.headVersion", "ref:#/components/parameters/wow.tailVersion" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/{id}/event/{headVersion}/{tailVersion}", "requestBody" : false, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1202,7 +1283,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.id" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/{id}/snapshot", "requestBody" : false, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_with_tenant_id" ] }, { "accept" : [ "application/json" ], @@ -1274,7 +1355,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/event/count", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1283,7 +1364,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/event/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -1292,7 +1373,16 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/event/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], + "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] +}, { + "accept" : [ "application/json" ], + "id" : "tck.modeling_command_aggregate_without_ctor_parameters.tenant.snapshot.analyze", + "method" : "POST", + "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], + "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/snapshot/analyze", + "requestBody" : true, + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -1301,7 +1391,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/snapshot/count", "requestBody" : true, - "responseCodes" : [ "200", "408", "429" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1310,7 +1400,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/snapshot/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1319,7 +1409,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/snapshot/list/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -1328,7 +1418,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/snapshot/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -1337,7 +1427,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/snapshot/paged/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -1346,7 +1436,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/snapshot/single", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -1355,7 +1445,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/snapshot/single/state", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1373,7 +1463,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.id", "ref:#/components/parameters/wow.headVersion", "ref:#/components/parameters/wow.tailVersion" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/{id}/event/{headVersion}/{tailVersion}", "requestBody" : false, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1391,7 +1481,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.id" ], "path" : "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/{id}/snapshot", "requestBody" : false, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] }, { "accept" : [ "application/json" ], @@ -1517,7 +1607,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.ownerId", "ref:#/components/parameters/wow.id", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/tenant/{tenantId}/owner/{ownerId}/sales-order/{id}/snapshot", "requestBody" : false, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -1562,7 +1652,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/tenant/{tenantId}/sales-order/event/count", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1571,7 +1661,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/tenant/{tenantId}/sales-order/event/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -1580,7 +1670,16 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/tenant/{tenantId}/sales-order/event/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], + "tagNames" : [ "example.order" ] +}, { + "accept" : [ "application/json" ], + "id" : "example.order.tenant.snapshot.analyze", + "method" : "POST", + "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.Wow-Space-Id" ], + "path" : "/tenant/{tenantId}/sales-order/snapshot/analyze", + "requestBody" : true, + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -1589,7 +1688,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/tenant/{tenantId}/sales-order/snapshot/count", "requestBody" : true, - "responseCodes" : [ "200", "408", "429" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1598,7 +1697,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/tenant/{tenantId}/sales-order/snapshot/list", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1607,7 +1706,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/tenant/{tenantId}/sales-order/snapshot/list/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -1616,7 +1715,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/tenant/{tenantId}/sales-order/snapshot/paged", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -1625,7 +1724,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/tenant/{tenantId}/sales-order/snapshot/paged/state", "requestBody" : true, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -1634,7 +1733,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/tenant/{tenantId}/sales-order/snapshot/single", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json" ], @@ -1643,7 +1742,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.Wow-Space-Id" ], "path" : "/tenant/{tenantId}/sales-order/snapshot/single/state", "requestBody" : true, - "responseCodes" : [ "200", "404" ], + "responseCodes" : [ "200", "400", "403", "404", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json", "text/event-stream" ], @@ -1652,7 +1751,7 @@ "parameterNames" : [ "ref:#/components/parameters/wow.tenantId", "ref:#/components/parameters/wow.id", "ref:#/components/parameters/wow.Wow-Space-Id", "ref:#/components/parameters/wow.headVersion", "ref:#/components/parameters/wow.tailVersion" ], "path" : "/tenant/{tenantId}/sales-order/{id}/event/{headVersion}/{tailVersion}", "requestBody" : false, - "responseCodes" : [ "200" ], + "responseCodes" : [ "200", "400", "403", "408", "429", "500", "502", "503", "504" ], "tagNames" : [ "example.order" ] }, { "accept" : [ "application/json", "text/event-stream" ], diff --git a/wow-openapi/src/test/resources/openapi/example-domain-openapi.snapshot.json b/wow-openapi/src/test/resources/openapi/example-domain-openapi.snapshot.json index 49324444820..0eaded9447e 100644 --- a/wow-openapi/src/test/resources/openapi/example-domain-openapi.snapshot.json +++ b/wow-openapi/src/test/resources/openapi/example-domain-openapi.snapshot.json @@ -447,6 +447,199 @@ } } }, + "wow.AnalyticsQuery" : { + "content" : { + "application/json" : { + "schema" : { + "oneOf" : [ { + "additionalProperties" : false, + "properties" : { + "completeness" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsCompleteness" + }, + "condition" : { + "$ref" : "#/components/schemas/wow.api.query.Condition" + }, + "consistency" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsConsistency" + }, + "grouping" : { + "additionalProperties" : false, + "properties" : { + "dimensions" : { + "items" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsDimension" + }, + "maxItems" : 0, + "type" : "array" + }, + "kind" : { + "const" : "GLOBAL", + "type" : "string" + } + }, + "required" : [ "kind" ], + "type" : "object" + }, + "metrics" : { + "items" : { + "oneOf" : [ { + "additionalProperties" : false, + "properties" : { + "alias" : { + "maxLength" : 128, + "minLength" : 1, + "type" : "string" + }, + "kind" : { + "const" : "DOCUMENT_COUNT", + "type" : "string" + } + }, + "required" : [ "alias", "kind" ], + "type" : "object" + }, { + "additionalProperties" : false, + "properties" : { + "alias" : { + "maxLength" : 128, + "minLength" : 1, + "type" : "string" + }, + "field" : { + "maxLength" : 512, + "minLength" : 1, + "type" : "string" + }, + "kind" : { + "enum" : [ "MIN", "MAX", "SUM", "AVERAGE" ], + "type" : "string" + } + }, + "required" : [ "alias", "field", "kind" ], + "type" : "object" + } ] + }, + "minItems" : 1, + "type" : "array" + }, + "numericPolicy" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsNumericPolicy" + }, + "window" : { + "additionalProperties" : false, + "properties" : { + "limit" : { + "format" : "int32", + "maximum" : 1, + "minimum" : 1, + "type" : "integer" + } + }, + "required" : [ "limit" ], + "type" : "object" + } + }, + "required" : [ "grouping", "metrics", "window" ], + "type" : "object" + }, { + "additionalProperties" : false, + "properties" : { + "completeness" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsCompleteness" + }, + "condition" : { + "$ref" : "#/components/schemas/wow.api.query.Condition" + }, + "consistency" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsConsistency" + }, + "grouping" : { + "additionalProperties" : false, + "properties" : { + "dimensions" : { + "items" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsDimension" + }, + "minItems" : 1, + "type" : "array" + }, + "kind" : { + "const" : "BY", + "type" : "string" + } + }, + "required" : [ "dimensions", "kind" ], + "type" : "object" + }, + "metrics" : { + "items" : { + "oneOf" : [ { + "additionalProperties" : false, + "properties" : { + "alias" : { + "maxLength" : 128, + "minLength" : 1, + "type" : "string" + }, + "kind" : { + "const" : "DOCUMENT_COUNT", + "type" : "string" + } + }, + "required" : [ "alias", "kind" ], + "type" : "object" + }, { + "additionalProperties" : false, + "properties" : { + "alias" : { + "maxLength" : 128, + "minLength" : 1, + "type" : "string" + }, + "field" : { + "maxLength" : 512, + "minLength" : 1, + "type" : "string" + }, + "kind" : { + "enum" : [ "MIN", "MAX", "SUM", "AVERAGE" ], + "type" : "string" + } + }, + "required" : [ "alias", "field", "kind" ], + "type" : "object" + } ] + }, + "minItems" : 1, + "type" : "array" + }, + "numericPolicy" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsNumericPolicy" + }, + "window" : { + "additionalProperties" : false, + "properties" : { + "cursor" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsCursor" + }, + "limit" : { + "format" : "int32", + "minimum" : 1, + "type" : "integer" + } + }, + "required" : [ "limit" ], + "type" : "object" + } + }, + "required" : [ "grouping", "metrics", "window" ], + "type" : "object" + } ] + } + } + } + }, "wow.CompensationTarget" : { "content" : { "application/json" : { @@ -485,6 +678,20 @@ } }, "responses" : { + "wow.AnalyticsPage" : { + "content" : { + "application/json" : { + "schema" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsPage" + } + } + }, + "headers" : { + "Wow-Error-Code" : { + "$ref" : "#/components/headers/wow.Wow-Error-Code" + } + } + }, "wow.BadRequest" : { "content" : { "application/json" : { @@ -665,6 +872,76 @@ } } }, + "wow.QueryAccessDenied" : { + "content" : { + "application/json" : { + "schema" : { + "$ref" : "#/components/schemas/wow.api.DefaultErrorInfo" + } + } + }, + "headers" : { + "Wow-Error-Code" : { + "$ref" : "#/components/headers/wow.Wow-Error-Code" + } + } + }, + "wow.QueryBackendTimeout" : { + "content" : { + "application/json" : { + "schema" : { + "$ref" : "#/components/schemas/wow.api.DefaultErrorInfo" + } + } + }, + "headers" : { + "Wow-Error-Code" : { + "$ref" : "#/components/headers/wow.Wow-Error-Code" + } + } + }, + "wow.QueryBackendUnavailable" : { + "content" : { + "application/json" : { + "schema" : { + "$ref" : "#/components/schemas/wow.api.DefaultErrorInfo" + } + } + }, + "headers" : { + "Wow-Error-Code" : { + "$ref" : "#/components/headers/wow.Wow-Error-Code" + } + } + }, + "wow.QueryIncompleteResult" : { + "content" : { + "application/json" : { + "schema" : { + "$ref" : "#/components/schemas/wow.api.DefaultErrorInfo" + } + } + }, + "headers" : { + "Wow-Error-Code" : { + "$ref" : "#/components/headers/wow.Wow-Error-Code" + } + } + }, + "wow.QueryInternalFailure" : { + "content" : { + "application/json" : { + "schema" : { + "$ref" : "#/components/schemas/wow.api.DefaultErrorInfo" + } + } + }, + "headers" : { + "Wow-Error-Code" : { + "$ref" : "#/components/headers/wow.Wow-Error-Code" + } + } + }, "wow.RequestTimeout" : { "content" : { "application/json" : { @@ -709,6 +986,10 @@ } }, "schemas" : { + "example.RoundingMode" : { + "enum" : [ "UP", "DOWN", "CEILING", "FLOOR", "HALF_UP", "HALF_DOWN", "HALF_EVEN", "UNNECESSARY" ], + "type" : "string" + }, "example.StringObjectMap" : { "type" : "object" }, @@ -4537,6 +4818,134 @@ "required" : [ "aggregateId", "aggregateName", "contextName", "tenantId" ], "type" : "object" }, + "wow.api.query.AnalyticsBucket" : { + "properties" : { + "keys" : { + "$ref" : "#/components/schemas/wow.api.query.StringAnalyticsValueMap" + }, + "metrics" : { + "$ref" : "#/components/schemas/wow.api.query.StringAnalyticsValueMap" + } + }, + "required" : [ "keys", "metrics" ], + "type" : "object" + }, + "wow.api.query.AnalyticsCompleteness" : { + "const" : "EXACT", + "type" : "string" + }, + "wow.api.query.AnalyticsConsistency" : { + "enum" : [ "EVENTUAL", "SNAPSHOT" ], + "type" : "string" + }, + "wow.api.query.AnalyticsCursor" : { + "maxLength" : 256, + "pattern" : "^[A-Za-z0-9._-]+$", + "type" : "string" + }, + "wow.api.query.AnalyticsDimension" : { + "properties" : { + "alias" : { + "maxLength" : 128, + "minLength" : 1, + "type" : "string" + }, + "field" : { + "maxLength" : 512, + "minLength" : 1, + "type" : "string" + }, + "missingPolicy" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsMissingPolicy" + } + }, + "required" : [ "alias", "field" ], + "type" : "object" + }, + "wow.api.query.AnalyticsMissingPolicy" : { + "enum" : [ "EXCLUDE", "AS_NULL_BUCKET" ], + "type" : "string" + }, + "wow.api.query.AnalyticsNumericPolicy" : { + "properties" : { + "overflowPolicy" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsOverflowPolicy" + }, + "precision" : { + "format" : "int32", + "maximum" : 34, + "minimum" : 1, + "type" : "integer" + }, + "promotion" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsNumericPromotion" + }, + "roundingMode" : { + "$ref" : "#/components/schemas/example.RoundingMode" + }, + "scale" : { + "format" : "int32", + "maximum" : 34, + "minimum" : 0, + "type" : "integer" + } + }, + "required" : [ "scale" ], + "type" : "object" + }, + "wow.api.query.AnalyticsNumericPromotion" : { + "const" : "DECIMAL128", + "type" : "string" + }, + "wow.api.query.AnalyticsOverflowPolicy" : { + "const" : "REJECT", + "type" : "string" + }, + "wow.api.query.AnalyticsPage" : { + "properties" : { + "buckets" : { + "items" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsBucket" + }, + "type" : "array" + }, + "completeness" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsCompleteness" + }, + "consistency" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsConsistency" + }, + "nextCursor" : { + "anyOf" : [ { + "type" : "null" + }, { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsCursor" + } ] + } + }, + "required" : [ "buckets", "completeness", "consistency", "nextCursor" ], + "type" : "object" + }, + "wow.api.query.AnalyticsValue" : { + "properties" : { + "type" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsValueType" + }, + "value" : { + "anyOf" : [ { + "type" : "null" + }, { + "type" : "string" + } ] + } + }, + "required" : [ "type", "value" ], + "type" : "object" + }, + "wow.api.query.AnalyticsValueType" : { + "enum" : [ "NULL", "BOOLEAN", "TEXT", "INT64", "DECIMAL", "INSTANT" ], + "type" : "string" + }, "wow.api.query.Condition" : { "properties" : { "children" : { @@ -4677,6 +5086,12 @@ "enum" : [ "ASC", "DESC" ], "type" : "string" }, + "wow.api.query.StringAnalyticsValueMap" : { + "additionalProperties" : { + "$ref" : "#/components/schemas/wow.api.query.AnalyticsValue" + }, + "type" : "object" + }, "wow.command.CommandResult" : { "properties" : { "aggregateId" : { @@ -5121,6 +5536,30 @@ "responses" : { "200" : { "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -5158,6 +5597,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -5184,33 +5647,114 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] } }, - "/cart/snapshot/count" : { + "/cart/snapshot/analyze" : { "post" : { - "operationId" : "example.cart.snapshot.count", + "operationId" : "example.cart.snapshot.analyze", "parameters" : [ ], "requestBody" : { - "$ref" : "#/components/requestBodies/example.cart.CountQuery" + "$ref" : "#/components/requestBodies/wow.AnalyticsQuery" }, "responses" : { "200" : { - "$ref" : "#/components/responses/wow.CountQuery" + "$ref" : "#/components/responses/wow.AnalyticsPage" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" }, "408" : { "$ref" : "#/components/responses/wow.RequestTimeout" }, "429" : { "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] } }, - "/cart/snapshot/list" : { + "/cart/snapshot/count" : { + "post" : { + "operationId" : "example.cart.snapshot.count", + "parameters" : [ ], + "requestBody" : { + "$ref" : "#/components/requestBodies/example.cart.CountQuery" + }, + "responses" : { + "200" : { + "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" + } + }, + "tags" : [ "example.cart", "customer" ] + } + }, + "/cart/snapshot/list" : { "post" : { "operationId" : "example.cart.snapshot.list_query", "parameters" : [ ], @@ -5242,6 +5786,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -5279,6 +5847,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -5305,6 +5897,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -5331,6 +5947,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -5358,8 +5998,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -5387,8 +6051,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -5467,6 +6155,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -5762,6 +6474,30 @@ "responses" : { "200" : { "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -5801,6 +6537,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -5829,6 +6589,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -6062,49 +6846,132 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] } }, - "/owner/{ownerId}/cart/snapshot/count" : { + "/owner/{ownerId}/cart/snapshot/analyze" : { "post" : { - "operationId" : "example.cart.owner.snapshot.count", + "operationId" : "example.cart.owner.snapshot.analyze", "parameters" : [ { "$ref" : "#/components/parameters/wow.ownerId" } ], "requestBody" : { - "$ref" : "#/components/requestBodies/example.cart.CountQuery" + "$ref" : "#/components/requestBodies/wow.AnalyticsQuery" }, "responses" : { "200" : { - "$ref" : "#/components/responses/wow.CountQuery" + "$ref" : "#/components/responses/wow.AnalyticsPage" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" }, "408" : { "$ref" : "#/components/responses/wow.RequestTimeout" }, "429" : { "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] } }, - "/owner/{ownerId}/cart/snapshot/list" : { + "/owner/{ownerId}/cart/snapshot/count" : { "post" : { - "operationId" : "example.cart.owner.snapshot.list_query", + "operationId" : "example.cart.owner.snapshot.count", "parameters" : [ { "$ref" : "#/components/parameters/wow.ownerId" } ], "requestBody" : { - "$ref" : "#/components/requestBodies/example.cart.ListQuery" + "$ref" : "#/components/requestBodies/example.cart.CountQuery" }, "responses" : { "200" : { - "content" : { - "application/json" : { + "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" + } + }, + "tags" : [ "example.cart", "customer" ] + } + }, + "/owner/{ownerId}/cart/snapshot/list" : { + "post" : { + "operationId" : "example.cart.owner.snapshot.list_query", + "parameters" : [ { + "$ref" : "#/components/parameters/wow.ownerId" + } ], + "requestBody" : { + "$ref" : "#/components/requestBodies/example.cart.ListQuery" + }, + "responses" : { + "200" : { + "content" : { + "application/json" : { "schema" : { "items" : { "$ref" : "#/components/schemas/example.cart.CartStateMaterializedSnapshot" @@ -6126,6 +6993,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -6165,6 +7056,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -6193,6 +7108,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -6221,6 +7160,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -6250,8 +7213,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -6281,8 +7268,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.cart", "customer" ] @@ -6537,6 +7548,30 @@ "responses" : { "200" : { "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6578,6 +7613,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6608,6 +7667,73 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" + } + }, + "tags" : [ "example.order" ] + } + }, + "/owner/{ownerId}/sales-order/snapshot/analyze" : { + "post" : { + "operationId" : "example.order.owner.snapshot.analyze", + "parameters" : [ { + "$ref" : "#/components/parameters/wow.Wow-Space-Id" + }, { + "$ref" : "#/components/parameters/wow.ownerId" + } ], + "requestBody" : { + "$ref" : "#/components/requestBodies/wow.AnalyticsQuery" + }, + "responses" : { + "200" : { + "$ref" : "#/components/responses/wow.AnalyticsPage" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6628,11 +7754,29 @@ "200" : { "$ref" : "#/components/responses/wow.CountQuery" }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "408" : { "$ref" : "#/components/responses/wow.RequestTimeout" }, "429" : { "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6674,18 +7818,42 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } - } - }, - "tags" : [ "example.order" ] - } - }, - "/owner/{ownerId}/sales-order/snapshot/list/state" : { - "post" : { - "operationId" : "example.order.owner.snapshot_state.list_query", - "parameters" : [ { - "$ref" : "#/components/parameters/wow.Wow-Space-Id" - }, { - "$ref" : "#/components/parameters/wow.ownerId" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" + } + }, + "tags" : [ "example.order" ] + } + }, + "/owner/{ownerId}/sales-order/snapshot/list/state" : { + "post" : { + "operationId" : "example.order.owner.snapshot_state.list_query", + "parameters" : [ { + "$ref" : "#/components/parameters/wow.Wow-Space-Id" + }, { + "$ref" : "#/components/parameters/wow.ownerId" } ], "requestBody" : { "$ref" : "#/components/requestBodies/example.order.ListQuery" @@ -6715,6 +7883,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6745,6 +7937,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6775,6 +7991,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6806,8 +8046,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6839,8 +8103,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6858,6 +8146,30 @@ "responses" : { "200" : { "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6897,6 +8209,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6925,6 +8261,71 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" + } + }, + "tags" : [ "example.order" ] + } + }, + "/sales-order/snapshot/analyze" : { + "post" : { + "operationId" : "example.order.snapshot.analyze", + "parameters" : [ { + "$ref" : "#/components/parameters/wow.Wow-Space-Id" + } ], + "requestBody" : { + "$ref" : "#/components/requestBodies/wow.AnalyticsQuery" + }, + "responses" : { + "200" : { + "$ref" : "#/components/responses/wow.AnalyticsPage" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6943,11 +8344,29 @@ "200" : { "$ref" : "#/components/responses/wow.CountQuery" }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "408" : { "$ref" : "#/components/responses/wow.RequestTimeout" }, "429" : { "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -6987,6 +8406,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -7026,6 +8469,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -7054,6 +8521,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -7082,6 +8573,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -7111,8 +8626,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -7142,8 +8681,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -7201,6 +8764,30 @@ "responses" : { "200" : { "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -7238,32 +8825,119 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" + } + }, + "tags" : [ "tck.mock_aggregate" ] + } + }, + "/tck/mock_aggregate/event/paged" : { + "post" : { + "operationId" : "tck.mock_aggregate.event.paged_query", + "parameters" : [ ], + "requestBody" : { + "$ref" : "#/components/requestBodies/wow.PagedQuery" + }, + "responses" : { + "200" : { + "content" : { + "application/json" : { + "schema" : { + "$ref" : "#/components/schemas/tck.mock_aggregate.MockCommandAggregateAggregatedDomainEventStreamPagedList" + } + } + }, + "headers" : { + "Wow-Error-Code" : { + "$ref" : "#/components/headers/wow.Wow-Error-Code" + } + } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] } }, - "/tck/mock_aggregate/event/paged" : { + "/tck/mock_aggregate/snapshot/analyze" : { "post" : { - "operationId" : "tck.mock_aggregate.event.paged_query", + "operationId" : "tck.mock_aggregate.snapshot.analyze", "parameters" : [ ], "requestBody" : { - "$ref" : "#/components/requestBodies/wow.PagedQuery" + "$ref" : "#/components/requestBodies/wow.AnalyticsQuery" }, "responses" : { "200" : { - "content" : { - "application/json" : { - "schema" : { - "$ref" : "#/components/schemas/tck.mock_aggregate.MockCommandAggregateAggregatedDomainEventStreamPagedList" - } - } - }, - "headers" : { - "Wow-Error-Code" : { - "$ref" : "#/components/headers/wow.Wow-Error-Code" - } - } + "$ref" : "#/components/responses/wow.AnalyticsPage" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -7280,11 +8954,29 @@ "200" : { "$ref" : "#/components/responses/wow.CountQuery" }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "408" : { "$ref" : "#/components/responses/wow.RequestTimeout" }, "429" : { "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -7322,6 +9014,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -7359,6 +9075,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -7385,6 +9125,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -7411,6 +9175,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -7438,8 +9226,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -7467,8 +9279,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -7522,6 +9358,30 @@ "responses" : { "200" : { "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -7559,6 +9419,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -7585,6 +9469,69 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" + } + }, + "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] + } + }, + "/tck/modeling_command_aggregate_with_tenant_id/snapshot/analyze" : { + "post" : { + "operationId" : "tck.modeling_command_aggregate_with_tenant_id.snapshot.analyze", + "parameters" : [ ], + "requestBody" : { + "$ref" : "#/components/requestBodies/wow.AnalyticsQuery" + }, + "responses" : { + "200" : { + "$ref" : "#/components/responses/wow.AnalyticsPage" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -7601,11 +9548,29 @@ "200" : { "$ref" : "#/components/responses/wow.CountQuery" }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "408" : { "$ref" : "#/components/responses/wow.RequestTimeout" }, "429" : { "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -7643,6 +9608,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -7680,6 +9669,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -7706,6 +9719,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -7732,6 +9769,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -7759,8 +9820,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -7788,8 +9873,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -7843,6 +9952,30 @@ "responses" : { "200" : { "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -7880,6 +10013,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -7906,6 +10063,69 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" + } + }, + "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] + } + }, + "/tck/modeling_command_aggregate_without_ctor_parameters/snapshot/analyze" : { + "post" : { + "operationId" : "tck.modeling_command_aggregate_without_ctor_parameters.snapshot.analyze", + "parameters" : [ ], + "requestBody" : { + "$ref" : "#/components/requestBodies/wow.AnalyticsQuery" + }, + "responses" : { + "200" : { + "$ref" : "#/components/responses/wow.AnalyticsPage" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -7922,11 +10142,29 @@ "200" : { "$ref" : "#/components/responses/wow.CountQuery" }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "408" : { "$ref" : "#/components/responses/wow.RequestTimeout" }, "429" : { "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -7964,6 +10202,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -8001,6 +10263,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -8027,6 +10313,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -8053,6 +10363,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -8080,8 +10414,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -8109,8 +10467,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -8166,6 +10548,30 @@ "responses" : { "200" : { "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -8205,6 +10611,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -8233,6 +10663,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -8376,6 +10830,47 @@ "tags" : [ "tck.mock_aggregate" ] } }, + "/tck/tenant/{tenantId}/mock_aggregate/snapshot/analyze" : { + "post" : { + "operationId" : "tck.mock_aggregate.tenant.snapshot.analyze", + "parameters" : [ { + "$ref" : "#/components/parameters/wow.tenantId" + } ], + "requestBody" : { + "$ref" : "#/components/requestBodies/wow.AnalyticsQuery" + }, + "responses" : { + "200" : { + "$ref" : "#/components/responses/wow.AnalyticsPage" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" + } + }, + "tags" : [ "tck.mock_aggregate" ] + } + }, "/tck/tenant/{tenantId}/mock_aggregate/snapshot/count" : { "post" : { "operationId" : "tck.mock_aggregate.tenant.snapshot.count", @@ -8389,11 +10884,29 @@ "200" : { "$ref" : "#/components/responses/wow.CountQuery" }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "408" : { "$ref" : "#/components/responses/wow.RequestTimeout" }, "429" : { "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -8433,6 +10946,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -8472,6 +11009,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -8500,6 +11061,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -8528,6 +11113,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -8557,8 +11166,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -8588,8 +11221,32 @@ } } }, - "404" : { - "$ref" : "#/components/responses/wow.NotFound" + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "404" : { + "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -8703,6 +11360,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -8873,8 +11554,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.mock_aggregate" ] @@ -9141,6 +11846,30 @@ "responses" : { "200" : { "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -9180,6 +11909,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -9208,6 +11961,71 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" + } + }, + "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] + } + }, + "/tck/tenant/{tenantId}/modeling_command_aggregate_with_tenant_id/snapshot/analyze" : { + "post" : { + "operationId" : "tck.modeling_command_aggregate_with_tenant_id.tenant.snapshot.analyze", + "parameters" : [ { + "$ref" : "#/components/parameters/wow.tenantId" + } ], + "requestBody" : { + "$ref" : "#/components/requestBodies/wow.AnalyticsQuery" + }, + "responses" : { + "200" : { + "$ref" : "#/components/responses/wow.AnalyticsPage" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -9226,11 +12044,29 @@ "200" : { "$ref" : "#/components/responses/wow.CountQuery" }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "408" : { "$ref" : "#/components/responses/wow.RequestTimeout" }, "429" : { "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -9270,6 +12106,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -9309,6 +12169,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -9337,6 +12221,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -9365,6 +12273,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -9394,8 +12326,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -9425,8 +12381,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -9540,6 +12520,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -9639,8 +12643,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_with_tenant_id" ] @@ -9907,6 +12935,30 @@ "responses" : { "200" : { "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -9946,6 +12998,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -9974,6 +13050,71 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" + } + }, + "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] + } + }, + "/tck/tenant/{tenantId}/modeling_command_aggregate_without_ctor_parameters/snapshot/analyze" : { + "post" : { + "operationId" : "tck.modeling_command_aggregate_without_ctor_parameters.tenant.snapshot.analyze", + "parameters" : [ { + "$ref" : "#/components/parameters/wow.tenantId" + } ], + "requestBody" : { + "$ref" : "#/components/requestBodies/wow.AnalyticsQuery" + }, + "responses" : { + "200" : { + "$ref" : "#/components/responses/wow.AnalyticsPage" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -9992,11 +13133,29 @@ "200" : { "$ref" : "#/components/responses/wow.CountQuery" }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "408" : { "$ref" : "#/components/responses/wow.RequestTimeout" }, "429" : { "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -10036,6 +13195,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -10075,6 +13258,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -10103,6 +13310,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -10131,6 +13362,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -10160,8 +13415,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -10191,8 +13470,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -10306,6 +13609,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -10405,8 +13732,32 @@ } } }, - "404" : { - "$ref" : "#/components/responses/wow.NotFound" + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "404" : { + "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "tck.modeling_command_aggregate_without_ctor_parameters" ] @@ -11168,8 +14519,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -11379,6 +14754,30 @@ "responses" : { "200" : { "$ref" : "#/components/responses/wow.CountQuery" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -11420,6 +14819,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -11450,6 +14873,73 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" + } + }, + "tags" : [ "example.order" ] + } + }, + "/tenant/{tenantId}/sales-order/snapshot/analyze" : { + "post" : { + "operationId" : "example.order.tenant.snapshot.analyze", + "parameters" : [ { + "$ref" : "#/components/parameters/wow.Wow-Space-Id" + }, { + "$ref" : "#/components/parameters/wow.tenantId" + } ], + "requestBody" : { + "$ref" : "#/components/requestBodies/wow.AnalyticsQuery" + }, + "responses" : { + "200" : { + "$ref" : "#/components/responses/wow.AnalyticsPage" + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -11470,11 +14960,29 @@ "200" : { "$ref" : "#/components/responses/wow.CountQuery" }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "408" : { "$ref" : "#/components/responses/wow.RequestTimeout" }, "429" : { "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -11516,6 +15024,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -11557,6 +15089,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -11587,6 +15143,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -11617,6 +15197,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -11648,8 +15252,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -11681,8 +15309,32 @@ } } }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, "404" : { "$ref" : "#/components/responses/wow.NotFound" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] @@ -11727,6 +15379,30 @@ "$ref" : "#/components/headers/wow.Wow-Error-Code" } } + }, + "400" : { + "$ref" : "#/components/responses/wow.BadRequest" + }, + "403" : { + "$ref" : "#/components/responses/wow.QueryAccessDenied" + }, + "408" : { + "$ref" : "#/components/responses/wow.RequestTimeout" + }, + "429" : { + "$ref" : "#/components/responses/wow.TooManyRequests" + }, + "500" : { + "$ref" : "#/components/responses/wow.QueryInternalFailure" + }, + "502" : { + "$ref" : "#/components/responses/wow.QueryIncompleteResult" + }, + "503" : { + "$ref" : "#/components/responses/wow.QueryBackendUnavailable" + }, + "504" : { + "$ref" : "#/components/responses/wow.QueryBackendTimeout" } }, "tags" : [ "example.order" ] diff --git a/wow-query/build.gradle.kts b/wow-query/build.gradle.kts index c9bfd01a7a3..15ed9c09011 100644 --- a/wow-query/build.gradle.kts +++ b/wow-query/build.gradle.kts @@ -2,4 +2,11 @@ dependencies { api(project(":wow-core")) testImplementation("io.projectreactor:reactor-test") testImplementation(project(":wow-tck")) -} \ No newline at end of file +} + +kotlin { + compilerOptions { + optIn.add("me.ahoo.wow.query.backend.ExperimentalQueryBackendApi") + optIn.add("me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi") + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/analytics/AnalyticsQueryService.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/analytics/AnalyticsQueryService.kt new file mode 100644 index 00000000000..6949e14de2a --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/analytics/AnalyticsQueryService.kt @@ -0,0 +1,61 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.analytics + +import me.ahoo.wow.api.modeling.NamedAggregateDecorator +import me.ahoo.wow.api.query.analytics.AnalyticsPage +import me.ahoo.wow.api.query.analytics.AnalyticsQuery +import me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi +import me.ahoo.wow.query.gateway.QueryExecutionMode +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.gateway.QueryTrustedContext +import me.ahoo.wow.query.gateway.QueryValidationMode +import reactor.core.publisher.Mono + +/** Target-bound Snapshot analytics service. It is intentionally separate from the seven-method QueryService. */ +interface AnalyticsQueryService : NamedAggregateDecorator { + fun analyze(query: AnalyticsQuery): Mono +} + +interface AnalyticsQueryServiceFactory { + fun create(namedAggregate: me.ahoo.wow.api.modeling.NamedAggregate): AnalyticsQueryService +} + +@ExperimentalQueryGatewayApi +data class AnalyticsQueryTrustedContextRequest( + val target: QueryTarget, + val executionMode: QueryExecutionMode, + val validationMode: QueryValidationMode, +) + +@ExperimentalQueryGatewayApi +fun interface AnalyticsQueryTrustedContextResolver { + fun resolve(request: AnalyticsQueryTrustedContextRequest): Mono +} + +@ExperimentalQueryGatewayApi +class CompositeAnalyticsQueryTrustedContextResolver( + resolvers: Iterable, +) : AnalyticsQueryTrustedContextResolver { + private val resolvers = resolvers.toList() + + init { + require(this.resolvers.isNotEmpty()) { "At least one Analytics trusted context resolver is required." } + } + + override fun resolve(request: AnalyticsQueryTrustedContextRequest): Mono = + reactor.core.publisher.Flux.fromIterable(resolvers) + .concatMap { resolver -> Mono.defer { resolver.resolve(request) } } + .next() +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/AnalyticsQueryBackend.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/AnalyticsQueryBackend.kt new file mode 100644 index 00000000000..3a2f3b18b82 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/AnalyticsQueryBackend.kt @@ -0,0 +1,309 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.backend + +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryTarget +import reactor.core.publisher.Mono +import java.math.RoundingMode +import java.util.Collections +import java.util.LinkedHashMap + +@ExperimentalQueryBackendApi +@JvmInline +value class AnalyticsAlias(val value: String) { + init { + require(value.isNotBlank()) { "Analytics alias must not be blank." } + require(value.length <= MAX_ALIAS_LENGTH) { "Analytics alias must not exceed $MAX_ALIAS_LENGTH characters." } + require(value.none(Char::isISOControl)) { "Analytics alias must not contain control characters." } + require('.' !in value && '$' !in value) { "Analytics alias must be a safe backend field name." } + } + + private companion object { + const val MAX_ALIAS_LENGTH = 128 + } +} + +@ExperimentalQueryBackendApi +enum class BackendAnalyticsMissingPolicy { + EXCLUDE, + AS_NULL_BUCKET, +} + +@ExperimentalQueryBackendApi +data class BackendAnalyticsDimension( + val alias: AnalyticsAlias, + val field: QueryFieldId, + val missingPolicy: BackendAnalyticsMissingPolicy, +) + +@ExperimentalQueryBackendApi +sealed interface BackendAnalyticsGrouping { + data object Global : BackendAnalyticsGrouping + + class By(dimensions: Iterable) : BackendAnalyticsGrouping { + val dimensions: List = immutableNonEmpty(dimensions, "Analytics dimensions") + + override fun equals(other: Any?): Boolean = this === other || other is By && dimensions == other.dimensions + + override fun hashCode(): Int = dimensions.hashCode() + } +} + +@ExperimentalQueryBackendApi +sealed interface BackendAnalyticsMetric { + val alias: AnalyticsAlias + + data class DocumentCount(override val alias: AnalyticsAlias) : BackendAnalyticsMetric + + data class Min(override val alias: AnalyticsAlias, val field: QueryFieldId) : BackendAnalyticsMetric + + data class Max(override val alias: AnalyticsAlias, val field: QueryFieldId) : BackendAnalyticsMetric + + data class Sum(override val alias: AnalyticsAlias, val field: QueryFieldId) : BackendAnalyticsMetric + + data class Average(override val alias: AnalyticsAlias, val field: QueryFieldId) : BackendAnalyticsMetric +} + +@ExperimentalQueryBackendApi +sealed interface BackendAnalyticsCondition { + data object All : BackendAnalyticsCondition +} + +@ExperimentalQueryBackendApi +enum class BackendAnalyticsNullPlacement { + FIRST, +} + +@ExperimentalQueryBackendApi +enum class BackendAnalyticsTextCollation { + BINARY, +} + +@ExperimentalQueryBackendApi +sealed interface BackendAnalyticsBucketOrder { + data object Global : BackendAnalyticsBucketOrder + + data class DimensionKeyAscending( + val nullPlacement: BackendAnalyticsNullPlacement, + val textCollation: BackendAnalyticsTextCollation, + ) : BackendAnalyticsBucketOrder +} + +@ExperimentalQueryBackendApi +class BackendAnalyticsPageWindow( + val limit: Int, + afterKey: Iterable? = null, +) { + val afterKey: List? = afterKey?.let { Collections.unmodifiableList(it.toList()) } + + init { + require(limit > 0) { "Analytics bucket limit must be positive." } + require(this.afterKey == null || this.afterKey.isNotEmpty()) { "Analytics after-key must not be empty." } + } + + override fun equals(other: Any?): Boolean = + this === other || other is BackendAnalyticsPageWindow && limit == other.limit && afterKey == other.afterKey + + override fun hashCode(): Int = 31 * limit + (afterKey?.hashCode() ?: 0) +} + +@ExperimentalQueryBackendApi +enum class BackendAnalyticsConsistency { + EVENTUAL, + SNAPSHOT, +} + +@ExperimentalQueryBackendApi +enum class BackendAnalyticsCompleteness { + EXACT, + APPROXIMATE, +} + +@ExperimentalQueryBackendApi +enum class BackendAnalyticsOverflowPolicy { + REJECT, +} + +@ExperimentalQueryBackendApi +enum class BackendAnalyticsNumericPromotion { + DECIMAL128, +} + +@ExperimentalQueryBackendApi +data class BackendAnalyticsNumericPolicy( + val promotion: BackendAnalyticsNumericPromotion, + val precision: Int, + val scale: Int, + val roundingMode: RoundingMode, + val overflowPolicy: BackendAnalyticsOverflowPolicy, +) { + init { + require(precision in 1..34) { "Analytics Decimal128 precision must be between 1 and 34." } + require(scale in 0..precision) { "Analytics numeric scale must be between zero and precision." } + } +} + +@ExperimentalQueryBackendApi +class BackendAnalyticsQueryPlan( + val target: QueryTarget, + val schemaContractId: SchemaContractId, + val filter: BackendEnforcedFilter, + val grouping: BackendAnalyticsGrouping, + metrics: Iterable, + val having: BackendAnalyticsCondition, + val bucketOrder: BackendAnalyticsBucketOrder, + val bucketWindow: BackendAnalyticsPageWindow, + val numericPolicy: BackendAnalyticsNumericPolicy?, + val requiredConsistency: BackendAnalyticsConsistency, + val requiredCompleteness: BackendAnalyticsCompleteness, + val requiredCapabilities: BackendRequiredCapabilities, + val semanticTier: SemanticTier, + val fingerprint: PlanFingerprint, +) { + val operation: QueryOperation = QueryOperation.ANALYZE + val metrics: List = immutableNonEmpty(metrics, "Analytics metrics") + + init { + val aliases = grouping.aliases() + this.metrics.map(BackendAnalyticsMetric::alias) + require(aliases.distinct().size == aliases.size) { "Analytics aliases must be unique." } + when (val currentGrouping = grouping) { + BackendAnalyticsGrouping.Global -> require( + bucketOrder == BackendAnalyticsBucketOrder.Global && + bucketWindow.afterKey == null && + bucketWindow.limit == 1, + ) { + "Global analytics must use limit one and global ordering without a cursor." + } + + is BackendAnalyticsGrouping.By -> { + require(bucketOrder is BackendAnalyticsBucketOrder.DimensionKeyAscending) { + "Grouped analytics must use dimension-key ordering." + } + require( + bucketWindow.afterKey == null || bucketWindow.afterKey.size == currentGrouping.dimensions.size, + ) { + "Analytics after-key arity must match the grouping dimensions." + } + } + } + } +} + +@ExperimentalQueryBackendApi +class BackendAnalyticsBucket( + keys: Map, + metrics: Map, +) { + val keys: Map = immutableAnalyticsValues(keys) + val metrics: Map = immutableAnalyticsValues(metrics) + + override fun equals(other: Any?): Boolean = + this === other || other is BackendAnalyticsBucket && keys == other.keys && metrics == other.metrics + + override fun hashCode(): Int = 31 * keys.hashCode() + metrics.hashCode() +} + +@ExperimentalQueryBackendApi +class BackendAnalyticsPage @JvmOverloads constructor( + buckets: Iterable, + afterKey: Iterable?, + val consistency: BackendAnalyticsConsistency, + val completeness: BackendAnalyticsCompleteness, + val cursorState: BackendAnalyticsCursorState? = null, +) { + val buckets: List = Collections.unmodifiableList(buckets.toList()) + val afterKey: List? = afterKey?.let { Collections.unmodifiableList(it.toList()) } + + override fun equals(other: Any?): Boolean = + this === other || + other is BackendAnalyticsPage && + buckets == other.buckets && + afterKey == other.afterKey && + consistency == other.consistency && + completeness == other.completeness && + cursorState == other.cursorState + + override fun hashCode(): Int { + var result = buckets.hashCode() + result = 31 * result + (afterKey?.hashCode() ?: 0) + result = 31 * result + consistency.hashCode() + result = 31 * result + completeness.hashCode() + result = 31 * result + (cursorState?.hashCode() ?: 0) + return result + } +} + +/** Opaque, backend-owned continuation state. It is persisted server-side and never embedded in a public cursor. */ +@ExperimentalQueryBackendApi +class BackendAnalyticsCursorState(payload: ByteArray) { + private val frozenPayload = payload.copyOf() + + init { + require(frozenPayload.isNotEmpty()) { "Analytics cursor state must not be empty." } + } + + fun payload(): ByteArray = frozenPayload.copyOf() + + override fun equals(other: Any?): Boolean = + this === other || other is BackendAnalyticsCursorState && frozenPayload.contentEquals(other.frozenPayload) + + override fun hashCode(): Int = frozenPayload.contentHashCode() +} + +@ExperimentalQueryBackendApi +fun interface AnalyticsQueryBackend { + fun analyze(plan: BackendAnalyticsQueryPlan, options: QueryBackendExecutionOptions): Mono + + /** + * Continues a backend-owned snapshot when [cursorState] is present. Stateless backends reject instead of silently + * ignoring physical continuation state. + */ + fun analyze( + plan: BackendAnalyticsQueryPlan, + options: QueryBackendExecutionOptions, + cursorState: BackendAnalyticsCursorState?, + ): Mono = if (cursorState == null) { + analyze(plan, options) + } else { + Mono.error(QueryBackendException(QueryBackendFailureKind.UNSUPPORTED)) + } +} + +/** Closes backend-owned analytics cursor state transferred to the Gateway. Close must be idempotent. */ +@ExperimentalQueryBackendApi +fun interface AnalyticsQueryCursorLifecycle { + fun close(cursorState: BackendAnalyticsCursorState): Mono +} + +private fun BackendAnalyticsGrouping.aliases(): List = + when (this) { + BackendAnalyticsGrouping.Global -> emptyList() + is BackendAnalyticsGrouping.By -> dimensions.map(BackendAnalyticsDimension::alias) + } + +private fun immutableNonEmpty(values: Iterable, name: String): List = + Collections.unmodifiableList(values.toList()).also { result -> + require(result.isNotEmpty()) { "$name must not be empty." } + } + +private fun immutableAnalyticsValues( + values: Map, +): Map { + val copy = LinkedHashMap(values.size) + values.entries.sortedBy { entry -> entry.key.value }.forEach { entry -> copy[entry.key] = entry.value } + return Collections.unmodifiableMap(copy) +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/ExperimentalQueryBackendApi.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/ExperimentalQueryBackendApi.kt new file mode 100644 index 00000000000..3d17133c0d0 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/ExperimentalQueryBackendApi.kt @@ -0,0 +1,28 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.backend + +/** + * Marks the additive Backend-facing Query Plan and execution SPI. + * + * Backends receive only immutable, validated logical plans. They must not accept wire query DTOs, driver-native input, + * or infer physical fields from logical names. + */ +@RequiresOptIn( + level = RequiresOptIn.Level.WARNING, + message = "The Query Backend SPI is experimental and may evolve before the next major release.", +) +@Retention(AnnotationRetention.BINARY) +@Target(AnnotationTarget.CLASS, AnnotationTarget.FUNCTION, AnnotationTarget.PROPERTY, AnnotationTarget.CONSTRUCTOR) +annotation class ExperimentalQueryBackendApi diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/NormalizedValue.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/NormalizedValue.kt new file mode 100644 index 00000000000..c13dd1c9f2c --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/NormalizedValue.kt @@ -0,0 +1,81 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.backend + +import java.math.BigDecimal +import java.time.Instant +import java.util.Collections +import java.util.LinkedHashMap + +/** Backend-neutral, deeply immutable value admitted into a validated Query Plan. */ +@ExperimentalQueryBackendApi +sealed interface NormalizedValue { + data object Null : NormalizedValue + + data class BooleanValue(val value: Boolean) : NormalizedValue + + data class Text(val value: String) : NormalizedValue + + data class Int64(val value: Long) : NormalizedValue + + class Decimal(value: BigDecimal) : NormalizedValue { + val value: BigDecimal = value.stripTrailingZeros() + + override fun equals(other: Any?): Boolean = + this === other || other is Decimal && value == other.value + + override fun hashCode(): Int = value.hashCode() + + override fun toString(): String = "Decimal(value=$value)" + } + + data class InstantValue(val value: Instant) : NormalizedValue + + class Bytes(value: ByteArray) : NormalizedValue { + private val value: ByteArray = value.copyOf() + + fun toByteArray(): ByteArray = value.copyOf() + + override fun equals(other: Any?): Boolean = + this === other || other is Bytes && value.contentEquals(other.value) + + override fun hashCode(): Int = value.contentHashCode() + + override fun toString(): String = "Bytes(size=${value.size})" + } + + class ListValue(values: Iterable) : NormalizedValue { + val values: List = Collections.unmodifiableList(values.toList()) + + override fun equals(other: Any?): Boolean = + this === other || other is ListValue && values == other.values + + override fun hashCode(): Int = values.hashCode() + + override fun toString(): String = values.toString() + } + + class ObjectValue(values: Map) : NormalizedValue { + val values: Map = Collections.unmodifiableMap(LinkedHashMap(values)) + private val orderedEntries: List> = + this.values.map { entry -> entry.key to entry.value } + + override fun equals(other: Any?): Boolean = + this === other || other is ObjectValue && orderedEntries == other.orderedEntries + + override fun hashCode(): Int = orderedEntries.hashCode() + + override fun toString(): String = values.toString() + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/QueryBackendComposition.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/QueryBackendComposition.kt new file mode 100644 index 00000000000..16e98c9e18d --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/QueryBackendComposition.kt @@ -0,0 +1,193 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.backend + +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryTarget +import java.util.Collections +import java.util.LinkedHashMap + +@ExperimentalQueryBackendApi +enum class BackendStreamSupport { + NONE, + BOUNDED_ONLY, +} + +/** One atomic, statically ready schema/backend registration. */ +@ExperimentalQueryBackendApi +class RecordQueryBackendContribution( + val schema: QueryDocumentSchema, + val backendId: BackendId, + supportedOperations: Set, + val streamSupport: BackendStreamSupport, + semanticTiers: Set, + fieldCapabilities: Map>, + searchScopes: Set = emptySet(), + val backend: RecordQueryBackend, + val analyticsBackend: AnalyticsQueryBackend? = null, + val mappingGenerationDigest: String = schema.contractId.value, +) { + val supportedOperations: Set = Collections.unmodifiableSet( + LinkedHashSet(supportedOperations.sortedBy(QueryOperation::name)), + ) + val semanticTiers: Set = Collections.unmodifiableSet( + LinkedHashSet(semanticTiers.sortedBy(SemanticTier::name)), + ) + val fieldCapabilities: Map> = immutableCapabilities(fieldCapabilities) + val searchScopes: Set = Collections.unmodifiableSet( + LinkedHashSet(searchScopes.sortedBy(SearchScopeId::value)), + ) + + init { + require(this.supportedOperations.isNotEmpty()) { "Record backend operations must not be empty." } + require(this.supportedOperations.all { operation -> operation in QUERY_OPERATIONS }) { + "Query backend contribution supports record operations and ANALYZE only." + } + require((QueryOperation.ANALYZE in this.supportedOperations) == (analyticsBackend != null)) { + "ANALYZE requires exactly one analytics backend." + } + require(this.semanticTiers.isNotEmpty()) { "Record backend semantic tiers must not be empty." } + require(QueryOperation.STREAM in this.supportedOperations || streamSupport == BackendStreamSupport.NONE) { + "Stream support must be NONE when STREAM is not registered." + } + require(QueryOperation.STREAM !in this.supportedOperations || streamSupport != BackendStreamSupport.NONE) { + "STREAM requires an explicit stream support level." + } + require( + mappingGenerationDigest.length == SHA_256_HEX_LENGTH && + mappingGenerationDigest.all { character -> character in HEX_CHARACTERS }, + ) { + "Mapping generation digest must be lowercase SHA-256 hex." + } + this.fieldCapabilities.forEach { (field, capabilities) -> + val schemaField = requireNotNull(schema.fields[field]) { + "Backend field capability $field is not declared by the logical schema." + } + require(schemaField.capabilities.containsAll(capabilities)) { + "Backend field capability $field exceeds the logical schema contract." + } + } + require(schema.searchScopes.keys.containsAll(this.searchScopes)) { + "Backend search scopes must be declared by the logical schema." + } + } + + private fun immutableCapabilities( + capabilities: Map>, + ): Map> { + val copy = LinkedHashMap>(capabilities.size) + capabilities.entries.sortedWith(compareBy(QUERY_FIELD_ID_COMPARATOR) { entry -> entry.key }).forEach { entry -> + copy[entry.key] = Collections.unmodifiableSet( + LinkedHashSet(entry.value.sortedBy(FieldCapability::name)), + ) + } + return Collections.unmodifiableMap(copy) + } + + private companion object { + const val SHA_256_HEX_LENGTH = 64 + const val HEX_CHARACTERS = "0123456789abcdef" + val QUERY_OPERATIONS = setOf( + QueryOperation.SINGLE, + QueryOperation.STREAM, + QueryOperation.PAGE, + QueryOperation.COUNT, + QueryOperation.ANALYZE, + ) + } +} + +/** A configured Backend route whose logical schema is known but whose static readiness has not been attested. */ +@ExperimentalQueryBackendApi +data class RecordQueryBackendNotReady( + val schema: QueryDocumentSchema, + val backendId: BackendId, +) + +@ExperimentalQueryBackendApi +class QueryBackendComposition( + contributions: Iterable, + notReadyBackends: Iterable, + defaultRoutes: Map, +) { + val contributions: List + val notReadyBackends: List + val defaultRoutes: Map + + init { + val contributionList = contributions.toList() + val notReadyList = notReadyBackends.toList() + require( + contributionList.map { contribution -> contribution.schema.target to contribution.backendId } + .distinct().size == contributionList.size, + ) { + "Query backend contribution keys must be unique." + } + require( + notReadyList.map { backend -> backend.schema.target to backend.backendId }.distinct().size == + notReadyList.size, + ) { + "Not-ready Query backend keys must be unique." + } + require( + contributionList.map { contribution -> contribution.schema.target to contribution.backendId }.toSet() + .intersect(notReadyList.map { backend -> backend.schema.target to backend.backendId }.toSet()) + .isEmpty(), + ) { + "A Query backend key cannot be ready and not-ready at the same time." + } + this.contributions = Collections.unmodifiableList( + contributionList.sortedWith( + compareBy { it.schema.target.namedAggregate.contextName } + .thenBy { it.schema.target.namedAggregate.aggregateName } + .thenBy { it.schema.target.documentKind.name } + .thenBy { it.backendId.value }, + ), + ) + this.notReadyBackends = Collections.unmodifiableList( + notReadyList.sortedWith( + compareBy { it.schema.target.namedAggregate.contextName } + .thenBy { it.schema.target.namedAggregate.aggregateName } + .thenBy { it.schema.target.documentKind.name } + .thenBy { it.backendId.value }, + ), + ) + val routes = LinkedHashMap(defaultRoutes.size) + defaultRoutes.entries.sortedWith( + compareBy> { it.key.namedAggregate.contextName } + .thenBy { it.key.namedAggregate.aggregateName } + .thenBy { it.key.documentKind.name }, + ).forEach { entry -> routes[entry.key] = entry.value } + this.defaultRoutes = Collections.unmodifiableMap(routes) + this.defaultRoutes.forEach { (target, backendId) -> + require( + this.contributions.any { it.schema.target == target && it.backendId == backendId } || + this.notReadyBackends.any { it.schema.target == target && it.backendId == backendId }, + ) { + "Default query backend route $target/$backendId is not registered." + } + } + } + + constructor( + contributions: Iterable, + defaultRoutes: Map, + ) : this(contributions, emptyList(), defaultRoutes) + + companion object { + val EMPTY = QueryBackendComposition(emptyList(), emptyList(), emptyMap()) + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/QueryBackendTypes.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/QueryBackendTypes.kt new file mode 100644 index 00000000000..2ccb372bd0f --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/QueryBackendTypes.kt @@ -0,0 +1,114 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.backend + +@ExperimentalQueryBackendApi +enum class QueryResultShape { + TYPED, + DYNAMIC, + COUNT, + ANALYTICS, +} + +@ExperimentalQueryBackendApi +enum class RecordResultShape { + TYPED, + DYNAMIC, +} + +@ExperimentalQueryBackendApi +enum class SystemFieldKind { + IDENTITY, + AGGREGATE_ID, + TENANT_ID, + OWNER_ID, + SPACE_ID, + DELETED, +} + +@ExperimentalQueryBackendApi +enum class JunctionOperator { + AND, + OR, + NOR, +} + +@ExperimentalQueryBackendApi +enum class PredicateOperator(val requiresValue: Boolean) { + EQ(true), + NE(true), + GT(true), + LT(true), + GTE(true), + LTE(true), + CONTAINS(true), + IN(true), + NOT_IN(true), + BETWEEN(true), + ALL_IN(true), + STARTS_WITH(true), + ENDS_WITH(true), + IS_NULL(false), + NOT_NULL(false), + IS_TRUE(false), + IS_FALSE(false), + EXISTS(true), +} + +@ExperimentalQueryBackendApi +enum class CaseSensitivity { + SENSITIVE, + INSENSITIVE, +} + +@ExperimentalQueryBackendApi +data class NormalizedPredicateOptions( + val caseSensitivity: CaseSensitivity = CaseSensitivity.SENSITIVE, +) + +@ExperimentalQueryBackendApi +@JvmInline +value class SearchScopeId(val value: String) { + init { + require(value.isNotBlank()) { + "Search scope id must not be blank." + } + } +} + +@ExperimentalQueryBackendApi +@JvmInline +value class BackendId(val value: String) { + init { + require(value.isNotBlank()) { + "Backend id must not be blank." + } + } +} + +@ExperimentalQueryBackendApi +@JvmInline +value class Utf8Json(val value: String) { + init { + require(value.isNotBlank()) { + "Native JSON must not be blank." + } + } +} + +@ExperimentalQueryBackendApi +enum class NormalizedSortDirection { + ASC, + DESC, +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/QueryDocumentSchema.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/QueryDocumentSchema.kt new file mode 100644 index 00000000000..04e4e1d9fe5 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/QueryDocumentSchema.kt @@ -0,0 +1,262 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.backend + +import me.ahoo.wow.query.gateway.QueryTarget +import java.io.ByteArrayOutputStream +import java.io.DataOutputStream +import java.security.MessageDigest +import java.util.Collections +import java.util.LinkedHashMap + +@JvmInline +@ExperimentalQueryBackendApi +value class SchemaContractId(val value: String) { + init { + require(value.matches(HEX_PATTERN)) { + "Schema contract id must be a SHA-256 hex string." + } + } + + private companion object { + val HEX_PATTERN = Regex("[0-9a-f]{64}") + } +} + +@ExperimentalQueryBackendApi +class QueryDocumentSchema( + val target: QueryTarget, + fields: Iterable, + searchScopes: Iterable, +) { + val fields: Map + private val fieldAliases: Map + val searchScopes: Map + val contractId: SchemaContractId + + init { + val fieldList = fields.toList() + require(fieldList.map(QueryFieldSchema::id).distinct().size == fieldList.size) { + "Query field ids must be unique." + } + val sortedFields = fieldList.sortedWith(compareBy(QUERY_FIELD_ID_COMPARATOR, QueryFieldSchema::id)) + val fieldMap = LinkedHashMap(sortedFields.size) + sortedFields.forEach { fieldMap[it.id] = it } + validatePathPrefixes(fieldMap) + this.fields = Collections.unmodifiableMap(fieldMap) + fieldAliases = buildFieldAliases(sortedFields, fieldMap) + + val searchScopeList = searchScopes.toList() + require(searchScopeList.map(QuerySearchScopeDefinition::id).distinct().size == searchScopeList.size) { + "Search scope ids must be unique." + } + val scopeMap = LinkedHashMap(searchScopeList.size) + val legacyKeys = mutableSetOf>() + searchScopeList.sortedBy { it.id.value }.forEach { definition -> + validateSearchScope(definition, fieldMap) + definition.legacyAliases.forEach { alias -> + require(legacyKeys.add(definition.owner to alias)) { + "Legacy search scope alias $alias is ambiguous for owner ${definition.owner}." + } + } + scopeMap[definition.id] = definition + } + this.searchScopes = Collections.unmodifiableMap(scopeMap) + contractId = SchemaContractId(SchemaContractEncoder.encode(this)) + } + + fun resolveLegacySearchScope( + owner: QueryFieldId.Path?, + alias: QueryFieldId.Path, + ): QuerySearchScopeDefinition? = searchScopes.values.singleOrNull { definition -> + definition.owner == owner && alias in definition.legacyAliases + } + + fun resolveField(id: QueryFieldId): QueryFieldId? = + when { + id in fields -> id + id is QueryFieldId.Path -> fieldAliases[id] + else -> null + } + + fun elementOwner(field: QueryFieldId.Path): QueryFieldId.Path? = nearestElementOwner(field, fields) + + private fun buildFieldAliases( + fields: List, + fieldMap: Map, + ): Map { + val aliases = LinkedHashMap() + fields.forEach { field -> + field.logicalAliases.forEach { alias -> + require(alias !in fieldMap) { + "Query field alias $alias conflicts with a canonical field." + } + require(aliases.put(alias, field.id) == null) { + "Query field alias $alias must be unique." + } + if (field.id is QueryFieldId.Path) { + require(nearestElementOwner(alias, fieldMap) == nearestElementOwner(field.id, fieldMap)) { + "Query field alias $alias must belong to the same element owner as ${field.id}." + } + } else { + require(nearestElementOwner(alias, fieldMap) == null) { + "System field alias $alias must remain at the root scope." + } + } + } + } + return Collections.unmodifiableMap(aliases) + } + + private fun validatePathPrefixes(fields: Map) { + fields.keys.filterIsInstance().forEach { path -> + for (prefixLength in 1 until path.segments.size) { + val prefix = QueryFieldId.Path(path.segments.take(prefixLength)) + val prefixSchema = requireNotNull(fields[prefix]) { + "Missing query field prefix $prefix for $path." + } + require(prefixSchema.type.isObjectContainer()) { + "Query field prefix $prefix must be an object container." + } + } + } + } + + private fun validateSearchScope( + definition: QuerySearchScopeDefinition, + fields: Map, + ) { + definition.owner?.let { owner -> + val ownerSchema = requireNotNull(fields[owner]) { "Search scope owner $owner is not declared." } + require(FieldCapability.ELEMENT_MATCH in ownerSchema.capabilities) { + "Search scope owner $owner must declare ELEMENT_MATCH." + } + } + definition.fields.forEach { field -> + val schema = requireNotNull(fields[field]) { + "Search scope field $field is not declared." + } + require(FieldCapability.FULL_TEXT in schema.capabilities) { + "Search scope field $field must declare FULL_TEXT." + } + require(nearestElementOwner(field, fields) == definition.owner) { + "Search scope field $field does not belong to owner ${definition.owner}." + } + } + definition.legacyAliases.forEach { alias -> + require(alias in fields) { + "Legacy search scope alias $alias is not a declared field." + } + require(nearestElementOwner(alias, fields) == definition.owner) { + "Legacy search scope alias $alias does not belong to owner ${definition.owner}." + } + } + } + + private fun nearestElementOwner( + field: QueryFieldId.Path, + fields: Map, + ): QueryFieldId.Path? = + (1 until field.segments.size) + .map { size -> QueryFieldId.Path(field.segments.take(size)) } + .lastOrNull { candidate -> FieldCapability.ELEMENT_MATCH in fields[candidate]?.capabilities.orEmpty() } +} + +private fun LogicalFieldType.isObjectContainer(): Boolean = + this == LogicalFieldType.Object || this is LogicalFieldType.Array && elementType == LogicalFieldType.Object + +private object SchemaContractEncoder { + fun encode(schema: QueryDocumentSchema): String { + val bytes = ByteArrayOutputStream() + DataOutputStream(bytes).use { output -> + output.writeUtf8("query-schema-v1") + output.writeUtf8(schema.target.namedAggregate.contextName) + output.writeUtf8(schema.target.namedAggregate.aggregateName) + output.writeUtf8(schema.target.documentKind.name) + output.writeInt(schema.fields.size) + schema.fields.values.forEach { field -> output.writeField(field) } + output.writeInt(schema.searchScopes.size) + schema.searchScopes.values.forEach { scope -> output.writeScope(scope) } + } + return MessageDigest.getInstance("SHA-256").digest(bytes.toByteArray()).toHex() + } + + private fun DataOutputStream.writeField(field: QueryFieldSchema) { + writeFieldId(field.id) + writeType(field.type) + writeUtf8(field.presence.name) + writeUtf8(field.nullability.name) + writeStrings(field.allowedOperators.map { it.name }.sorted()) + writeStrings(field.capabilities.map { it.name }.sorted()) + writeFieldIds(field.logicalAliases.sortedWith(QUERY_FIELD_PATH_COMPARATOR)) + } + + private fun DataOutputStream.writeScope(scope: QuerySearchScopeDefinition) { + writeUtf8(scope.id.value) + writeBoolean(scope.owner != null) + scope.owner?.let { owner -> writeFieldId(owner) } + writeFieldIds(scope.fields) + writeFieldIds(scope.legacyAliases.sortedWith(QUERY_FIELD_PATH_COMPARATOR)) + } + + private fun DataOutputStream.writeType(type: LogicalFieldType) { + when (type) { + LogicalFieldType.Text -> writeUtf8("text") + LogicalFieldType.Boolean -> writeUtf8("boolean") + LogicalFieldType.Int64 -> writeUtf8("int64") + LogicalFieldType.Decimal -> writeUtf8("decimal") + LogicalFieldType.Instant -> writeUtf8("instant") + LogicalFieldType.Bytes -> writeUtf8("bytes") + LogicalFieldType.Object -> writeUtf8("object") + is LogicalFieldType.Array -> { + writeUtf8("array") + writeType(type.elementType) + writeUtf8(type.elementNullability.name) + writeUtf8(type.emptySemantics.name) + } + } + } + + private fun DataOutputStream.writeFieldIds(ids: List) { + writeInt(ids.size) + ids.forEach { id -> writeFieldId(id) } + } + + private fun DataOutputStream.writeFieldId(id: QueryFieldId) { + when (id) { + is QueryFieldId.System -> { + writeByte(0) + writeUtf8(id.kind.name) + } + + is QueryFieldId.Path -> { + writeByte(1) + writeStrings(id.segments) + } + } + } + + private fun DataOutputStream.writeStrings(values: List) { + writeInt(values.size) + values.forEach { value -> writeUtf8(value) } + } + + private fun DataOutputStream.writeUtf8(value: String) { + val bytes = value.toByteArray(Charsets.UTF_8) + writeInt(bytes.size) + write(bytes) + } + + private fun ByteArray.toHex(): String = joinToString("") { byte -> "%02x".format(byte) } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/QueryFieldSchema.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/QueryFieldSchema.kt new file mode 100644 index 00000000000..95bfde57612 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/QueryFieldSchema.kt @@ -0,0 +1,336 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.backend + +import java.util.Collections + +@ExperimentalQueryBackendApi +sealed interface QueryFieldId { + data class System(val kind: SystemFieldKind) : QueryFieldId + + class Path(segments: Iterable) : QueryFieldId { + val segments: List = Collections.unmodifiableList(segments.toList()) + + init { + require(this.segments.isNotEmpty()) { + "Query field path must not be empty." + } + require(this.segments.none(String::isBlank)) { + "Query field path segments must not be blank." + } + require(this.segments.none { segment -> segment.any(Char::isISOControl) }) { + "Query field path segments must not contain control characters." + } + } + + override fun equals(other: Any?): Boolean = this === other || other is Path && segments == other.segments + + override fun hashCode(): Int = segments.hashCode() + + override fun toString(): String = segments.joinToString(".") + } +} + +@ExperimentalQueryBackendApi +enum class Presence { + REQUIRED, + OPTIONAL, +} + +@ExperimentalQueryBackendApi +enum class Nullability { + NON_NULL, + NULLABLE, +} + +@ExperimentalQueryBackendApi +enum class EmptyArraySemantics { + DISTINCT, + COLLAPSES_TO_MISSING, +} + +@ExperimentalQueryBackendApi +sealed interface LogicalFieldType { + data object Text : LogicalFieldType + + data object Boolean : LogicalFieldType + + data object Int64 : LogicalFieldType + + data object Decimal : LogicalFieldType + + data object Instant : LogicalFieldType + + data object Bytes : LogicalFieldType + + data object Object : LogicalFieldType + + data class Array( + val elementType: LogicalFieldType, + val elementNullability: Nullability, + val emptySemantics: EmptyArraySemantics, + ) : LogicalFieldType +} + +@ExperimentalQueryBackendApi +enum class FieldCapability { + EXACT, + PRESENCE, + RANGE, + FULL_TEXT, + LITERAL_PATTERN, + SORTABLE, + PROJECTABLE, + AGGREGATABLE, + ELEMENT_MATCH, + ELEMENT_NULL, +} + +@ExperimentalQueryBackendApi +class QueryFieldSchema( + val id: QueryFieldId, + val type: LogicalFieldType, + val presence: Presence, + val nullability: Nullability, + allowedOperators: Iterable, + capabilities: Iterable, + logicalAliases: Iterable = emptyList(), +) { + val allowedOperators: Set = immutableEnumSet(allowedOperators) + val capabilities: Set = immutableEnumSet(capabilities) + val logicalAliases: Set = Collections.unmodifiableSet( + LinkedHashSet(logicalAliases.sortedWith(QUERY_FIELD_PATH_COMPARATOR)), + ) + + init { + if (FieldCapability.ELEMENT_MATCH in this.capabilities) { + require(type is LogicalFieldType.Array && type.elementType == LogicalFieldType.Object) { + "ELEMENT_MATCH requires an array of objects." + } + } + if (FieldCapability.FULL_TEXT in this.capabilities) { + require(type.operandType() == LogicalFieldType.Text) { + "FULL_TEXT requires a text field." + } + } + if (FieldCapability.LITERAL_PATTERN in this.capabilities) { + require(type.operandType() == LogicalFieldType.Text) { + "LITERAL_PATTERN requires a text field." + } + } + if (type.containsObjectValue()) { + require(FieldCapability.EXACT !in this.capabilities) { + "Object values cannot declare EXACT capability." + } + } + if (FieldCapability.RANGE in this.capabilities) { + require(type.operandType().isOrderedScalar()) { + "RANGE requires a numeric or instant field." + } + } + if (FieldCapability.SORTABLE in this.capabilities || FieldCapability.AGGREGATABLE in this.capabilities) { + require(type.isPortableScalar()) { + "SORTABLE and AGGREGATABLE require a portable scalar field." + } + } + this.allowedOperators.forEach { operator -> + require(operator.requiredCapability() in this.capabilities) { + "Operator $operator requires ${operator.requiredCapability()}." + } + if (operator == PredicateOperator.ALL_IN) { + require(type is LogicalFieldType.Array) { + "ALL_IN requires an array field." + } + } + if (operator == PredicateOperator.BETWEEN) { + require(type.operandType().isOrderedScalar()) { + "BETWEEN requires an ordered scalar or array element type." + } + } + } + } + + override fun equals(other: Any?): Boolean = + this === other || + other is QueryFieldSchema && + id == other.id && + type == other.type && + presence == other.presence && + nullability == other.nullability && + allowedOperators == other.allowedOperators && + capabilities == other.capabilities && + logicalAliases == other.logicalAliases + + override fun hashCode(): Int { + var result = id.hashCode() + result = 31 * result + type.hashCode() + result = 31 * result + presence.hashCode() + result = 31 * result + nullability.hashCode() + result = 31 * result + allowedOperators.hashCode() + result = 31 * result + capabilities.hashCode() + result = 31 * result + logicalAliases.hashCode() + return result + } +} + +internal fun QueryFieldSchema.accepts(value: NormalizedValue): Boolean = + if (value == NormalizedValue.Null) { + nullability == Nullability.NULLABLE + } else { + type.accepts(value) + } + +internal fun LogicalFieldType.accepts(value: NormalizedValue): Boolean = + when (this) { + LogicalFieldType.Text -> value is NormalizedValue.Text + LogicalFieldType.Boolean -> value is NormalizedValue.BooleanValue + LogicalFieldType.Int64 -> value is NormalizedValue.Int64 + LogicalFieldType.Decimal -> value is NormalizedValue.Decimal || value is NormalizedValue.Int64 + LogicalFieldType.Instant -> value is NormalizedValue.InstantValue + LogicalFieldType.Bytes -> value is NormalizedValue.Bytes + LogicalFieldType.Object -> value is NormalizedValue.ObjectValue + is LogicalFieldType.Array -> + value is NormalizedValue.ListValue && value.values.all(::acceptsElement) + } + +internal fun QueryFieldSchema.acceptsOperand( + operator: PredicateOperator, + value: NormalizedValue, +): Boolean { + if (value == NormalizedValue.Null) { + return when (operator) { + PredicateOperator.EQ, + PredicateOperator.NE, + PredicateOperator.IN, + PredicateOperator.NOT_IN, + PredicateOperator.ALL_IN, + -> true + + else -> false + } + } + val operandType = if (type is LogicalFieldType.Array) type.elementType else type + return operandType.accepts(value) +} + +internal fun QueryFieldSchema.hasOperandType(expected: LogicalFieldType): Boolean = type.operandType() == expected + +private fun LogicalFieldType.Array.acceptsElement(value: NormalizedValue): Boolean = + if (value == NormalizedValue.Null) { + elementNullability == Nullability.NULLABLE + } else { + elementType.accepts(value) + } + +@ExperimentalQueryBackendApi +class QuerySearchScopeDefinition( + val id: SearchScopeId, + val owner: QueryFieldId.Path?, + fields: Iterable, + legacyAliases: Iterable, +) { + val fields: List + val legacyAliases: Set = Collections.unmodifiableSet( + LinkedHashSet(legacyAliases.sortedWith(QUERY_FIELD_PATH_COMPARATOR)), + ) + + init { + val materializedFields = fields.toList() + require(materializedFields.isNotEmpty()) { + "Search scope fields must not be empty." + } + require(materializedFields.distinct().size == materializedFields.size) { + "Search scope fields must be unique." + } + this.fields = Collections.unmodifiableList( + materializedFields.sortedWith(QUERY_FIELD_PATH_COMPARATOR), + ) + } + + override fun equals(other: Any?): Boolean = + this === other || + other is QuerySearchScopeDefinition && + id == other.id && + owner == other.owner && + fields == other.fields && + legacyAliases == other.legacyAliases + + override fun hashCode(): Int { + var result = id.hashCode() + result = 31 * result + (owner?.hashCode() ?: 0) + result = 31 * result + fields.hashCode() + result = 31 * result + legacyAliases.hashCode() + return result + } +} + +private fun > immutableEnumSet(values: Iterable): Set = + Collections.unmodifiableSet(LinkedHashSet(values.sortedBy(Enum::name))) + +private fun LogicalFieldType.isOrderedScalar(): Boolean = + this == LogicalFieldType.Int64 || this == LogicalFieldType.Decimal || this == LogicalFieldType.Instant + +private fun LogicalFieldType.containsObjectValue(): Boolean = + this == LogicalFieldType.Object || this is LogicalFieldType.Array && elementType.containsObjectValue() + +private fun LogicalFieldType.isPortableScalar(): Boolean = + this == LogicalFieldType.Text || this == LogicalFieldType.Boolean || isOrderedScalar() + +private fun LogicalFieldType.operandType(): LogicalFieldType = + if (this is LogicalFieldType.Array) elementType else this + +internal val QUERY_FIELD_PATH_COMPARATOR: Comparator = Comparator { left, right -> + compareSegments(left.segments, right.segments) +} + +internal val QUERY_FIELD_ID_COMPARATOR: Comparator = Comparator { left, right -> + when { + left is QueryFieldId.System && right is QueryFieldId.System -> left.kind.name.compareTo(right.kind.name) + left is QueryFieldId.System -> -1 + right is QueryFieldId.System -> 1 + else -> QUERY_FIELD_PATH_COMPARATOR.compare(left as QueryFieldId.Path, right as QueryFieldId.Path) + } +} + +private fun compareSegments(left: List, right: List): Int { + for (index in 0 until minOf(left.size, right.size)) { + val comparison = left[index].compareTo(right[index]) + if (comparison != 0) { + return comparison + } + } + return left.size.compareTo(right.size) +} + +private fun PredicateOperator.requiredCapability(): FieldCapability = + when (this) { + PredicateOperator.IS_NULL, + PredicateOperator.NOT_NULL, + PredicateOperator.EXISTS, + -> FieldCapability.PRESENCE + + PredicateOperator.GT, + PredicateOperator.LT, + PredicateOperator.GTE, + PredicateOperator.LTE, + PredicateOperator.BETWEEN, + -> FieldCapability.RANGE + + PredicateOperator.CONTAINS, + PredicateOperator.STARTS_WITH, + PredicateOperator.ENDS_WITH, + -> FieldCapability.LITERAL_PATTERN + + else -> FieldCapability.EXACT + } diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/RecordQueryBackend.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/RecordQueryBackend.kt new file mode 100644 index 00000000000..1ebf997f431 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/RecordQueryBackend.kt @@ -0,0 +1,153 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.backend + +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Instant +import java.util.Collections + +@ExperimentalQueryBackendApi +data class QueryBackendExecutionOptions( + val deadline: Instant?, + val maxReturnedRecords: Long?, + val maxScannedRecords: Long? = null, + val maxPageWindow: Long? = null, + val maxCandidateBuckets: Int? = null, + val maxReturnedBuckets: Int? = null, + val maxCursorPages: Int? = null, + val allowDiskUse: Boolean = false, +) { + init { + require(maxReturnedRecords == null || maxReturnedRecords > 0) { + "Maximum returned records must be positive." + } + require(maxScannedRecords == null || maxScannedRecords > 0) { + "Maximum scanned records must be positive." + } + require(maxPageWindow == null || maxPageWindow > 0) { + "Maximum page window must be positive." + } + require(maxCandidateBuckets == null || maxCandidateBuckets > 0) { + "Maximum candidate buckets must be positive." + } + require(maxReturnedBuckets == null || maxReturnedBuckets > 0) { + "Maximum returned buckets must be positive." + } + require(maxCursorPages == null || maxCursorPages > 0) { + "Maximum cursor pages must be positive." + } + } + + constructor(deadline: Instant?, maxReturnedRecords: Long?) : this( + deadline, + maxReturnedRecords, + null, + null, + null, + null, + null, + false, + ) +} + +@ExperimentalQueryBackendApi +data class BackendRecord( + val identity: String, + val document: NormalizedValue.ObjectValue, + val completeness: BackendRecordCompleteness, +) { + init { + require(identity.isNotBlank()) { "Backend record identity must not be blank." } + } +} + +@ExperimentalQueryBackendApi +enum class BackendRecordCompleteness { + COMPLETE, + UNKNOWN, +} + +@ExperimentalQueryBackendApi +enum class BackendTotalRelation { + EXACT, + LOWER_BOUND, + UNKNOWN, +} + +@ExperimentalQueryBackendApi +enum class BackendPageConsistency { + SAME_INPUT, + INDEPENDENT, + UNKNOWN, +} + +@ExperimentalQueryBackendApi +class BackendPage( + records: Iterable, + val total: Long, + val totalRelation: BackendTotalRelation, + val consistency: BackendPageConsistency, +) { + val records: List = Collections.unmodifiableList(records.toList()) + + init { + require(total >= 0) { "Backend page total must not be negative." } + } + + override fun equals(other: Any?): Boolean = + this === other || + other is BackendPage && + records == other.records && + total == other.total && + totalRelation == other.totalRelation && + consistency == other.consistency + + override fun hashCode(): Int { + var result = records.hashCode() + result = 31 * result + total.hashCode() + result = 31 * result + totalRelation.hashCode() + result = 31 * result + consistency.hashCode() + return result + } +} + +/** Storage SPI for validated, backend-neutral record plans. */ +@ExperimentalQueryBackendApi +interface RecordQueryBackend { + fun single(plan: BackendSingleQueryPlan, options: QueryBackendExecutionOptions): Mono + + fun stream(plan: BackendStreamQueryPlan, options: QueryBackendExecutionOptions): Flux + + fun page(plan: BackendPageQueryPlan, options: QueryBackendExecutionOptions): Mono = + Mono.error(QueryBackendException(QueryBackendFailureKind.UNSUPPORTED)) + + fun count(plan: BackendCountQueryPlan, options: QueryBackendExecutionOptions): Mono +} + +@ExperimentalQueryBackendApi +class QueryBackendException( + val kind: QueryBackendFailureKind, + cause: Throwable? = null, +) : IllegalStateException("Query backend failure: $kind", cause) + +@ExperimentalQueryBackendApi +enum class QueryBackendFailureKind { + UNAVAILABLE, + TIMEOUT, + BUDGET_EXCEEDED, + INCOMPLETE_RESULT, + MAPPING_FAILURE, + UNSUPPORTED, +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/RecordQueryPlan.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/RecordQueryPlan.kt new file mode 100644 index 00000000000..8309d6ef169 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/backend/RecordQueryPlan.kt @@ -0,0 +1,282 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.backend + +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryTarget +import java.util.Collections +import java.util.LinkedHashMap + +@ExperimentalQueryBackendApi +@JvmInline +value class PlanFingerprint(val value: String) { + init { + require(value.matches(HEX_PATTERN)) { "Plan fingerprint must be a SHA-256 hex string." } + } + + private companion object { + val HEX_PATTERN = Regex("[0-9a-f]{64}") + } +} + +@ExperimentalQueryBackendApi +enum class SemanticTier { + PORTABLE, + SEARCH, + NATIVE, +} + +@ExperimentalQueryBackendApi +sealed interface BackendPlannedCondition { + data object All : BackendPlannedCondition + + data object None : BackendPlannedCondition + + class Junction( + val operator: JunctionOperator, + children: Iterable, + ) : BackendPlannedCondition { + val children: List = Collections.unmodifiableList(children.toList()) + + init { + require(this.children.isNotEmpty()) { "Backend junction children must not be empty." } + } + + override fun equals(other: Any?): Boolean = + this === other || other is Junction && operator == other.operator && children == other.children + + override fun hashCode(): Int = 31 * operator.hashCode() + children.hashCode() + } + + data class Predicate( + val field: QueryFieldId, + val operator: PredicateOperator, + val value: NormalizedValue? = null, + val options: NormalizedPredicateOptions = NormalizedPredicateOptions(), + ) : BackendPlannedCondition { + init { + require(operator.requiresValue == (value != null)) { "Predicate value does not match operator $operator." } + } + } + + data class ElementMatch( + val field: QueryFieldId.Path, + val condition: BackendPlannedCondition, + ) : BackendPlannedCondition + + data class Search(val scope: SearchScopeId, val text: String) : BackendPlannedCondition { + init { + require(text.isNotBlank()) { "Search text must not be blank." } + } + } + + data class Native(val backendId: BackendId, val payload: Utf8Json) : BackendPlannedCondition +} + +@ExperimentalQueryBackendApi +data class BackendEnforcedFilter( + val user: BackendPlannedCondition, + val mandatory: BackendPlannedCondition, +) { + val condition: BackendPlannedCondition = BackendPlannedCondition.Junction( + JunctionOperator.AND, + listOf(user, mandatory), + ) +} + +@ExperimentalQueryBackendApi +class BackendRequiredCapabilities( + fieldRequirements: Map> = emptyMap(), + searchRequirements: Set = emptySet(), + val nativeBackend: BackendId? = null, +) { + val fieldRequirements: Map> = immutableRequirements(fieldRequirements) + val searchRequirements: Set = Collections.unmodifiableSet( + LinkedHashSet(searchRequirements.sortedBy(SearchScopeId::value)), + ) + + override fun equals(other: Any?): Boolean = + this === other || + other is BackendRequiredCapabilities && + fieldRequirements == other.fieldRequirements && + searchRequirements == other.searchRequirements && + nativeBackend == other.nativeBackend + + override fun hashCode(): Int = + 31 * (31 * fieldRequirements.hashCode() + searchRequirements.hashCode()) + (nativeBackend?.hashCode() ?: 0) + + private fun immutableRequirements( + requirements: Map>, + ): Map> { + val copy = LinkedHashMap>(requirements.size) + requirements.entries.sortedBy { entry -> entry.key.stableKey() }.forEach { (field, capabilities) -> + copy[field] = Collections.unmodifiableSet(LinkedHashSet(capabilities.sortedBy(FieldCapability::name))) + } + return Collections.unmodifiableMap(copy) + } +} + +@ExperimentalQueryBackendApi +sealed interface BackendProjection { + data object All : BackendProjection + + class Include(fields: Iterable) : BackendProjection { + val fields: List = immutableNonEmptyFields(fields) + + override fun equals(other: Any?): Boolean = this === other || other is Include && fields == other.fields + + override fun hashCode(): Int = fields.hashCode() + } + + class Exclude(fields: Iterable) : BackendProjection { + val fields: List = immutableNonEmptyFields(fields) + + override fun equals(other: Any?): Boolean = this === other || other is Exclude && fields == other.fields + + override fun hashCode(): Int = fields.hashCode() + } +} + +@ExperimentalQueryBackendApi +enum class BackendSortOrigin { + USER, + STABILITY_TIE_BREAKER, +} + +@ExperimentalQueryBackendApi +data class BackendSort( + val field: QueryFieldId, + val direction: NormalizedSortDirection, + val origin: BackendSortOrigin, +) + +@ExperimentalQueryBackendApi +sealed interface BackendRecordQueryPlan { + val target: QueryTarget + val operation: QueryOperation + val schemaContractId: SchemaContractId + val filter: BackendEnforcedFilter + val requiredCapabilities: BackendRequiredCapabilities + val semanticTier: SemanticTier + val fingerprint: PlanFingerprint +} + +@ExperimentalQueryBackendApi +sealed interface BackendRecordResultPlan : BackendRecordQueryPlan { + val resultShape: RecordResultShape + val projection: BackendProjection + val sort: List +} + +@ExperimentalQueryBackendApi +class BackendSingleQueryPlan( + override val target: QueryTarget, + override val schemaContractId: SchemaContractId, + override val filter: BackendEnforcedFilter, + override val resultShape: RecordResultShape, + override val projection: BackendProjection, + sort: Iterable, + override val requiredCapabilities: BackendRequiredCapabilities, + override val semanticTier: SemanticTier, + override val fingerprint: PlanFingerprint, +) : BackendRecordResultPlan { + override val operation: QueryOperation = QueryOperation.SINGLE + override val sort: List = Collections.unmodifiableList(sort.toList()) +} + +@ExperimentalQueryBackendApi +class BackendStreamQueryPlan( + override val target: QueryTarget, + override val schemaContractId: SchemaContractId, + override val filter: BackendEnforcedFilter, + override val resultShape: RecordResultShape, + override val projection: BackendProjection, + sort: Iterable, + val limit: Int, + override val requiredCapabilities: BackendRequiredCapabilities, + override val semanticTier: SemanticTier, + override val fingerprint: PlanFingerprint, +) : BackendRecordResultPlan { + override val operation: QueryOperation = QueryOperation.STREAM + override val sort: List = Collections.unmodifiableList(sort.toList()) + + init { + require(limit > 0) { "Backend stream limit must be positive." } + } +} + +@ExperimentalQueryBackendApi +data class BackendPageWindow( + val offset: Long, + val size: Int, +) { + init { + require(offset >= 0) { "Backend page offset must not be negative." } + require(size > 0) { "Backend page size must be positive." } + } +} + +@ExperimentalQueryBackendApi +enum class BackendTotalMode { + EXACT, +} + +@ExperimentalQueryBackendApi +enum class BackendRequiredConsistency { + SAME_INPUT, +} + +@ExperimentalQueryBackendApi +class BackendPageQueryPlan( + override val target: QueryTarget, + override val schemaContractId: SchemaContractId, + override val filter: BackendEnforcedFilter, + override val resultShape: RecordResultShape, + override val projection: BackendProjection, + sort: Iterable, + val page: BackendPageWindow, + val totalMode: BackendTotalMode, + val requiredConsistency: BackendRequiredConsistency, + override val requiredCapabilities: BackendRequiredCapabilities, + override val semanticTier: SemanticTier, + override val fingerprint: PlanFingerprint, +) : BackendRecordResultPlan { + override val operation: QueryOperation = QueryOperation.PAGE + override val sort: List = Collections.unmodifiableList(sort.toList()) +} + +@ExperimentalQueryBackendApi +class BackendCountQueryPlan( + override val target: QueryTarget, + override val schemaContractId: SchemaContractId, + override val filter: BackendEnforcedFilter, + override val requiredCapabilities: BackendRequiredCapabilities, + override val semanticTier: SemanticTier, + override val fingerprint: PlanFingerprint, +) : BackendRecordQueryPlan { + override val operation: QueryOperation = QueryOperation.COUNT +} + +private fun immutableNonEmptyFields(fields: Iterable): List = + Collections.unmodifiableList(fields.toList()).also { copy -> + require(copy.isNotEmpty()) { "Backend projection fields must not be empty." } + } + +private fun QueryFieldId.stableKey(): String = + when (this) { + is QueryFieldId.System -> "0:${kind.name}" + is QueryFieldId.Path -> "1:${segments.joinToString("\u0000") { segment -> "${segment.length}:$segment" }}" + } diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/cursor/QueryCursorLeaseStore.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/cursor/QueryCursorLeaseStore.kt new file mode 100644 index 00000000000..722f0f40a23 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/cursor/QueryCursorLeaseStore.kt @@ -0,0 +1,213 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.cursor + +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Duration +import java.time.Instant +import java.util.Collections + +/** Marks the additive persistent cursor-store SPI while its operational implementations mature. */ +@RequiresOptIn( + level = RequiresOptIn.Level.WARNING, + message = "The Query cursor-store SPI is experimental and may evolve before the next major release.", +) +@Retention(AnnotationRetention.BINARY) +@Target(AnnotationTarget.CLASS, AnnotationTarget.FUNCTION, AnnotationTarget.PROPERTY, AnnotationTarget.CONSTRUCTOR) +annotation class ExperimentalQueryCursorApi + +@ExperimentalQueryCursorApi +class QueryCursorHmacKey( + val id: Int, + secret: ByteArray, +) { + private val frozenSecret = secret.copyOf() + + init { + require(id in 1..UByte.MAX_VALUE.toInt()) { "Query cursor signing key id must fit one non-zero byte." } + require(frozenSecret.size >= MINIMUM_SECRET_BYTES) { + "Query cursor HMAC secret must contain at least 256 bits." + } + } + + @JvmSynthetic + internal fun secretCopy(): ByteArray = frozenSecret.copyOf() + + private companion object { + const val MINIMUM_SECRET_BYTES = 32 + } +} + +@ExperimentalQueryCursorApi +class QueryCursorSigningKeys( + val current: QueryCursorHmacKey, + previous: Iterable = emptyList(), +) { + val previous: List = Collections.unmodifiableList(previous.toList()) + + init { + val all = listOf(current) + this.previous + require(all.size <= MAX_KEYS) { "Query cursor signing key ring exceeds its bounded key count." } + require(all.map(QueryCursorHmacKey::id).distinct().size == all.size) { + "Query cursor signing key ids must be unique." + } + } + + private companion object { + const val MAX_KEYS = 4 + } +} + +@ExperimentalQueryCursorApi +data class QueryCursorLeaseConfiguration( + val store: QueryCursorLeaseStore, + val signingKeys: QueryCursorSigningKeys, + val leaseTtl: Duration = Duration.ofMinutes(2), + val maxCursorTtl: Duration = Duration.ofMinutes(5), + val maxBackendStateBytes: Int = 4096, +) { + init { + require(!leaseTtl.isZero && !leaseTtl.isNegative) { "Query cursor lease TTL must be positive." } + require(!maxCursorTtl.isZero && !maxCursorTtl.isNegative) { "Query cursor maximum TTL must be positive." } + require(leaseTtl <= maxCursorTtl) { "Query cursor lease TTL must not exceed its maximum TTL." } + require(maxBackendStateBytes > 0) { "Query cursor backend state limit must be positive." } + require(maxBackendStateBytes <= MAX_BACKEND_STATE_BYTES) { + "Query cursor backend state limit exceeds its supported maximum." + } + } + + companion object { + const val MAX_BACKEND_STATE_BYTES: Int = 1024 * 1024 + } +} + +@ExperimentalQueryCursorApi +class QueryCursorLeaseId(val value: String) { + init { + require(value.isNotBlank()) { "Query cursor lease id must not be blank." } + require(value.length <= MAX_ID_LENGTH) { "Query cursor lease id must not exceed $MAX_ID_LENGTH characters." } + require(URL_SAFE.matches(value)) { "Query cursor lease id must be URL-safe without padding." } + } + + override fun equals(other: Any?): Boolean = this === other || other is QueryCursorLeaseId && value == other.value + + override fun hashCode(): Int = value.hashCode() + + override fun toString(): String = value + + private companion object { + const val MAX_ID_LENGTH = 128 + val URL_SAFE = Regex("^[A-Za-z0-9_-]+$") + } +} + +@ExperimentalQueryCursorApi +class QueryCursorStoreRevision(val value: String) { + init { + require(value.isNotBlank()) { "Query cursor store revision must not be blank." } + require(value.length <= MAX_REVISION_LENGTH) { + "Query cursor store revision must not exceed $MAX_REVISION_LENGTH characters." + } + require(value.none(Char::isISOControl)) { "Query cursor store revision must not contain control characters." } + } + + override fun equals(other: Any?): Boolean = + this === other || other is QueryCursorStoreRevision && value == other.value + + override fun hashCode(): Int = value.hashCode() + + override fun toString(): String = value + + private companion object { + const val MAX_REVISION_LENGTH = 256 + } +} + +@ExperimentalQueryCursorApi +enum class QueryCursorPayloadFormat { + WOW_QUERY_CURSOR_V1, +} + +@ExperimentalQueryCursorApi +class QueryCursorLeaseEntry( + val id: QueryCursorLeaseId, + val expiresAt: Instant, + val payloadFormat: QueryCursorPayloadFormat, + payload: ByteArray, +) { + private val frozenPayload = payload.copyOf() + + init { + require(frozenPayload.isNotEmpty()) { "Query cursor payload must not be empty." } + } + + fun payload(): ByteArray = frozenPayload.copyOf() + + override fun equals(other: Any?): Boolean = + this === other || + other is QueryCursorLeaseEntry && + id == other.id && + expiresAt == other.expiresAt && + payloadFormat == other.payloadFormat && + frozenPayload.contentEquals(other.frozenPayload) + + override fun hashCode(): Int { + var result = id.hashCode() + result = 31 * result + expiresAt.hashCode() + result = 31 * result + payloadFormat.hashCode() + result = 31 * result + frozenPayload.contentHashCode() + return result + } +} + +@ExperimentalQueryCursorApi +enum class QueryCursorLeaseCreateResult { + CREATED, + COLLISION, + CAPACITY_EXCEEDED, +} + +@ExperimentalQueryCursorApi +class StoredQueryCursorLease( + val entry: QueryCursorLeaseEntry, + val revision: QueryCursorStoreRevision, +) { + override fun equals(other: Any?): Boolean = + this === other || other is StoredQueryCursorLease && entry == other.entry && revision == other.revision + + override fun hashCode(): Int = 31 * entry.hashCode() + revision.hashCode() +} + +/** + * Persistent, cross-node cursor ownership store. + * + * Implementations must make [compareAndDelete] atomic for the exact [StoredQueryCursorLease.revision]. A successful + * delete transfers one-time ownership to that caller. [scanExpired] is a bounded keyset scan ordered by lease id. + */ +@ExperimentalQueryCursorApi +interface QueryCursorLeaseStore { + fun create(entry: QueryCursorLeaseEntry): Mono + + /** Empty means the lease id is absent. */ + fun load(id: QueryCursorLeaseId): Mono + + fun compareAndDelete(expected: StoredQueryCursorLease): Mono + + fun scanExpired( + before: Instant, + afterId: QueryCursorLeaseId? = null, + limit: Int, + ): Flux +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/event/EventStreamQueryServiceFactory.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/event/EventStreamQueryServiceFactory.kt index b99a4b50dfc..e375fb6f161 100644 --- a/wow-query/src/main/kotlin/me/ahoo/wow/query/event/EventStreamQueryServiceFactory.kt +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/event/EventStreamQueryServiceFactory.kt @@ -14,16 +14,18 @@ package me.ahoo.wow.query.event import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.modeling.materialize +import java.util.concurrent.ConcurrentHashMap fun interface EventStreamQueryServiceFactory { fun create(namedAggregate: NamedAggregate): EventStreamQueryService } abstract class AbstractEventStreamQueryServiceFactory : EventStreamQueryServiceFactory { - private val queryServiceCache = mutableMapOf() + private val queryServiceCache = ConcurrentHashMap() override fun create(namedAggregate: NamedAggregate): EventStreamQueryService { - return queryServiceCache.computeIfAbsent(namedAggregate) { + return queryServiceCache.computeIfAbsent(namedAggregate.materialize()) { createQueryService(it) } } diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/filter/MaskingDynamicDocumentQueryFilter.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/filter/MaskingDynamicDocumentQueryFilter.kt index 20dd6672897..940ed05f0af 100644 --- a/wow-query/src/main/kotlin/me/ahoo/wow/query/filter/MaskingDynamicDocumentQueryFilter.kt +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/filter/MaskingDynamicDocumentQueryFilter.kt @@ -11,13 +11,18 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.query.filter import me.ahoo.wow.api.query.DynamicDocument import me.ahoo.wow.filter.FilterChain +import me.ahoo.wow.query.gateway.QueryErrorCategory +import me.ahoo.wow.query.gateway.QueryExecutionException import me.ahoo.wow.query.mask.AggregateDynamicDocumentMasker import me.ahoo.wow.query.mask.DataMaskerRegistry import me.ahoo.wow.query.mask.mask +import me.ahoo.wow.serialization.MessageRecords import reactor.core.publisher.Mono abstract class MaskingDynamicDocumentQueryFilter( @@ -48,7 +53,7 @@ abstract class MaskingDynamicDocumentQueryFilter { context.asSingleQuery().rewriteResult { result -> result.map { - aggregateDataMasker.mask(it) + aggregateDataMasker.maskPreservingSystemFields(it) } } } @@ -56,15 +61,20 @@ abstract class MaskingDynamicDocumentQueryFilter { context.asListQuery().rewriteResult { result -> result.map { - aggregateDataMasker.mask(it) + aggregateDataMasker.maskPreservingSystemFields(it) } } } QueryType.DYNAMIC_PAGED -> { context.asPagedQuery().rewriteResult { result -> - result.map { - aggregateDataMasker.mask(it) + result.map { page -> + me.ahoo.wow.api.query.PagedList( + page.total, + page.list.map { document -> + aggregateDataMasker.maskPreservingSystemFields(document) + }, + ) } } } @@ -73,4 +83,35 @@ abstract class MaskingDynamicDocumentQueryFilter.maskPreservingSystemFields( + source: DynamicDocument, + ): DynamicDocument { + val protectedFields = SYSTEM_FIELDS.associateWith { field -> + ProtectedField(source.containsKey(field), source[field]) + } + val masked = mask(source) + protectedFields.forEach { (field, original) -> + if (masked.containsKey(field) != original.present || masked[field] != original.value) { + throw QueryExecutionException( + category = QueryErrorCategory.INTERNAL_FAILURE, + path = "$.result.$field", + code = "RESULT_MASKING_SYSTEM_FIELD_VIOLATION", + ) + } + } + return masked + } + + private data class ProtectedField(val present: Boolean, val value: Any?) + + private companion object { + val SYSTEM_FIELDS = listOf( + MessageRecords.ID, + MessageRecords.AGGREGATE_ID, + MessageRecords.TENANT_ID, + MessageRecords.OWNER_ID, + MessageRecords.SPACE_ID, + ) + } } diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/filter/QueryFilter.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/filter/QueryFilter.kt index a7535614d98..17397bf9968 100644 --- a/wow-query/src/main/kotlin/me/ahoo/wow/query/filter/QueryFilter.kt +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/filter/QueryFilter.kt @@ -18,3 +18,12 @@ import me.ahoo.wow.filter.FilterType @FilterType(QueryHandler::class) interface QueryFilter> : Filter + +/** + * Compatibility request filter that completes before Query Gateway admission and policy evaluation. + * + * Any result written by this phase is discarded. Implementations may only rewrite the request query and attributes + * needed by another pre-admission filter. Result masking is provided by the framework's cardinality-preserving + * masking phase. + */ +interface PreAdmissionQueryFilter diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/filter/QueryHandler.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/filter/QueryHandler.kt index fa6f4aed7ce..97ff4355095 100644 --- a/wow-query/src/main/kotlin/me/ahoo/wow/query/filter/QueryHandler.kt +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/filter/QueryHandler.kt @@ -42,109 +42,90 @@ abstract class AbstractQueryHandler( private val errorHandler: ErrorHandler> ) : QueryHandler { override fun handle(context: QueryContext<*, *>): Mono { - return chain.filter(context) - .onErrorResume { - if (context is ErrorAccessor) { - context.setError(it) - } - errorHandler.handle(context, it) - } + return Mono.defer { chain.filter(context) } + .failClosed(context) } override fun single(namedAggregate: NamedAggregate, singleQuery: ISingleQuery): Mono { - val context = DefaultQueryContext>( - namedAggregate = namedAggregate, - queryType = QueryType.SINGLE - ).setQuery(singleQuery) - return handle(context) - .then( - Mono.defer { - context.getRequiredResult() - } - ) + return executeMono(namedAggregate, QueryType.SINGLE, singleQuery) } override fun dynamicSingle(namedAggregate: NamedAggregate, singleQuery: ISingleQuery): Mono { - val context = DefaultQueryContext>( - namedAggregate = namedAggregate, - queryType = QueryType.DYNAMIC_SINGLE - ).setQuery(singleQuery) - return handle(context) - .then( - Mono.defer { - context.getRequiredResult() - } - ) + return executeMono(namedAggregate, QueryType.DYNAMIC_SINGLE, singleQuery) } override fun list(namedAggregate: NamedAggregate, listQuery: IListQuery): Flux { - val context = DefaultQueryContext>( - namedAggregate = namedAggregate, - queryType = QueryType.LIST - ).setQuery(listQuery) - return handle(context) - .thenMany( - Flux.defer { - context.getRequiredResult() - } - ) + return executeFlux(namedAggregate, QueryType.LIST, listQuery) } override fun dynamicList(namedAggregate: NamedAggregate, listQuery: IListQuery): Flux { - val context = DefaultQueryContext>( - namedAggregate = namedAggregate, - queryType = QueryType.DYNAMIC_LIST - ).setQuery(listQuery) - return handle(context) - .thenMany( - Flux.defer { - context.getRequiredResult() - } - ) + return executeFlux(namedAggregate, QueryType.DYNAMIC_LIST, listQuery) } override fun paged( namedAggregate: NamedAggregate, pagedQuery: IPagedQuery ): Mono> { - val context = DefaultQueryContext>>( - namedAggregate = namedAggregate, - queryType = QueryType.PAGED - ).setQuery(pagedQuery) - return handle(context) - .then( - Mono.defer { - context.getRequiredResult() - } - ) + return executeMono(namedAggregate, QueryType.PAGED, pagedQuery) } override fun dynamicPaged( namedAggregate: NamedAggregate, pagedQuery: IPagedQuery ): Mono> { - val context = DefaultQueryContext>>( - namedAggregate = namedAggregate, - queryType = QueryType.DYNAMIC_PAGED - ).setQuery(pagedQuery) - return handle(context) - .then( - Mono.defer { - context.getRequiredResult() - } - ) + return executeMono(namedAggregate, QueryType.DYNAMIC_PAGED, pagedQuery) } override fun count(namedAggregate: NamedAggregate, condition: Condition): Mono { - val context = DefaultQueryContext>( - namedAggregate = namedAggregate, - queryType = QueryType.COUNT - ).setQuery(condition) - return handle(context) - .then( - Mono.defer { - context.getRequiredResult() - } - ) + return executeMono(namedAggregate, QueryType.COUNT, condition) + } + + private fun handleError(context: QueryContext<*, *>, throwable: Throwable): Mono { + if (context is ErrorAccessor) { + context.setError(throwable) + } + return errorHandler.handle(context, throwable) + } + + private fun executeMono( + namedAggregate: NamedAggregate, + queryType: QueryType, + query: Q, + ): Mono { + return Mono.defer { + val context = DefaultQueryContext>( + namedAggregate = namedAggregate, + queryType = queryType, + ).setQuery(query) + Mono.defer { handle(context) } + .then(Mono.defer { context.getRequiredResult() }.failClosed(context)) + } + } + + private fun executeFlux( + namedAggregate: NamedAggregate, + queryType: QueryType, + query: Q, + ): Flux { + return Flux.defer { + val context = DefaultQueryContext>( + namedAggregate = namedAggregate, + queryType = queryType, + ).setQuery(query) + Mono.defer { handle(context) } + .thenMany(Flux.defer { context.getRequiredResult() }.failClosed(context)) + } + } + + private fun Mono.failClosed(context: QueryContext<*, *>): Mono { + return onErrorResume { throwable -> + handleError(context, throwable).then(Mono.error(throwable)) + } + } + + private fun Flux.failClosed(context: QueryContext<*, *>): Flux { + return onErrorResume { throwable -> + handleError(context, throwable).thenMany(Flux.error(throwable)) + } } } diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/gateway/AnalyticsQueryGateway.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/gateway/AnalyticsQueryGateway.kt new file mode 100644 index 00000000000..984b66ed72c --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/gateway/AnalyticsQueryGateway.kt @@ -0,0 +1,24 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.gateway + +import me.ahoo.wow.api.query.analytics.AnalyticsPage +import me.ahoo.wow.api.query.analytics.AnalyticsQuery +import reactor.core.publisher.Mono + +/** Public Analytics port. Kept separate so QueryGateway's existing JVM descriptor remains unchanged. */ +@ExperimentalQueryGatewayApi +fun interface AnalyticsQueryGateway { + fun analyze(call: QueryCall, query: AnalyticsQuery): Mono +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/gateway/QueryGateway.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/gateway/QueryGateway.kt new file mode 100644 index 00000000000..861b641f75e --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/gateway/QueryGateway.kt @@ -0,0 +1,611 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.gateway + +import me.ahoo.wow.api.exception.BindingError +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DynamicDocument +import me.ahoo.wow.api.query.IListQuery +import me.ahoo.wow.api.query.IPagedQuery +import me.ahoo.wow.api.query.ISingleQuery +import me.ahoo.wow.api.query.PagedList +import me.ahoo.wow.exception.WowException +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.modeling.materialize +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Duration +import java.time.Instant +import java.util.Collections +import java.util.LinkedHashMap + +/** + * Marks the additive, evolving Query Gateway composition API. + * + * Existing [me.ahoo.wow.query.QueryService] contracts remain the supported compatibility surface. Applications should + * normally obtain Gateway-backed services from Spring rather than construct the runtime directly. + */ +@RequiresOptIn( + level = RequiresOptIn.Level.WARNING, + message = "The Query Gateway composition API is experimental and may evolve before the next major release.", +) +@Retention(AnnotationRetention.BINARY) +@Target(AnnotationTarget.CLASS, AnnotationTarget.FUNCTION, AnnotationTarget.PROPERTY, AnnotationTarget.CONSTRUCTOR) +annotation class ExperimentalQueryGatewayApi + +@ExperimentalQueryGatewayApi +enum class QueryDocumentKind { + SNAPSHOT, + EVENT_STREAM, +} + +@ExperimentalQueryGatewayApi +enum class QueryExecutionMode { + LEGACY, + SHADOW, + PLANNED, +} + +@ExperimentalQueryGatewayApi +enum class QueryValidationMode { + COMPATIBLE, + STRICT, +} + +@ExperimentalQueryGatewayApi +enum class QueryOperation { + SINGLE, + STREAM, + PAGE, + COUNT, + ANALYZE, +} + +@ExperimentalQueryGatewayApi +class QueryTarget( + namedAggregate: NamedAggregate, + val documentKind: QueryDocumentKind, +) { + val namedAggregate: MaterializedNamedAggregate = namedAggregate.materialize() + + override fun equals(other: Any?): Boolean = + this === other || + other is QueryTarget && namedAggregate == other.namedAggregate && documentKind == other.documentKind + + override fun hashCode(): Int = 31 * namedAggregate.hashCode() + documentKind.hashCode() + + override fun toString(): String = "QueryTarget(namedAggregate=$namedAggregate, documentKind=$documentKind)" +} + +@ExperimentalQueryGatewayApi +class QueryPurpose(val value: String) { + init { + requireIdentifier(value, "Query purpose") + } + + override fun equals(other: Any?): Boolean = this === other || other is QueryPurpose && value == other.value + + override fun hashCode(): Int = value.hashCode() + + override fun toString(): String = value +} + +@ExperimentalQueryGatewayApi +data class QueryResourceScope( + val tenantId: String? = null, + val ownerId: String? = null, + val spaceId: String? = null, +) { + init { + tenantId?.let { requireIdentifier(it, "Resource tenant id") } + ownerId?.let { requireIdentifier(it, "Resource owner id") } + spaceId?.let { requireIdentifier(it, "Resource space id") } + } +} + +@ExperimentalQueryGatewayApi +data class QueryExecutionBudget( + val maxReturnedRecords: Long? = null, + val maxScannedRecords: Long? = null, + val maxPageWindow: Long? = null, + val maxCandidateBuckets: Int? = null, + val maxReturnedBuckets: Int? = null, + val maxCursorPages: Int? = null, + val allowDiskUse: Boolean = false, +) { + init { + require(maxReturnedRecords == null || maxReturnedRecords > 0) + require(maxScannedRecords == null || maxScannedRecords > 0) + require(maxPageWindow == null || maxPageWindow > 0) + require(maxCandidateBuckets == null || maxCandidateBuckets > 0) + require(maxReturnedBuckets == null || maxReturnedBuckets > 0) + require(maxCursorPages == null || maxCursorPages > 0) + } + + constructor(maxReturnedRecords: Long?) : this( + maxReturnedRecords, + null, + null, + null, + null, + null, + false, + ) +} + +@ExperimentalQueryGatewayApi +data class QueryCall( + val target: QueryTarget, + val purpose: QueryPurpose, + val resourceScope: QueryResourceScope = QueryResourceScope(), + val deadline: Instant? = null, + val budget: QueryExecutionBudget = QueryExecutionBudget(), +) + +@ExperimentalQueryGatewayApi +data class QueryGatewayConfiguration( + val executionMode: QueryExecutionMode = QueryExecutionMode.LEGACY, + val validationMode: QueryValidationMode = QueryValidationMode.COMPATIBLE, +) + +@ExperimentalQueryGatewayApi +data class QueryExecutionProfile( + val executionMode: QueryExecutionMode, + val validationMode: QueryValidationMode, +) + +@ExperimentalQueryGatewayApi +data class QueryOperationProfileKey( + val target: QueryTarget, + val operation: QueryOperation, +) + +/** Immutable rollout profile: operation override, then target override, then the default profile. */ +@ExperimentalQueryGatewayApi +class QueryExecutionProfiles( + val defaultProfile: QueryExecutionProfile = QueryExecutionProfile( + QueryExecutionMode.LEGACY, + QueryValidationMode.COMPATIBLE, + ), + targetProfiles: Map = emptyMap(), + operationProfiles: Map = emptyMap(), +) { + val targetProfiles: Map = Collections.unmodifiableMap( + LinkedHashMap(targetProfiles), + ) + val operationProfiles: Map = Collections.unmodifiableMap( + LinkedHashMap(operationProfiles), + ) + + fun resolve(target: QueryTarget, operation: QueryOperation): QueryExecutionProfile = + operationProfiles[QueryOperationProfileKey(target, operation)] ?: targetProfiles[target] ?: defaultProfile + + companion object { + fun fixed(configuration: QueryGatewayConfiguration): QueryExecutionProfiles = QueryExecutionProfiles( + QueryExecutionProfile(configuration.executionMode, configuration.validationMode), + ) + } +} + +@ExperimentalQueryGatewayApi +data class QueryShadowConfiguration( + val maxConcurrentProbes: Int = 4, + val maxComparedRecords: Int = 1_000, + val probeTimeout: Duration = Duration.ofSeconds(30), +) { + init { + require(maxConcurrentProbes > 0) + require(maxComparedRecords > 0) + require(!probeTimeout.isZero && !probeTimeout.isNegative) + } +} + +@ExperimentalQueryGatewayApi +enum class QueryShadowOutcome { + MATCH, + VALUE_MISMATCH, + PROBE_ERROR, + PRIMARY_ERROR, + CANCELLED, + SKIPPED, + SATURATED, +} + +@ExperimentalQueryGatewayApi +data class QueryShadowObservation( + val target: QueryTarget, + val operation: QueryOperation, + val planFingerprint: String?, + val outcome: QueryShadowOutcome, + val reasonCode: String? = null, +) + +@ExperimentalQueryGatewayApi +fun interface QueryShadowObserver { + fun onObservation(observation: QueryShadowObservation) + + companion object { + val NONE = QueryShadowObserver { } + } +} + +@ExperimentalQueryGatewayApi +enum class QueryRuntimeHealthKind { + FALLBACK, + SHADOW_SUPERVISOR_FAILURE, + CURSOR_CLEANUP_FAILURE, +} + +/** Descriptor-only runtime health signal. It deliberately excludes authority, query values and backend causes. */ +@ExperimentalQueryGatewayApi +data class QueryRuntimeHealthObservation( + val target: QueryTarget, + val operation: QueryOperation, + val kind: QueryRuntimeHealthKind, + val reasonCode: String, +) + +@ExperimentalQueryGatewayApi +fun interface QueryRuntimeHealthObserver { + fun onObservation(observation: QueryRuntimeHealthObservation) + + companion object { + val NONE = QueryRuntimeHealthObserver { } + } +} + +@ExperimentalQueryGatewayApi +sealed interface QueryOwnerGrant { + data object Unrestricted : QueryOwnerGrant + + data class Only(val ownerId: String) : QueryOwnerGrant { + init { + requireIdentifier(ownerId, "Authority owner id") + } + } +} + +@ExperimentalQueryGatewayApi +sealed interface QuerySpaceGrant { + data object Unrestricted : QuerySpaceGrant + + data object DenyAll : QuerySpaceGrant + + class AllowList(spaceIds: Iterable) : QuerySpaceGrant { + val spaceIds: Set = immutableIdentifiers(spaceIds, "Authority space id") + + init { + require(this.spaceIds.isNotEmpty()) { + "Authority space allow-list must not be empty." + } + } + + override fun equals(other: Any?): Boolean = this === other || other is AllowList && spaceIds == other.spaceIds + + override fun hashCode(): Int = spaceIds.hashCode() + } +} + +@ExperimentalQueryGatewayApi +sealed interface QueryAuthority { + val principalId: String + + class Subject( + val subjectId: String, + val tenantId: String, + val ownerGrant: QueryOwnerGrant = QueryOwnerGrant.Unrestricted, + val spaceGrant: QuerySpaceGrant = QuerySpaceGrant.Unrestricted, + ) : QueryAuthority { + override val principalId: String = subjectId + + init { + requireIdentifier(subjectId, "Authority subject id") + requireIdentifier(tenantId, "Authority tenant id") + } + + override fun equals(other: Any?): Boolean = + this === other || + other is Subject && + subjectId == other.subjectId && + tenantId == other.tenantId && + ownerGrant == other.ownerGrant && + spaceGrant == other.spaceGrant + + override fun hashCode(): Int { + var result = subjectId.hashCode() + result = 31 * result + tenantId.hashCode() + result = 31 * result + ownerGrant.hashCode() + result = 31 * result + spaceGrant.hashCode() + return result + } + } + + class Service( + val serviceId: String, + val tenantId: String, + purposes: Iterable, + ) : QueryAuthority { + override val principalId: String = serviceId + val purposes: Set = Collections.unmodifiableSet( + LinkedHashSet(purposes.sortedBy(QueryPurpose::value)), + ) + + init { + requireIdentifier(serviceId, "Authority service id") + requireIdentifier(tenantId, "Authority tenant id") + require(this.purposes.isNotEmpty()) { + "Service authority purposes must not be empty." + } + } + + override fun equals(other: Any?): Boolean = + this === other || + other is Service && + serviceId == other.serviceId && + tenantId == other.tenantId && + purposes == other.purposes + + override fun hashCode(): Int { + var result = serviceId.hashCode() + result = 31 * result + tenantId.hashCode() + result = 31 * result + purposes.hashCode() + return result + } + } + + data class System( + override val principalId: String, + val justification: String, + ) : QueryAuthority { + init { + requireIdentifier(principalId, "Authority principal id") + requireIdentifier(justification, "System authority justification") + } + } + + data class Legacy(val grant: QueryLegacyGrant) : QueryAuthority { + override val principalId: String = grant.callerId + } +} + +/** Exact, pre-registered compatibility grant for a trusted process-internal caller. */ +@ExperimentalQueryGatewayApi +data class QueryLegacyGrant( + val callerId: String, + val target: QueryTarget, + val purpose: QueryPurpose, + val executionMode: QueryExecutionMode, + val resourceScope: QueryResourceScope, +) { + init { + requireIdentifier(callerId, "Legacy query caller id") + } +} + +@ExperimentalQueryGatewayApi +data class QueryAuthorityRequest( + val call: QueryCall, + val executionMode: QueryExecutionMode, + val validationMode: QueryValidationMode, +) + +@ExperimentalQueryGatewayApi +data class QueryTrustedContextRequest( + val callRequest: QueryCallResolutionRequest, + val executionMode: QueryExecutionMode, + val validationMode: QueryValidationMode, +) + +@ExperimentalQueryGatewayApi +data class QueryTrustedContext( + val call: QueryCall, + val authority: QueryAuthority, +) + +@ExperimentalQueryGatewayApi +fun interface QueryAuthorityResolver { + /** Called once for every subscription. Empty and error signals are fail-closed by the Gateway. */ + fun resolve(request: QueryAuthorityRequest): Mono +} + +/** A trusted context source that atomically resolves both halves of one compatibility-facade subscription. */ +@ExperimentalQueryGatewayApi +fun interface QueryTrustedContextResolver { + fun resolve(request: QueryTrustedContextRequest): Mono +} + +/** Ordered composition. An error from an applicable resolver is fail-closed and never falls through. */ +@ExperimentalQueryGatewayApi +class CompositeQueryTrustedContextResolver(resolvers: Iterable) : + QueryTrustedContextResolver { + private val resolvers = resolvers.toList() + + init { + require(this.resolvers.isNotEmpty()) { + "At least one trusted query context resolver is required." + } + } + + override fun resolve(request: QueryTrustedContextRequest): Mono = + Flux.fromIterable(resolvers) + .concatMap { resolver -> Mono.defer { resolver.resolve(request) } } + .next() +} + +/** + * Resolves an exact registered [QueryLegacyGrant] from a trusted process-internal Reactor context marker. + * + * This is a one-version migration bridge. The caller marker is not a System authority and cannot change the grant's + * target, purpose, execution mode or resource scope. + */ +@ExperimentalQueryGatewayApi +class QueryLegacyContextResolver(grants: Iterable) : + QueryTrustedContextResolver, + me.ahoo.wow.query.analytics.AnalyticsQueryTrustedContextResolver { + private val grantsByCallerAndTarget: Map + + init { + val materialized = grants.toList() + val indexed = materialized.associateBy { LegacyGrantKey(it.callerId, it.target) } + require(indexed.size == materialized.size) { + "Legacy query grants must be unique by caller id and target." + } + grantsByCallerAndTarget = Collections.unmodifiableMap(LinkedHashMap(indexed)) + } + + override fun resolve(request: QueryTrustedContextRequest): Mono = Mono.deferContextual { context -> + val callerId = context.getOrDefault(LEGACY_CALLER_CONTEXT_KEY, null) + ?: return@deferContextual Mono.empty() + val grant = grantsByCallerAndTarget[LegacyGrantKey(callerId, request.callRequest.target)] + ?: return@deferContextual Mono.error(legacyGrantRejected()) + if (request.executionMode != grant.executionMode || + request.callRequest.target != grant.target + ) { + return@deferContextual Mono.error(legacyGrantRejected()) + } + val call = QueryCall(grant.target, grant.purpose, grant.resourceScope) + Mono.just(QueryTrustedContext(call, QueryAuthority.Legacy(grant))) + } + + override fun resolve( + request: me.ahoo.wow.query.analytics.AnalyticsQueryTrustedContextRequest, + ): Mono = Mono.deferContextual { context -> + val callerId = context.getOrDefault(LEGACY_CALLER_CONTEXT_KEY, null) + ?: return@deferContextual Mono.empty() + val grant = grantsByCallerAndTarget[LegacyGrantKey(callerId, request.target)] + ?: return@deferContextual Mono.error(legacyGrantRejected()) + if (request.executionMode != grant.executionMode || request.target != grant.target) { + return@deferContextual Mono.error(legacyGrantRejected()) + } + Mono.just( + QueryTrustedContext( + QueryCall(grant.target, grant.purpose, grant.resourceScope), + QueryAuthority.Legacy(grant), + ), + ) + } +} + +@ExperimentalQueryGatewayApi +fun Mono.withLegacyQueryCaller(callerId: String): Mono { + requireIdentifier(callerId, "Legacy query caller id") + return contextWrite { context -> context.put(LEGACY_CALLER_CONTEXT_KEY, callerId) } +} + +@ExperimentalQueryGatewayApi +fun Flux.withLegacyQueryCaller(callerId: String): Flux { + requireIdentifier(callerId, "Legacy query caller id") + return contextWrite { context -> context.put(LEGACY_CALLER_CONTEXT_KEY, callerId) } +} + +@ExperimentalQueryGatewayApi +interface QueryGateway { + fun single(call: QueryCall, query: ISingleQuery): Mono + + fun single(call: QueryCall, query: ISingleQuery, resultType: Class): Mono + + fun stream(call: QueryCall, query: IListQuery): Flux + + fun stream(call: QueryCall, query: IListQuery, resultType: Class): Flux + + fun page(call: QueryCall, query: IPagedQuery): Mono> + + fun page(call: QueryCall, query: IPagedQuery, resultType: Class): Mono> + + fun count(call: QueryCall, condition: Condition): Mono +} + +/** A target-bound typed materializer. Registration is unique per exact [QueryTarget]. */ +@ExperimentalQueryGatewayApi +class QueryResultMaterializer( + val target: QueryTarget, + val resultType: Class, + private val materialize: (identity: String, document: DynamicDocument) -> R, +) { + fun materialize(identity: String, document: DynamicDocument): R = materialize.invoke(identity, document) +} + +@ExperimentalQueryGatewayApi +enum class QueryErrorCategory { + ACCESS_DENIED, + INVALID_QUERY, + INVALID_CURSOR, + BUDGET_EXCEEDED, + UNSUPPORTED_FEATURE, + BACKEND_UNAVAILABLE, + BACKEND_TIMEOUT, + INCOMPLETE_RESULT, + MAPPING_FAILURE, + INTERNAL_FAILURE, +} + +@ExperimentalQueryGatewayApi +class QueryExecutionException( + val category: QueryErrorCategory, + val path: String, + val code: String, + cause: Throwable? = null, +) : WowException( + errorCode = "Query.${category.name}.$code", + errorMsg = category.safeMessage(), + cause = cause, + bindingErrors = listOf(BindingError(path, code)), +) + +private fun QueryErrorCategory.safeMessage(): String = + when (this) { + QueryErrorCategory.ACCESS_DENIED -> "Query access was denied." + QueryErrorCategory.INVALID_QUERY -> "The query is invalid." + QueryErrorCategory.INVALID_CURSOR -> "The query cursor is invalid." + QueryErrorCategory.BUDGET_EXCEEDED -> "The query budget was exceeded." + QueryErrorCategory.UNSUPPORTED_FEATURE -> "The query uses an unsupported feature." + QueryErrorCategory.BACKEND_UNAVAILABLE -> "The query backend is unavailable." + QueryErrorCategory.BACKEND_TIMEOUT -> "The query backend timed out." + QueryErrorCategory.INCOMPLETE_RESULT -> "The query result is incomplete." + QueryErrorCategory.MAPPING_FAILURE -> "The query result could not be mapped." + QueryErrorCategory.INTERNAL_FAILURE -> "The query failed unexpectedly." + } + +private fun requireIdentifier(value: String, name: String) { + require(value.isNotBlank()) { + "$name must not be blank." + } + require(value.none(Char::isISOControl)) { + "$name must not contain control characters." + } + require(value.length <= MAX_IDENTIFIER_LENGTH) { + "$name must not exceed $MAX_IDENTIFIER_LENGTH characters." + } +} + +private fun immutableIdentifiers(values: Iterable, name: String): Set { + val materialized = values.toList() + materialized.forEach { requireIdentifier(it, name) } + return Collections.unmodifiableSet(LinkedHashSet(materialized.sorted())) +} + +private data class LegacyGrantKey(val callerId: String, val target: QueryTarget) + +private fun legacyGrantRejected(): QueryExecutionException = QueryExecutionException( + category = QueryErrorCategory.ACCESS_DENIED, + path = "$.executionContext.legacyGrant", + code = "LEGACY_CALLER_NOT_ALLOWED", +) + +private const val LEGACY_CALLER_CONTEXT_KEY = "me.ahoo.wow.query.legacy.caller" + +private const val MAX_IDENTIFIER_LENGTH = 512 diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/gateway/QueryGatewayRuntime.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/gateway/QueryGatewayRuntime.kt new file mode 100644 index 00000000000..40dab681abd --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/gateway/QueryGatewayRuntime.kt @@ -0,0 +1,63 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.gateway + +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.query.event.EventStreamQueryService +import me.ahoo.wow.query.snapshot.SnapshotQueryService + +@ExperimentalQueryGatewayApi +enum class QueryElementPathMode { + /** MongoDB `$elemMatch`: child fields are relative to the current array element. */ + CURRENT_ELEMENT_RELATIVE, + + /** Elasticsearch `nested`: child fields retain the complete root-qualified path. */ + ROOT_QUALIFIED, +} + +@ExperimentalQueryGatewayApi +enum class QueryMatchScopeMode { + /** The legacy backend treats MATCH as collection/document-wide full-text search. */ + DOCUMENT, + + /** The legacy backend binds MATCH to the supplied field. */ + FIELD, +} + +@ExperimentalQueryGatewayApi +data class QueryLegacyDialect( + val elementPathMode: QueryElementPathMode, + val matchScopeMode: QueryMatchScopeMode, +) + +@ExperimentalQueryGatewayApi +fun interface QueryLegacyDialectResolver { + /** Must return the dialect of the raw storage route selected for this exact target. */ + fun resolve(target: QueryTarget): QueryLegacyDialect +} + +/** + * Resolves raw storage services without sharing the public application-facade factory type. + * + * Spring integrations must implement this from storage bindings. A Gateway-backed facade can therefore never be + * selected recursively as its own raw backend. + */ +@ExperimentalQueryGatewayApi +interface QueryRawServiceSource { + fun snapshot(namedAggregate: NamedAggregate): SnapshotQueryService<*> + + fun eventStream(namedAggregate: NamedAggregate): EventStreamQueryService +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/gateway/QueryServiceFacade.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/gateway/QueryServiceFacade.kt new file mode 100644 index 00000000000..b7594c3a385 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/gateway/QueryServiceFacade.kt @@ -0,0 +1,549 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(ExperimentalQueryCursorApi::class, ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.gateway + +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DynamicDocument +import me.ahoo.wow.api.query.IListQuery +import me.ahoo.wow.api.query.IPagedQuery +import me.ahoo.wow.api.query.ISingleQuery +import me.ahoo.wow.api.query.MaterializedSnapshot +import me.ahoo.wow.api.query.PagedList +import me.ahoo.wow.event.DomainEventStream +import me.ahoo.wow.modeling.materialize +import me.ahoo.wow.query.analytics.AnalyticsQueryService +import me.ahoo.wow.query.analytics.AnalyticsQueryServiceFactory +import me.ahoo.wow.query.analytics.AnalyticsQueryTrustedContextRequest +import me.ahoo.wow.query.analytics.AnalyticsQueryTrustedContextResolver +import me.ahoo.wow.query.backend.ExperimentalQueryBackendApi +import me.ahoo.wow.query.backend.QueryBackendComposition +import me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi +import me.ahoo.wow.query.cursor.QueryCursorLeaseConfiguration +import me.ahoo.wow.query.event.AbstractEventStreamQueryServiceFactory +import me.ahoo.wow.query.event.EventStreamQueryService +import me.ahoo.wow.query.filter.QueryType +import me.ahoo.wow.query.internal.gateway.QueryGatewayRuntimeBuilder +import me.ahoo.wow.query.internal.gateway.TrustedAuthorityChannel +import me.ahoo.wow.query.snapshot.AbstractSnapshotQueryServiceFactory +import me.ahoo.wow.query.snapshot.SnapshotQueryService +import me.ahoo.wow.serialization.JsonSerializer +import me.ahoo.wow.serialization.convert +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.core.scheduler.Scheduler +import reactor.core.scheduler.Schedulers +import java.time.Clock + +@ExperimentalQueryGatewayApi +data class QueryCallResolutionRequest( + val target: QueryTarget, + val queryType: QueryType, +) + +@ExperimentalQueryGatewayApi +fun interface QueryCallResolver { + /** Called once for each subscription made through a compatibility QueryService facade. */ + fun resolve(request: QueryCallResolutionRequest): Mono +} + +@ExperimentalQueryGatewayApi +class GatewaySnapshotQueryServiceFactory( + private val gateway: QueryGateway, + callResolver: QueryCallResolver, +) : AbstractSnapshotQueryServiceFactory() { + private val facadeContextResolver = callResolver.asFacadeContextResolver() + + override fun createQueryService(namedAggregate: NamedAggregate): SnapshotQueryService<*> = + GatewaySnapshotQueryService(namedAggregate.materialize(), gateway, facadeContextResolver) +} + +@ExperimentalQueryGatewayApi +class GatewayEventStreamQueryServiceFactory( + private val gateway: QueryGateway, + callResolver: QueryCallResolver, +) : AbstractEventStreamQueryServiceFactory() { + private val facadeContextResolver = callResolver.asFacadeContextResolver() + + override fun createQueryService(namedAggregate: NamedAggregate): EventStreamQueryService = + GatewayEventStreamQueryService(namedAggregate.materialize(), gateway, facadeContextResolver) +} + +private class GatewaySnapshotQueryService( + override val namedAggregate: NamedAggregate, + private val gateway: QueryGateway, + private val facadeContextResolver: FacadeContextResolver, +) : SnapshotQueryService { + override val name: String = GATEWAY_QUERY_SERVICE_NAME + + override fun single(singleQuery: ISingleQuery): Mono> = + callMono(QueryType.SINGLE) { call -> gateway.single(call, singleQuery, MATERIALIZED_SNAPSHOT_TYPE) } + .map(MaterializedSnapshot<*>::eraseStateType) + + override fun dynamicSingle(singleQuery: ISingleQuery): Mono = + callMono(QueryType.DYNAMIC_SINGLE) { call -> gateway.single(call, singleQuery) } + + override fun list(listQuery: IListQuery): Flux> = + callFlux(QueryType.LIST) { call -> gateway.stream(call, listQuery, MATERIALIZED_SNAPSHOT_TYPE) } + .map(MaterializedSnapshot<*>::eraseStateType) + + override fun dynamicList(listQuery: IListQuery): Flux = + callFlux(QueryType.DYNAMIC_LIST) { call -> gateway.stream(call, listQuery) } + + override fun paged(pagedQuery: IPagedQuery): Mono>> = + callMono(QueryType.PAGED) { call -> gateway.page(call, pagedQuery, MATERIALIZED_SNAPSHOT_TYPE) } + .map { page -> + PagedList(page.total, page.list.map(MaterializedSnapshot<*>::eraseStateType)) + } + + override fun dynamicPaged(pagedQuery: IPagedQuery): Mono> = + callMono(QueryType.DYNAMIC_PAGED) { call -> gateway.page(call, pagedQuery) } + + override fun count(condition: Condition): Mono = + callMono(QueryType.COUNT) { call -> gateway.count(call, condition) } + + private fun callMono(queryType: QueryType, source: (QueryCall) -> Mono): Mono = + resolveFacadeContext(facadeContextResolver, target(), queryType) + .flatMap { context -> context.applyTo(source(context.call)) } + + private fun callFlux(queryType: QueryType, source: (QueryCall) -> Flux): Flux = + resolveFacadeContext(facadeContextResolver, target(), queryType) + .flatMapMany { context -> context.applyTo(source(context.call)) } + + private fun target(): QueryTarget = QueryTarget(namedAggregate, QueryDocumentKind.SNAPSHOT) +} + +private class GatewayEventStreamQueryService( + override val namedAggregate: NamedAggregate, + private val gateway: QueryGateway, + private val facadeContextResolver: FacadeContextResolver, +) : EventStreamQueryService { + override fun single(singleQuery: ISingleQuery): Mono = + callMono(QueryType.SINGLE) { call -> gateway.single(call, singleQuery, DomainEventStream::class.java) } + + override fun dynamicSingle(singleQuery: ISingleQuery): Mono = + callMono(QueryType.DYNAMIC_SINGLE) { call -> gateway.single(call, singleQuery) } + + override fun list(listQuery: IListQuery): Flux = + callFlux(QueryType.LIST) { call -> gateway.stream(call, listQuery, DomainEventStream::class.java) } + + override fun dynamicList(listQuery: IListQuery): Flux = + callFlux(QueryType.DYNAMIC_LIST) { call -> gateway.stream(call, listQuery) } + + override fun paged(pagedQuery: IPagedQuery): Mono> = + callMono(QueryType.PAGED) { call -> gateway.page(call, pagedQuery, DomainEventStream::class.java) } + + override fun dynamicPaged(pagedQuery: IPagedQuery): Mono> = + callMono(QueryType.DYNAMIC_PAGED) { call -> gateway.page(call, pagedQuery) } + + override fun count(condition: Condition): Mono = + callMono(QueryType.COUNT) { call -> gateway.count(call, condition) } + + private fun callMono(queryType: QueryType, source: (QueryCall) -> Mono): Mono = + resolveFacadeContext(facadeContextResolver, target(), queryType) + .flatMap { context -> context.applyTo(source(context.call)) } + + private fun callFlux(queryType: QueryType, source: (QueryCall) -> Flux): Flux = + resolveFacadeContext(facadeContextResolver, target(), queryType) + .flatMapMany { context -> context.applyTo(source(context.call)) } + + private fun target(): QueryTarget = QueryTarget(namedAggregate, QueryDocumentKind.EVENT_STREAM) +} + +private fun resolveFacadeContext( + resolver: FacadeContextResolver, + target: QueryTarget, + queryType: QueryType, +): Mono = Mono.defer { + resolver.resolveContext(QueryCallResolutionRequest(target, queryType)) +}.onErrorMap { error -> + if (error is QueryExecutionException) { + error + } else { + queryCallError("QUERY_CALL_RESOLUTION_FAILED", error) + } +}.switchIfEmpty(Mono.error(queryCallError("QUERY_CALL_REQUIRED"))) + .map { context -> + if (context.call.target != target) { + throw queryCallError("QUERY_CALL_TARGET_MISMATCH") + } + context + } + +private fun interface FacadeContextResolver { + fun resolveContext(request: QueryCallResolutionRequest): Mono +} + +private data class ResolvedFacadeContext( + val call: QueryCall, + val authority: QueryAuthority?, + val trustedAuthorityChannel: TrustedAuthorityChannel?, +) { + fun applyTo(source: Mono): Mono = + authority?.let { trustedAuthorityChannel!!.bind(source, it) } ?: source + + fun applyTo(source: Flux): Flux = + authority?.let { trustedAuthorityChannel!!.bind(source, it) } ?: source +} + +private class CallOnlyFacadeContextResolver( + private val callResolver: QueryCallResolver, +) : FacadeContextResolver { + override fun resolveContext(request: QueryCallResolutionRequest): Mono = + callResolver.resolve(request).map { call -> ResolvedFacadeContext(call, null, null) } +} + +private class TrustedFacadeContextResolver( + private val trustedContextResolver: QueryTrustedContextResolver, + private val executionProfiles: QueryExecutionProfiles, + private val trustedAuthorityChannel: TrustedAuthorityChannel, +) : QueryCallResolver, FacadeContextResolver { + override fun resolve(request: QueryCallResolutionRequest): Mono = + resolveContext(request).map(ResolvedFacadeContext::call) + + override fun resolveContext(request: QueryCallResolutionRequest): Mono = + executionProfiles.resolve(request.target, request.queryType.toOperation()).let { profile -> + trustedContextResolver.resolve( + QueryTrustedContextRequest( + request, + profile.executionMode, + profile.validationMode, + ), + ) + }.map { context -> ResolvedFacadeContext(context.call, context.authority, trustedAuthorityChannel) } +} + +private fun QueryType.toOperation(): QueryOperation = + when (this) { + QueryType.SINGLE, + QueryType.DYNAMIC_SINGLE, + -> QueryOperation.SINGLE + + QueryType.LIST, + QueryType.DYNAMIC_LIST, + -> QueryOperation.STREAM + + QueryType.PAGED, + QueryType.DYNAMIC_PAGED, + -> QueryOperation.PAGE + + QueryType.COUNT -> QueryOperation.COUNT + } + +private fun QueryCallResolver.asFacadeContextResolver(): FacadeContextResolver = + this as? FacadeContextResolver ?: CallOnlyFacadeContextResolver(this) + +/** + * Owns one immutable Gateway runtime for the supplied aggregate set. + * + * The raw source has a distinct type from the public application factories, so the runtime cannot recursively resolve + * its own facade. The trusted resolver and its per-runtime authority capability are frozen at construction time. + */ +@ExperimentalQueryGatewayApi +class QueryGatewayRuntime private constructor(private val state: RuntimeState) { + val gateway: QueryGateway + get() = state.gateway + + val analyticsGateway: AnalyticsQueryGateway + get() = state.analyticsGateway + + /** Reaps one bounded batch of expired cursor leases and closes their backend-owned resources. */ + @ExperimentalQueryCursorApi + fun reapExpiredQueryCursors(batchSize: Int = 100): Mono = state.cursorReaper(batchSize) + + fun snapshotQueryServiceFactory(): GatewaySnapshotQueryServiceFactory = GatewaySnapshotQueryServiceFactory( + state.gateway, + state.trustedCallResolver, + ) + + fun eventStreamQueryServiceFactory(): GatewayEventStreamQueryServiceFactory = GatewayEventStreamQueryServiceFactory( + state.gateway, + state.trustedCallResolver, + ) + + fun analyticsQueryServiceFactory(): AnalyticsQueryServiceFactory = GatewayAnalyticsQueryServiceFactory( + state.analyticsGateway, + state.executionProfiles, + state.trustedAuthorityChannel, + state.analyticsTrustedContextResolver, + ) + + private class RuntimeState( + val gateway: QueryGateway, + val analyticsGateway: AnalyticsQueryGateway, + val trustedCallResolver: QueryCallResolver, + val executionProfiles: QueryExecutionProfiles, + val trustedAuthorityChannel: TrustedAuthorityChannel, + val analyticsTrustedContextResolver: AnalyticsQueryTrustedContextResolver, + val cursorReaper: (Int) -> Mono, + ) + + companion object { + fun create( + namedAggregates: Iterable, + rawServiceSource: QueryRawServiceSource, + dialectResolver: QueryLegacyDialectResolver, + authorityResolver: QueryAuthorityResolver, + trustedContextResolver: QueryTrustedContextResolver = QueryTrustedContextResolver { Mono.empty() }, + resultMaterializers: Iterable> = emptyList(), + configuration: QueryGatewayConfiguration = QueryGatewayConfiguration(), + clock: Clock = Clock.systemUTC(), + scheduler: Scheduler = Schedulers.parallel(), + ): QueryGatewayRuntime = createRuntime( + namedAggregates, + rawServiceSource, + dialectResolver, + authorityResolver, + trustedContextResolver, + resultMaterializers, + clock, + scheduler, + QueryBackendComposition.EMPTY, + QueryExecutionProfiles.fixed(configuration), + QueryShadowConfiguration(), + QueryShadowObserver.NONE, + QueryRuntimeHealthObserver.NONE, + null, + ) + + @ExperimentalQueryBackendApi + fun create( + namedAggregates: Iterable, + backendComposition: QueryBackendComposition, + rawServiceSource: QueryRawServiceSource, + dialectResolver: QueryLegacyDialectResolver, + authorityResolver: QueryAuthorityResolver, + trustedContextResolver: QueryTrustedContextResolver = QueryTrustedContextResolver { Mono.empty() }, + resultMaterializers: Iterable> = emptyList(), + configuration: QueryGatewayConfiguration = QueryGatewayConfiguration(), + executionProfiles: QueryExecutionProfiles = QueryExecutionProfiles.fixed(configuration), + shadowConfiguration: QueryShadowConfiguration = QueryShadowConfiguration(), + shadowObserver: QueryShadowObserver = QueryShadowObserver.NONE, + runtimeHealthObserver: QueryRuntimeHealthObserver = QueryRuntimeHealthObserver.NONE, + clock: Clock = Clock.systemUTC(), + scheduler: Scheduler = Schedulers.parallel(), + ): QueryGatewayRuntime = createRuntime( + namedAggregates, + rawServiceSource, + dialectResolver, + authorityResolver, + trustedContextResolver, + resultMaterializers, + clock, + scheduler, + backendComposition, + executionProfiles, + shadowConfiguration, + shadowObserver, + runtimeHealthObserver, + null, + ) + + @ExperimentalQueryBackendApi + @ExperimentalQueryCursorApi + fun create( + namedAggregates: Iterable, + backendComposition: QueryBackendComposition, + cursorLeaseConfiguration: QueryCursorLeaseConfiguration, + rawServiceSource: QueryRawServiceSource, + dialectResolver: QueryLegacyDialectResolver, + authorityResolver: QueryAuthorityResolver, + trustedContextResolver: QueryTrustedContextResolver = QueryTrustedContextResolver { Mono.empty() }, + resultMaterializers: Iterable> = emptyList(), + configuration: QueryGatewayConfiguration = QueryGatewayConfiguration(), + executionProfiles: QueryExecutionProfiles = QueryExecutionProfiles.fixed(configuration), + shadowConfiguration: QueryShadowConfiguration = QueryShadowConfiguration(), + shadowObserver: QueryShadowObserver = QueryShadowObserver.NONE, + runtimeHealthObserver: QueryRuntimeHealthObserver = QueryRuntimeHealthObserver.NONE, + clock: Clock = Clock.systemUTC(), + scheduler: Scheduler = Schedulers.parallel(), + ): QueryGatewayRuntime = createRuntime( + namedAggregates, + rawServiceSource, + dialectResolver, + authorityResolver, + trustedContextResolver, + resultMaterializers, + clock, + scheduler, + backendComposition, + executionProfiles, + shadowConfiguration, + shadowObserver, + runtimeHealthObserver, + cursorLeaseConfiguration, + ) + + @OptIn(ExperimentalQueryBackendApi::class) + private fun createRuntime( + namedAggregates: Iterable, + rawServiceSource: QueryRawServiceSource, + dialectResolver: QueryLegacyDialectResolver, + authorityResolver: QueryAuthorityResolver, + trustedContextResolver: QueryTrustedContextResolver, + resultMaterializers: Iterable>, + clock: Clock, + scheduler: Scheduler, + backendComposition: QueryBackendComposition, + executionProfiles: QueryExecutionProfiles, + shadowConfiguration: QueryShadowConfiguration, + shadowObserver: QueryShadowObserver, + runtimeHealthObserver: QueryRuntimeHealthObserver, + cursorLeaseConfiguration: QueryCursorLeaseConfiguration?, + ): QueryGatewayRuntime { + val analyticsTrustedContextResolver = + (trustedContextResolver as? AnalyticsQueryTrustedContextResolver) + ?: AnalyticsQueryTrustedContextResolver { Mono.empty() } + val components = QueryGatewayRuntimeBuilder.build( + namedAggregates, + rawServiceSource, + resultMaterializers, + dialectResolver, + authorityResolver, + clock, + scheduler, + backendComposition, + executionProfiles, + shadowConfiguration, + shadowObserver, + runtimeHealthObserver, + cursorLeaseConfiguration, + ) + return QueryGatewayRuntime( + RuntimeState( + components.gateway, + components.analyticsGateway, + TrustedFacadeContextResolver( + trustedContextResolver, + executionProfiles, + components.trustedAuthorityChannel, + ), + executionProfiles, + components.trustedAuthorityChannel, + analyticsTrustedContextResolver, + components.cursorReaper, + ), + ) + } + } +} + +private class GatewayAnalyticsQueryServiceFactory( + private val gateway: AnalyticsQueryGateway, + private val executionProfiles: QueryExecutionProfiles, + private val trustedAuthorityChannel: TrustedAuthorityChannel, + private val resolver: AnalyticsQueryTrustedContextResolver, +) : AnalyticsQueryServiceFactory { + override fun create(namedAggregate: NamedAggregate): AnalyticsQueryService = GatewayAnalyticsQueryService( + namedAggregate.materialize(), + gateway, + executionProfiles, + trustedAuthorityChannel, + resolver, + ) +} + +private class GatewayAnalyticsQueryService( + override val namedAggregate: NamedAggregate, + private val gateway: AnalyticsQueryGateway, + private val executionProfiles: QueryExecutionProfiles, + private val trustedAuthorityChannel: TrustedAuthorityChannel, + private val resolver: AnalyticsQueryTrustedContextResolver, +) : AnalyticsQueryService { + override fun analyze( + query: me.ahoo.wow.api.query.analytics.AnalyticsQuery + ): Mono = + Mono.defer { + val target = QueryTarget(namedAggregate, QueryDocumentKind.SNAPSHOT) + val profile = executionProfiles.resolve(target, QueryOperation.ANALYZE) + Mono.defer { + resolver.resolve( + AnalyticsQueryTrustedContextRequest(target, profile.executionMode, profile.validationMode), + ) + }.onErrorMap { error -> + if (error is QueryExecutionException) error else queryCallError("QUERY_CALL_RESOLUTION_FAILED", error) + }.switchIfEmpty(Mono.error(queryCallError("QUERY_CALL_REQUIRED"))) + .flatMap { context -> + if (context.call.target != target) { + return@flatMap Mono.error(queryCallError("QUERY_CALL_TARGET_MISMATCH")) + } + trustedAuthorityChannel.bind(gateway.analyze(context.call, query), context.authority) + } + } +} + +@ExperimentalQueryGatewayApi +object QueryResultMaterializers { + fun snapshot(target: QueryTarget, stateType: Class<*>): QueryResultMaterializer> { + require(target.documentKind == QueryDocumentKind.SNAPSHOT) { + "Snapshot materializer requires a SNAPSHOT target." + } + val snapshotType = JsonSerializer.typeFactory.constructParametricType( + MaterializedSnapshot::class.java, + stateType, + ) + return QueryResultMaterializer(target, MATERIALIZED_SNAPSHOT_TYPE) { identity, document -> + document.convert>(snapshotType).also { snapshot -> + require(snapshot.aggregateId == identity) { + "Materialized snapshot identity does not match the backend record identity." + } + require(stateType.isInstance(snapshot.state)) { + "Materialized snapshot state does not match the registered state type." + } + } + } + } + + fun eventStream(target: QueryTarget): QueryResultMaterializer { + require(target.documentKind == QueryDocumentKind.EVENT_STREAM) { + "Event-stream materializer requires an EVENT_STREAM target." + } + return QueryResultMaterializer(target, DomainEventStream::class.java) { identity, document -> + document.convert(DomainEventStream::class.java).also { eventStream -> + require(eventStream.id == identity) { + "Materialized event-stream identity does not match the backend record identity." + } + } + } + } +} + +private fun queryCallError(code: String, cause: Throwable? = null): QueryExecutionException = + QueryExecutionException(QueryErrorCategory.ACCESS_DENIED, "$.executionContext.call", code, cause) + +private fun MaterializedSnapshot<*>.eraseStateType(): MaterializedSnapshot = MaterializedSnapshot( + contextName = contextName, + aggregateName = aggregateName, + tenantId = tenantId, + ownerId = ownerId, + spaceId = spaceId, + aggregateId = aggregateId, + version = version, + eventId = eventId, + firstOperator = firstOperator, + operator = operator, + firstEventTime = firstEventTime, + eventTime = eventTime, + state = state, + snapshotTime = snapshotTime, + tags = tags, + deleted = deleted, +) + +private val MATERIALIZED_SNAPSHOT_TYPE: Class> = + MaterializedSnapshot::class.java + +private const val GATEWAY_QUERY_SERVICE_NAME = "QueryGateway" diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/AdmissionBudget.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/AdmissionBudget.kt new file mode 100644 index 00000000000..36648090c1a --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/AdmissionBudget.kt @@ -0,0 +1,86 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.admission + +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import java.math.BigDecimal +import java.math.BigInteger +import java.nio.charset.StandardCharsets + +/** Per-admission cumulative budget shared by every condition value, field and option. */ +internal class AdmissionBudget( + private val limits: QueryAdmissionLimits, +) { + private var valueNodes: Int = 0 + private var payloadBytes: Long = 0 + + fun enterValue(path: QueryRejectionPath) { + if (valueNodes == limits.maxValueNodes) { + rejectBudget(path, QueryRejectionCode.VALUE_NODE_LIMIT_EXCEEDED) + } + valueNodes++ + } + + fun consumeString(value: String, path: QueryRejectionPath) { + if (value.length > limits.maxStringLength) { + rejectBudget(path, QueryRejectionCode.STRING_LIMIT_EXCEEDED) + } + consumeUtf8(value, path) + } + + fun consumeUtf8(value: String, path: QueryRejectionPath) { + consumePayload(value.toByteArray(StandardCharsets.UTF_8).size.toLong(), path) + } + + fun consumeBytes(size: Int, path: QueryRejectionPath) { + if (size > limits.maxByteArrayLength) { + rejectBudget(path, QueryRejectionCode.BYTE_ARRAY_LIMIT_EXCEEDED) + } + consumePayload(size.toLong(), path) + } + + fun consumeNumber(number: Number, path: QueryRejectionPath): String { + when (number) { + is BigDecimal -> if (number.precision() > limits.maxNumericPrecision) { + rejectBudget(path, QueryRejectionCode.NUMERIC_PRECISION_LIMIT_EXCEEDED) + } + is BigInteger -> { + val maxBits = limits.maxNumericPrecision.toLong() * 4 + 1 + if (number.abs().bitLength().toLong() > maxBits) { + rejectBudget(path, QueryRejectionCode.NUMERIC_PRECISION_LIMIT_EXCEEDED) + } + } + } + val text = number.toString() + val precision = text.trimStart('-').count(Char::isDigit) + if (precision > limits.maxNumericPrecision) { + rejectBudget(path, QueryRejectionCode.NUMERIC_PRECISION_LIMIT_EXCEEDED) + } + consumeString(text, path) + return text + } + + private fun consumePayload(size: Long, path: QueryRejectionPath) { + if (size > limits.maxValuePayloadBytes - payloadBytes) { + rejectBudget(path, QueryRejectionCode.PAYLOAD_LIMIT_EXCEEDED) + } + payloadBytes += size + } + + private fun rejectBudget(path: QueryRejectionPath, code: QueryRejectionCode): Nothing = + rejectQuery(QueryRejectionCategory.BUDGET_EXCEEDED, path, code) +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/AdmittedQuery.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/AdmittedQuery.kt new file mode 100644 index 00000000000..4edcec9569c --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/AdmittedQuery.kt @@ -0,0 +1,232 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.admission + +import me.ahoo.wow.api.query.DeletionState +import me.ahoo.wow.api.query.Operator +import me.ahoo.wow.api.query.Sort +import me.ahoo.wow.api.query.analytics.AnalyticsBucketWindow +import me.ahoo.wow.api.query.analytics.AnalyticsCompleteness +import me.ahoo.wow.api.query.analytics.AnalyticsConsistency +import me.ahoo.wow.api.query.analytics.AnalyticsGrouping +import me.ahoo.wow.api.query.analytics.AnalyticsMetric +import me.ahoo.wow.api.query.analytics.AnalyticsNumericPolicy +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.normalization.CaseSensitivity +import java.time.LocalTime +import java.time.ZoneId +import java.time.format.DateTimeFormatter +import java.util.Collections + +internal data class QueryAdmissionLimits( + val maxConditionDepth: Int = 32, + val maxConditionNodes: Int = 1024, + val maxChildrenPerNode: Int = 128, + val maxFieldLength: Int = 512, + val maxStringLength: Int = 65_536, + val maxCollectionSize: Int = 1024, + val maxObjectFields: Int = 256, + val maxValueDepth: Int = 16, + val maxValueNodes: Int = 16_384, + val maxNumericPrecision: Int = 1024, + val maxByteArrayLength: Int = 65_536, + val maxValuePayloadBytes: Long = 4L * 1024 * 1024, + val maxProjectionFields: Int = 128, + val maxSortFields: Int = 32, + val maxOptions: Int = 8, +) { + init { + require( + listOf( + maxConditionDepth, + maxConditionNodes, + maxChildrenPerNode, + maxFieldLength, + maxStringLength, + maxCollectionSize, + maxObjectFields, + maxValueDepth, + maxValueNodes, + maxNumericPrecision, + maxByteArrayLength, + maxProjectionFields, + maxSortFields, + maxOptions, + ).all { it > 0 }, + ) { + "Query admission limits must be positive." + } + require(maxValuePayloadBytes > 0) { + "Query value payload limit must be positive." + } + } + + companion object { + val DEFAULT: QueryAdmissionLimits = QueryAdmissionLimits() + } +} + +internal data class AdmittedQueryInvocation( + val target: QueryTarget, + val operation: QueryOperation, + val resultShape: QueryResultShape, + val input: AdmittedQueryInput, +) + +internal sealed interface AdmittedQueryInput { + data class Single(val query: AdmittedRecordQuery) : AdmittedQueryInput + + data class Stream( + val query: AdmittedRecordQuery, + val limit: Int, + ) : AdmittedQueryInput + + data class Page( + val query: AdmittedRecordQuery, + val page: AdmittedPage, + ) : AdmittedQueryInput + + data class Count(val condition: AdmittedCondition) : AdmittedQueryInput + + data class Analytics(val query: AnalyticsQuery) : AdmittedQueryInput + + data class AnalyticsWire(val query: AdmittedAnalyticsQuery) : AdmittedQueryInput +} + +internal class AdmittedAnalyticsQuery( + val condition: AdmittedCondition, + val grouping: AnalyticsGrouping, + metrics: Iterable, + val window: AnalyticsBucketWindow, + val numericPolicy: AnalyticsNumericPolicy?, + val consistency: AnalyticsConsistency, + val completeness: AnalyticsCompleteness, +) { + val metrics: List = Collections.unmodifiableList(metrics.toList()) +} + +internal class AdmittedRecordQuery( + val condition: AdmittedCondition, + val projection: AdmittedProjection, + sort: Iterable, +) { + val sort: List = Collections.unmodifiableList(sort.toList()) + + override fun equals(other: Any?): Boolean = + this === other || + other is AdmittedRecordQuery && + condition == other.condition && + projection == other.projection && + sort == other.sort + + override fun hashCode(): Int = 31 * (31 * condition.hashCode() + projection.hashCode()) + sort.hashCode() +} + +internal class AdmittedProjection( + include: Iterable, + exclude: Iterable, +) { + val include: List = Collections.unmodifiableList(include.toList()) + val exclude: List = Collections.unmodifiableList(exclude.toList()) + + override fun equals(other: Any?): Boolean = + this === other || + other is AdmittedProjection && + include == other.include && + exclude == other.exclude + + override fun hashCode(): Int = 31 * include.hashCode() + exclude.hashCode() +} + +internal data class AdmittedSort( + val field: String, + val direction: Sort.Direction, +) + +internal data class AdmittedPage( + val index: Int, + val size: Int, + val offset: Long, +) + +internal sealed interface AdmittedConditionValue { + data object Absent : AdmittedConditionValue + + /** Legacy RAW marker; no driver object crosses the admission boundary. */ + data object NativeUnbound : AdmittedConditionValue + + data class QueryValue(val value: NormalizedValue) : AdmittedConditionValue + + data class TimeOfDay(val value: LocalTime) : AdmittedConditionValue + + data class Deletion(val value: DeletionState) : AdmittedConditionValue +} + +internal data class AdmittedConditionOptions( + val caseSensitivity: CaseSensitivity = CaseSensitivity.SENSITIVE, + val zoneId: ZoneId? = null, + val datePattern: AdmittedDatePattern? = null, +) + +internal class AdmittedDatePattern( + val formatter: DateTimeFormatter, + descriptor: String, +) { + private val signature: List = listOf( + descriptor, + formatter.locale, + formatter.decimalStyle, + formatter.resolverStyle, + formatter.chronology, + formatter.zone, + formatter.resolverFields, + ) + + override fun equals(other: Any?): Boolean = + this === other || other is AdmittedDatePattern && signature == other.signature + + override fun hashCode(): Int = signature.hashCode() +} + +internal class AdmittedCondition( + val field: String, + val operator: Operator, + val value: AdmittedConditionValue, + children: Iterable, + val options: AdmittedConditionOptions, +) { + val children: List = Collections.unmodifiableList(children.toList()) + + override fun equals(other: Any?): Boolean = + this === other || + other is AdmittedCondition && + field == other.field && + operator == other.operator && + value == other.value && + children == other.children && + options == other.options + + override fun hashCode(): Int { + var result = field.hashCode() + result = 31 * result + operator.hashCode() + result = 31 * result + value.hashCode() + result = 31 * result + children.hashCode() + result = 31 * result + options.hashCode() + return result + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/AnalyticsAdmissionGuard.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/AnalyticsAdmissionGuard.kt new file mode 100644 index 00000000000..3ef0635bc8d --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/AnalyticsAdmissionGuard.kt @@ -0,0 +1,86 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.admission + +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.analytics.AnalyticsGroupingKind +import me.ahoo.wow.api.query.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery + +internal class AnalyticsAdmissionGuard( + private val limits: QueryAdmissionLimits, +) { + fun admit( + query: AnalyticsQuery, + path: QueryRejectionPath, + budget: AdmissionBudget, + conditionAdmission: (Condition, QueryRejectionPath) -> AdmittedCondition, + ): AdmittedAnalyticsQuery { + val dimensions = query.grouping.dimensions + val metrics = query.metrics + if (dimensions.size > limits.maxCollectionSize || metrics.size > limits.maxCollectionSize) { + rejectBudget(path, QueryRejectionCode.COLLECTION_LIMIT_EXCEEDED) + } + dimensions.forEachIndexed { index, dimension -> + val dimensionPath = path.property("grouping").property("dimensions").index(index) + budget.consumeString(dimension.alias, dimensionPath.property("alias")) + validateField(dimension.field, dimensionPath.property("field")) + budget.consumeUtf8(dimension.field, dimensionPath.property("field")) + } + metrics.forEachIndexed { index, metric -> + val metricPath = path.property("metrics").index(index) + budget.consumeString(metric.alias, metricPath.property("alias")) + metric.field?.let { field -> + validateField(field, metricPath.property("field")) + budget.consumeUtf8(field, metricPath.property("field")) + } + } + query.window.cursor?.let { cursor -> + budget.consumeString(cursor.value, path.property("window").property("cursor")) + } + if (query.grouping.kind == AnalyticsGroupingKind.GLOBAL && query.window.cursor != null) { + rejectInvalid(path.property("window").property("cursor"), QueryRejectionCode.INVALID_CURSOR_BINDING) + } + return AdmittedAnalyticsQuery( + condition = conditionAdmission(query.condition, path.property("condition")), + grouping = query.grouping, + metrics = metrics, + window = query.window, + numericPolicy = query.numericPolicy, + consistency = query.consistency, + completeness = query.completeness, + ) + } + + private fun validateField(field: String, path: QueryRejectionPath) { + if (field.isBlank()) { + rejectInvalid(path, QueryRejectionCode.FIELD_REQUIRED) + } + if (field.length > limits.maxFieldLength) { + rejectBudget(path, QueryRejectionCode.STRING_LIMIT_EXCEEDED) + } + if (field.split('.').any { it.isBlank() || it.any(Char::isISOControl) }) { + rejectInvalid(path, QueryRejectionCode.INVALID_FIELD) + } + } + + private fun rejectInvalid(path: QueryRejectionPath, code: QueryRejectionCode): Nothing = + rejectQuery(QueryRejectionCategory.INVALID_QUERY, path, code) + + private fun rejectBudget(path: QueryRejectionPath, code: QueryRejectionCode): Nothing = + rejectQuery(QueryRejectionCategory.BUDGET_EXCEEDED, path, code) +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/RawAdmissionGuard.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/RawAdmissionGuard.kt new file mode 100644 index 00000000000..f37bb6ffacb --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/RawAdmissionGuard.kt @@ -0,0 +1,720 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.admission + +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DeletionState +import me.ahoo.wow.api.query.IListQuery +import me.ahoo.wow.api.query.IPagedQuery +import me.ahoo.wow.api.query.ISingleQuery +import me.ahoo.wow.api.query.Operator +import me.ahoo.wow.api.query.Pagination +import me.ahoo.wow.api.query.Projection +import me.ahoo.wow.api.query.Sort +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.model.QueryInput +import me.ahoo.wow.query.internal.model.QueryInvocation +import me.ahoo.wow.query.internal.normalization.CaseSensitivity +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import java.time.DateTimeException +import java.time.LocalTime +import java.time.ZoneId +import java.time.format.DateTimeFormatter +import java.time.format.DateTimeParseException +import java.util.Collections +import java.util.IdentityHashMap + +internal class RawAdmissionGuard( + private val limits: QueryAdmissionLimits, +) { + private val valueSnapshotter = RawValueSnapshotter(limits) + private val analyticsAdmissionGuard = AnalyticsAdmissionGuard(limits) + + fun admit(invocation: QueryInvocation): AdmittedQueryInvocation { + val session = AdmissionSession(limits) + val inputPath = QueryRejectionPath.ROOT.property("input") + val admittedInput = + when (val input = invocation.input) { + is QueryInput.Single -> AdmittedQueryInput.Single( + admitRecordQuery(input.query, inputPath.property("query"), session), + ) + + is QueryInput.Stream -> admitStream(input.query, inputPath.property("query"), session) + is QueryInput.Page -> admitPage(input.query, inputPath.property("query"), session) + is QueryInput.Count -> AdmittedQueryInput.Count( + admitCondition(input.condition, inputPath.property("condition"), 1, session), + ) + + is QueryInput.Analytics -> AdmittedQueryInput.Analytics(input.query) + is QueryInput.AnalyticsWire -> AdmittedQueryInput.AnalyticsWire( + analyticsAdmissionGuard.admit( + input.query, + inputPath.property("query"), + session.budget, + ) { condition, path -> admitCondition(condition, path, 1, session) }, + ) + } + return AdmittedQueryInvocation( + target = invocation.target, + operation = invocation.operation, + resultShape = invocation.resultShape, + input = admittedInput, + ) + } + + private fun admitStream( + query: IListQuery, + path: QueryRejectionPath, + session: AdmissionSession, + ): AdmittedQueryInput.Stream { + val condition: Condition? = query.condition + val projection: Projection? = query.projection + val sort: List? = query.sort + val limit = query.limit + requireRecordQuery(condition, projection, sort, path) + if (limit < 0) { + rejectInvalid(path.property("limit"), QueryRejectionCode.INVALID_LIMIT) + } + return AdmittedQueryInput.Stream( + query = admitRecordQuery( + checkNotNull(condition), + checkNotNull(projection), + checkNotNull(sort), + path, + session, + ), + limit = limit, + ) + } + + private fun admitPage( + query: IPagedQuery, + path: QueryRejectionPath, + session: AdmissionSession, + ): AdmittedQueryInput.Page { + val condition: Condition? = query.condition + val projection: Projection? = query.projection + val sort: List? = query.sort + val pagination: Pagination? = query.pagination + requireRecordQuery(condition, projection, sort, path) + if (pagination == null) { + rejectInvalid(path.property("pagination"), QueryRejectionCode.INVALID_PAGE) + } + val index = pagination.index + val size = pagination.size + if (index < 1 || size <= 0) { + rejectInvalid(path.property("pagination"), QueryRejectionCode.INVALID_PAGE) + } + val offset = Math.multiplyExact(index.toLong() - 1, size.toLong()) + return AdmittedQueryInput.Page( + query = admitRecordQuery( + checkNotNull(condition), + checkNotNull(projection), + checkNotNull(sort), + path, + session, + ), + page = AdmittedPage(index, size, offset), + ) + } + + private fun admitRecordQuery( + query: ISingleQuery, + path: QueryRejectionPath, + session: AdmissionSession, + ): AdmittedRecordQuery { + val condition: Condition? = query.condition + val projection: Projection? = query.projection + val sort: List? = query.sort + requireRecordQuery(condition, projection, sort, path) + return admitRecordQuery( + checkNotNull(condition), + checkNotNull(projection), + checkNotNull(sort), + path, + session, + ) + } + + private fun requireRecordQuery( + condition: Condition?, + projection: Projection?, + sort: List?, + path: QueryRejectionPath, + ) { + if (condition == null) { + rejectInvalid(path.property("condition"), QueryRejectionCode.INVALID_VALUE_TYPE) + } + if (projection == null) { + rejectInvalid(path.property("projection"), QueryRejectionCode.INVALID_PROJECTION) + } + if (sort == null) { + rejectInvalid(path.property("sort"), QueryRejectionCode.INVALID_SORT) + } + } + + private fun admitRecordQuery( + condition: Condition, + projection: Projection, + sort: List, + path: QueryRejectionPath, + session: AdmissionSession, + ): AdmittedRecordQuery = + AdmittedRecordQuery( + condition = admitCondition(condition, path.property("condition"), 1, session), + projection = admitProjection(projection, path.property("projection"), session), + sort = admitSort(sort, path.property("sort"), session), + ) + + private fun admitProjection( + projection: Projection, + path: QueryRejectionPath, + session: AdmissionSession, + ): AdmittedProjection { + val include = projection.include + val exclude = projection.exclude + val admittedInclude = admitFieldList( + include, + path.property("include"), + limits.maxProjectionFields, + session, + ) + return AdmittedProjection( + include = admittedInclude, + exclude = admitFieldList( + exclude, + path.property("exclude"), + limits.maxProjectionFields - admittedInclude.size, + session, + ), + ) + } + + private fun admitSort( + sort: Iterable<*>, + path: QueryRejectionPath, + session: AdmissionSession, + ): List { + val result = ArrayList() + val iterator = sort.iterator() + while (iterator.hasNext()) { + if (result.size == limits.maxSortFields) { + rejectBudget(path, QueryRejectionCode.SORT_LIMIT_EXCEEDED) + } + val index = result.size + val item = iterator.next() + if (item !is Sort) { + rejectInvalid(path.index(index), QueryRejectionCode.INVALID_SORT) + } + val fieldPath = path.index(index).property("field") + validateField(item.field, fieldPath) + session.budget.consumeUtf8(item.field, fieldPath) + result += AdmittedSort(item.field, item.direction) + } + return Collections.unmodifiableList(result) + } + + private fun admitFieldList( + fields: Iterable<*>, + path: QueryRejectionPath, + limit: Int, + session: AdmissionSession, + ): List { + val result = ArrayList() + val iterator = fields.iterator() + while (iterator.hasNext()) { + if (result.size == limit) { + rejectBudget(path, QueryRejectionCode.PROJECTION_LIMIT_EXCEEDED) + } + val index = result.size + val field = iterator.next() + if (field !is String) { + rejectInvalid(path.index(index), QueryRejectionCode.INVALID_PROJECTION) + } + validateField(field, path.index(index)) + session.budget.consumeUtf8(field, path.index(index)) + result += field + } + return Collections.unmodifiableList(result) + } + + @Suppress("CyclomaticComplexMethod", "LongMethod") + private fun admitCondition( + condition: Condition, + path: QueryRejectionPath, + depth: Int, + session: AdmissionSession, + ): AdmittedCondition { + if (depth > limits.maxConditionDepth) { + rejectBudget(path, QueryRejectionCode.CONDITION_DEPTH_LIMIT_EXCEEDED) + } + if (session.conditionNodes == limits.maxConditionNodes) { + rejectBudget(path, QueryRejectionCode.CONDITION_NODE_LIMIT_EXCEEDED) + } + session.conditionNodes++ + if (session.conditionActive.put(condition, Unit) != null) { + rejectInvalid(path, QueryRejectionCode.CYCLIC_INPUT) + } + try { + val field = condition.field + val operator = condition.operator + val rawValue = condition.value + val rawChildren = condition.children + val rawOptions = condition.options + val fieldPath = path.property("field") + validateConditionField(field, operator, fieldPath) + session.budget.consumeUtf8(field, fieldPath) + val children = materializeChildren(rawChildren, path.property("children")) + validateChildren(operator, children, path.property("children")) + val options = admitOptions(operator, rawOptions, path.property("options"), session) + val value = admitConditionValue(operator, rawValue, path.property("value"), session) + val admittedChildren = children.mapIndexed { index, child -> + admitCondition(child, path.property("children").index(index), depth + 1, session) + } + return AdmittedCondition(field, operator, value, admittedChildren, options) + } finally { + session.conditionActive.remove(condition) + } + } + + private fun materializeChildren( + children: List<*>, + path: QueryRejectionPath, + ): List { + val result = ArrayList() + val iterator = children.iterator() + while (iterator.hasNext()) { + if (result.size == limits.maxChildrenPerNode) { + rejectBudget(path, QueryRejectionCode.CHILDREN_LIMIT_EXCEEDED) + } + val index = result.size + val child = iterator.next() + if (child !is Condition) { + rejectInvalid(path.index(index), QueryRejectionCode.INVALID_CHILDREN) + } + result += child + } + return result + } + + private fun validateChildren( + operator: Operator, + children: List, + path: QueryRejectionPath, + ) { + when (operator) { + Operator.AND, + Operator.OR, + Operator.NOR, + -> if (children.isEmpty()) { + rejectInvalid(path, QueryRejectionCode.INVALID_CHILDREN) + } + + Operator.ELEM_MATCH -> if (children.size != 1) { + rejectInvalid(path, QueryRejectionCode.INVALID_CHILDREN) + } + + else -> if (children.isNotEmpty()) { + rejectInvalid(path, QueryRejectionCode.INVALID_CHILDREN) + } + } + } + + @Suppress("CyclomaticComplexMethod", "LongMethod") + private fun admitConditionValue( + operator: Operator, + rawValue: Any?, + path: QueryRejectionPath, + session: AdmissionSession, + ): AdmittedConditionValue = + when (operator) { + Operator.AND, + Operator.OR, + Operator.NOR, + Operator.ALL, + Operator.ELEM_MATCH, + Operator.NULL, + Operator.NOT_NULL, + Operator.TRUE, + Operator.FALSE, + Operator.TODAY, + Operator.TOMORROW, + Operator.THIS_WEEK, + Operator.NEXT_WEEK, + Operator.LAST_WEEK, + Operator.THIS_MONTH, + Operator.LAST_MONTH, + -> AdmittedConditionValue.Absent + + Operator.ID, + Operator.AGGREGATE_ID, + Operator.TENANT_ID, + Operator.OWNER_ID, + Operator.SPACE_ID, + Operator.CONTAINS, + Operator.STARTS_WITH, + Operator.ENDS_WITH, + Operator.MATCH, + -> AdmittedConditionValue.QueryValue( + normalizeRequiredString(rawValue, path, session, requireNonBlank = operator == Operator.MATCH), + ) + + Operator.IDS, + Operator.AGGREGATE_IDS, + -> AdmittedConditionValue.QueryValue( + valueSnapshotter.snapshotRequiredStringIterable(rawValue, path, session.budget), + ) + + Operator.IN, + Operator.NOT_IN, + Operator.ALL_IN, + -> AdmittedConditionValue.QueryValue( + valueSnapshotter.snapshotRequiredIterable(rawValue, path, session.budget), + ) + + Operator.BETWEEN -> { + val value = valueSnapshotter.snapshotRequiredIterable(rawValue, path, session.budget) + if (value.values.size != 2) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_ARITY) + } + AdmittedConditionValue.QueryValue(value) + } + + Operator.EXISTS -> { + session.budget.enterValue(path) + if (rawValue !is Boolean) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE) + } + AdmittedConditionValue.QueryValue(NormalizedValue.BooleanValue(rawValue)) + } + + Operator.DELETED -> AdmittedConditionValue.Deletion(normalizeDeletionState(rawValue, path, session)) + Operator.BEFORE_TODAY -> AdmittedConditionValue.TimeOfDay(normalizeTimeOfDay(rawValue, path, session)) + Operator.RECENT_DAYS, + Operator.EARLIER_DAYS, + -> AdmittedConditionValue.QueryValue( + NormalizedValue.Int64(normalizePositiveWholeNumber(rawValue, path, session)), + ) + + Operator.EQ, + Operator.NE, + Operator.GT, + Operator.LT, + Operator.GTE, + Operator.LTE, + -> AdmittedConditionValue.QueryValue(valueSnapshotter.snapshot(rawValue, path, session.budget)) + + Operator.RAW -> AdmittedConditionValue.NativeUnbound + } + + private fun normalizeRequiredString( + rawValue: Any?, + path: QueryRejectionPath, + session: AdmissionSession, + requireNonBlank: Boolean, + ): NormalizedValue.Text { + session.budget.enterValue(path) + if (rawValue !is String) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE) + } + session.budget.consumeString(rawValue, path) + if (requireNonBlank && rawValue.isBlank()) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE) + } + return NormalizedValue.Text(rawValue) + } + + private fun normalizeDeletionState( + rawValue: Any?, + path: QueryRejectionPath, + session: AdmissionSession, + ): DeletionState { + session.budget.enterValue(path) + return when (rawValue) { + is DeletionState -> rawValue + is Boolean -> if (rawValue) DeletionState.DELETED else DeletionState.ACTIVE + is String -> { + session.budget.consumeString(rawValue, path) + try { + DeletionState.valueOf(rawValue.uppercase()) + } catch (error: IllegalArgumentException) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE, error) + } + } + else -> rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE) + } + } + + private fun normalizeTimeOfDay( + rawValue: Any?, + path: QueryRejectionPath, + session: AdmissionSession, + ): LocalTime { + session.budget.enterValue(path) + return when (rawValue) { + is LocalTime -> rawValue + is String -> { + session.budget.consumeString(rawValue, path) + try { + LocalTime.parse(rawValue) + } catch (error: DateTimeParseException) { + rejectInvalid(path, QueryRejectionCode.INVALID_TIME_VALUE, error) + } + } + is Number -> { + val seconds = exactLong(rawValue, path, QueryRejectionCode.INVALID_TIME_VALUE, session) + if (seconds !in 0..86_399) { + rejectInvalid(path, QueryRejectionCode.INVALID_TIME_VALUE) + } + LocalTime.ofSecondOfDay(seconds) + } + else -> rejectInvalid(path, QueryRejectionCode.INVALID_TIME_VALUE) + } + } + + private fun normalizePositiveWholeNumber( + rawValue: Any?, + path: QueryRejectionPath, + session: AdmissionSession, + ): Long { + session.budget.enterValue(path) + if (rawValue !is Number) { + rejectInvalid(path, QueryRejectionCode.INVALID_TIME_VALUE) + } + val value = exactLong(rawValue, path, QueryRejectionCode.INVALID_TIME_VALUE, session) + if (value <= 0) { + rejectInvalid(path, QueryRejectionCode.INVALID_TIME_VALUE) + } + return value + } + + private fun exactLong( + number: Number, + path: QueryRejectionPath, + code: QueryRejectionCode, + session: AdmissionSession, + ): Long { + if (!number.isSupported()) { + rejectInvalid(path, code) + } + val numberText = session.budget.consumeNumber(number, path) + return try { + numberText.toBigDecimal().longValueExact() + } catch (error: NumberFormatException) { + rejectInvalid(path, code, error) + } catch (error: ArithmeticException) { + rejectInvalid(path, code, error) + } + } + + private fun Number.isSupported(): Boolean = + when (this) { + is Byte, + is Short, + is Int, + is Long, + is Float, + is Double, + is java.math.BigDecimal, + is java.math.BigInteger, + -> true + else -> false + } + + private fun admitOptions( + operator: Operator, + options: Map<*, *>, + path: QueryRejectionPath, + session: AdmissionSession, + ): AdmittedConditionOptions { + var caseSensitivity = CaseSensitivity.SENSITIVE + var zoneId: ZoneId? = null + var datePattern: AdmittedDatePattern? = null + var count = 0 + val seenKeys = HashSet() + val iterator = options.entries.iterator() + while (iterator.hasNext()) { + if (count == limits.maxOptions) { + rejectBudget(path, QueryRejectionCode.OPTIONS_LIMIT_EXCEEDED) + } + val entry = iterator.next() + count++ + val key = entry.key + if (key !is String) { + rejectInvalid(path, QueryRejectionCode.INVALID_OPTION_TYPE) + } + val optionPath = path.key(key) + session.budget.consumeString(key, optionPath) + if (!seenKeys.add(key)) { + rejectInvalid(optionPath, QueryRejectionCode.DUPLICATE_OBJECT_KEY) + } + val value = entry.value + session.budget.enterValue(optionPath) + when (key) { + Condition.IGNORE_CASE_OPTION_KEY -> { + ensureOptionAllowed(operator, STRING_OPTION_OPERATORS, optionPath) + if (value !is Boolean) { + rejectInvalid(optionPath, QueryRejectionCode.INVALID_OPTION_TYPE) + } + caseSensitivity = if (value) CaseSensitivity.INSENSITIVE else CaseSensitivity.SENSITIVE + } + + Condition.ZONE_ID_OPTION_KEY -> { + ensureOptionAllowed(operator, TIME_OPERATORS, optionPath) + zoneId = normalizeZoneId(value, optionPath, session) + } + + Condition.DATE_PATTERN_OPTION_KEY -> { + ensureOptionAllowed(operator, TIME_OPERATORS, optionPath) + datePattern = normalizeDatePattern(value, optionPath, session) + } + + else -> rejectInvalid(optionPath, QueryRejectionCode.UNKNOWN_OPTION) + } + } + return AdmittedConditionOptions(caseSensitivity, zoneId, datePattern) + } + + private fun ensureOptionAllowed( + operator: Operator, + allowed: Set, + path: QueryRejectionPath, + ) { + if (operator !in allowed) { + rejectInvalid(path, QueryRejectionCode.OPTION_NOT_ALLOWED) + } + } + + private fun normalizeZoneId( + value: Any?, + path: QueryRejectionPath, + session: AdmissionSession, + ): ZoneId = + when (value) { + is ZoneId -> value + is String -> { + session.budget.consumeString(value, path) + try { + ZoneId.of(value) + } catch (error: DateTimeException) { + rejectInvalid(path, QueryRejectionCode.INVALID_OPTION_VALUE, error) + } + } + else -> rejectInvalid(path, QueryRejectionCode.INVALID_OPTION_TYPE) + } + + private fun normalizeDatePattern( + value: Any?, + path: QueryRejectionPath, + session: AdmissionSession, + ): AdmittedDatePattern = + when (value) { + is DateTimeFormatter -> admitDateTimeFormatter(value, path, session) + is String -> { + session.budget.consumeString(value, path) + try { + val formatter = DateTimeFormatter.ofPattern(value) + AdmittedDatePattern(formatter, formatter.toString()) + } catch (error: IllegalArgumentException) { + rejectInvalid(path, QueryRejectionCode.INVALID_OPTION_VALUE, error) + } + } + else -> rejectInvalid(path, QueryRejectionCode.INVALID_OPTION_TYPE) + } + + private fun admitDateTimeFormatter( + formatter: DateTimeFormatter, + path: QueryRejectionPath, + session: AdmissionSession, + ): AdmittedDatePattern { + val descriptor = formatter.toString() + session.budget.consumeString(descriptor, path) + return AdmittedDatePattern(formatter, descriptor) + } + + private fun validateConditionField(field: String, operator: Operator, path: QueryRejectionPath) { + if (operator in FIELDLESS_OPERATORS) { + if (field.isNotEmpty()) { + rejectInvalid(path, QueryRejectionCode.INVALID_FIELD) + } + return + } + validateField(field, path) + } + + private fun validateField(field: String, path: QueryRejectionPath) { + if (field.isBlank()) { + rejectInvalid(path, QueryRejectionCode.FIELD_REQUIRED) + } + if (field.length > limits.maxFieldLength) { + rejectBudget(path, QueryRejectionCode.STRING_LIMIT_EXCEEDED) + } + if (field.split('.').any { it.isBlank() || it.any(Char::isISOControl) }) { + rejectInvalid(path, QueryRejectionCode.INVALID_FIELD) + } + } + + private fun rejectInvalid( + path: QueryRejectionPath, + code: QueryRejectionCode, + cause: Throwable? = null, + ): Nothing = rejectQuery(QueryRejectionCategory.INVALID_QUERY, path, code, cause) + + private fun rejectBudget(path: QueryRejectionPath, code: QueryRejectionCode): Nothing = + rejectQuery(QueryRejectionCategory.BUDGET_EXCEEDED, path, code) + + private class AdmissionSession(limits: QueryAdmissionLimits) { + var conditionNodes: Int = 0 + val conditionActive: IdentityHashMap = IdentityHashMap() + val budget: AdmissionBudget = AdmissionBudget(limits) + } + + companion object { + private val FIELDLESS_OPERATORS = setOf( + Operator.AND, + Operator.OR, + Operator.NOR, + Operator.ID, + Operator.IDS, + Operator.AGGREGATE_ID, + Operator.AGGREGATE_IDS, + Operator.TENANT_ID, + Operator.OWNER_ID, + Operator.SPACE_ID, + Operator.DELETED, + Operator.ALL, + Operator.RAW, + ) + private val STRING_OPTION_OPERATORS = setOf( + Operator.CONTAINS, + Operator.STARTS_WITH, + Operator.ENDS_WITH, + ) + private val TIME_OPERATORS = setOf( + Operator.TODAY, + Operator.BEFORE_TODAY, + Operator.TOMORROW, + Operator.THIS_WEEK, + Operator.NEXT_WEEK, + Operator.LAST_WEEK, + Operator.THIS_MONTH, + Operator.LAST_MONTH, + Operator.RECENT_DAYS, + Operator.EARLIER_DAYS, + ) + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/RawValueSnapshotter.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/RawValueSnapshotter.kt new file mode 100644 index 00000000000..7490a3dbb85 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/admission/RawValueSnapshotter.kt @@ -0,0 +1,309 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.admission + +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import java.math.BigDecimal +import java.math.BigInteger +import java.time.Instant +import java.time.OffsetDateTime +import java.time.ZonedDateTime +import java.util.Date +import java.util.IdentityHashMap +import java.util.LinkedHashMap +import java.util.UUID + +/** + * Converts legacy `Any` values into the closed, deeply immutable normalized value algebra. + * + * Each public operation owns its identity session, so this stateless component is safe to reuse concurrently. + */ +internal class RawValueSnapshotter( + private val limits: QueryAdmissionLimits, +) { + fun snapshot( + rawValue: Any?, + path: QueryRejectionPath, + budget: AdmissionBudget, + ): NormalizedValue = snapshotValue(rawValue, path, depth = 1, ValueSession(), budget) + + fun snapshotRequiredIterable( + rawValue: Any?, + path: QueryRejectionPath, + budget: AdmissionBudget, + ): NormalizedValue.ListValue { + if (rawValue !is Iterable<*>) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE) + } + budget.enterValue(path) + return snapshotIterable(rawValue, path, depth = 1, ValueSession(), budget) + } + + fun snapshotRequiredStringIterable( + rawValue: Any?, + path: QueryRejectionPath, + budget: AdmissionBudget, + ): NormalizedValue.ListValue { + if (rawValue !is Iterable<*>) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE) + } + budget.enterValue(path) + val session = ValueSession() + enterValue(rawValue, path, session) + try { + val result = ArrayList() + val iterator = rawValue.iterator() + while (iterator.hasNext()) { + if (result.size == limits.maxCollectionSize) { + rejectBudget(path, QueryRejectionCode.COLLECTION_LIMIT_EXCEEDED) + } + val itemPath = path.index(result.size) + val item = iterator.next() + budget.enterValue(itemPath) + if (item !is String) { + rejectInvalid(itemPath, QueryRejectionCode.INVALID_VALUE_TYPE) + } + budget.consumeString(item, itemPath) + result += NormalizedValue.Text(item) + } + return NormalizedValue.ListValue(result) + } finally { + session.active.remove(rawValue) + } + } + + @Suppress("CyclomaticComplexMethod", "LongMethod") + private fun snapshotValue( + rawValue: Any?, + path: QueryRejectionPath, + depth: Int, + session: ValueSession, + budget: AdmissionBudget, + ): NormalizedValue { + if (depth > limits.maxValueDepth) { + rejectBudget(path, QueryRejectionCode.VALUE_DEPTH_LIMIT_EXCEEDED) + } + budget.enterValue(path) + return when (rawValue) { + null -> NormalizedValue.Null + is NormalizedValue -> rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE) + is Boolean -> NormalizedValue.BooleanValue(rawValue) + is String -> { + budget.consumeString(rawValue, path) + NormalizedValue.Text(rawValue) + } + is Char -> normalizedText(rawValue.toString(), path, budget) + is UUID -> normalizedText(rawValue.toString(), path, budget) + is Enum<*> -> normalizedText(rawValue.name, path, budget) + is Number -> normalizeNumber(rawValue, path, budget) + is Instant -> NormalizedValue.InstantValue(rawValue) + is Date -> { + val instant = try { + rawValue.toInstant() + } catch (error: UnsupportedOperationException) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE, error) + } + NormalizedValue.InstantValue(instant) + } + is OffsetDateTime -> NormalizedValue.InstantValue(rawValue.toInstant()) + is ZonedDateTime -> NormalizedValue.InstantValue(rawValue.toInstant()) + is ByteArray -> { + budget.consumeBytes(rawValue.size, path) + NormalizedValue.Bytes(rawValue) + } + is Map<*, *> -> snapshotMap(rawValue, path, depth, session, budget) + is Iterable<*> -> snapshotIterable(rawValue, path, depth, session, budget) + is Array<*> -> snapshotArray(rawValue, path, depth, session, budget) + else -> rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE) + } + } + + private fun normalizedText( + value: String, + path: QueryRejectionPath, + budget: AdmissionBudget, + ): NormalizedValue.Text { + budget.consumeString(value, path) + return NormalizedValue.Text(value) + } + + @Suppress("CyclomaticComplexMethod") + private fun normalizeNumber( + number: Number, + path: QueryRejectionPath, + budget: AdmissionBudget, + ): NormalizedValue { + if (!number.isSupported()) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE) + } + budget.consumeNumber(number, path) + val decimal = try { + when (number) { + is BigDecimal -> number + is BigInteger -> number.toBigDecimal() + is Byte, + is Short, + is Int, + is Long, + -> BigDecimal.valueOf(number.toLong()) + is Float -> { + if (!number.isFinite()) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE) + } + BigDecimal.valueOf(number.toDouble()) + } + is Double -> { + if (!number.isFinite()) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE) + } + BigDecimal.valueOf(number) + } + else -> error("Supported numeric types are exhaustive.") + } + } catch (error: NumberFormatException) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE, error) + } + val exactLong = try { + decimal.longValueExact() + } catch (_: ArithmeticException) { + null + } + if (exactLong != null) { + return NormalizedValue.Int64(exactLong) + } + return try { + NormalizedValue.Decimal(decimal) + } catch (error: ArithmeticException) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE, error) + } + } + + private fun Number.isSupported(): Boolean = + when (this) { + is BigDecimal, + is BigInteger, + is Byte, + is Short, + is Int, + is Long, + is Float, + is Double, + -> true + else -> false + } + + private fun snapshotIterable( + values: Iterable<*>, + path: QueryRejectionPath, + depth: Int, + session: ValueSession, + budget: AdmissionBudget, + ): NormalizedValue.ListValue { + enterValue(values, path, session) + try { + val result = ArrayList() + val iterator = values.iterator() + while (iterator.hasNext()) { + if (result.size == limits.maxCollectionSize) { + rejectBudget(path, QueryRejectionCode.COLLECTION_LIMIT_EXCEEDED) + } + val index = result.size + result += snapshotValue(iterator.next(), path.index(index), depth + 1, session, budget) + } + return NormalizedValue.ListValue(result) + } finally { + session.active.remove(values) + } + } + + private fun snapshotArray( + values: Array<*>, + path: QueryRejectionPath, + depth: Int, + session: ValueSession, + budget: AdmissionBudget, + ): NormalizedValue.ListValue { + if (values.size > limits.maxCollectionSize) { + rejectBudget(path, QueryRejectionCode.COLLECTION_LIMIT_EXCEEDED) + } + enterValue(values, path, session) + try { + return NormalizedValue.ListValue( + values.mapIndexed { index, value -> + snapshotValue(value, path.index(index), depth + 1, session, budget) + }, + ) + } finally { + session.active.remove(values) + } + } + + private fun snapshotMap( + values: Map<*, *>, + path: QueryRejectionPath, + depth: Int, + session: ValueSession, + budget: AdmissionBudget, + ): NormalizedValue.ObjectValue { + enterValue(values, path, session) + try { + val result = LinkedHashMap() + val iterator = values.entries.iterator() + var entryCount = 0 + while (iterator.hasNext()) { + if (entryCount == limits.maxObjectFields) { + rejectBudget(path, QueryRejectionCode.OBJECT_LIMIT_EXCEEDED) + } + val entry = iterator.next() + entryCount++ + val key = entry.key + if (key !is String) { + rejectInvalid(path, QueryRejectionCode.INVALID_VALUE_TYPE) + } + val keyPath = path.key(key) + budget.consumeString(key, keyPath) + if (result.containsKey(key)) { + rejectInvalid(keyPath, QueryRejectionCode.DUPLICATE_OBJECT_KEY) + } + result[key] = snapshotValue(entry.value, keyPath, depth + 1, session, budget) + } + return NormalizedValue.ObjectValue(result) + } finally { + session.active.remove(values) + } + } + + private fun enterValue(value: Any, path: QueryRejectionPath, session: ValueSession) { + if (session.active.put(value, Unit) != null) { + rejectInvalid(path, QueryRejectionCode.CYCLIC_INPUT) + } + } + + private fun rejectInvalid( + path: QueryRejectionPath, + code: QueryRejectionCode, + cause: Throwable? = null, + ): Nothing = rejectQuery(QueryRejectionCategory.INVALID_QUERY, path, code, cause) + + private fun rejectBudget(path: QueryRejectionPath, code: QueryRejectionCode): Nothing = + rejectQuery(QueryRejectionCategory.BUDGET_EXCEEDED, path, code) + + private class ValueSession { + val active: IdentityHashMap = IdentityHashMap() + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/analytics/AnalyticsModel.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/analytics/AnalyticsModel.kt new file mode 100644 index 00000000000..194e8c90ead --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/analytics/AnalyticsModel.kt @@ -0,0 +1,196 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.analytics + +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.cursor.QueryCursorToken +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.plan.PlanFingerprint +import me.ahoo.wow.query.internal.value.NonEmptyList +import java.math.RoundingMode + +@JvmInline +internal value class AnalyticsAlias(val value: String) { + init { + require(value.isNotBlank()) { "Analytics alias must not be blank." } + require(value.length <= MAX_ALIAS_LENGTH) { + "Analytics alias must not exceed $MAX_ALIAS_LENGTH characters." + } + require(value.none(Char::isISOControl)) { "Analytics alias must not contain control characters." } + require('.' !in value && '$' !in value) { "Analytics alias must be a safe backend field name." } + } + + private companion object { + const val MAX_ALIAS_LENGTH = 128 + } +} + +internal enum class AnalyticsMissingPolicy { + EXCLUDE, + AS_NULL_BUCKET, +} + +internal data class AnalyticsDimension( + val alias: AnalyticsAlias, + val field: LogicalField, + val missingPolicy: AnalyticsMissingPolicy = AnalyticsMissingPolicy.EXCLUDE, +) + +internal sealed interface AnalyticsGrouping { + data object Global : AnalyticsGrouping + + data class By(val dimensions: NonEmptyList) : AnalyticsGrouping +} + +internal sealed interface AnalyticsMetric { + val alias: AnalyticsAlias + + data class DocumentCount( + override val alias: AnalyticsAlias, + ) : AnalyticsMetric + + data class Min( + override val alias: AnalyticsAlias, + val field: LogicalField, + ) : AnalyticsMetric + + data class Max( + override val alias: AnalyticsAlias, + val field: LogicalField, + ) : AnalyticsMetric + + data class Sum( + override val alias: AnalyticsAlias, + val field: LogicalField, + ) : AnalyticsMetric + + data class Average( + override val alias: AnalyticsAlias, + val field: LogicalField, + ) : AnalyticsMetric +} + +internal sealed interface AnalyticsCondition { + data object All : AnalyticsCondition + + /** Reserved logical alias reference. The first portable planner rejects it deterministically. */ + data class Predicate(val alias: AnalyticsAlias) : AnalyticsCondition +} + +internal sealed interface AnalyticsBucketOrder { + data object Default : AnalyticsBucketOrder + + data object DimensionKeyAscending : AnalyticsBucketOrder + + data class MetricDescending(val alias: AnalyticsAlias) : AnalyticsBucketOrder +} + +internal sealed interface AnalyticsBucketWindow { + val limit: Int + + data class First(override val limit: Int) : AnalyticsBucketWindow { + init { + require(limit > 0) { + "Analytics bucket limit must be positive." + } + } + } + + data class After( + override val limit: Int, + val cursor: DecodedAnalyticsCursor, + ) : AnalyticsBucketWindow { + init { + require(limit > 0) { + "Analytics bucket limit must be positive." + } + } + } +} + +internal enum class AnalyticsConsistency { + EVENTUAL, + SNAPSHOT, +} + +internal enum class AnalyticsCompleteness { + EXACT, + APPROXIMATE, +} + +internal enum class AnalyticsOverflowPolicy { + REJECT, +} + +internal enum class AnalyticsNumericPromotion { + DECIMAL128, +} + +internal enum class AnalyticsNullPlacement { + FIRST, +} + +internal enum class AnalyticsTextCollation { + BINARY, +} + +internal data class AnalyticsNumericPolicy( + val promotion: AnalyticsNumericPromotion, + val precision: Int, + val scale: Int, + val roundingMode: RoundingMode, + val overflowPolicy: AnalyticsOverflowPolicy, +) { + init { + require(precision > 0) { + "Analytics numeric precision must be positive." + } + require(scale in 0..precision) { + "Analytics numeric scale must be between zero and precision." + } + } +} + +/** + * Decoded semantic cursor state. Token encoding, signing and expiry are deliberately outside Phase 1. + */ +internal data class DecodedAnalyticsCursor( + val target: QueryTarget, + val planFingerprint: PlanFingerprint, + val dimensionAliases: NonEmptyList, + val afterKey: NonEmptyList, +) + +/** + * Backend-independent analytics input after its future wire adapter has normalized field and value semantics. + */ +internal data class AnalyticsQuery( + val userCondition: NormalizedCondition, + val grouping: AnalyticsGrouping, + val metrics: NonEmptyList, + val having: AnalyticsCondition = AnalyticsCondition.All, + val bucketOrder: AnalyticsBucketOrder = AnalyticsBucketOrder.Default, + val bucketWindow: AnalyticsBucketWindow = AnalyticsBucketWindow.First(DEFAULT_BUCKET_LIMIT), + val numericPolicy: AnalyticsNumericPolicy? = null, + val requiredConsistency: AnalyticsConsistency = AnalyticsConsistency.EVENTUAL, + val requiredCompleteness: AnalyticsCompleteness = AnalyticsCompleteness.EXACT, + /** Opaque public token awaiting asynchronous store resolution inside the Gateway. */ + val cursorToken: QueryCursorToken? = null, +) { + private companion object { + const val DEFAULT_BUCKET_LIMIT = 100 + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/PersistentQueryCursorLeaseCoordinator.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/PersistentQueryCursorLeaseCoordinator.kt new file mode 100644 index 00000000000..bc26a430dba --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/PersistentQueryCursorLeaseCoordinator.kt @@ -0,0 +1,157 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class) + +package me.ahoo.wow.query.internal.cursor + +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import reactor.core.publisher.Mono +import java.time.Instant + +/** Coordinates the persistent CAS lease store with backend-owned resource cleanup. */ +internal class PersistentQueryCursorLeaseCoordinator( + private val manager: PersistentQueryCursorLeaseManager, + registrations: Iterable, + private val observer: QueryCursorLeaseObserver = QueryCursorLeaseObserver.NOOP, +) { + private val closers: Map = registrations.toList().let { values -> + require( + values.map { registration -> registration.target to registration.backendId }.distinct().size == values.size, + ) { + "Duplicate cursor backend lease registration." + } + values.associate { registration -> + CursorBackendKey(registration.target, registration.backendId) to registration.closer + } + } + + fun load(token: QueryCursorToken): Mono = manager.load(token) + + fun acquire( + loaded: LoadedQueryCursorLease, + expectedBinding: QueryCursorLeaseBinding, + ): Mono = manager.acquire(loaded, expectedBinding) + + fun issue(envelope: QueryCursorEnvelope): Mono { + requireCloser(envelope) + return manager.issue(envelope) + } + + fun supports(target: QueryTarget, backendId: BackendId): Boolean = + closers.containsKey(CursorBackendKey(target, backendId)) + + fun close(envelope: QueryCursorEnvelope, reason: QueryCursorCleanupReason): Mono = + closeBestEffort(envelope, reason) + + fun close( + state: QueryCursorBackendState, + descriptor: QueryCursorLeaseDescriptor, + reason: QueryCursorCleanupReason, + ): Mono = closeBestEffort(state, descriptor, reason) + + /** Reaps one bounded, stable-keyset batch. Scheduling and repetition remain an application responsibility. */ + fun reapExpired(before: Instant, batchSize: Int): Mono { + require(batchSize > 0) { "Query cursor reaper batch size must be positive." } + return manager.reapExpired(before, limit = batchSize) + .concatMap { envelope -> + closeForReaper(envelope).map { closed -> if (closed) 1L else 0L } + } + .reduce(0L, Long::plus) + } + + private fun closeForReaper(envelope: QueryCursorEnvelope): Mono { + val state = envelope.backendState ?: return Mono.just(true) + val descriptor = QueryCursorLeaseDescriptor(envelope.target, state.backendId, envelope.mappingGenerationDigest) + val closer = closers[CursorBackendKey(descriptor.target, state.backendId)] ?: return Mono.fromSupplier { + notifyFailure( + descriptor, + QueryCursorCleanupReason.ABANDONED, + IllegalStateException("Cursor backend closer is not registered."), + ) + false + } + return Mono.defer { closer.close(state) } + .thenReturn(true) + .onErrorResume { error -> + notifyFailure(descriptor, QueryCursorCleanupReason.ABANDONED, error) + Mono.just(false) + } + } + + private fun requireCloser(envelope: QueryCursorEnvelope) { + val state = envelope.backendState ?: return + if (closers[CursorBackendKey(envelope.target, state.backendId)] == null) { + rejectQuery( + QueryRejectionCategory.BACKEND_UNAVAILABLE, + CURSOR_PATH, + QueryRejectionCode.BACKEND_NOT_REGISTERED, + ) + } + } + + private fun closeBestEffort( + envelope: QueryCursorEnvelope, + reason: QueryCursorCleanupReason, + ): Mono = envelope.backendState?.let { state -> + closeBestEffort( + state, + QueryCursorLeaseDescriptor(envelope.target, state.backendId, envelope.mappingGenerationDigest), + reason, + ) + } ?: Mono.empty() + + private fun closeBestEffort( + state: QueryCursorBackendState, + descriptor: QueryCursorLeaseDescriptor, + reason: QueryCursorCleanupReason, + ): Mono { + val closer = closers[CursorBackendKey(descriptor.target, state.backendId)] ?: return Mono.fromRunnable { + notifyFailure(descriptor, reason, IllegalStateException("Cursor backend closer is not registered.")) + } + return Mono.defer { closer.close(state) } + .onErrorResume { error -> + notifyFailure(descriptor, reason, error) + Mono.empty() + } + } + + private fun notifyFailure( + descriptor: QueryCursorLeaseDescriptor, + reason: QueryCursorCleanupReason, + error: Throwable, + ) { + try { + observer.onCleanupFailure(descriptor, reason, error) + } catch (_: RuntimeException) { + // Observability must never replace the query or cleanup outcome. + } + } + + private companion object { + val CURSOR_PATH: QueryRejectionPath = QueryRejectionPath.ROOT.property("cursor") + } + + private data class CursorBackendKey(val target: QueryTarget, val backendId: BackendId) +} + +internal data class PersistentQueryCursorBackendLeaseRegistration( + val target: QueryTarget, + val backendId: BackendId, + val closer: QueryCursorBackendLeaseCloser, +) diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/PersistentQueryCursorLeaseManager.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/PersistentQueryCursorLeaseManager.kt new file mode 100644 index 00000000000..9352868791c --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/PersistentQueryCursorLeaseManager.kt @@ -0,0 +1,516 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.query.internal.cursor + +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.cursor.QueryCursorLeaseCreateResult +import me.ahoo.wow.query.cursor.QueryCursorLeaseEntry +import me.ahoo.wow.query.cursor.QueryCursorLeaseId +import me.ahoo.wow.query.cursor.QueryCursorLeaseStore +import me.ahoo.wow.query.cursor.QueryCursorPayloadFormat +import me.ahoo.wow.query.cursor.StoredQueryCursorLease +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.io.ByteArrayInputStream +import java.io.ByteArrayOutputStream +import java.io.DataInputStream +import java.io.DataOutputStream +import java.io.IOException +import java.math.BigDecimal +import java.security.MessageDigest +import java.security.SecureRandom +import java.time.Clock +import java.time.DateTimeException +import java.time.Instant +import java.time.temporal.ChronoUnit +import java.util.Base64 + +internal class PersistentQueryCursorLeaseManager( + private val store: QueryCursorLeaseStore, + signingKeyRing: QueryCursorSigningKeyRing, + private val clock: Clock = Clock.systemUTC(), + private val limits: QueryCursorLeaseLimits = QueryCursorLeaseLimits(), +) { + private val tokenCodec = QueryCursorTokenCodec(signingKeyRing) + private val envelopeCodec = QueryCursorEnvelopeCodec(signingKeyRing, limits.maxBackendStateBytes) + private val random = SecureRandom() + + fun issue(envelope: QueryCursorEnvelope): Mono = Mono.defer { + val normalized = envelope.copy(expiresAt = envelope.expiresAt.truncatedTo(ChronoUnit.MILLIS)) + validateEnvelope(normalized) + create(normalized, attempt = 0) + } + + /** Loads and verifies the immutable envelope without consuming its store revision. */ + fun load(token: QueryCursorToken): Mono = Mono.defer { + val claims = tokenCodec.decode(token) + val now = clock.instant() + if (!claims.expiresAt.isAfter(now)) rejectExpiredCursor() + store.load(QueryCursorLeaseId(claims.id)) + .switchIfEmpty(Mono.error(invalidCursorException())) + .map { stored -> decodeStored(stored, claims, now) } + } + + /** Validates the full semantic binding before atomically transferring one-time ownership. */ + fun acquire( + loaded: LoadedQueryCursorLease, + expectedBinding: QueryCursorLeaseBinding, + ): Mono = Mono.defer { + if (loaded.envelope.binding() != expectedBinding) { + rejectQuery( + QueryRejectionCategory.INVALID_CURSOR, + CURSOR_PATH, + QueryRejectionCode.INVALID_CURSOR_BINDING, + ) + } + store.compareAndDelete(loaded.stored) + .flatMap { deleted -> + if (deleted) Mono.just(loaded.envelope) else Mono.error(invalidCursorException()) + } + } + + /** Transfers only leases whose exact revision is still owned by this reaper. */ + fun reapExpired( + before: Instant, + afterId: QueryCursorLeaseId? = null, + limit: Int, + ): Flux { + require(limit > 0) { "Query cursor reaper limit must be positive." } + return Flux.defer { + store.scanExpired(before, afterId, limit) + .concatMap { stored -> + Mono.defer { + val envelope = envelopeCodec.decode(stored.entry.payload()) + if (envelope.expiresAt != stored.entry.expiresAt || envelope.expiresAt.isAfter(before)) { + return@defer Mono.error(invalidCursorException()) + } + store.compareAndDelete(stored).filter { it }.map { envelope } + } + } + } + } + + private fun create(envelope: QueryCursorEnvelope, attempt: Int): Mono { + if (attempt == MAX_COLLISION_ATTEMPTS) { + return Mono.error(invalidCursorException()) + } + val idBytes = ByteArray(CURSOR_ID_BYTES).also(random::nextBytes) + val id = URL_ENCODER.encodeToString(idBytes) + val token = tokenCodec.encode(id, envelope.expiresAt) + val entry = QueryCursorLeaseEntry( + QueryCursorLeaseId(id), + envelope.expiresAt, + QueryCursorPayloadFormat.WOW_QUERY_CURSOR_V1, + envelopeCodec.encode(envelope), + ) + return store.create(entry).flatMap { result -> + when (result) { + QueryCursorLeaseCreateResult.CREATED -> Mono.just(token) + QueryCursorLeaseCreateResult.COLLISION -> create(envelope, attempt + 1) + QueryCursorLeaseCreateResult.CAPACITY_EXCEEDED -> Mono.error(cursorCapacityException()) + } + } + } + + private fun decodeStored( + stored: StoredQueryCursorLease, + claims: QueryCursorTokenCodec.TokenClaims, + now: Instant, + ): LoadedQueryCursorLease { + requireMatchingStoredClaims(stored, claims) + requireUnexpiredStoredEntry(stored, now) + val envelope = envelopeCodec.decode(stored.entry.payload()) + requireMatchingStoredEnvelope(stored, envelope) + return LoadedQueryCursorLease(stored, envelope) + } + + private fun requireMatchingStoredClaims( + stored: StoredQueryCursorLease, + claims: QueryCursorTokenCodec.TokenClaims, + ) { + if (stored.entry.id.value != claims.id || stored.entry.expiresAt != claims.expiresAt) { + throw invalidCursorException() + } + } + + private fun requireUnexpiredStoredEntry(stored: StoredQueryCursorLease, now: Instant) { + if (!stored.entry.expiresAt.isAfter(now)) throw expiredCursorException() + } + + private fun requireMatchingStoredEnvelope( + stored: StoredQueryCursorLease, + envelope: QueryCursorEnvelope, + ) { + if (envelope.expiresAt != stored.entry.expiresAt) throw invalidCursorException() + } + + private fun validateEnvelope(envelope: QueryCursorEnvelope) { + val now = clock.instant() + if (!envelope.expiresAt.isAfter(now)) rejectExpiredCursor() + val maximumExpiry = maximumExpiry(now) + if (envelope.expiresAt.isAfter(maximumExpiry) || + envelope.backendState?.let { state -> state.size > limits.maxBackendStateBytes } == true + ) { + rejectCursorCapacity() + } + } + + private fun maximumExpiry(now: Instant): Instant = try { + now.plus(limits.maxTtl) + } catch (error: DateTimeException) { + throw IllegalStateException("Cursor TTL cannot be represented.", error) + } catch (error: ArithmeticException) { + throw IllegalStateException("Cursor TTL cannot be represented.", error) + } + + private companion object { + const val CURSOR_ID_BYTES = 32 + const val MAX_COLLISION_ATTEMPTS = 8 + val URL_ENCODER: Base64.Encoder = Base64.getUrlEncoder().withoutPadding() + val CURSOR_PATH: QueryRejectionPath = QueryRejectionPath.ROOT.property("cursor") + } +} + +internal data class LoadedQueryCursorLease( + val stored: StoredQueryCursorLease, + val envelope: QueryCursorEnvelope, +) + +private class QueryCursorEnvelopeCodec( + private val keyRing: QueryCursorSigningKeyRing, + private val maxBackendStateBytes: Int, +) { + private val maxBodyBytes: Int = Math.addExact( + FIXED_BODY_BYTES, + maxOf(DEFAULT_BACKEND_STATE_BYTES, maxBackendStateBytes), + ) + private val maxPayloadBytes: Int = Math.addExact(HEADER_BYTES + HMAC_BYTES, maxBodyBytes) + + fun encode(envelope: QueryCursorEnvelope): ByteArray { + val body = ByteArrayOutputStream().use { bytes -> + DataOutputStream(bytes).use { output -> output.writeEnvelope(envelope) } + bytes.toByteArray() + } + require(body.size <= maxBodyBytes) { "Query cursor envelope exceeds its encoded size limit." } + val headerAndBody = ByteArrayOutputStream(HEADER_BYTES + body.size).use { bytes -> + DataOutputStream(bytes).use { output -> + output.writeInt(MAGIC) + output.writeByte(FORMAT_VERSION) + output.writeByte(keyRing.current.id) + output.writeInt(body.size) + output.write(body) + } + bytes.toByteArray() + } + return headerAndBody + keyRing.current.sign(headerAndBody) + } + + fun decode(payload: ByteArray): QueryCursorEnvelope { + if (payload.size !in (HEADER_BYTES + HMAC_BYTES + 1)..maxPayloadBytes) rejectInvalidCursor() + val signed = payload.copyOf(payload.size - HMAC_BYTES) + val signature = payload.copyOfRange(payload.size - HMAC_BYTES, payload.size) + val key = signingKey(signed) + if (!MessageDigest.isEqual(signature, key.sign(signed))) rejectInvalidCursor() + return translateMalformedCursor { decodeSigned(signed) } + } + + private fun decodeSigned(signed: ByteArray): QueryCursorEnvelope = + DataInputStream(ByteArrayInputStream(signed)).use { input -> + if (input.readInt() != MAGIC || input.readUnsignedByte() != FORMAT_VERSION) rejectInvalidCursor() + if (keyRing.resolve(input.readUnsignedByte()) == null) rejectInvalidCursor() + val bodySize = input.readInt() + if (bodySize <= 0 || bodySize > maxBodyBytes || bodySize != input.available()) rejectInvalidCursor() + val envelope = input.readEnvelope() + if (input.available() != 0) rejectInvalidCursor() + envelope + } + + private inline fun translateMalformedCursor(block: () -> T): T = try { + block() + } catch (error: QueryRejectedException) { + throw error + } catch (error: IOException) { + rejectInvalidCursor(error) + } catch (error: DateTimeException) { + rejectInvalidCursor(error) + } catch (error: ArithmeticException) { + rejectInvalidCursor(error) + } catch (error: IllegalArgumentException) { + rejectInvalidCursor(error) + } + + private fun signingKey(signed: ByteArray): QueryCursorSigningKey = try { + DataInputStream(ByteArrayInputStream(signed)).use { input -> + if (input.readInt() != MAGIC || input.readUnsignedByte() != FORMAT_VERSION) rejectInvalidCursor() + keyRing.resolve(input.readUnsignedByte()) ?: rejectInvalidCursor() + } + } catch (error: QueryRejectedException) { + throw error + } catch (error: IOException) { + rejectInvalidCursor(error) + } + + private fun DataOutputStream.writeEnvelope(envelope: QueryCursorEnvelope) { + writeBoundedUtf8(envelope.target.namedAggregate.contextName) + writeBoundedUtf8(envelope.target.namedAggregate.aggregateName) + writeByte(envelope.target.documentKind.ordinal) + writeBoundedUtf8(envelope.planFingerprint.value) + writeBoundedUtf8(envelope.mappingGenerationDigest.value) + writeBoundedUtf8(envelope.securityContextDigest.value) + writeBoundedUtf8(envelope.backendId.value) + when (val position = envelope.position) { + is QueryCursorPosition.Record -> { + writeByte(RECORD_POSITION) + writeValues(position.sortKey) + } + + is QueryCursorPosition.Analytics -> { + writeByte(ANALYTICS_POSITION) + writeInt(position.dimensionAliases.size) + position.dimensionAliases.forEach { alias -> writeBoundedUtf8(alias.value) } + writeValues(position.afterKey) + } + } + writeLong(envelope.expiresAt.epochSecond) + writeInt(envelope.expiresAt.nano) + writeInt(envelope.pageNumber) + writeBudgetCeiling(envelope.budgetCeiling) + writeBoolean(envelope.backendState != null) + envelope.backendState?.let { state -> + writeBoundedUtf8(state.backendId.value) + writeBytes(state.payload()) + } + } + + private fun DataInputStream.readEnvelope(): QueryCursorEnvelope { + val target = QueryTarget( + MaterializedNamedAggregate(readBoundedUtf8(), readBoundedUtf8()), + QueryDocumentKind.entries.getOrNull(readUnsignedByte()) ?: rejectInvalidCursor(), + ) + val fingerprint = PlanFingerprint(readBoundedUtf8()) + val mappingDigest = QueryCursorMappingDigest(readBoundedUtf8()) + val securityDigest = QueryCursorSecurityContextDigest(readBoundedUtf8()) + val backendId = BackendId(readBoundedUtf8()) + val position = when (readUnsignedByte()) { + RECORD_POSITION -> QueryCursorPosition.Record(readValues()) + ANALYTICS_POSITION -> { + val aliasCount = readCount(MAX_POSITION_VALUES) + val aliases = List(aliasCount) { AnalyticsAlias(readBoundedUtf8()) } + QueryCursorPosition.Analytics(aliases, readValues()) + } + + else -> rejectInvalidCursor() + } + val expiresAt = Instant.ofEpochSecond(readLong(), readInt().toLong()) + val pageNumber = readInt() + val budgetCeiling = readBudgetCeiling() + val backendState = if (readBoolean()) { + QueryCursorBackendState(BackendId(readBoundedUtf8()), readBytes()) + } else { + null + } + return QueryCursorEnvelope( + target, + fingerprint, + mappingDigest, + securityDigest, + position, + expiresAt, + backendState, + pageNumber, + budgetCeiling, + backendId, + ) + } + + private fun DataOutputStream.writeBudgetCeiling(budget: QueryCursorBudgetCeiling) { + writeNullableLong(budget.maxScannedRecords) + writeNullableLong(budget.maxReturnedRecords) + writeNullableLong(budget.maxPageWindow) + writeNullableInt(budget.maxCandidateBuckets) + writeNullableInt(budget.maxReturnedBuckets) + writeNullableInt(budget.maxCursorPages) + writeBoolean(budget.allowDiskUse) + } + + private fun DataInputStream.readBudgetCeiling(): QueryCursorBudgetCeiling = QueryCursorBudgetCeiling( + maxScannedRecords = readNullableLong(), + maxReturnedRecords = readNullableLong(), + maxPageWindow = readNullableLong(), + maxCandidateBuckets = readNullableInt(), + maxReturnedBuckets = readNullableInt(), + maxCursorPages = readNullableInt(), + allowDiskUse = readBoolean(), + ) + + private fun DataOutputStream.writeNullableLong(value: Long?) { + writeBoolean(value != null) + value?.let(::writeLong) + } + + private fun DataInputStream.readNullableLong(): Long? = if (readBoolean()) readLong() else null + + private fun DataOutputStream.writeNullableInt(value: Int?) { + writeBoolean(value != null) + value?.let(::writeInt) + } + + private fun DataInputStream.readNullableInt(): Int? = if (readBoolean()) readInt() else null + + private fun DataOutputStream.writeValues(values: List) { + writeInt(values.size) + values.forEach { value -> + when (value) { + NormalizedValue.Null -> writeByte(NULL_VALUE) + is NormalizedValue.BooleanValue -> { + writeByte(BOOLEAN_VALUE) + writeBoolean(value.value) + } + + is NormalizedValue.Text -> { + writeByte(TEXT_VALUE) + writeBoundedUtf8(value.value) + } + + is NormalizedValue.Int64 -> { + writeByte(INT64_VALUE) + writeLong(value.value) + } + + is NormalizedValue.Decimal -> { + writeByte(DECIMAL_VALUE) + writeBoundedUtf8(value.value.toPlainString()) + } + + is NormalizedValue.InstantValue -> { + writeByte(INSTANT_VALUE) + writeLong(value.value.epochSecond) + writeInt(value.value.nano) + } + + is NormalizedValue.Bytes, + is NormalizedValue.ListValue, + is NormalizedValue.ObjectValue, + -> rejectInvalidCursor() + } + } + } + + private fun DataInputStream.readValues(): List { + val count = readCount(MAX_POSITION_VALUES) + return List(count) { + when (readUnsignedByte()) { + NULL_VALUE -> NormalizedValue.Null + BOOLEAN_VALUE -> NormalizedValue.BooleanValue(readBoolean()) + TEXT_VALUE -> NormalizedValue.Text(readBoundedUtf8()) + INT64_VALUE -> NormalizedValue.Int64(readLong()) + DECIMAL_VALUE -> NormalizedValue.Decimal(BigDecimal(readBoundedUtf8())) + INSTANT_VALUE -> NormalizedValue.InstantValue(Instant.ofEpochSecond(readLong(), readInt().toLong())) + else -> rejectInvalidCursor() + } + } + } + + private fun DataOutputStream.writeBoundedUtf8(value: String) { + val bytes = value.toByteArray(Charsets.UTF_8) + require(bytes.size <= MAX_STRING_BYTES) { "Query cursor text exceeds its encoded size limit." } + writeInt(bytes.size) + write(bytes) + } + + private fun DataInputStream.readBoundedUtf8(): String { + val size = readCount(MAX_STRING_BYTES) + val bytes = ByteArray(size) + readFully(bytes) + return bytes.toString(Charsets.UTF_8) + } + + private fun DataOutputStream.writeBytes(value: ByteArray) { + require(value.size <= maxBackendStateBytes) { "Query cursor backend state exceeds its encoded size limit." } + writeInt(value.size) + write(value) + } + + private fun DataInputStream.readBytes(): ByteArray { + val size = readCount(maxBackendStateBytes) + val value = ByteArray(size) + readFully(value) + return value + } + + private fun DataInputStream.readCount(maximum: Int): Int = readInt().also { value -> + if (value !in 0..maximum) rejectInvalidCursor() + } + + private companion object { + const val MAGIC = 0x57514345 + const val FORMAT_VERSION = 3 + const val HMAC_BYTES = 32 + const val HEADER_BYTES = Int.SIZE_BYTES + Byte.SIZE_BYTES + Byte.SIZE_BYTES + Int.SIZE_BYTES + const val FIXED_BODY_BYTES = 4 * 1024 + const val DEFAULT_BACKEND_STATE_BYTES = 4 * 1024 + const val MAX_STRING_BYTES = 1024 + const val MAX_POSITION_VALUES = 32 + const val RECORD_POSITION = 1 + const val ANALYTICS_POSITION = 2 + const val NULL_VALUE = 0 + const val BOOLEAN_VALUE = 1 + const val TEXT_VALUE = 2 + const val INT64_VALUE = 3 + const val DECIMAL_VALUE = 4 + const val INSTANT_VALUE = 5 + } +} + +private fun invalidCursorException(cause: Throwable? = null): QueryRejectedException = + me.ahoo.wow.query.internal.policy.rejectedException( + QueryRejectionCategory.INVALID_CURSOR, + QueryRejectionPath.ROOT.property("cursor"), + QueryRejectionCode.INVALID_CURSOR_TOKEN, + cause, + ) + +private fun expiredCursorException(): QueryRejectedException = + me.ahoo.wow.query.internal.policy.rejectedException( + QueryRejectionCategory.INVALID_CURSOR, + QueryRejectionPath.ROOT.property("cursor"), + QueryRejectionCode.CURSOR_EXPIRED, + ) + +private fun cursorCapacityException(): QueryRejectedException = + me.ahoo.wow.query.internal.policy.rejectedException( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionPath.ROOT.property("cursor"), + QueryRejectionCode.CURSOR_CAPACITY_EXCEEDED, + ) + +private fun rejectExpiredCursor(): Nothing = throw expiredCursorException() + +private fun rejectCursorCapacity(): Nothing = throw cursorCapacityException() diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorLeaseCoordinator.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorLeaseCoordinator.kt new file mode 100644 index 00000000000..3a549e8bff8 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorLeaseCoordinator.kt @@ -0,0 +1,181 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.cursor + +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono + +internal fun interface QueryCursorBackendLeaseCloser { + fun close(state: QueryCursorBackendState): Mono +} + +internal data class QueryCursorBackendLeaseRegistration( + val backendId: BackendId, + val closer: QueryCursorBackendLeaseCloser, +) + +internal enum class QueryCursorCleanupReason { + TERMINAL, + ABANDONED, +} + +internal data class QueryCursorLeaseDescriptor( + val target: QueryTarget, + val backendId: BackendId, + val mappingGenerationDigest: QueryCursorMappingDigest, +) + +internal fun interface QueryCursorLeaseObserver { + fun onCleanupFailure( + descriptor: QueryCursorLeaseDescriptor, + reason: QueryCursorCleanupReason, + error: Throwable, + ) + + companion object { + val NOOP: QueryCursorLeaseObserver = QueryCursorLeaseObserver { _, _, _ -> } + } +} + +/** Coordinates one-time cursor ownership with backend resource cleanup. */ +internal class QueryCursorLeaseCoordinator( + private val manager: InMemoryQueryCursorLeaseManager, + registrations: Iterable, + private val observer: QueryCursorLeaseObserver = QueryCursorLeaseObserver.NOOP, +) { + private val closers: Map = registrations.toList().let { values -> + require(values.map(QueryCursorBackendLeaseRegistration::backendId).distinct().size == values.size) { + "Duplicate cursor backend lease registration." + } + values.associate { registration -> registration.backendId to registration.closer } + } + + fun issue(envelope: QueryCursorEnvelope): QueryCursorToken { + requireCloser(envelope) + return manager.issue(envelope) + } + + fun acquire(token: QueryCursorToken, expectedBinding: QueryCursorLeaseBinding): AcquiredQueryCursorLease = + AcquiredQueryCursorLease(this, manager.acquire(token, expectedBinding)) + + fun reapExpired(): Mono = Flux.fromIterable(manager.reapExpired()) + .flatMapSequential( + { envelope -> closeBestEffort(envelope, QueryCursorCleanupReason.ABANDONED) }, + CLEANUP_CONCURRENCY, + ).then() + + internal fun transfer( + current: QueryCursorEnvelope, + position: QueryCursorPosition, + expiresAt: java.time.Instant, + backendState: QueryCursorBackendState?, + ): QueryCursorToken { + if (current.backendState?.backendId != backendState?.backendId) { + rejectQuery( + QueryRejectionCategory.INVALID_CURSOR, + CURSOR_PATH, + QueryRejectionCode.INVALID_CURSOR_BINDING, + ) + } + return issue(current.copy(position = position, expiresAt = expiresAt, backendState = backendState)) + } + + internal fun closeTerminal(envelope: QueryCursorEnvelope): Mono = + closeBestEffort(envelope, QueryCursorCleanupReason.TERMINAL) + + private fun requireCloser(envelope: QueryCursorEnvelope) { + val state = envelope.backendState ?: return + if (closers[state.backendId] == null) { + rejectQuery( + QueryRejectionCategory.BACKEND_UNAVAILABLE, + CURSOR_PATH, + QueryRejectionCode.BACKEND_NOT_REGISTERED, + ) + } + } + + private fun closeBestEffort( + envelope: QueryCursorEnvelope, + reason: QueryCursorCleanupReason, + ): Mono { + val state = envelope.backendState ?: return Mono.empty() + val closer = closers[state.backendId] ?: return Mono.fromRunnable { + notifyFailure(envelope, reason, IllegalStateException("Cursor backend closer is not registered.")) + } + return Mono.defer { closer.close(state) } + .onErrorResume { error -> + notifyFailure(envelope, reason, error) + Mono.empty() + } + } + + private fun notifyFailure( + envelope: QueryCursorEnvelope, + reason: QueryCursorCleanupReason, + error: Throwable, + ) { + val state = envelope.backendState ?: return + try { + observer.onCleanupFailure( + QueryCursorLeaseDescriptor(envelope.target, state.backendId, envelope.mappingGenerationDigest), + reason, + error, + ) + } catch (_: RuntimeException) { + // Observability must never replace the query or cleanup outcome. + } + } + + private companion object { + const val CLEANUP_CONCURRENCY = 8 + val CURSOR_PATH: QueryRejectionPath = QueryRejectionPath.ROOT.property("cursor") + } +} + +internal class AcquiredQueryCursorLease internal constructor( + private val coordinator: QueryCursorLeaseCoordinator, + val envelope: QueryCursorEnvelope, +) { + private var settled = false + + @Synchronized + fun transfer( + position: QueryCursorPosition, + expiresAt: java.time.Instant, + backendState: QueryCursorBackendState? = envelope.backendState, + ): QueryCursorToken { + check(!settled) { "Cursor lease has already been settled." } + val token = coordinator.transfer(envelope, position, expiresAt, backendState) + settled = true + return token + } + + fun close(): Mono = Mono.defer { + val shouldClose = synchronized(this) { + if (settled) { + false + } else { + settled = true + true + } + } + if (shouldClose) coordinator.closeTerminal(envelope) else Mono.empty() + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorLeaseManager.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorLeaseManager.kt new file mode 100644 index 00000000000..2b63a8285d8 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorLeaseManager.kt @@ -0,0 +1,348 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class) + +package me.ahoo.wow.query.internal.cursor + +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import java.security.SecureRandom +import java.time.Clock +import java.time.DateTimeException +import java.time.Duration +import java.time.Instant +import java.util.Base64 +import java.util.Collections +import java.util.concurrent.ConcurrentHashMap + +@JvmInline +internal value class QueryCursorToken(val value: String) + +@JvmInline +internal value class QueryCursorMappingDigest(val value: String) { + init { + requireCursorDigest(value, "Cursor mapping generation digest") + } +} + +@JvmInline +internal value class QueryCursorSecurityContextDigest(val value: String) { + init { + requireCursorDigest(value, "Cursor security-context digest") + } +} + +internal class QueryCursorBackendState( + val backendId: BackendId, + payload: ByteArray, +) { + private val frozenPayload = payload.copyOf() + + val size: Int + get() = frozenPayload.size + + fun payload(): ByteArray = frozenPayload.copyOf() + + override fun equals(other: Any?): Boolean = + this === other || + other is QueryCursorBackendState && + backendId == other.backendId && + frozenPayload.contentEquals(other.frozenPayload) + + override fun hashCode(): Int = 31 * backendId.hashCode() + frozenPayload.contentHashCode() +} + +internal sealed interface QueryCursorPosition { + class Record(sortKey: Iterable) : QueryCursorPosition { + val sortKey: List = immutableCursorValues(sortKey, "Record cursor sort key") + + override fun equals(other: Any?): Boolean = + this === other || other is Record && sortKey == other.sortKey + + override fun hashCode(): Int = sortKey.hashCode() + } + + class Analytics( + dimensionAliases: Iterable, + afterKey: Iterable, + ) : QueryCursorPosition { + val dimensionAliases: List = Collections.unmodifiableList(dimensionAliases.toList()) + val afterKey: List = immutableCursorValues(afterKey, "Analytics cursor after-key") + + init { + require(this.dimensionAliases.isNotEmpty()) { "Analytics cursor dimensions must not be empty." } + require(this.dimensionAliases.distinct().size == this.dimensionAliases.size) { + "Analytics cursor dimensions must be unique." + } + require(this.dimensionAliases.size == this.afterKey.size) { + "Analytics cursor dimension and key arity must match." + } + } + + override fun equals(other: Any?): Boolean = + this === other || + other is Analytics && + dimensionAliases == other.dimensionAliases && + afterKey == other.afterKey + + override fun hashCode(): Int = 31 * dimensionAliases.hashCode() + afterKey.hashCode() + } +} + +internal data class QueryCursorEnvelope( + val target: QueryTarget, + val planFingerprint: PlanFingerprint, + val mappingGenerationDigest: QueryCursorMappingDigest, + val securityContextDigest: QueryCursorSecurityContextDigest, + val position: QueryCursorPosition, + val expiresAt: Instant, + val backendState: QueryCursorBackendState? = null, + /** One-based page that this lease resumes. */ + val pageNumber: Int = 2, + /** Initial execution ceiling. Continuations may keep or tighten it, but never remove or relax it. */ + val budgetCeiling: QueryCursorBudgetCeiling = QueryCursorBudgetCeiling(), + val backendId: BackendId = backendState?.backendId ?: BackendId(UNSPECIFIED_BACKEND_ID), +) { + init { + require(pageNumber > 1) { "A continuation cursor must resume page two or later." } + } +} + +internal data class QueryCursorBudgetCeiling( + val maxScannedRecords: Long? = null, + val maxReturnedRecords: Long? = null, + val maxPageWindow: Long? = null, + val maxCandidateBuckets: Int? = null, + val maxReturnedBuckets: Int? = null, + val maxCursorPages: Int? = null, + val allowDiskUse: Boolean = false, +) { + init { + require(maxScannedRecords == null || maxScannedRecords > 0) + require(maxReturnedRecords == null || maxReturnedRecords > 0) + require(maxPageWindow == null || maxPageWindow > 0) + require(maxCandidateBuckets == null || maxCandidateBuckets > 0) + require(maxReturnedBuckets == null || maxReturnedBuckets > 0) + require(maxCursorPages == null || maxCursorPages > 0) + } +} + +internal data class QueryCursorLeaseBinding( + val target: QueryTarget, + val planFingerprint: PlanFingerprint, + val mappingGenerationDigest: QueryCursorMappingDigest, + val securityContextDigest: QueryCursorSecurityContextDigest, + val backendId: BackendId, +) + +internal fun QueryCursorEnvelope.binding(): QueryCursorLeaseBinding = QueryCursorLeaseBinding( + target, + planFingerprint, + mappingGenerationDigest, + securityContextDigest, + backendId, +) + +internal data class QueryCursorLeaseLimits( + val maxEntries: Int = 10_000, + val maxTtl: Duration = Duration.ofMinutes(5), + val maxBackendStateBytes: Int = 4_096, +) { + init { + require(maxEntries > 0) + require(!maxTtl.isZero && !maxTtl.isNegative) + require(maxBackendStateBytes > 0) + require(maxBackendStateBytes <= me.ahoo.wow.query.cursor.QueryCursorLeaseConfiguration.MAX_BACKEND_STATE_BYTES) + } +} + +/** + * A bounded, one-time cursor lease registry. + * + * The token contains only a random lease id, format/signing-key ids and expiry. Semantic cursor keys and backend state + * (for example a PIT id) remain server-side. [acquire] atomically transfers ownership to the caller by removing the + * entry. The caller must either issue the next lease after successful continuation or close the backend state on + * terminal, error or cancellation. [reapExpired] transfers abandoned entries to the caller for best-effort resource + * cleanup. + */ +internal class InMemoryQueryCursorLeaseManager( + signingKeyRing: QueryCursorSigningKeyRing, + private val clock: Clock = Clock.systemUTC(), + private val limits: QueryCursorLeaseLimits = QueryCursorLeaseLimits(), +) { + private val codec = QueryCursorTokenCodec(signingKeyRing) + private val random = SecureRandom() + private val entries = ConcurrentHashMap() + private val issueLock = Any() + + val size: Int + get() = entries.size + + constructor( + secret: ByteArray, + clock: Clock = Clock.systemUTC(), + limits: QueryCursorLeaseLimits = QueryCursorLeaseLimits(), + keyId: Int = CURRENT_SIGNING_KEY_ID, + ) : this(QueryCursorSigningKeyRing(QueryCursorSigningKey(keyId, secret)), clock, limits) + + fun issue(envelope: QueryCursorEnvelope): QueryCursorToken { + validateEnvelope(envelope) + val id = ByteArray(CURSOR_ID_BYTES) + return synchronized(issueLock) { + if (entries.size >= limits.maxEntries) { + rejectQuery( + QueryRejectionCategory.BUDGET_EXCEEDED, + CURSOR_PATH, + QueryRejectionCode.CURSOR_CAPACITY_EXCEEDED, + ) + } + var candidate: String + do { + random.nextBytes(id) + candidate = URL_ENCODER.encodeToString(id) + } while (entries.containsKey(candidate)) + val token = codec.encode(candidate, envelope.expiresAt) + entries[candidate] = LeaseEntry(envelope) + token + } + } + + fun acquire(token: QueryCursorToken, expectedBinding: QueryCursorLeaseBinding): QueryCursorEnvelope { + val claims = codec.decode(token) + val now = clock.instant() + if (!claims.expiresAt.isAfter(now)) { + rejectQuery( + QueryRejectionCategory.INVALID_CURSOR, + CURSOR_PATH, + QueryRejectionCode.CURSOR_EXPIRED, + ) + } + val entry = entries[claims.id] + if (entry == null || entry.envelope.expiresAt != claims.expiresAt) { + rejectInvalidCursor() + } + if (!entry.envelope.expiresAt.isAfter(now)) { + rejectQuery( + QueryRejectionCategory.INVALID_CURSOR, + CURSOR_PATH, + QueryRejectionCode.CURSOR_EXPIRED, + ) + } + if (entry.envelope.binding() != expectedBinding) { + rejectQuery( + QueryRejectionCategory.INVALID_CURSOR, + CURSOR_PATH, + QueryRejectionCode.INVALID_CURSOR_BINDING, + ) + } + if (!entries.remove(claims.id, entry)) { + rejectInvalidCursor() + } + return entry.envelope + } + + fun reapExpired(): List { + val now = clock.instant() + val expired = mutableListOf() + entries.forEach { (id, entry) -> + if (!entry.envelope.expiresAt.isAfter(now) && entries.remove(id, entry)) { + expired += entry.envelope + } + } + return Collections.unmodifiableList(expired) + } + + private fun validateEnvelope(envelope: QueryCursorEnvelope) { + val now = clock.instant() + if (!envelope.expiresAt.isAfter(now)) { + rejectQuery( + QueryRejectionCategory.INVALID_CURSOR, + CURSOR_PATH, + QueryRejectionCode.CURSOR_EXPIRED, + ) + } + val maximumExpiry = try { + now.plus(limits.maxTtl) + } catch (error: DateTimeException) { + throw IllegalStateException("Cursor TTL cannot be represented.", error) + } catch (error: ArithmeticException) { + throw IllegalStateException("Cursor TTL cannot be represented.", error) + } + if (envelope.expiresAt.isAfter(maximumExpiry) || + envelope.backendState?.let { state -> state.size > limits.maxBackendStateBytes } == true + ) { + rejectQuery( + QueryRejectionCategory.BUDGET_EXCEEDED, + CURSOR_PATH, + QueryRejectionCode.CURSOR_CAPACITY_EXCEEDED, + ) + } + } + + private data class LeaseEntry(val envelope: QueryCursorEnvelope) + + private companion object { + const val CURRENT_SIGNING_KEY_ID = 1 + const val CURSOR_ID_BYTES = 32 + val CURSOR_PATH: QueryRejectionPath = QueryRejectionPath.ROOT.property("cursor") + val URL_ENCODER: Base64.Encoder = Base64.getUrlEncoder().withoutPadding() + } +} + +private fun requireCursorDigest(value: String, name: String) { + require(value.length == SHA_256_HEX_LENGTH && value.all { character -> character in HEX_CHARACTERS }) { + "$name must be lowercase SHA-256 hex." + } +} + +private const val SHA_256_HEX_LENGTH = 64 +private const val HEX_CHARACTERS = "0123456789abcdef" +private const val UNSPECIFIED_BACKEND_ID = "unspecified" + +private fun immutableCursorValues(values: Iterable, name: String): List = + Collections.unmodifiableList(values.toList()).also { frozen -> + require(frozen.isNotEmpty()) { "$name must not be empty." } + require(frozen.all(NormalizedValue::isPortableCursorScalar)) { + "$name contains a non-scalar value." + } + } + +private fun NormalizedValue.isPortableCursorScalar(): Boolean = when (this) { + NormalizedValue.Null, + is NormalizedValue.BooleanValue, + is NormalizedValue.Text, + is NormalizedValue.Int64, + is NormalizedValue.Decimal, + is NormalizedValue.InstantValue, + -> true + + is NormalizedValue.Bytes, + is NormalizedValue.ListValue, + is NormalizedValue.ObjectValue, + -> false +} + +internal fun rejectInvalidCursor(cause: Throwable? = null): Nothing = rejectQuery( + QueryRejectionCategory.INVALID_CURSOR, + QueryRejectionPath.ROOT.property("cursor"), + QueryRejectionCode.INVALID_CURSOR_TOKEN, + cause, +) diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorSecurityDigest.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorSecurityDigest.kt new file mode 100644 index 00000000000..7f45594c88e --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorSecurityDigest.kt @@ -0,0 +1,120 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.cursor + +import me.ahoo.wow.query.internal.policy.QueryAuthority +import me.ahoo.wow.query.internal.policy.QueryExecutionContext +import me.ahoo.wow.query.internal.policy.QueryOwnerGrant +import me.ahoo.wow.query.internal.policy.QuerySpaceGrant +import java.io.ByteArrayOutputStream +import java.io.DataOutputStream +import java.security.MessageDigest + +internal object QueryCursorSecurityDigest { + fun compute(context: QueryExecutionContext): QueryCursorSecurityContextDigest { + val bytes = ByteArrayOutputStream().use { buffer -> + DataOutputStream(buffer).use { output -> output.writeContext(context) } + buffer.toByteArray() + } + return QueryCursorSecurityContextDigest(MessageDigest.getInstance("SHA-256").digest(bytes).toHex()) + } + + private fun DataOutputStream.writeContext(context: QueryExecutionContext) { + writeUtf8("wow.query.cursor.security.v1") + writeUtf8(context.purpose.value) + writeUtf8(context.executionMode.name) + writeUtf8(context.validationMode.name) + writeNullableUtf8(context.resourceScope.tenantId) + writeNullableUtf8(context.resourceScope.ownerId) + writeNullableUtf8(context.resourceScope.spaceId) + writeAuthority(context.authority) + } + + private fun DataOutputStream.writeAuthority(authority: QueryAuthority) { + when (authority) { + is QueryAuthority.Subject -> writeSubject(authority) + is QueryAuthority.Service -> writeService(authority) + is QueryAuthority.System -> { + writeByte(SYSTEM_AUTHORITY) + writeUtf8(authority.principalId) + writeUtf8(authority.justification) + } + + is QueryAuthority.Legacy -> { + writeByte(LEGACY_AUTHORITY) + writeUtf8(authority.grant.callerId.value) + writeUtf8(authority.grant.target.namedAggregate.contextName) + writeUtf8(authority.grant.target.namedAggregate.aggregateName) + writeUtf8(authority.grant.target.documentKind.name) + writeUtf8(authority.grant.purpose.value) + writeUtf8(authority.grant.executionMode.name) + writeNullableUtf8(authority.grant.resourceScope.tenantId) + writeNullableUtf8(authority.grant.resourceScope.ownerId) + writeNullableUtf8(authority.grant.resourceScope.spaceId) + } + } + } + + private fun DataOutputStream.writeSubject(authority: QueryAuthority.Subject) { + writeByte(SUBJECT_AUTHORITY) + writeUtf8(authority.subjectId) + writeUtf8(authority.tenantId) + when (val owner = authority.ownerGrant) { + QueryOwnerGrant.Unrestricted -> writeByte(UNRESTRICTED_GRANT) + is QueryOwnerGrant.Only -> { + writeByte(ONLY_GRANT) + writeUtf8(owner.ownerId) + } + } + when (val space = authority.spaceGrant) { + QuerySpaceGrant.Unrestricted -> writeByte(UNRESTRICTED_GRANT) + QuerySpaceGrant.DenyAll -> writeByte(DENY_ALL_GRANT) + is QuerySpaceGrant.AllowList -> { + writeByte(ALLOW_LIST_GRANT) + writeInt(space.spaceIds.size) + space.spaceIds.sorted().forEach { spaceId -> writeUtf8(spaceId) } + } + } + } + + private fun DataOutputStream.writeService(authority: QueryAuthority.Service) { + writeByte(SERVICE_AUTHORITY) + writeUtf8(authority.serviceId) + writeUtf8(authority.tenantId) + writeInt(authority.purposes.size) + authority.purposes.sortedBy { purpose -> purpose.value }.forEach { purpose -> writeUtf8(purpose.value) } + } + + private fun DataOutputStream.writeNullableUtf8(value: String?) { + writeBoolean(value != null) + value?.let { current -> writeUtf8(current) } + } + + private fun DataOutputStream.writeUtf8(value: String) { + val bytes = value.toByteArray(Charsets.UTF_8) + writeInt(bytes.size) + write(bytes) + } + + private fun ByteArray.toHex(): String = joinToString(separator = "") { byte -> "%02x".format(byte) } + + private const val SUBJECT_AUTHORITY = 0 + private const val SERVICE_AUTHORITY = 1 + private const val SYSTEM_AUTHORITY = 2 + private const val LEGACY_AUTHORITY = 3 + private const val UNRESTRICTED_GRANT = 0 + private const val ONLY_GRANT = 1 + private const val DENY_ALL_GRANT = 1 + private const val ALLOW_LIST_GRANT = 2 +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorTokenCodec.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorTokenCodec.kt new file mode 100644 index 00000000000..ac9288f87eb --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorTokenCodec.kt @@ -0,0 +1,180 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.cursor + +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import java.io.ByteArrayInputStream +import java.io.ByteArrayOutputStream +import java.io.DataInputStream +import java.io.DataOutputStream +import java.io.IOException +import java.security.MessageDigest +import java.time.DateTimeException +import java.time.Instant +import java.util.Base64 +import java.util.Collections +import java.util.LinkedHashMap +import javax.crypto.Mac +import javax.crypto.spec.SecretKeySpec + +internal class QueryCursorSigningKey( + val id: Int, + secret: ByteArray, +) { + private val key = secret.copyOf().also { frozen -> + require(frozen.size >= MINIMUM_SECRET_BYTES) { "Cursor HMAC secret must contain at least 256 bits." } + } + + init { + require(id in 1..UByte.MAX_VALUE.toInt()) { "Cursor signing key id must fit one non-zero byte." } + } + + fun sign(payload: ByteArray): ByteArray = Mac.getInstance(HMAC_ALGORITHM).run { + init(SecretKeySpec(key, HMAC_ALGORITHM)) + doFinal(payload) + } + + private companion object { + const val MINIMUM_SECRET_BYTES = 32 + const val HMAC_ALGORITHM = "HmacSHA256" + } +} + +internal class QueryCursorSigningKeyRing( + val current: QueryCursorSigningKey, + previous: Iterable = emptyList(), +) { + private val keys: Map + + init { + val all = listOf(current) + previous.toList() + require(all.size <= MAX_SIGNING_KEYS) { "Cursor signing key ring exceeds its bounded key count." } + require(all.map(QueryCursorSigningKey::id).distinct().size == all.size) { + "Cursor signing key ids must be unique." + } + keys = Collections.unmodifiableMap( + LinkedHashMap(all.size).also { copy -> + all.forEach { key -> copy[key.id] = key } + }, + ) + } + + fun resolve(id: Int): QueryCursorSigningKey? = keys[id] + + private companion object { + const val MAX_SIGNING_KEYS = 4 + } +} + +internal class QueryCursorTokenCodec( + private val keyRing: QueryCursorSigningKeyRing, + private val formatVersion: Int = CURRENT_FORMAT_VERSION, +) { + init { + require(formatVersion in 1..UByte.MAX_VALUE.toInt()) + } + + fun encode(id: String, expiresAt: Instant): QueryCursorToken { + val idBytes = try { + URL_DECODER.decode(id) + } catch (error: IllegalArgumentException) { + rejectInvalidCursor(error) + } + if (idBytes.size != CURSOR_ID_BYTES) rejectInvalidCursor() + val payload = ByteArrayOutputStream(PAYLOAD_BYTES).use { bytes -> + DataOutputStream(bytes).use { output -> + output.writeInt(MAGIC) + output.writeByte(formatVersion) + output.writeByte(keyRing.current.id) + output.write(idBytes) + output.writeLong(expiresAt.epochSecond) + output.writeInt(expiresAt.nano) + } + bytes.toByteArray() + } + val signature = keyRing.current.sign(payload) + return QueryCursorToken("${URL_ENCODER.encodeToString(payload)}.${URL_ENCODER.encodeToString(signature)}") + } + + fun decode(token: QueryCursorToken): TokenClaims = parseClaims(verifiedPayload(token)) + + private fun verifiedPayload(token: QueryCursorToken): ByteArray { + if (token.value.length > MAX_TOKEN_LENGTH) rejectInvalidCursor() + val separator = token.value.indexOf('.') + if (separator <= 0 || separator != token.value.lastIndexOf('.')) rejectInvalidCursor() + val payload = decodePart(token.value.substring(0, separator)) + val signature = decodePart(token.value.substring(separator + 1)) + val signingKey = signingKey(payload) + if (payload.size != PAYLOAD_BYTES || signature.size != HMAC_BYTES || + !MessageDigest.isEqual(signature, signingKey.sign(payload)) + ) { + rejectInvalidCursor() + } + return payload + } + + private fun signingKey(payload: ByteArray): QueryCursorSigningKey { + if (payload.size != PAYLOAD_BYTES) rejectInvalidCursor() + return try { + DataInputStream(ByteArrayInputStream(payload)).use { input -> + if (input.readInt() != MAGIC || input.readUnsignedByte() != formatVersion) rejectInvalidCursor() + keyRing.resolve(input.readUnsignedByte()) ?: rejectInvalidCursor() + } + } catch (error: QueryRejectedException) { + throw error + } catch (error: IOException) { + rejectInvalidCursor(error) + } + } + + private fun parseClaims(payload: ByteArray): TokenClaims = try { + DataInputStream(ByteArrayInputStream(payload)).use { input -> + if (input.readInt() != MAGIC || input.readUnsignedByte() != formatVersion) rejectInvalidCursor() + if (keyRing.resolve(input.readUnsignedByte()) == null) rejectInvalidCursor() + val idBytes = ByteArray(CURSOR_ID_BYTES) + input.readFully(idBytes) + val expiresAt = Instant.ofEpochSecond(input.readLong(), input.readInt().toLong()) + if (input.available() != 0) rejectInvalidCursor() + TokenClaims(URL_ENCODER.encodeToString(idBytes), expiresAt) + } + } catch (error: QueryRejectedException) { + throw error + } catch (error: DateTimeException) { + rejectInvalidCursor(error) + } catch (error: ArithmeticException) { + rejectInvalidCursor(error) + } catch (error: IOException) { + rejectInvalidCursor(error) + } + + private fun decodePart(value: String): ByteArray = try { + URL_DECODER.decode(value) + } catch (error: IllegalArgumentException) { + rejectInvalidCursor(error) + } + + data class TokenClaims(val id: String, val expiresAt: Instant) + + private companion object { + const val MAGIC = 0x57514352 + const val CURRENT_FORMAT_VERSION = 1 + const val CURSOR_ID_BYTES = 32 + const val HMAC_BYTES = 32 + const val PAYLOAD_BYTES = + Int.SIZE_BYTES + Byte.SIZE_BYTES + Byte.SIZE_BYTES + CURSOR_ID_BYTES + Long.SIZE_BYTES + Int.SIZE_BYTES + const val MAX_TOKEN_LENGTH = 256 + val URL_ENCODER: Base64.Encoder = Base64.getUrlEncoder().withoutPadding() + val URL_DECODER: Base64.Decoder = Base64.getUrlDecoder() + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/BoundedQueryShadowSupervisor.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/BoundedQueryShadowSupervisor.kt new file mode 100644 index 00000000000..9d3a56a51dc --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/BoundedQueryShadowSupervisor.kt @@ -0,0 +1,293 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryShadowConfiguration +import me.ahoo.wow.query.gateway.QueryShadowObservation +import me.ahoo.wow.query.gateway.QueryShadowObserver +import me.ahoo.wow.query.gateway.QueryShadowOutcome +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejection +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import reactor.core.Disposable +import reactor.core.publisher.Flux +import java.util.concurrent.atomic.AtomicBoolean +import java.util.concurrent.atomic.AtomicInteger + +/** Bounded runtime owner for cold shadow publishers. Result values never leave this internal comparison boundary. */ +internal class BoundedQueryShadowSupervisor( + private val configuration: QueryShadowConfiguration, + private val observer: QueryShadowObserver, +) : QueryShadowSupervisor { + private val active = AtomicInteger() + + @Suppress("TooGenericExceptionCaught") + override fun submit(task: QueryShadowTask): QueryShadowSubmission { + if (active.incrementAndGet() > configuration.maxConcurrentProbes) { + active.decrementAndGet() + val issue = saturatedIssue() + observe(task.observation(QueryShadowOutcome.SATURATED, issue.code.name)) + return QueryShadowSubmission.Rejected(issue) + } + val comparison = ShadowComparison( + descriptor = task.descriptor(), + maxComparedRecords = configuration.maxComparedRecords, + observer = observer, + release = { active.decrementAndGet() }, + ) + return try { + comparison.subscribe(task) + QueryShadowSubmission.Accepted(comparison) + } catch (error: RuntimeException) { + comparison.failSubscription(error) + QueryShadowSubmission.Rejected(supervisorUnavailableIssue()) + } + } + + override fun onSkipped(skip: QueryShadowSkip) { + val issue = skip.issues.values.first() + observe( + QueryShadowObservation( + skip.target.toPublic(), + skip.operation, + null, + QueryShadowOutcome.SKIPPED, + issue.code.name, + ), + ) + } + + private fun observe(observation: QueryShadowObservation) { + try { + observer.onObservation(observation) + } catch (_: RuntimeException) { + // Shadow observability cannot alter the primary result. + } + } +} + +private class ShadowComparison( + private val descriptor: QueryShadowDescriptor, + private val maxComparedRecords: Int, + private val observer: QueryShadowObserver, + private val release: () -> Unit, +) : QueryShadowHandle { + private val finished = AtomicBoolean() + private val primaryValues = mutableListOf() + private val probeValues = mutableListOf() + private var primaryTerminal: PrimaryTerminal? = null + private var probeTerminal: ProbeTerminal? = null + private var primaryLimitExceeded = false + private var subscription: Disposable? = null + + fun subscribe(task: QueryShadowTask) { + subscription = Flux.from(task.publisher) + .take(maxComparedRecords.toLong() + 1) + .subscribe(::onProbeValue, ::onProbeError, ::onProbeComplete) + } + + fun failSubscription(error: RuntimeException) { + synchronized(this) { + probeTerminal = ProbeTerminal.Error(error) + primaryTerminal = PrimaryTerminal.Cancelled + } + finishIfReady() + } + + override fun onPrimary(signal: QueryShadowPrimarySignal) { + var cancelProbe = false + synchronized(this) { + when (signal) { + is QueryShadowPrimarySignal.RecordValue -> cancelProbe = addPrimary(signal.value.comparable()) + is QueryShadowPrimarySignal.PageValue -> cancelProbe = addPrimary(signal.value.comparable()) + is QueryShadowPrimarySignal.CountValue -> cancelProbe = addPrimary(signal.value) + QueryShadowPrimarySignal.Complete -> primaryTerminal = PrimaryTerminal.Complete + is QueryShadowPrimarySignal.Error -> primaryTerminal = PrimaryTerminal.Error(signal.error) + QueryShadowPrimarySignal.Cancelled -> primaryTerminal = PrimaryTerminal.Cancelled + } + } + if (cancelProbe) subscription?.dispose() + finishIfReady() + } + + private fun addPrimary(value: Any): Boolean { + if (primaryValues.size < maxComparedRecords) { + primaryValues += value + return false + } + primaryLimitExceeded = true + if (probeTerminal == null) { + probeTerminal = ProbeTerminal.LimitExceeded + return true + } + return false + } + + override fun cancelProbe() { + subscription?.dispose() + synchronized(this) { + if (probeTerminal == null) { + probeTerminal = ProbeTerminal.Cancelled + } + } + finishIfReady() + } + + private fun onProbeValue(value: Any) { + synchronized(this) { + if (probeValues.size >= maxComparedRecords) { + probeTerminal = ProbeTerminal.LimitExceeded + } else { + probeValues += value.comparable() + } + } + } + + private fun onProbeError(error: Throwable) { + synchronized(this) { + if (probeTerminal == null) { + probeTerminal = ProbeTerminal.Error(error) + } + } + finishIfReady() + } + + private fun onProbeComplete() { + synchronized(this) { + if (probeTerminal == null) { + probeTerminal = ProbeTerminal.Complete + } + } + finishIfReady() + } + + private fun finishIfReady() { + val observation = synchronized(this) { + val primary = primaryTerminal ?: return + val probe = probeTerminal ?: return + if (!finished.compareAndSet(false, true)) return + observation(primary, probe) + } + release() + try { + observer.onObservation(observation) + } catch (_: RuntimeException) { + // Shadow observability cannot alter the primary result. + } + } + + private fun observation(primary: PrimaryTerminal, probe: ProbeTerminal): QueryShadowObservation { + val (outcome, reason) = when { + primary is PrimaryTerminal.Error -> QueryShadowOutcome.PRIMARY_ERROR to primary.error.rejection.code.name + primary == PrimaryTerminal.Cancelled -> QueryShadowOutcome.CANCELLED to null + probe is ProbeTerminal.Error -> QueryShadowOutcome.PROBE_ERROR to probe.error.reasonCode() + probe == ProbeTerminal.Cancelled -> QueryShadowOutcome.CANCELLED to null + primaryLimitExceeded || probe == ProbeTerminal.LimitExceeded -> + QueryShadowOutcome.PROBE_ERROR to QueryRejectionCode.RESULT_LIMIT_EXCEEDED.name + + primaryValues == probeValues -> QueryShadowOutcome.MATCH to null + else -> QueryShadowOutcome.VALUE_MISMATCH to null + } + return descriptor.observation(outcome, reason) + } +} + +private sealed interface PrimaryTerminal { + data object Complete : PrimaryTerminal + + data class Error(val error: QueryRejectedException) : PrimaryTerminal + + data object Cancelled : PrimaryTerminal +} + +private sealed interface ProbeTerminal { + data object Complete : ProbeTerminal + + data class Error(val error: Throwable) : ProbeTerminal + + data object LimitExceeded : ProbeTerminal + + data object Cancelled : ProbeTerminal +} + +private data class ComparableRecord( + val identity: String, + val document: me.ahoo.wow.query.backend.NormalizedValue.ObjectValue, +) + +private data class ComparablePage( + val records: List, + val total: Long, +) + +private fun Any.comparable(): Any = + when (this) { + is BackendRecord -> comparable() + is BackendPage -> comparable() + else -> this + } + +private fun BackendRecord.comparable(): ComparableRecord = ComparableRecord(identity, document) + +private fun BackendPage.comparable(): ComparablePage = ComparablePage(records.map(BackendRecord::comparable), total) + +private fun Throwable.reasonCode(): String = + (this as? QueryRejectedException)?.rejection?.code?.name ?: QueryRejectionCode.UNEXPECTED_QUERY_FAILURE.name + +private fun QueryShadowTask.descriptor(): QueryShadowDescriptor = QueryShadowDescriptor( + target, + fingerprint, + semanticTier, + operation, +) + +private fun QueryShadowTask.observation(outcome: QueryShadowOutcome, reasonCode: String?): QueryShadowObservation = + descriptor().observation(outcome, reasonCode) + +private fun QueryShadowDescriptor.observation( + outcome: QueryShadowOutcome, + reasonCode: String?, +): QueryShadowObservation = QueryShadowObservation( + target.toPublic(), + operation, + fingerprint.value, + outcome, + reasonCode, +) + +private fun me.ahoo.wow.query.internal.model.QueryTarget.toPublic(): QueryTarget = QueryTarget( + namedAggregate, + when (documentKind) { + me.ahoo.wow.query.internal.model.QueryDocumentKind.SNAPSHOT -> QueryDocumentKind.SNAPSHOT + me.ahoo.wow.query.internal.model.QueryDocumentKind.EVENT_STREAM -> QueryDocumentKind.EVENT_STREAM + }, +) + +private fun saturatedIssue(): QueryRejection = QueryRejection( + QueryRejectionCategory.BACKEND_UNAVAILABLE, + QueryRejectionPath.ROOT.property("shadow").property("supervisor"), + QueryRejectionCode.SHADOW_SUPERVISOR_SATURATED, +) + +private fun supervisorUnavailableIssue(): QueryRejection = QueryRejection( + QueryRejectionCategory.BACKEND_UNAVAILABLE, + QueryRejectionPath.ROOT.property("shadow").property("supervisor"), + QueryRejectionCode.SHADOW_SUPERVISOR_UNAVAILABLE, +) diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/ExperimentalAnalyticsBackendAdapter.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/ExperimentalAnalyticsBackendAdapter.kt new file mode 100644 index 00000000000..086221ed8e9 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/ExperimentalAnalyticsBackendAdapter.kt @@ -0,0 +1,237 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.BackendAnalyticsBucketOrder +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsCondition +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsCursorState +import me.ahoo.wow.query.backend.BackendAnalyticsDimension +import me.ahoo.wow.query.backend.BackendAnalyticsGrouping +import me.ahoo.wow.query.backend.BackendAnalyticsMetric +import me.ahoo.wow.query.backend.BackendAnalyticsMissingPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNullPlacement +import me.ahoo.wow.query.backend.BackendAnalyticsNumericPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsNumericPromotion +import me.ahoo.wow.query.backend.BackendAnalyticsOverflowPolicy +import me.ahoo.wow.query.backend.BackendAnalyticsPageWindow +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.BackendAnalyticsTextCollation +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.accepts +import me.ahoo.wow.query.internal.analytics.AnalyticsCompleteness +import me.ahoo.wow.query.internal.analytics.AnalyticsConsistency +import me.ahoo.wow.query.internal.plan.AnalyticsQueryPlan +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsBucketOrder +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsCondition +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsGrouping +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsMetric +import reactor.core.publisher.Mono +import me.ahoo.wow.query.backend.AnalyticsQueryBackend as ExperimentalAnalyticsQueryBackend +import me.ahoo.wow.query.backend.BackendAnalyticsPage as ExperimentalAnalyticsPage +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias as InternalAnalyticsAlias + +internal class ExperimentalAnalyticsBackendAdapter( + private val delegate: ExperimentalAnalyticsQueryBackend, + private val schema: QueryDocumentSchema, +) : AnalyticsQueryBackend { + override fun analyze(plan: AnalyticsQueryPlan, options: QueryExecutionOptions): Mono = + delegate.analyze(plan.toBackendPlan(), options.toBackendOptions()).map { page -> page.toInternal(plan, schema) } + + override fun analyze( + plan: AnalyticsQueryPlan, + options: QueryExecutionOptions, + cursorState: ByteArray?, + ): Mono = delegate.analyze( + plan.toBackendPlan(), + options.toBackendOptions(), + cursorState?.let(::BackendAnalyticsCursorState), + ).map { page -> page.toInternal(plan, schema) } +} + +private fun AnalyticsQueryPlan.toBackendPlan(): BackendAnalyticsQueryPlan = BackendAnalyticsQueryPlan( + target, + schemaContractId, + filter.toBackendFilter(), + grouping.toBackendGrouping(), + metrics.values.map(PlannedAnalyticsMetric::toBackendMetric), + when (having) { + PlannedAnalyticsCondition.All -> BackendAnalyticsCondition.All + }, + bucketOrder.toBackendOrder(), + BackendAnalyticsPageWindow(bucketWindow.limit, bucketWindow.afterKey?.values), + numericPolicy?.let { policy -> + BackendAnalyticsNumericPolicy( + BackendAnalyticsNumericPromotion.valueOf(policy.promotion.name), + policy.precision, + policy.scale, + policy.roundingMode, + BackendAnalyticsOverflowPolicy.valueOf(policy.overflowPolicy.name), + ) + }, + BackendAnalyticsConsistency.valueOf(requiredConsistency.name), + BackendAnalyticsCompleteness.valueOf(requiredCompleteness.name), + requiredCapabilities.toBackendCapabilities(), + semanticTier.toBackendTier(), + PlanFingerprint(fingerprint.value), +) + +private fun PlannedAnalyticsGrouping.toBackendGrouping(): BackendAnalyticsGrouping = + when (this) { + PlannedAnalyticsGrouping.Global -> BackendAnalyticsGrouping.Global + is PlannedAnalyticsGrouping.By -> BackendAnalyticsGrouping.By( + dimensions.values.map { dimension -> + BackendAnalyticsDimension( + dimension.alias.toBackendAlias(), + dimension.field, + BackendAnalyticsMissingPolicy.valueOf(dimension.missingPolicy.name), + ) + }, + ) + } + +private fun PlannedAnalyticsMetric.toBackendMetric(): BackendAnalyticsMetric = + when (this) { + is PlannedAnalyticsMetric.DocumentCount -> BackendAnalyticsMetric.DocumentCount(alias.toBackendAlias()) + is PlannedAnalyticsMetric.Min -> BackendAnalyticsMetric.Min(alias.toBackendAlias(), field) + is PlannedAnalyticsMetric.Max -> BackendAnalyticsMetric.Max(alias.toBackendAlias(), field) + is PlannedAnalyticsMetric.Sum -> BackendAnalyticsMetric.Sum(alias.toBackendAlias(), field) + is PlannedAnalyticsMetric.Average -> BackendAnalyticsMetric.Average(alias.toBackendAlias(), field) + } + +private fun PlannedAnalyticsBucketOrder.toBackendOrder(): BackendAnalyticsBucketOrder = + when (this) { + PlannedAnalyticsBucketOrder.Global -> BackendAnalyticsBucketOrder.Global + is PlannedAnalyticsBucketOrder.DimensionKeyAscending -> BackendAnalyticsBucketOrder.DimensionKeyAscending( + BackendAnalyticsNullPlacement.valueOf(nullPlacement.name), + BackendAnalyticsTextCollation.valueOf(textCollation.name), + ) + } + +private fun ExperimentalAnalyticsPage.toInternal( + plan: AnalyticsQueryPlan, + schema: QueryDocumentSchema, +): BackendAnalyticsPage { + validateResult(plan, schema) + return BackendAnalyticsPage( + buckets.map { bucket -> + BackendAnalyticsBucket( + bucket.keys.mapKeys { entry -> entry.key.toInternalAlias() }, + bucket.metrics.mapKeys { entry -> entry.key.toInternalAlias() }, + ) + }, + afterKey, + when (consistency) { + BackendAnalyticsConsistency.EVENTUAL -> AnalyticsConsistency.EVENTUAL + BackendAnalyticsConsistency.SNAPSHOT -> AnalyticsConsistency.SNAPSHOT + }, + when (completeness) { + BackendAnalyticsCompleteness.EXACT -> AnalyticsCompleteness.EXACT + BackendAnalyticsCompleteness.APPROXIMATE -> AnalyticsCompleteness.APPROXIMATE + }, + cursorState?.payload(), + ) +} + +private fun ExperimentalAnalyticsPage.validateResult(plan: AnalyticsQueryPlan, schema: QueryDocumentSchema) { + val dimensions = when (val grouping = plan.grouping) { + PlannedAnalyticsGrouping.Global -> emptyList() + is PlannedAnalyticsGrouping.By -> grouping.dimensions.values + } + val dimensionsByAlias = dimensions.associateBy { dimension -> dimension.alias.value } + val metricsByAlias = plan.metrics.values.associateBy { metric -> metric.alias.value } + buckets.forEach { bucket -> + validateBucket(schema, bucket, dimensionsByAlias, metricsByAlias) + } + afterKey?.forEachIndexed { index, value -> + requireDimensionValue(schema, dimensions.getOrNull(index) ?: mappingFailure(), value) + } +} + +private fun validateBucket( + schema: QueryDocumentSchema, + bucket: me.ahoo.wow.query.backend.BackendAnalyticsBucket, + dimensionsByAlias: Map, + metricsByAlias: Map, +) { + bucket.keys.forEach { (alias, value) -> + requireDimensionValue(schema, dimensionsByAlias[alias.value] ?: mappingFailure(), value) + } + bucket.metrics.forEach { (alias, value) -> + requireMetricValue(schema, metricsByAlias[alias.value] ?: mappingFailure(), value) + } +} + +private fun requireMetricValue( + schema: QueryDocumentSchema, + metric: PlannedAnalyticsMetric, + value: NormalizedValue, +) { + when (metric) { + is PlannedAnalyticsMetric.DocumentCount -> + if (value !is NormalizedValue.Int64 || value.value < 0) mappingFailure() + + is PlannedAnalyticsMetric.Min -> requireFieldValue(schema, metric.field, value, allowNull = true) + is PlannedAnalyticsMetric.Max -> requireFieldValue(schema, metric.field, value, allowNull = true) + is PlannedAnalyticsMetric.Sum, + is PlannedAnalyticsMetric.Average, + -> requireNumericMetricValue(value) + } +} + +private fun requireNumericMetricValue(value: NormalizedValue) { + if (value != NormalizedValue.Null && + value !is NormalizedValue.Int64 && + value !is NormalizedValue.Decimal + ) { + mappingFailure() + } +} + +private fun requireDimensionValue( + schema: QueryDocumentSchema, + dimension: me.ahoo.wow.query.internal.plan.PlannedAnalyticsDimension, + value: NormalizedValue, +) { + if (value == NormalizedValue.Null) { + if (dimension.missingPolicy.name != "AS_NULL_BUCKET") mappingFailure() + return + } + requireFieldValue(schema, dimension.field, value, allowNull = false) +} + +private fun requireFieldValue( + schema: QueryDocumentSchema, + fieldId: me.ahoo.wow.query.backend.QueryFieldId, + value: NormalizedValue, + allowNull: Boolean, +) { + if (allowNull && value == NormalizedValue.Null) return + val field = schema.fields[fieldId] ?: mappingFailure() + if (!field.type.accepts(value)) mappingFailure() +} + +private fun mappingFailure(): Nothing = throw QueryBackendException(QueryBackendFailureKind.MAPPING_FAILURE) + +private fun InternalAnalyticsAlias.toBackendAlias(): AnalyticsAlias = AnalyticsAlias(value) + +private fun AnalyticsAlias.toInternalAlias(): InternalAnalyticsAlias = InternalAnalyticsAlias(value) diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/ExperimentalRecordBackendAdapter.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/ExperimentalRecordBackendAdapter.kt new file mode 100644 index 00000000000..47299109517 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/ExperimentalRecordBackendAdapter.kt @@ -0,0 +1,220 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendEnforcedFilter +import me.ahoo.wow.query.backend.BackendPageConsistency +import me.ahoo.wow.query.backend.BackendPageQueryPlan +import me.ahoo.wow.query.backend.BackendPageWindow +import me.ahoo.wow.query.backend.BackendPlannedCondition +import me.ahoo.wow.query.backend.BackendProjection +import me.ahoo.wow.query.backend.BackendRequiredCapabilities +import me.ahoo.wow.query.backend.BackendRequiredConsistency +import me.ahoo.wow.query.backend.BackendSingleQueryPlan +import me.ahoo.wow.query.backend.BackendSort +import me.ahoo.wow.query.backend.BackendSortOrigin +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.BackendTotalMode +import me.ahoo.wow.query.backend.BackendTotalRelation +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.internal.plan.CountQueryPlan +import me.ahoo.wow.query.internal.plan.EnforcedFilter +import me.ahoo.wow.query.internal.plan.PlannedCondition +import me.ahoo.wow.query.internal.plan.PlannedProjection +import me.ahoo.wow.query.internal.plan.PlannedSort +import me.ahoo.wow.query.internal.plan.PlannedSortOrigin +import me.ahoo.wow.query.internal.plan.QueryPlan +import me.ahoo.wow.query.internal.plan.RequiredCapabilities +import me.ahoo.wow.query.internal.plan.RequiredConsistency +import me.ahoo.wow.query.internal.plan.SingleQueryPlan +import me.ahoo.wow.query.internal.plan.StreamLimit +import me.ahoo.wow.query.internal.plan.StreamQueryPlan +import me.ahoo.wow.query.internal.plan.TotalMode +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import me.ahoo.wow.query.backend.BackendPage as ExperimentalBackendPage +import me.ahoo.wow.query.backend.BackendRecord as ExperimentalBackendRecord +import me.ahoo.wow.query.backend.BackendRecordCompleteness as ExperimentalRecordCompleteness +import me.ahoo.wow.query.backend.RecordQueryBackend as ExperimentalRecordQueryBackend + +internal class ExperimentalRecordBackendAdapter( + private val delegate: ExperimentalRecordQueryBackend, +) : RecordQueryBackend { + override fun single(plan: SingleQueryPlan, options: QueryExecutionOptions): Mono = + delegate.single(plan.toBackendPlan(), options.toBackendOptions()).map(ExperimentalBackendRecord::toInternal) + + override fun stream(plan: StreamQueryPlan, options: QueryExecutionOptions): Flux { + val backendPlan = plan.toBackendPlan() + return delegate.stream(backendPlan, options.toBackendOptions()).map(ExperimentalBackendRecord::toInternal) + } + + override fun page( + plan: me.ahoo.wow.query.internal.plan.PageQueryPlan, + options: QueryExecutionOptions, + ): Mono = delegate.page(plan.toBackendPlan(), options.toBackendOptions()) + .map(ExperimentalBackendPage::toInternal) + + override fun count(plan: CountQueryPlan, options: QueryExecutionOptions): Mono = + delegate.count(plan.toBackendPlan(), options.toBackendOptions()) +} + +private fun SingleQueryPlan.toBackendPlan(): BackendSingleQueryPlan = BackendSingleQueryPlan( + target, + schemaContractId, + filter.toBackendFilter(), + resultShape, + projection.toBackendProjection(), + sort.map(PlannedSort::toBackendSort), + requiredCapabilities.toBackendCapabilities(), + semanticTier.toBackendTier(), + PlanFingerprint(fingerprint.value), +) + +private fun StreamQueryPlan.toBackendPlan(): BackendStreamQueryPlan = BackendStreamQueryPlan( + target, + schemaContractId, + filter.toBackendFilter(), + resultShape, + projection.toBackendProjection(), + sort.map(PlannedSort::toBackendSort), + when (val streamLimit = limit) { + is StreamLimit.Bounded -> streamLimit.value + StreamLimit.Unbounded -> throw QueryBackendException(QueryBackendFailureKind.UNSUPPORTED) + }, + requiredCapabilities.toBackendCapabilities(), + semanticTier.toBackendTier(), + PlanFingerprint(fingerprint.value), +) + +private fun CountQueryPlan.toBackendPlan(): BackendCountQueryPlan = BackendCountQueryPlan( + target, + schemaContractId, + filter.toBackendFilter(), + requiredCapabilities.toBackendCapabilities(), + semanticTier.toBackendTier(), + PlanFingerprint(fingerprint.value), +) + +private fun me.ahoo.wow.query.internal.plan.PageQueryPlan.toBackendPlan(): BackendPageQueryPlan = + BackendPageQueryPlan( + target, + schemaContractId, + filter.toBackendFilter(), + resultShape, + projection.toBackendProjection(), + sort.map(PlannedSort::toBackendSort), + BackendPageWindow(page.offset, page.size), + when (totalMode) { + TotalMode.EXACT -> BackendTotalMode.EXACT + }, + when (requiredConsistency) { + RequiredConsistency.SAME_INPUT -> BackendRequiredConsistency.SAME_INPUT + }, + requiredCapabilities.toBackendCapabilities(), + semanticTier.toBackendTier(), + PlanFingerprint(fingerprint.value), + ) + +internal fun EnforcedFilter.toBackendFilter(): BackendEnforcedFilter = BackendEnforcedFilter( + user.toBackendCondition(), + mandatory.toBackendCondition(), +) + +internal fun PlannedCondition.toBackendCondition(): BackendPlannedCondition = + when (this) { + PlannedCondition.All -> BackendPlannedCondition.All + PlannedCondition.None -> BackendPlannedCondition.None + is PlannedCondition.Junction -> BackendPlannedCondition.Junction( + operator, + children.values.map(PlannedCondition::toBackendCondition), + ) + + is PlannedCondition.Predicate -> BackendPlannedCondition.Predicate(field, operator, value, options) + is PlannedCondition.ElementMatch -> BackendPlannedCondition.ElementMatch(field, condition.toBackendCondition()) + is PlannedCondition.Search -> BackendPlannedCondition.Search(scope, text) + is PlannedCondition.Native -> BackendPlannedCondition.Native(backendId, payload) + } + +private fun PlannedProjection.toBackendProjection(): BackendProjection = + when (this) { + PlannedProjection.All -> BackendProjection.All + is PlannedProjection.Include -> BackendProjection.Include(fields.values) + is PlannedProjection.Exclude -> BackendProjection.Exclude(fields.values) + } + +private fun PlannedSort.toBackendSort(): BackendSort = BackendSort( + field, + direction, + when (origin) { + PlannedSortOrigin.USER -> BackendSortOrigin.USER + PlannedSortOrigin.STABILITY_TIE_BREAKER -> BackendSortOrigin.STABILITY_TIE_BREAKER + }, +) + +internal fun RequiredCapabilities.toBackendCapabilities(): BackendRequiredCapabilities = BackendRequiredCapabilities( + fieldRequirements, + searchRequirements, + nativeBackend, +) + +internal fun me.ahoo.wow.query.internal.plan.SemanticTier.toBackendTier(): me.ahoo.wow.query.backend.SemanticTier = + me.ahoo.wow.query.backend.SemanticTier.valueOf(name) + +internal fun QueryExecutionOptions.toBackendOptions(): QueryBackendExecutionOptions = + QueryBackendExecutionOptions( + deadline = deadline, + maxReturnedRecords = budget.maxReturnedRecords, + maxScannedRecords = budget.maxScannedRecords, + maxPageWindow = budget.maxPageWindow, + maxCandidateBuckets = budget.maxCandidateBuckets, + maxReturnedBuckets = budget.maxReturnedBuckets, + maxCursorPages = budget.maxCursorPages, + allowDiskUse = budget.allowDiskUse, + ) + +private fun ExperimentalBackendRecord.toInternal(): BackendRecord = BackendRecord( + identity, + document, + when (completeness) { + ExperimentalRecordCompleteness.COMPLETE -> BackendRecordCompleteness.COMPLETE + ExperimentalRecordCompleteness.UNKNOWN -> BackendRecordCompleteness.UNKNOWN + }, +) + +private fun ExperimentalBackendPage.toInternal(): BackendPage = BackendPage( + records.map(ExperimentalBackendRecord::toInternal), + total, + when (totalRelation) { + BackendTotalRelation.EXACT -> me.ahoo.wow.query.internal.execution.BackendTotalRelation.EXACT + BackendTotalRelation.LOWER_BOUND -> me.ahoo.wow.query.internal.execution.BackendTotalRelation.LOWER_BOUND + BackendTotalRelation.UNKNOWN -> me.ahoo.wow.query.internal.execution.BackendTotalRelation.UNKNOWN + }, + when (consistency) { + BackendPageConsistency.SAME_INPUT -> me.ahoo.wow.query.internal.execution.BackendPageConsistency.SAME_INPUT + BackendPageConsistency.INDEPENDENT -> me.ahoo.wow.query.internal.execution.BackendPageConsistency.INDEPENDENT + BackendPageConsistency.UNKNOWN -> me.ahoo.wow.query.internal.execution.BackendPageConsistency.UNKNOWN + }, +) + +internal fun QueryPlan.isExperimentalRecordPlan(): Boolean = + this is SingleQueryPlan || + this is StreamQueryPlan && limit is StreamLimit.Bounded || + this is me.ahoo.wow.query.internal.plan.PageQueryPlan || + this is CountQueryPlan diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/LegacyQueryExecution.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/LegacyQueryExecution.kt new file mode 100644 index 00000000000..c8511bb6f72 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/LegacyQueryExecution.kt @@ -0,0 +1,321 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.normalization.NormalizedDeletionScope +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.plan.AnalyticsQueryPlan +import me.ahoo.wow.query.internal.plan.CountQueryPlan +import me.ahoo.wow.query.internal.plan.PlannedCondition +import me.ahoo.wow.query.internal.plan.QueryPlan +import me.ahoo.wow.query.internal.plan.RecordQueryPlan +import me.ahoo.wow.query.internal.planning.PlanningDecision +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.util.Collections +import java.util.LinkedHashMap + +/** + * Immutable input to a target-specific legacy compiler. It never contains the original wire DTO. + */ +internal data class LegacyCompilationInput( + val invocation: NormalizedQueryInvocation, + val schema: QueryDocumentSchema, + val decision: PlanningDecision, +) { + private val sourceToken = LegacyCompilationToken() + + val enforcementRequirements: LegacyEnforcementRequirements = LegacyEnforcementRequirements( + deletionScope = invocation.deletionScope(), + mandatoryCondition = decision.mandatoryCondition(), + ) + + init { + require(invocation.target == schema.target) { + "Legacy compilation target must match the schema target." + } + require(invocation.hasConsistentShape()) { + "Legacy compilation invocation has an inconsistent operation or result shape." + } + when (decision) { + is PlanningDecision.Planned -> require( + decision.plan.target == invocation.target && + decision.plan.schemaContractId == schema.contractId && + decision.plan.operation == invocation.operation && + decision.plan.matchesResultShape(invocation.resultShape), + ) { + "Planned legacy compilation proof must match invocation, schema, operation and result shape." + } + + is PlanningDecision.LegacyFallback -> require( + decision.validatedMandatory.target == invocation.target && + decision.validatedMandatory.schemaContractId == schema.contractId, + ) { + "Legacy fallback mandatory proof must match invocation and schema." + } + } + } + + /** + * Creates a trusted compiler attestation that both the framework deletion rule and the validated mandatory + * condition were lowered. It is not an independent inspection of a physical backend query. The per-input token + * prevents a compiled query from being cached and replayed for another request. + */ + fun attestLowering( + deletionScope: NormalizedDeletionScope, + mandatoryCondition: PlannedCondition, + ): LegacyLoweringAttestation { + val attested = LegacyEnforcementRequirements(deletionScope, mandatoryCondition) + if (attested != enforcementRequirements) { + rejectLegacyMandatory() + } + return LegacyLoweringAttestation(sourceToken, attested) + } + + internal fun accepts(attestation: LegacyLoweringAttestation): Boolean = + attestation.sourceToken === sourceToken && attestation.requirements == enforcementRequirements +} + +internal data class LegacyEnforcementRequirements( + val deletionScope: NormalizedDeletionScope, + val mandatoryCondition: PlannedCondition, +) + +internal class LegacyLoweringAttestation internal constructor( + internal val sourceToken: LegacyCompilationToken, + internal val requirements: LegacyEnforcementRequirements, +) + +internal class LegacyCompilationToken + +internal interface LegacyCompiledQuery { + val target: QueryTarget + val operation: QueryOperation + val schemaContractId: SchemaContractId + val loweringAttestation: LegacyLoweringAttestation +} + +internal fun interface LegacyQueryCompiler { + fun compile(input: LegacyCompilationInput): C +} + +internal interface LegacyQueryBackend { + fun single(query: C, options: QueryExecutionOptions): Mono + + fun stream(query: C, options: QueryExecutionOptions): Flux + + fun page(query: C, options: QueryExecutionOptions): Mono + + fun count(query: C, options: QueryExecutionOptions): Mono +} + +/** + * Erased final binding constructed only from the typed compiler/backend pair below. Registry callers cannot provide an + * alternate implementation that bypasses per-input lowering attestation validation. + */ +internal class LegacyExecutionBinding private constructor( + val target: QueryTarget, + private val delegate: LegacyExecutionDelegate, +) { + fun single(input: LegacyCompilationInput, options: QueryExecutionOptions): Mono = + delegate.single(input, options) + + fun stream(input: LegacyCompilationInput, options: QueryExecutionOptions): Flux = + delegate.stream(input, options) + + fun page(input: LegacyCompilationInput, options: QueryExecutionOptions): Mono = + delegate.page(input, options) + + fun count(input: LegacyCompilationInput, options: QueryExecutionOptions): Mono = + delegate.count(input, options) + + companion object { + fun create( + target: QueryTarget, + compiler: LegacyQueryCompiler, + backend: LegacyQueryBackend, + errorBoundary: QueryErrorBoundary = QueryErrorBoundary(), + ): LegacyExecutionBinding = LegacyExecutionBinding( + target, + DefaultLegacyExecutionDelegate(target, compiler, backend, errorBoundary), + ) + } +} + +private interface LegacyExecutionDelegate { + fun single(input: LegacyCompilationInput, options: QueryExecutionOptions): Mono + + fun stream(input: LegacyCompilationInput, options: QueryExecutionOptions): Flux + + fun page(input: LegacyCompilationInput, options: QueryExecutionOptions): Mono + + fun count(input: LegacyCompilationInput, options: QueryExecutionOptions): Mono +} + +private class DefaultLegacyExecutionDelegate( + private val target: QueryTarget, + private val compiler: LegacyQueryCompiler, + private val backend: LegacyQueryBackend, + private val errorBoundary: QueryErrorBoundary = QueryErrorBoundary(), +) : LegacyExecutionDelegate { + override fun single(input: LegacyCompilationInput, options: QueryExecutionOptions): Mono = + Mono.defer { + requireSupportedBudget(options) + val compiled = compile(input, QueryOperation.SINGLE) + backendMono { backend.single(compiled, options) } + } + + override fun stream(input: LegacyCompilationInput, options: QueryExecutionOptions): Flux = + Flux.defer { + requireSupportedBudget(options) + val compiled = compile(input, QueryOperation.STREAM) + backendFlux { backend.stream(compiled, options) } + } + + override fun page(input: LegacyCompilationInput, options: QueryExecutionOptions): Mono = + Mono.defer { + requireSupportedBudget(options) + val compiled = compile(input, QueryOperation.PAGE) + backendMono { backend.page(compiled, options) } + } + + override fun count(input: LegacyCompilationInput, options: QueryExecutionOptions): Mono = + Mono.defer { + requireSupportedBudget(options) + val compiled = compile(input, QueryOperation.COUNT) + backendMono { backend.count(compiled, options) } + } + + private fun compile(input: LegacyCompilationInput, expectedOperation: QueryOperation): C { + if (input.invocation.target != target || input.invocation.operation != expectedOperation) { + rejectLegacy(QueryRejectionCode.LEGACY_LOWERING_UNSUPPORTED) + } + val compiled = compiler.compile(input) + val matchesInput = compiled.target == target && + compiled.operation == expectedOperation && + compiled.schemaContractId == input.schema.contractId && + input.accepts(compiled.loweringAttestation) + if (!matchesInput) { + rejectLegacy(QueryRejectionCode.LEGACY_LOWERING_UNSUPPORTED) + } + return compiled + } + + private fun requireSupportedBudget(options: QueryExecutionOptions) { + val budget = options.budget + val unsupported = sequenceOf( + budget.maxScannedRecords, + budget.maxCandidateBuckets, + budget.maxReturnedBuckets, + budget.maxCursorPages, + ).any { value -> value != null } || budget.allowDiskUse + if (unsupported) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionPath.ROOT.property("executionContext").property("budget"), + QueryRejectionCode.EXECUTION_BUDGET_UNSUPPORTED, + ) + } + } + + private fun backendMono(source: () -> Mono): Mono = + Mono.defer(source).onErrorMap(errorBoundary::normalizeBackend) + + private fun backendFlux(source: () -> Flux): Flux = + Flux.defer(source).onErrorMap(errorBoundary::normalizeBackend) +} + +private fun NormalizedQueryInvocation.hasConsistentShape(): Boolean = + when (operation) { + QueryOperation.SINGLE -> input is NormalizedQueryInput.Single && resultShape.isRecord() + QueryOperation.STREAM -> input is NormalizedQueryInput.Stream && resultShape.isRecord() + QueryOperation.PAGE -> input is NormalizedQueryInput.Page && resultShape.isRecord() + QueryOperation.COUNT -> input is NormalizedQueryInput.Count && resultShape == QueryResultShape.COUNT + QueryOperation.ANALYZE -> input is NormalizedQueryInput.Analytics && resultShape == QueryResultShape.ANALYTICS + } + +private fun QueryResultShape.isRecord(): Boolean = this == QueryResultShape.TYPED || this == QueryResultShape.DYNAMIC + +private fun QueryPlan.matchesResultShape(resultShape: QueryResultShape): Boolean = + when (this) { + is RecordQueryPlan -> this.resultShape.name == resultShape.name + is CountQueryPlan -> resultShape == QueryResultShape.COUNT + is AnalyticsQueryPlan -> resultShape == QueryResultShape.ANALYTICS + } + +private fun NormalizedQueryInvocation.deletionScope(): NormalizedDeletionScope = + when (val normalizedInput = input) { + is NormalizedQueryInput.Single -> normalizedInput.query.deletionScope + is NormalizedQueryInput.Stream -> normalizedInput.query.deletionScope + is NormalizedQueryInput.Page -> normalizedInput.query.deletionScope + is NormalizedQueryInput.Count -> normalizedInput.deletionScope + is NormalizedQueryInput.Analytics -> NormalizedDeletionScope.EXPLICIT + } + +private fun PlanningDecision.mandatoryCondition(): PlannedCondition = + when (this) { + is PlanningDecision.Planned -> plan.filter.mandatory + is PlanningDecision.LegacyFallback -> validatedMandatory.condition + } + +internal class LegacyBackendRegistry(bindings: Iterable) { + val bindings: Map + + init { + val bindingList = bindings.toList() + require(bindingList.map(LegacyExecutionBinding::target).distinct().size == bindingList.size) { + "Legacy backend targets must be unique." + } + val copy = LinkedHashMap(bindingList.size) + bindingList.sortedWith(LEGACY_BINDING_COMPARATOR).forEach { binding -> copy[binding.target] = binding } + this.bindings = Collections.unmodifiableMap(copy) + } + + fun resolve(target: QueryTarget): LegacyExecutionBinding = bindings[target] + ?: rejectQuery( + QueryRejectionCategory.BACKEND_UNAVAILABLE, + LEGACY_PATH, + QueryRejectionCode.LEGACY_BACKEND_NOT_REGISTERED, + ) +} + +internal fun rejectLegacy( + code: QueryRejectionCode, + category: QueryRejectionCategory = QueryRejectionCategory.UNSUPPORTED_FEATURE, + cause: Throwable? = null, +): Nothing = rejectQuery(category, LEGACY_PATH, code, cause) + +internal fun rejectLegacyMandatory(cause: Throwable? = null): Nothing = rejectQuery( + QueryRejectionCategory.ACCESS_DENIED, + QueryRejectionPath.ROOT.property("constraints").property("mandatoryCondition"), + QueryRejectionCode.MANDATORY_CONDITION_UNENFORCEABLE, + cause, +) + +private val LEGACY_BINDING_COMPARATOR: Comparator = + compareBy { binding -> binding.target.namedAggregate.contextName } + .thenBy { binding -> binding.target.namedAggregate.aggregateName } + .thenBy { binding -> binding.target.documentKind.name } + +private val LEGACY_PATH = QueryRejectionPath.ROOT.property("legacy") diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryBackend.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryBackend.kt new file mode 100644 index 00000000000..d1dd49816be --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryBackend.kt @@ -0,0 +1,186 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsCompleteness +import me.ahoo.wow.query.internal.analytics.AnalyticsConsistency +import me.ahoo.wow.query.internal.plan.AnalyticsQueryPlan +import me.ahoo.wow.query.internal.plan.CountQueryPlan +import me.ahoo.wow.query.internal.plan.PageQueryPlan +import me.ahoo.wow.query.internal.plan.SingleQueryPlan +import me.ahoo.wow.query.internal.plan.StreamQueryPlan +import me.ahoo.wow.query.internal.policy.QueryExecutionBudget +import me.ahoo.wow.query.internal.policy.QueryExecutionContext +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Instant +import java.util.Collections +import java.util.LinkedHashMap + +internal typealias QueryBackendException = me.ahoo.wow.query.backend.QueryBackendException +internal typealias QueryBackendFailureKind = me.ahoo.wow.query.backend.QueryBackendFailureKind + +internal data class QueryExecutionOptions( + val deadline: Instant?, + val budget: QueryExecutionBudget, +) { + companion object { + fun from(context: QueryExecutionContext): QueryExecutionOptions = + QueryExecutionOptions(context.deadline, context.budget) + } +} + +internal data class BackendRecord( + val identity: String, + val document: NormalizedValue.ObjectValue, + val completeness: BackendRecordCompleteness, +) { + init { + require(identity.isNotBlank()) { + "Backend record identity must not be blank." + } + } +} + +internal enum class BackendRecordCompleteness { + COMPLETE, + UNKNOWN, +} + +internal enum class BackendTotalRelation { + EXACT, + LOWER_BOUND, + UNKNOWN, +} + +internal enum class BackendPageConsistency { + SAME_INPUT, + INDEPENDENT, + UNKNOWN, +} + +internal class BackendPage( + records: Iterable, + val total: Long, + val totalRelation: BackendTotalRelation, + val consistency: BackendPageConsistency, +) { + val records: List = Collections.unmodifiableList(records.toList()) + + init { + require(total >= 0) { + "Backend page total must not be negative." + } + } + + override fun equals(other: Any?): Boolean = + this === other || + other is BackendPage && + records == other.records && + total == other.total && + totalRelation == other.totalRelation && + consistency == other.consistency + + override fun hashCode(): Int { + var result = records.hashCode() + result = 31 * result + total.hashCode() + result = 31 * result + totalRelation.hashCode() + result = 31 * result + consistency.hashCode() + return result + } +} + +internal class BackendAnalyticsBucket( + keys: Map, + metrics: Map, +) { + val keys: Map = immutableValues(keys) + val metrics: Map = immutableValues(metrics) + + override fun equals(other: Any?): Boolean = + this === other || other is BackendAnalyticsBucket && keys == other.keys && metrics == other.metrics + + override fun hashCode(): Int = 31 * keys.hashCode() + metrics.hashCode() + + private fun immutableValues(values: Map): Map { + val copy = LinkedHashMap(values.size) + values.entries.sortedBy { entry -> entry.key.value }.forEach { entry -> copy[entry.key] = entry.value } + return Collections.unmodifiableMap(copy) + } +} + +internal class BackendAnalyticsPage( + buckets: Iterable, + afterKey: List?, + val consistency: AnalyticsConsistency, + val completeness: AnalyticsCompleteness, + cursorState: ByteArray? = null, +) { + private val frozenCursorState: ByteArray? = cursorState?.copyOf() + val buckets: List = Collections.unmodifiableList(buckets.toList()) + val afterKey: List? = afterKey?.let { Collections.unmodifiableList(it.toList()) } + + fun cursorState(): ByteArray? = frozenCursorState?.copyOf() + + override fun equals(other: Any?): Boolean = + this === other || + other is BackendAnalyticsPage && + buckets == other.buckets && + afterKey == other.afterKey && + consistency == other.consistency && + completeness == other.completeness && + cursorStateContentEquals(other) + + override fun hashCode(): Int { + var result = buckets.hashCode() + result = 31 * result + (afterKey?.hashCode() ?: 0) + result = 31 * result + consistency.hashCode() + result = 31 * result + completeness.hashCode() + result = 31 * result + (frozenCursorState?.contentHashCode() ?: 0) + return result + } + + private fun cursorStateContentEquals(other: BackendAnalyticsPage): Boolean = + when { + frozenCursorState == null -> other.frozenCursorState == null + other.frozenCursorState == null -> false + else -> frozenCursorState.contentEquals(other.frozenCursorState) + } +} + +internal interface RecordQueryBackend { + fun single(plan: SingleQueryPlan, options: QueryExecutionOptions): Mono + + fun stream(plan: StreamQueryPlan, options: QueryExecutionOptions): Flux + + fun page(plan: PageQueryPlan, options: QueryExecutionOptions): Mono + + fun count(plan: CountQueryPlan, options: QueryExecutionOptions): Mono +} + +internal fun interface AnalyticsQueryBackend { + fun analyze(plan: AnalyticsQueryPlan, options: QueryExecutionOptions): Mono + + fun analyze( + plan: AnalyticsQueryPlan, + options: QueryExecutionOptions, + cursorState: ByteArray?, + ): Mono = if (cursorState == null) { + analyze(plan, options) + } else { + Mono.error(QueryBackendException(QueryBackendFailureKind.UNSUPPORTED)) + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryBackendRegistry.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryBackendRegistry.kt new file mode 100644 index 00000000000..0ecf973d09d --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryBackendRegistry.kt @@ -0,0 +1,231 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.wow.query.backend.ExperimentalQueryBackendApi +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.QUERY_FIELD_ID_COMPARATOR +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.SearchScopeId +import me.ahoo.wow.query.internal.plan.QueryPlan +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.plan.StreamLimit +import me.ahoo.wow.query.internal.plan.StreamQueryPlan +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import java.util.Collections +import java.util.LinkedHashMap +import me.ahoo.wow.query.backend.RecordQueryBackend as ExperimentalRecordQueryBackend + +internal data class QueryBackendKey( + val target: QueryTarget, + val backendId: BackendId, +) + +internal enum class QueryBackendStreamSupport { + NONE, + BOUNDED_ONLY, + UNBOUNDED, +} + +internal class QueryBackendDescriptor( + val key: QueryBackendKey, + val schemaContractId: SchemaContractId, + supportedOperations: Set, + semanticTiers: Set, + fieldCapabilities: Map>, + searchScopes: Set = emptySet(), + val mappingGenerationDigest: String = schemaContractId.value, + val streamSupport: QueryBackendStreamSupport = if (QueryOperation.STREAM in supportedOperations) { + QueryBackendStreamSupport.UNBOUNDED + } else { + QueryBackendStreamSupport.NONE + }, +) { + val supportedOperations: Set = Collections.unmodifiableSet( + LinkedHashSet(supportedOperations.sortedBy(QueryOperation::name)), + ) + val semanticTiers: Set = Collections.unmodifiableSet( + LinkedHashSet(semanticTiers.sortedBy(SemanticTier::name)), + ) + val fieldCapabilities: Map> = immutableCapabilities(fieldCapabilities) + val searchScopes: Set = Collections.unmodifiableSet( + LinkedHashSet(searchScopes.sortedBy(SearchScopeId::value)), + ) + + init { + require(this.supportedOperations.isNotEmpty()) { + "Query backend must support at least one operation." + } + require(this.semanticTiers.isNotEmpty()) { + "Query backend must support at least one semantic tier." + } + require(QueryOperation.STREAM in this.supportedOperations || streamSupport == QueryBackendStreamSupport.NONE) { + "Stream support must be NONE when STREAM is not registered." + } + require(QueryOperation.STREAM !in this.supportedOperations || streamSupport != QueryBackendStreamSupport.NONE) { + "STREAM requires an explicit stream support level." + } + require( + mappingGenerationDigest.length == SHA_256_HEX_LENGTH && + mappingGenerationDigest.all { character -> character in HEX_CHARACTERS }, + ) { + "Mapping generation digest must be lowercase SHA-256 hex." + } + } + + private fun immutableCapabilities( + capabilities: Map>, + ): Map> { + val copy = LinkedHashMap>(capabilities.size) + capabilities.entries.sortedWith(compareBy(QUERY_FIELD_ID_COMPARATOR) { entry -> entry.key }).forEach { entry -> + copy[entry.key] = Collections.unmodifiableSet( + LinkedHashSet(entry.value.sortedBy(FieldCapability::name)), + ) + } + return Collections.unmodifiableMap(copy) + } +} + +@OptIn(ExperimentalQueryBackendApi::class) +internal data class QueryBackendRegistration( + val descriptor: QueryBackendDescriptor, + val recordBackend: RecordQueryBackend? = null, + val experimentalRecordBackend: ExperimentalRecordQueryBackend? = null, + val analyticsBackend: AnalyticsQueryBackend? = null, +) { + init { + val recordOperations = descriptor.supportedOperations - QueryOperation.ANALYZE + require(recordOperations.isEmpty() || recordBackend != null || experimentalRecordBackend != null) { + "Record operations require a record query backend." + } + require(recordBackend == null || experimentalRecordBackend == null) { + "A query backend registration must have one record backend owner." + } + require(QueryOperation.ANALYZE !in descriptor.supportedOperations || analyticsBackend != null) { + "ANALYZE requires an analytics query backend." + } + } +} + +internal class QueryBackendRegistry( + registrations: Iterable, + defaultRoutes: Map, + notReadyKeys: Set = emptySet(), +) { + val registrations: Map + val defaultRoutes: Map + val notReadyKeys: Set + + init { + val registrationList = registrations.toList() + require( + registrationList.map { registration -> registration.descriptor.key }.distinct().size == + registrationList.size, + ) { + "Query backend registration keys must be unique." + } + val registrationCopy = LinkedHashMap(registrationList.size) + registrationList.sortedWith(QUERY_BACKEND_REGISTRATION_COMPARATOR).forEach { registration -> + registrationCopy[registration.descriptor.key] = registration + } + this.registrations = Collections.unmodifiableMap(registrationCopy) + val routeCopy = LinkedHashMap(defaultRoutes.size) + defaultRoutes.entries.sortedWith(QUERY_BACKEND_ROUTE_COMPARATOR).forEach { entry -> + routeCopy[entry.key] = entry.value + } + this.defaultRoutes = Collections.unmodifiableMap(routeCopy) + this.notReadyKeys = Collections.unmodifiableSet( + LinkedHashSet(notReadyKeys.sortedWith(QUERY_BACKEND_KEY_COMPARATOR)), + ) + require(this.registrations.keys.intersect(this.notReadyKeys).isEmpty()) { + "A Query backend registration cannot be ready and not-ready at the same time." + } + } + + fun resolve(plan: QueryPlan): QueryBackendRegistration { + val backendId = plan.requiredCapabilities.nativeBackend ?: defaultRoutes[plan.target] + ?: rejectBackend(QueryRejectionCode.BACKEND_NOT_REGISTERED) + val key = QueryBackendKey(plan.target, backendId) + val registration = registrations[key] ?: if (key in notReadyKeys) { + rejectBackend(QueryRejectionCode.BACKEND_NOT_READY) + } else { + rejectBackend(QueryRejectionCode.BACKEND_NOT_REGISTERED) + } + validate(plan, registration.descriptor) + return registration + } + + private fun validate(plan: QueryPlan, descriptor: QueryBackendDescriptor) { + if (plan.schemaContractId != descriptor.schemaContractId) { + rejectBackend(QueryRejectionCode.BACKEND_SCHEMA_MISMATCH) + } + if (plan.operation !in descriptor.supportedOperations) { + rejectBackend(QueryRejectionCode.BACKEND_OPERATION_UNSUPPORTED) + } + if ( + plan is StreamQueryPlan && + plan.limit == StreamLimit.Unbounded && + descriptor.streamSupport == QueryBackendStreamSupport.BOUNDED_ONLY + ) { + rejectBackend(QueryRejectionCode.BACKEND_OPERATION_UNSUPPORTED) + } + if (plan.semanticTier !in descriptor.semanticTiers) { + rejectBackend(QueryRejectionCode.BACKEND_CAPABILITY_MISMATCH) + } + val missingFieldCapability = plan.requiredCapabilities.fieldRequirements.any { (field, required) -> + !descriptor.fieldCapabilities[field].orEmpty().containsAll(required) + } + if (missingFieldCapability || !descriptor.searchScopes.containsAll(plan.requiredCapabilities.searchRequirements)) { + rejectBackend(QueryRejectionCode.BACKEND_CAPABILITY_MISMATCH) + } + val nativeBackend = plan.requiredCapabilities.nativeBackend + if (nativeBackend != null && nativeBackend != descriptor.key.backendId) { + rejectBackend(QueryRejectionCode.BACKEND_CAPABILITY_MISMATCH) + } + } + + private fun rejectBackend(code: QueryRejectionCode): Nothing = rejectQuery( + QueryRejectionCategory.BACKEND_UNAVAILABLE, + BACKEND_PATH, + code, + ) +} + +private val QUERY_BACKEND_REGISTRATION_COMPARATOR: Comparator = + compareBy { registration -> registration.descriptor.key.target.namedAggregate.contextName } + .thenBy { registration -> registration.descriptor.key.target.namedAggregate.aggregateName } + .thenBy { registration -> registration.descriptor.key.target.documentKind.name } + .thenBy { registration -> registration.descriptor.key.backendId.value } + +private val QUERY_BACKEND_ROUTE_COMPARATOR: Comparator> = + compareBy> { entry -> entry.key.namedAggregate.contextName } + .thenBy { entry -> entry.key.namedAggregate.aggregateName } + .thenBy { entry -> entry.key.documentKind.name } + +private val QUERY_BACKEND_KEY_COMPARATOR: Comparator = + compareBy { key -> key.target.namedAggregate.contextName } + .thenBy { key -> key.target.namedAggregate.aggregateName } + .thenBy { key -> key.target.documentKind.name } + .thenBy { key -> key.backendId.value } + +private val BACKEND_PATH = QueryRejectionPath.ROOT.property("backend") +private const val SHA_256_HEX_LENGTH = 64 +private const val HEX_CHARACTERS = "0123456789abcdef" diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutionLifecycle.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutionLifecycle.kt new file mode 100644 index 00000000000..d84d9252909 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutionLifecycle.kt @@ -0,0 +1,244 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.policy.QueryExecutionRequest +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejection +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.core.publisher.SignalType +import reactor.core.scheduler.Scheduler +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.util.concurrent.atomic.AtomicBoolean +import java.util.concurrent.atomic.AtomicLong +import java.util.concurrent.atomic.AtomicReference + +internal class QueryErrorBoundary { + fun normalize(error: Throwable): Throwable = + when (error) { + is QueryRejectedException -> error + is QueryBackendException -> error.toRejectedException() + else -> QueryRejectedException( + QueryRejection( + QueryRejectionCategory.INTERNAL_FAILURE, + QueryRejectionPath.ROOT, + QueryRejectionCode.UNEXPECTED_QUERY_FAILURE, + ), + error, + ) + } + + /** Normalizes failures emitted by an untrusted backend without allowing it to spoof gateway-stage rejections. */ + fun normalizeBackend(error: Throwable): Throwable = + when (error) { + is QueryBackendException -> error.toRejectedException() + else -> QueryRejectedException( + QueryRejection( + QueryRejectionCategory.INTERNAL_FAILURE, + QueryRejectionPath.ROOT.property("backend"), + QueryRejectionCode.BACKEND_EXECUTION_FAILED, + ), + error, + ) + } + + private fun QueryBackendException.toRejectedException(): QueryRejectedException { + val (category, path, code) = + when (kind) { + QueryBackendFailureKind.UNAVAILABLE -> Triple( + QueryRejectionCategory.BACKEND_UNAVAILABLE, + QueryRejectionPath.ROOT.property("backend"), + QueryRejectionCode.BACKEND_EXECUTION_FAILED, + ) + + QueryBackendFailureKind.TIMEOUT -> Triple( + QueryRejectionCategory.BACKEND_TIMEOUT, + QueryRejectionPath.ROOT.property("backend"), + QueryRejectionCode.BACKEND_TIMEOUT, + ) + + QueryBackendFailureKind.BUDGET_EXCEEDED -> Triple( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionPath.ROOT.property("executionContext").property("budget"), + QueryRejectionCode.BACKEND_BUDGET_EXCEEDED, + ) + + QueryBackendFailureKind.INCOMPLETE_RESULT -> Triple( + QueryRejectionCategory.INCOMPLETE_RESULT, + QueryRejectionPath.ROOT.property("backend").property("result"), + QueryRejectionCode.INCOMPLETE_RESULT, + ) + + QueryBackendFailureKind.MAPPING_FAILURE -> Triple( + QueryRejectionCategory.MAPPING_FAILURE, + QueryRejectionPath.ROOT.property("result"), + QueryRejectionCode.RESULT_MAPPING_FAILED, + ) + + QueryBackendFailureKind.UNSUPPORTED -> Triple( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionPath.ROOT.property("backend"), + QueryRejectionCode.BACKEND_OPERATION_UNSUPPORTED, + ) + } + return QueryRejectedException(QueryRejection(category, path, code), this) + } +} + +internal data class QueryLifecycleDescriptor( + val request: QueryExecutionRequest, + val operation: QueryOperation, +) + +internal enum class QueryTerminationKind { + COMPLETE, + ERROR, + CANCEL, +} + +internal data class QueryLifecycleTerminal( + val descriptor: QueryLifecycleDescriptor, + val kind: QueryTerminationKind, + val emitted: Long, + val error: QueryRejectedException?, +) + +internal interface QueryLifecycleObserver { + fun onStart(descriptor: QueryLifecycleDescriptor) = Unit + + fun onTerminal(terminal: QueryLifecycleTerminal) = Unit + + companion object { + val NONE: QueryLifecycleObserver = object : QueryLifecycleObserver {} + } +} + +internal class QueryLifecycleMonitor( + private val observer: QueryLifecycleObserver = QueryLifecycleObserver.NONE, +) { + fun observeMono(descriptor: QueryLifecycleDescriptor, source: Mono): Mono = Mono.defer { + observeStart(descriptor) + val state = ObservationState(descriptor) + source + .doOnNext { state.emitted.incrementAndGet() } + .doOnError(state.error::set) + .doFinally { signal -> state.terminate(signal) } + } + + fun observeFlux(descriptor: QueryLifecycleDescriptor, source: Flux): Flux = Flux.defer { + observeStart(descriptor) + val state = ObservationState(descriptor) + source + .doOnNext { state.emitted.incrementAndGet() } + .doOnError(state.error::set) + .doFinally { signal -> state.terminate(signal) } + } + + private fun observeStart(descriptor: QueryLifecycleDescriptor) { + try { + observer.onStart(descriptor) + } catch (_: RuntimeException) { + // Observability cannot alter query behavior. + } + } + + private inner class ObservationState( + private val descriptor: QueryLifecycleDescriptor, + ) { + val emitted = AtomicLong() + val error = AtomicReference() + private val terminated = AtomicBoolean() + + fun terminate(signal: SignalType) { + if (!terminated.compareAndSet(false, true)) { + return + } + val kind = + when (signal) { + SignalType.ON_COMPLETE -> QueryTerminationKind.COMPLETE + SignalType.CANCEL -> QueryTerminationKind.CANCEL + else -> QueryTerminationKind.ERROR + } + val normalizedError = error.get() as? QueryRejectedException + try { + observer.onTerminal(QueryLifecycleTerminal(descriptor, kind, emitted.get(), normalizedError)) + } catch (_: RuntimeException) { + // Observability cannot replace the terminal signal. + } + } + } +} + +internal class QueryDeadlineEnforcer( + private val clock: Clock, + private val scheduler: Scheduler, +) { + fun cappedDeadline(deadline: Instant?, maximumDuration: Duration): Instant { + require(!maximumDuration.isZero && !maximumDuration.isNegative) { + "Maximum deadline duration must be positive." + } + val maximumDeadline = clock.instant().plus(maximumDuration) + return if (deadline != null && deadline.isBefore(maximumDeadline)) deadline else maximumDeadline + } + + fun enforceMono(deadline: Instant?, source: () -> Mono): Mono = Mono.defer { + val remaining = remaining(deadline) ?: return@defer Mono.defer(source) + Mono.defer(source).timeout(remaining, Mono.error(deadlineExpired()), scheduler) + } + + fun enforceFlux(deadline: Instant?, source: () -> Flux): Flux = Flux.defer { + val remaining = remaining(deadline) ?: return@defer Flux.defer(source) + enforceDeadline(Flux.defer(source), remaining) + } + + private fun remaining(deadline: Instant?): Duration? { + deadline ?: return null + val remaining = Duration.between(clock.instant(), deadline) + if (remaining.isZero || remaining.isNegative) { + throw deadlineExpired() + } + return remaining + } + + private fun enforceDeadline(source: Flux, remaining: Duration): Flux = Flux.defer { + val expired = AtomicBoolean() + val timeout = Mono.delay(remaining, scheduler).doOnNext { expired.set(true) } + source.takeUntilOther(timeout).concatWith( + Flux.defer { + if (expired.get()) { + Flux.error(deadlineExpired()) + } else { + Flux.empty() + } + }, + ) + } + + private fun deadlineExpired(): QueryRejectedException = QueryRejectedException( + QueryRejection( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionPath.ROOT.property("executionContext").property("deadline"), + QueryRejectionCode.DEADLINE_EXPIRED, + ), + ) +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutionRoute.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutionRoute.kt new file mode 100644 index 00000000000..674251cf50d --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutionRoute.kt @@ -0,0 +1,186 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.internal.model.QueryExecutionMode +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.plan.QueryPlan +import me.ahoo.wow.query.internal.planning.PlanningDecision +import me.ahoo.wow.query.internal.policy.QueryExecutionContext +import me.ahoo.wow.query.internal.rejection.QueryRejection +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import me.ahoo.wow.query.internal.value.NonEmptyList + +internal data class QueryFallback( + val target: QueryTarget, + val operation: QueryOperation, + val executionMode: QueryExecutionMode, + val issues: NonEmptyList, +) + +internal sealed interface QueryExecutionRoute { + data class Planned( + val registry: QueryBackendRegistry, + val plan: QueryPlan, + ) : QueryExecutionRoute + + data class Legacy( + val binding: LegacyExecutionBinding, + val input: LegacyCompilationInput, + val fallback: QueryFallback? = null, + ) : QueryExecutionRoute + + data class Shadow( + val legacyBinding: LegacyExecutionBinding, + val legacyInput: LegacyCompilationInput, + val plannedRegistry: QueryBackendRegistry, + val plan: QueryPlan, + ) : QueryExecutionRoute +} + +internal class QueryExecutionRouteResolver( + private val plannedRegistry: QueryBackendRegistry, + private val legacyRegistry: LegacyBackendRegistry, +) { + fun resolve( + context: QueryExecutionContext, + invocation: NormalizedQueryInvocation, + schema: QueryDocumentSchema, + decision: PlanningDecision, + ): QueryExecutionRoute { + validateInput(context, invocation, schema, decision) + val legacyInput = LegacyCompilationInput(invocation, schema, decision) + return when (context.executionMode) { + QueryExecutionMode.LEGACY -> legacyRoute( + invocation, + legacyInput, + decision, + QueryExecutionMode.LEGACY, + ) + QueryExecutionMode.SHADOW -> shadowRoute(invocation, legacyInput, decision) + QueryExecutionMode.PLANNED -> plannedRoute(invocation, legacyInput, decision) + } + } + + private fun legacyRoute( + invocation: NormalizedQueryInvocation, + input: LegacyCompilationInput, + decision: PlanningDecision, + executionMode: QueryExecutionMode, + ): QueryExecutionRoute { + rejectLegacyAnalytics(invocation) + return QueryExecutionRoute.Legacy( + legacyRegistry.resolve(invocation.target), + input, + decision.fallback(invocation, executionMode), + ) + } + + private fun shadowRoute( + invocation: NormalizedQueryInvocation, + legacyInput: LegacyCompilationInput, + decision: PlanningDecision, + ): QueryExecutionRoute { + rejectLegacyAnalytics(invocation) + val legacy = legacyRegistry.resolve(invocation.target) + return when (decision) { + is PlanningDecision.Planned -> QueryExecutionRoute.Shadow( + legacy, + legacyInput, + plannedRegistry, + decision.plan, + ) + + is PlanningDecision.LegacyFallback -> legacyRoute( + invocation, + legacyInput, + decision, + QueryExecutionMode.SHADOW, + ) + } + } + + private fun plannedRoute( + invocation: NormalizedQueryInvocation, + legacyInput: LegacyCompilationInput, + decision: PlanningDecision, + ): QueryExecutionRoute = + when (decision) { + is PlanningDecision.Planned -> QueryExecutionRoute.Planned( + plannedRegistry, + decision.plan, + ) + + is PlanningDecision.LegacyFallback -> { + legacyRoute( + invocation, + legacyInput, + decision, + QueryExecutionMode.PLANNED, + ) + } + } + + private fun PlanningDecision.fallback( + invocation: NormalizedQueryInvocation, + executionMode: QueryExecutionMode, + ): QueryFallback? = when (this) { + is PlanningDecision.Planned -> null + is PlanningDecision.LegacyFallback -> QueryFallback( + invocation.target, + invocation.operation, + executionMode, + issues, + ) + } + + private fun validateInput( + context: QueryExecutionContext, + invocation: NormalizedQueryInvocation, + schema: QueryDocumentSchema, + decision: PlanningDecision, + ) { + if (context.target != invocation.target || schema.target != invocation.target) { + rejectQuery( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionPath.ROOT.property("target"), + QueryRejectionCode.TARGET_SCHEMA_MISMATCH, + ) + } + if (context.validationMode == QueryValidationMode.STRICT && decision is PlanningDecision.LegacyFallback) { + rejectQuery( + QueryRejectionCategory.INTERNAL_FAILURE, + QueryRejectionPath.ROOT.property("execution"), + QueryRejectionCode.EXECUTION_DECISION_INVALID, + ) + } + } + + private fun rejectLegacyAnalytics(invocation: NormalizedQueryInvocation) { + if (invocation.operation == QueryOperation.ANALYZE) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionPath.ROOT.property("executionContext").property("executionMode"), + QueryRejectionCode.EXECUTION_MODE_UNSUPPORTED, + ) + } + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutor.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutor.kt new file mode 100644 index 00000000000..ef3d2333243 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutor.kt @@ -0,0 +1,642 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsCompleteness +import me.ahoo.wow.query.internal.analytics.AnalyticsConsistency +import me.ahoo.wow.query.internal.model.QueryExecutionMode +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.plan.AnalyticsQueryPlan +import me.ahoo.wow.query.internal.plan.PageQueryPlan +import me.ahoo.wow.query.internal.plan.PlanFingerprint +import me.ahoo.wow.query.internal.plan.QueryPlan +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.plan.StreamLimit +import me.ahoo.wow.query.internal.plan.StreamQueryPlan +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejection +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import me.ahoo.wow.query.internal.value.NonEmptyList +import org.reactivestreams.Publisher +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Duration + +internal sealed interface QueryShadowTask { + val target: QueryTarget + val fingerprint: PlanFingerprint + val semanticTier: SemanticTier + val operation: QueryOperation + val publisher: Publisher<*> + + data class Single( + override val target: QueryTarget, + override val fingerprint: PlanFingerprint, + override val semanticTier: SemanticTier, + override val publisher: Mono, + ) : QueryShadowTask { + override val operation: QueryOperation = QueryOperation.SINGLE + } + + data class Stream( + override val target: QueryTarget, + override val fingerprint: PlanFingerprint, + override val semanticTier: SemanticTier, + override val publisher: Flux, + ) : QueryShadowTask { + override val operation: QueryOperation = QueryOperation.STREAM + } + + data class Page( + override val target: QueryTarget, + override val fingerprint: PlanFingerprint, + override val semanticTier: SemanticTier, + override val publisher: Mono, + ) : QueryShadowTask { + override val operation: QueryOperation = QueryOperation.PAGE + } + + data class Count( + override val target: QueryTarget, + override val fingerprint: PlanFingerprint, + override val semanticTier: SemanticTier, + override val publisher: Mono, + ) : QueryShadowTask { + override val operation: QueryOperation = QueryOperation.COUNT + } +} + +internal data class QueryShadowDescriptor( + val target: QueryTarget, + val fingerprint: PlanFingerprint, + val semanticTier: SemanticTier, + val operation: QueryOperation, +) + +private fun QueryShadowTask.descriptor(): QueryShadowDescriptor = QueryShadowDescriptor( + target, + fingerprint, + semanticTier, + operation, +) + +internal sealed interface QueryShadowPrimarySignal { + data class RecordValue(val value: BackendRecord) : QueryShadowPrimarySignal + + data class PageValue(val value: BackendPage) : QueryShadowPrimarySignal + + data class CountValue(val value: Long) : QueryShadowPrimarySignal + + data object Complete : QueryShadowPrimarySignal + + data class Error(val error: QueryRejectedException) : QueryShadowPrimarySignal + + data object Cancelled : QueryShadowPrimarySignal +} + +internal interface QueryShadowHandle { + fun onPrimary(signal: QueryShadowPrimarySignal) + + fun cancelProbe() + + companion object { + val NONE: QueryShadowHandle = object : QueryShadowHandle { + override fun onPrimary(signal: QueryShadowPrimarySignal) = Unit + + override fun cancelProbe() = Unit + } + } +} + +internal data class QueryShadowSkip( + val target: QueryTarget, + val operation: QueryOperation, + val issues: NonEmptyList, +) + +internal sealed interface QueryShadowSubmission { + data class Accepted(val handle: QueryShadowHandle) : QueryShadowSubmission + + data class Rejected(val issue: QueryRejection) : QueryShadowSubmission +} + +internal fun interface QueryShadowSupervisor { + fun submit(task: QueryShadowTask): QueryShadowSubmission + + fun onSkipped(skip: QueryShadowSkip) = Unit + + companion object { + val DISABLED: QueryShadowSupervisor = QueryShadowSupervisor { + QueryShadowSubmission.Rejected(shadowSupervisorUnavailable()) + } + } +} + +internal data class QueryShadowSupervisorFailure( + val task: QueryShadowDescriptor, + val issue: QueryRejection, + val cause: Throwable? = null, +) + +internal interface QueryDecisionObserver { + fun onFallback(fallback: QueryFallback) = Unit + + fun onShadowSupervisorFailure(failure: QueryShadowSupervisorFailure) = Unit + + companion object { + val NONE: QueryDecisionObserver = object : QueryDecisionObserver {} + } +} + +internal class QueryExecutor( + private val deadlineEnforcer: QueryDeadlineEnforcer, + private val errorBoundary: QueryErrorBoundary = QueryErrorBoundary(), + private val shadowSupervisor: QueryShadowSupervisor = QueryShadowSupervisor.DISABLED, + private val decisionObserver: QueryDecisionObserver = QueryDecisionObserver.NONE, + private val shadowProbeTimeout: Duration = Duration.ofSeconds(30), +) { + init { + require(!shadowProbeTimeout.isZero && !shadowProbeTimeout.isNegative) { + "Shadow probe timeout must be positive." + } + } + fun single(route: QueryExecutionRoute, options: QueryExecutionOptions): Mono = + when (route) { + is QueryExecutionRoute.Planned -> plannedSingle(route.registry, route.plan, options) + is QueryExecutionRoute.Legacy -> legacyMono(route) { route.binding.single(route.input, options) } + is QueryExecutionRoute.Shadow -> shadowMono( + route.legacyBinding.single(route.legacyInput, options), + plannedSingle(route.plannedRegistry, route.plan, options), + options, + QueryShadowPrimarySignal::RecordValue, + ) { probe -> + QueryShadowTask.Single(route.plan.target, route.plan.fingerprint, route.plan.semanticTier, probe) + } + } + + fun stream(route: QueryExecutionRoute, options: QueryExecutionOptions): Flux = + when (route) { + is QueryExecutionRoute.Planned -> plannedStream(route.registry, route.plan, options) + is QueryExecutionRoute.Legacy -> legacyFlux(route) { + enforceLegacyStreamLimit(route.input, route.binding.stream(route.input, options)) + } + + is QueryExecutionRoute.Shadow -> shadowStream(route, options) + } + + fun page(route: QueryExecutionRoute, options: QueryExecutionOptions): Mono { + val result = when (route) { + is QueryExecutionRoute.Planned -> plannedPage(route.registry, route.plan, options) + is QueryExecutionRoute.Legacy -> legacyMono(route) { + route.binding.page(route.input, options).map { page -> requireLegacyPage(route.input, page) } + } + + is QueryExecutionRoute.Shadow -> shadowMono( + route.legacyBinding.page(route.legacyInput, options) + .map { page -> requireLegacyPage(route.legacyInput, page) } + .switchIfEmpty(incompleteResult()), + plannedPage(route.plannedRegistry, route.plan, options), + options, + QueryShadowPrimarySignal::PageValue, + ) { probe -> + QueryShadowTask.Page(route.plan.target, route.plan.fingerprint, route.plan.semanticTier, probe) + } + } + return result.switchIfEmpty(incompleteResult()) + } + + fun count(route: QueryExecutionRoute, options: QueryExecutionOptions): Mono { + val result = when (route) { + is QueryExecutionRoute.Planned -> plannedCount(route.registry, route.plan, options) + is QueryExecutionRoute.Legacy -> legacyMono(route) { route.binding.count(route.input, options) } + is QueryExecutionRoute.Shadow -> shadowMono( + route.legacyBinding.count(route.legacyInput, options) + .map(::requireNonNegativeCount) + .switchIfEmpty(incompleteResult()), + plannedCount(route.plannedRegistry, route.plan, options), + options, + QueryShadowPrimarySignal::CountValue, + ) { probe -> + QueryShadowTask.Count(route.plan.target, route.plan.fingerprint, route.plan.semanticTier, probe) + } + } + return result.map(::requireNonNegativeCount).switchIfEmpty(incompleteResult()) + } + + fun analyze( + route: QueryExecutionRoute, + options: QueryExecutionOptions, + cursorState: ByteArray? = null, + ): Mono { + val result = when (route) { + is QueryExecutionRoute.Planned -> plannedAnalytics(route.registry, route.plan, options, cursorState) + is QueryExecutionRoute.Legacy, + is QueryExecutionRoute.Shadow, + -> rejectExecution(QueryRejectionCode.EXECUTION_MODE_UNSUPPORTED) + } + return result.switchIfEmpty(incompleteResult()) + } + + private fun plannedSingle( + registration: QueryBackendRegistration, + plan: QueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.defer { + val backend = requireRecordBackend(registration) + backendMono { backend.single(requirePlan(plan), options) }.map(::requireCompleteRecord) + } + + private fun plannedSingle( + registry: QueryBackendRegistry, + plan: QueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.defer { plannedSingle(registry.resolve(plan), plan, options) } + + private fun plannedStream( + registration: QueryBackendRegistration, + plan: QueryPlan, + options: QueryExecutionOptions, + ): Flux = Flux.defer { + val streamPlan = requirePlan(plan) + val backend = requireRecordBackend(registration) + val records = backendFlux { backend.stream(streamPlan, options) }.map(::requireCompleteRecord) + enforceStreamLimit(records, streamPlan.limit) + } + + private fun plannedStream( + registry: QueryBackendRegistry, + plan: QueryPlan, + options: QueryExecutionOptions, + ): Flux = Flux.defer { plannedStream(registry.resolve(plan), plan, options) } + + private fun plannedPage( + registration: QueryBackendRegistration, + plan: QueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.defer { + val pagePlan = requirePlan(plan) + val backend = requireRecordBackend(registration) + backendMono { backend.page(pagePlan, options) } + .map { page -> requireCompletePage(pagePlan, page) } + .switchIfEmpty(incompleteResult()) + } + + private fun plannedPage( + registry: QueryBackendRegistry, + plan: QueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.defer { plannedPage(registry.resolve(plan), plan, options) } + + private fun plannedCount( + registration: QueryBackendRegistration, + plan: QueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.defer { + val backend = requireRecordBackend(registration) + backendMono { backend.count(requirePlan(plan), options) } + .map(::requireNonNegativeCount) + .switchIfEmpty(incompleteResult()) + } + + private fun plannedCount( + registry: QueryBackendRegistry, + plan: QueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.defer { plannedCount(registry.resolve(plan), plan, options) } + + private fun plannedAnalytics( + registration: QueryBackendRegistration, + plan: QueryPlan, + options: QueryExecutionOptions, + cursorState: ByteArray?, + ): Mono = Mono.defer { + val analyticsPlan = requirePlan(plan) + val backend = registration.analyticsBackend + ?: rejectExecution(QueryRejectionCode.BACKEND_OPERATION_UNSUPPORTED) + backendMono { backend.analyze(analyticsPlan, options, cursorState) }.map { page -> + if (!page.isCompleteFor(analyticsPlan)) { + rejectIncomplete() + } + page + }.switchIfEmpty(incompleteResult()) + } + + private fun plannedAnalytics( + registry: QueryBackendRegistry, + plan: QueryPlan, + options: QueryExecutionOptions, + cursorState: ByteArray?, + ): Mono = Mono.defer { + plannedAnalytics(registry.resolve(plan), plan, options, cursorState) + } + + private fun shadowStream( + route: QueryExecutionRoute.Shadow, + options: QueryExecutionOptions, + ): Flux = Flux.defer { + val plan = requirePlan(route.plan) + val primary = enforceLegacyStreamLimit( + route.legacyInput, + route.legacyBinding.stream(route.legacyInput, options), + ) + if (plan.limit == StreamLimit.Unbounded) { + notifyShadowSkipped( + route.legacyInput.invocation.target, + route.legacyInput.invocation.operation, + NonEmptyList.of( + QueryRejection( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionPath.ROOT.property("shadow"), + QueryRejectionCode.SHADOW_PROBE_UNBOUNDED_STREAM, + ), + ), + ) + primary + } else { + shadowFlux( + plan, + primary, + plannedStream(route.plannedRegistry, route.plan, options), + options, + ) + } + } + + private fun shadowMono( + primary: Mono, + probe: Mono

, + options: QueryExecutionOptions, + signal: (T) -> QueryShadowPrimarySignal, + taskFactory: (Mono

) -> QueryShadowTask, + ): Mono = Mono.defer { + val handle = submitShadow(taskFactory(normalizeShadowMono(options, probe))) + primary + .doOnNext { value -> reportPrimary(handle, signal(value)) } + .doOnSuccess { reportPrimary(handle, QueryShadowPrimarySignal.Complete) } + .doOnError { error -> reportPrimaryError(handle, error) } + .doOnCancel { cancelShadow(handle) } + } + + private fun shadowFlux( + plan: QueryPlan, + primary: Flux, + probe: Flux, + options: QueryExecutionOptions, + ): Flux = Flux.defer { + val normalizedProbe = normalizeShadowFlux(options, probe) + val handle = submitShadow( + QueryShadowTask.Stream(plan.target, plan.fingerprint, plan.semanticTier, normalizedProbe), + ) + primary + .doOnNext { value -> reportPrimary(handle, QueryShadowPrimarySignal.RecordValue(value)) } + .doOnComplete { reportPrimary(handle, QueryShadowPrimarySignal.Complete) } + .doOnError { error -> reportPrimaryError(handle, error) } + .doOnCancel { cancelShadow(handle) } + } + + @Suppress("TooGenericExceptionCaught") + private fun submitShadow(task: QueryShadowTask): QueryShadowHandle = try { + when (val submission = shadowSupervisor.submit(task)) { + is QueryShadowSubmission.Accepted -> submission.handle + is QueryShadowSubmission.Rejected -> { + notifyShadowSupervisorFailure(task, submission.issue) + QueryShadowHandle.NONE + } + } + } catch (error: RuntimeException) { + notifyShadowSupervisorFailure(task, shadowSupervisorUnavailable(), error) + QueryShadowHandle.NONE + } + + private fun reportPrimary(handle: QueryShadowHandle, signal: QueryShadowPrimarySignal) { + try { + handle.onPrimary(signal) + } catch (_: RuntimeException) { + // Shadow comparison cannot alter the primary result. + } + } + + private fun reportPrimaryError(handle: QueryShadowHandle, error: Throwable) { + reportPrimary(handle, QueryShadowPrimarySignal.Error(errorBoundary.normalize(error) as QueryRejectedException)) + cancelProbe(handle) + } + + private fun cancelShadow(handle: QueryShadowHandle) { + reportPrimary(handle, QueryShadowPrimarySignal.Cancelled) + cancelProbe(handle) + } + + private fun cancelProbe(handle: QueryShadowHandle) { + try { + handle.cancelProbe() + } catch (_: RuntimeException) { + // Shadow cleanup cannot alter the primary result. + } + } + + private fun normalizeShadowMono(options: QueryExecutionOptions, probe: Mono): Mono = + deadlineEnforcer.enforceMono(shadowDeadline(options)) { probe }.onErrorMap(errorBoundary::normalize) + + private fun normalizeShadowFlux(options: QueryExecutionOptions, probe: Flux): Flux = + deadlineEnforcer.enforceFlux(shadowDeadline(options)) { probe }.onErrorMap(errorBoundary::normalize) + + private fun shadowDeadline(options: QueryExecutionOptions) = + deadlineEnforcer.cappedDeadline(options.deadline, shadowProbeTimeout) + + private fun backendMono(source: () -> Mono): Mono = + Mono.defer(source).onErrorMap(errorBoundary::normalizeBackend) + + private fun backendFlux(source: () -> Flux): Flux = + Flux.defer(source).onErrorMap(errorBoundary::normalizeBackend) + + private fun legacyMono(route: QueryExecutionRoute.Legacy, source: () -> Mono): Mono = Mono.defer { + notifyFallback(route) + source() + } + + private fun legacyFlux(route: QueryExecutionRoute.Legacy, source: () -> Flux): Flux = Flux.defer { + notifyFallback(route) + source() + } + + private fun notifyFallback(route: QueryExecutionRoute.Legacy) { + route.fallback?.let { fallback -> + try { + decisionObserver.onFallback(fallback) + } catch (_: RuntimeException) { + // Decision observability cannot alter the legacy result. + } + if (fallback.executionMode == QueryExecutionMode.SHADOW) { + notifyShadowSkipped(fallback.target, fallback.operation, fallback.issues) + } + } + } + + private fun notifyShadowSupervisorFailure( + task: QueryShadowTask, + issue: QueryRejection, + cause: Throwable? = null, + ) { + try { + decisionObserver.onShadowSupervisorFailure( + QueryShadowSupervisorFailure(task.descriptor(), issue, cause), + ) + } catch (_: RuntimeException) { + // Decision observability cannot alter the primary result. + } + } + + private fun notifyShadowSkipped( + target: QueryTarget, + operation: QueryOperation, + issues: NonEmptyList, + ) { + try { + shadowSupervisor.onSkipped(QueryShadowSkip(target, operation, issues)) + } catch (_: RuntimeException) { + // Shadow observability cannot alter the legacy result. + } + } + + private fun enforceLegacyStreamLimit( + input: LegacyCompilationInput, + source: Flux, + ): Flux { + val stream = input.invocation.input as? NormalizedQueryInput.Stream + ?: rejectExecution(QueryRejectionCode.EXECUTION_DECISION_INVALID) + val limit = if (stream.limit == 0) StreamLimit.Unbounded else StreamLimit.Bounded(stream.limit) + return enforceStreamLimit(source, limit) + } + + private fun enforceStreamLimit(source: Flux, limit: StreamLimit): Flux = + when (limit) { + StreamLimit.Unbounded -> source + is StreamLimit.Bounded -> source.index().map { indexed -> + if (indexed.t1 >= limit.value) { + rejectIncomplete() + } + indexed.t2 + } + } + + private fun requireCompletePage(plan: PageQueryPlan, page: BackendPage): BackendPage { + val expectedRecordCount = if (page.total <= plan.page.offset) { + 0 + } else { + minOf(plan.page.size.toLong(), page.total - plan.page.offset).toInt() + } + val completeEnvelope = page.totalRelation == BackendTotalRelation.EXACT && + page.consistency == BackendPageConsistency.SAME_INPUT && + page.records.size == expectedRecordCount && + page.records.size <= plan.page.size && + page.records.all { record -> record.completeness == BackendRecordCompleteness.COMPLETE } + if (!completeEnvelope) { + rejectIncomplete() + } + return page + } + + private fun requireLegacyPage(input: LegacyCompilationInput, page: BackendPage): BackendPage { + val pageInput = input.invocation.input as? NormalizedQueryInput.Page + ?: rejectExecution(QueryRejectionCode.EXECUTION_DECISION_INVALID) + if (page.records.size > pageInput.page.size) { + rejectIncomplete() + } + return page + } + + private fun requireRecordBackend(registration: QueryBackendRegistration): RecordQueryBackend = + registration.recordBackend + ?: registration.experimentalRecordBackend?.let(::ExperimentalRecordBackendAdapter) + ?: rejectExecution(QueryRejectionCode.BACKEND_OPERATION_UNSUPPORTED) + + private fun requireCompleteRecord(record: BackendRecord): BackendRecord { + if (record.completeness != BackendRecordCompleteness.COMPLETE) { + rejectIncomplete() + } + return record + } + + private fun requireNonNegativeCount(count: Long): Long { + if (count < 0) { + rejectIncomplete() + } + return count + } + + private fun AnalyticsConsistency.satisfies(required: AnalyticsConsistency): Boolean = + this == required || this == AnalyticsConsistency.SNAPSHOT && required == AnalyticsConsistency.EVENTUAL + + private fun AnalyticsCompleteness.satisfies(required: AnalyticsCompleteness): Boolean = + this == required || this == AnalyticsCompleteness.EXACT && required == AnalyticsCompleteness.APPROXIMATE + + private fun BackendAnalyticsPage.isCompleteFor(plan: AnalyticsQueryPlan): Boolean = + consistency.satisfies(plan.requiredConsistency) && + completeness.satisfies(plan.requiredCompleteness) && + buckets.size <= plan.bucketWindow.limit && + hasExpectedCursorState(plan) && + hasExpectedAnalyticsShape(plan) + + private fun BackendAnalyticsPage.hasExpectedCursorState(plan: AnalyticsQueryPlan): Boolean = when { + plan.grouping is me.ahoo.wow.query.internal.plan.PlannedAnalyticsGrouping.Global -> cursorState() == null + consistency == AnalyticsConsistency.EVENTUAL -> cursorState() == null + consistency == AnalyticsConsistency.SNAPSHOT -> cursorState() != null + else -> false + } + + private fun BackendAnalyticsPage.hasExpectedAnalyticsShape(plan: AnalyticsQueryPlan): Boolean { + val (dimensionAliases, cursorMatches) = when (val grouping = plan.grouping) { + me.ahoo.wow.query.internal.plan.PlannedAnalyticsGrouping.Global -> + emptySet() to (afterKey == null) + + is me.ahoo.wow.query.internal.plan.PlannedAnalyticsGrouping.By -> { + val aliases = grouping.dimensions.values.mapTo(LinkedHashSet()) { dimension -> dimension.alias } + aliases to (afterKey == null || afterKey.size == aliases.size) + } + } + val metricAliases = plan.metrics.values.mapTo(LinkedHashSet()) { metric -> metric.alias } + return cursorMatches && buckets.all { bucket -> + bucket.keys.keys == dimensionAliases && bucket.metrics.keys == metricAliases + } + } + + private inline fun requirePlan(plan: QueryPlan): P = + plan as? P ?: rejectExecution(QueryRejectionCode.EXECUTION_DECISION_INVALID) + + private fun rejectIncomplete(): Nothing = rejectQuery( + QueryRejectionCategory.INCOMPLETE_RESULT, + QueryRejectionPath.ROOT.property("backend").property("result"), + QueryRejectionCode.INCOMPLETE_RESULT, + ) + + private fun incompleteResult(): Mono = Mono.defer { rejectIncomplete() } + + private fun rejectExecution(code: QueryRejectionCode): Nothing = rejectQuery( + QueryRejectionCategory.INTERNAL_FAILURE, + QueryRejectionPath.ROOT.property("execution"), + code, + ) +} + +private fun shadowSupervisorUnavailable(): QueryRejection = QueryRejection( + QueryRejectionCategory.BACKEND_UNAVAILABLE, + QueryRejectionPath.ROOT.property("shadow").property("supervisor"), + QueryRejectionCode.SHADOW_SUPERVISOR_UNAVAILABLE, +) diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryGateway.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryGateway.kt new file mode 100644 index 00000000000..4795ee290d6 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/execution/QueryGateway.kt @@ -0,0 +1,656 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.wow.query.internal.admission.RawAdmissionGuard +import me.ahoo.wow.query.internal.analytics.AnalyticsBucketWindow +import me.ahoo.wow.query.internal.analytics.AnalyticsConsistency +import me.ahoo.wow.query.internal.analytics.DecodedAnalyticsCursor +import me.ahoo.wow.query.internal.cursor.PersistentQueryCursorLeaseCoordinator +import me.ahoo.wow.query.internal.cursor.QueryCursorBackendState +import me.ahoo.wow.query.internal.cursor.QueryCursorBudgetCeiling +import me.ahoo.wow.query.internal.cursor.QueryCursorCleanupReason +import me.ahoo.wow.query.internal.cursor.QueryCursorEnvelope +import me.ahoo.wow.query.internal.cursor.QueryCursorLeaseBinding +import me.ahoo.wow.query.internal.cursor.QueryCursorLeaseDescriptor +import me.ahoo.wow.query.internal.cursor.QueryCursorMappingDigest +import me.ahoo.wow.query.internal.cursor.QueryCursorPosition +import me.ahoo.wow.query.internal.cursor.QueryCursorSecurityContextDigest +import me.ahoo.wow.query.internal.cursor.QueryCursorSecurityDigest +import me.ahoo.wow.query.internal.cursor.QueryCursorToken +import me.ahoo.wow.query.internal.model.QueryInvocation +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.normalization.QueryNormalizer +import me.ahoo.wow.query.internal.plan.AnalyticsQueryPlan +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsGrouping +import me.ahoo.wow.query.internal.planning.PlanningConstraints +import me.ahoo.wow.query.internal.planning.PlanningDecision +import me.ahoo.wow.query.internal.planning.QueryPlanner +import me.ahoo.wow.query.internal.planning.ResultPlanningConstraint +import me.ahoo.wow.query.internal.policy.QueryExecutionBudget +import me.ahoo.wow.query.internal.policy.QueryExecutionContextFactory +import me.ahoo.wow.query.internal.policy.QueryExecutionRequest +import me.ahoo.wow.query.internal.policy.QueryPolicyEnforcer +import me.ahoo.wow.query.internal.policy.QueryPolicyInput +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import me.ahoo.wow.query.internal.schema.QuerySchemaRegistry +import me.ahoo.wow.query.internal.value.NonEmptyList +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.DateTimeException +import java.time.Duration +import me.ahoo.wow.query.backend.AnalyticsAlias as BackendAnalyticsAlias +import me.ahoo.wow.query.backend.PlanFingerprint as BackendPlanFingerprint +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias as InternalAnalyticsAlias +import me.ahoo.wow.query.internal.plan.PlanFingerprint as InternalPlanFingerprint + +internal fun interface QueryInvocationFactory { + fun create(): QueryInvocation +} + +internal data class AnalyticsExecutionPage( + val page: BackendAnalyticsPage, + val nextCursor: QueryCursorToken?, +) + +internal data class AnalyticsCursorRuntime( + val coordinator: PersistentQueryCursorLeaseCoordinator, + val leaseTtl: Duration, + val clock: Clock, +) { + init { + require(!leaseTtl.isZero && !leaseTtl.isNegative) + } +} + +internal class QueryGateway( + private val admissionGuard: RawAdmissionGuard, + private val normalizer: QueryNormalizer, + private val schemaRegistry: QuerySchemaRegistry, + private val contextFactory: QueryExecutionContextFactory, + private val policyEnforcer: QueryPolicyEnforcer, + private val planner: QueryPlanner, + private val routeResolver: QueryExecutionRouteResolver, + private val executor: QueryExecutor, + private val deadlineEnforcer: QueryDeadlineEnforcer, + private val analyticsCursorRuntime: AnalyticsCursorRuntime? = null, + private val errorBoundary: QueryErrorBoundary = QueryErrorBoundary(), + private val lifecycleMonitor: QueryLifecycleMonitor = QueryLifecycleMonitor(), +) { + fun single( + request: QueryExecutionRequest, + invocationFactory: QueryInvocationFactory, + ): Mono = singleResult(request, invocationFactory) { record -> record } + + fun singleResult( + request: QueryExecutionRequest, + invocationFactory: QueryInvocationFactory, + materialize: (BackendRecord) -> T, + ): Mono = observeMono(request, QueryOperation.SINGLE) { + prepare(request, QueryOperation.SINGLE, invocationFactory) + .flatMap { prepared -> executor.single(prepared.route, prepared.options) } + .map(materialize) + } + + fun stream( + request: QueryExecutionRequest, + invocationFactory: QueryInvocationFactory, + ): Flux = streamResult(request, invocationFactory) { record -> record } + + fun streamResult( + request: QueryExecutionRequest, + invocationFactory: QueryInvocationFactory, + materialize: (BackendRecord) -> T, + ): Flux = observeFlux(request, QueryOperation.STREAM) { + prepare(request, QueryOperation.STREAM, invocationFactory) + .flatMapMany { prepared -> executor.stream(prepared.route, prepared.options) } + .map(materialize) + } + + fun page( + request: QueryExecutionRequest, + invocationFactory: QueryInvocationFactory, + ): Mono = pageResult(request, invocationFactory) { page -> page } + + fun pageResult( + request: QueryExecutionRequest, + invocationFactory: QueryInvocationFactory, + materialize: (BackendPage) -> T, + ): Mono = observeMono(request, QueryOperation.PAGE) { + prepare(request, QueryOperation.PAGE, invocationFactory) + .flatMap { prepared -> executor.page(prepared.route, prepared.options) } + .map(materialize) + } + + fun count( + request: QueryExecutionRequest, + invocationFactory: QueryInvocationFactory, + ): Mono = observeMono(request, QueryOperation.COUNT) { + prepare(request, QueryOperation.COUNT, invocationFactory) + .flatMap { prepared -> executor.count(prepared.route, prepared.options) } + } + + fun analyze( + request: QueryExecutionRequest, + invocationFactory: QueryInvocationFactory, + ): Mono = observeMono(request, QueryOperation.ANALYZE) { + prepare(request, QueryOperation.ANALYZE, invocationFactory) + .flatMap { prepared -> executor.analyze(prepared.route, prepared.options) } + } + + fun analyzePublic( + request: QueryExecutionRequest, + invocationFactory: QueryInvocationFactory, + ): Mono = observeMono(request, QueryOperation.ANALYZE) { + preparePlanning(request, QueryOperation.ANALYZE, invocationFactory).flatMap(::executePublicAnalytics) + } + + fun reapExpiredAnalyticsCursors(batchSize: Int): Mono { + require(batchSize > 0) { "Query cursor reaper batch size must be positive." } + val runtime = analyticsCursorRuntime + ?: return Mono.error(IllegalStateException("Persistent Query cursor runtime is not configured.")) + return runtime.coordinator.reapExpired(runtime.clock.instant(), batchSize) + } + + private fun executePublicAnalytics(prepared: PreparedPlanning): Mono { + val input = prepared.normalized.input as? NormalizedQueryInput.Analytics ?: rejectInvalidInvocation() + val plan = requireAnalyticsPlan(prepared) + val cursorRuntime = analyticsCursorRuntime + requireAnalyticsCursorStore(plan, cursorRuntime) + val route = resolvePlannedRoute(prepared, plan) + val registration = route.registry.resolve(plan) + requireSnapshotCursorLifecycle(plan, registration, cursorRuntime) + val mappingDigest = QueryCursorMappingDigest(registration.descriptor.mappingGenerationDigest) + val securityDigest = QueryCursorSecurityDigest.compute(prepared.context) + val token = input.query.cursorToken + return if (token == null) { + withCursorSession( + CursorExecutionSession( + cursorRuntime?.coordinator, + null, + QueryCursorLeaseDescriptor( + plan.target, + registration.descriptor.key.backendId, + mappingDigest, + ), + ), + ) { session -> + executeAnalyticsPage( + prepared, + route, + plan, + mappingDigest, + securityDigest, + currentPage = 1, + session, + ) + } + } else { + continuePublicAnalytics( + prepared, + plan, + mappingDigest, + securityDigest, + registration, + requireNotNull(cursorRuntime), + token, + input.query.bucketWindow.limit, + ) + } + } + + private fun requireAnalyticsCursorStore( + plan: AnalyticsQueryPlan, + cursorRuntime: AnalyticsCursorRuntime?, + ) { + if (plan.grouping is PlannedAnalyticsGrouping.By && cursorRuntime == null) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + CURSOR_PATH, + QueryRejectionCode.CURSOR_STORE_REQUIRED, + ) + } + } + + private fun requireSnapshotCursorLifecycle( + plan: AnalyticsQueryPlan, + registration: QueryBackendRegistration, + cursorRuntime: AnalyticsCursorRuntime?, + ) { + val requiresLifecycle = + plan.requiredConsistency == AnalyticsConsistency.SNAPSHOT && plan.grouping is PlannedAnalyticsGrouping.By + val supportsLifecycle = cursorRuntime?.coordinator + ?.supports(plan.target, registration.descriptor.key.backendId) == true + if (requiresLifecycle && !supportsLifecycle) { + rejectQuery( + QueryRejectionCategory.BACKEND_UNAVAILABLE, + QueryRejectionPath.ROOT.property("backend"), + QueryRejectionCode.BACKEND_OPERATION_UNSUPPORTED, + ) + } + } + + private fun continuePublicAnalytics( + prepared: PreparedPlanning, + basePlan: AnalyticsQueryPlan, + mappingDigest: QueryCursorMappingDigest, + securityDigest: QueryCursorSecurityContextDigest, + registration: QueryBackendRegistration, + cursorRuntime: AnalyticsCursorRuntime, + token: QueryCursorToken, + limit: Int, + ): Mono = cursorRuntime.coordinator.load(token).flatMap { loaded -> + val envelope = loaded.envelope + enforceCursorContinuationBudget(envelope, prepared.context.budget, envelope.pageNumber) + val expectedBinding = QueryCursorLeaseBinding( + basePlan.target, + BackendPlanFingerprint(basePlan.fingerprint.value), + mappingDigest, + securityDigest, + registration.descriptor.key.backendId, + ) + val continued = continueAnalytics(prepared, envelope, limit) + val continuedPlan = requireAnalyticsPlan(continued) + val continuedRoute = resolvePlannedRoute(continued, continuedPlan) + val continuedRegistration = continuedRoute.registry.resolve(continuedPlan) + if (continuedRegistration.descriptor.key != registration.descriptor.key || + continuedRegistration.descriptor.mappingGenerationDigest != mappingDigest.value + ) { + rejectInvalidCursorBinding() + } + cursorRuntime.coordinator.acquire(loaded, expectedBinding).flatMap { acquired -> + withCursorSession( + CursorExecutionSession( + cursorRuntime.coordinator, + acquired, + QueryCursorLeaseDescriptor( + continuedPlan.target, + continuedRegistration.descriptor.key.backendId, + mappingDigest, + ), + ), + ) { session -> + executeAnalyticsPage( + continued, + continuedRoute, + continuedPlan, + mappingDigest, + securityDigest, + envelope.pageNumber, + session, + ) + } + } + } + + private fun prepare( + request: QueryExecutionRequest, + expectedOperation: QueryOperation, + invocationFactory: QueryInvocationFactory, + ): Mono = preparePlanning(request, expectedOperation, invocationFactory).map { prepared -> + PreparedExecution(resolveRoute(prepared), QueryExecutionOptions.from(prepared.context)) + } + + private fun preparePlanning( + request: QueryExecutionRequest, + expectedOperation: QueryOperation, + invocationFactory: QueryInvocationFactory, + ): Mono = Mono.defer { + val normalized = freezeAndNormalize(request, expectedOperation, invocationFactory) + contextFactory.resolve(request).flatMap { context -> + val schema = schemaRegistry[normalized.target] + ?: rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionPath.ROOT.property("target"), + QueryRejectionCode.SCHEMA_NOT_REGISTERED, + ) + policyEnforcer.authorize(QueryPolicyInput(context, normalized, schema)).map { constraints -> + val effectiveConstraints = constraints.constrain(context.budget) + PreparedPlanning( + context, + normalized, + schema, + effectiveConstraints, + planner.plan(normalized, schema, effectiveConstraints), + ) + } + } + } + + private fun freezeAndNormalize( + request: QueryExecutionRequest, + expectedOperation: QueryOperation, + invocationFactory: QueryInvocationFactory, + ): NormalizedQueryInvocation { + val invocation = invocationFactory.create() + if (invocation.target != request.target || invocation.operation != expectedOperation) { + rejectQuery( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionPath.ROOT.property("input"), + QueryRejectionCode.INVALID_INVOCATION, + ) + } + return normalizer.normalize(admissionGuard.admit(invocation)) + } + + private fun observeMono( + request: QueryExecutionRequest, + operation: QueryOperation, + source: () -> Mono, + ): Mono { + val normalized = deadlineEnforcer.enforceMono(request.deadline, source) + .onErrorMap(errorBoundary::normalize) + return lifecycleMonitor.observeMono(QueryLifecycleDescriptor(request, operation), normalized) + } + + private fun observeFlux( + request: QueryExecutionRequest, + operation: QueryOperation, + source: () -> Flux, + ): Flux { + val normalized = deadlineEnforcer.enforceFlux(request.deadline, source) + .onErrorMap(errorBoundary::normalize) + return lifecycleMonitor.observeFlux(QueryLifecycleDescriptor(request, operation), normalized) + } + + private data class PreparedExecution( + val route: QueryExecutionRoute, + val options: QueryExecutionOptions, + ) + + private data class PreparedPlanning( + val context: me.ahoo.wow.query.internal.policy.QueryExecutionContext, + val normalized: NormalizedQueryInvocation, + val schema: me.ahoo.wow.query.backend.QueryDocumentSchema, + val constraints: PlanningConstraints, + val decision: PlanningDecision, + ) + + private fun resolveRoute( + prepared: PreparedPlanning, + plan: AnalyticsQueryPlan? = null, + ): QueryExecutionRoute = routeResolver.resolve( + prepared.context, + prepared.normalized, + prepared.schema, + plan?.let { PlanningDecision.Planned(it) } ?: prepared.decision, + ) + + private fun resolvePlannedRoute( + prepared: PreparedPlanning, + plan: AnalyticsQueryPlan, + ): QueryExecutionRoute.Planned = + resolveRoute(prepared, plan) as? QueryExecutionRoute.Planned ?: rejectInvalidExecutionDecision() + + private fun requireAnalyticsPlan(prepared: PreparedPlanning): AnalyticsQueryPlan = + (prepared.decision as? PlanningDecision.Planned)?.plan as? AnalyticsQueryPlan + ?: rejectInvalidExecutionDecision() + + private fun continueAnalytics( + prepared: PreparedPlanning, + envelope: QueryCursorEnvelope, + limit: Int, + ): PreparedPlanning { + val currentInput = prepared.normalized.input as? NormalizedQueryInput.Analytics ?: rejectInvalidInvocation() + val position = envelope.position as? QueryCursorPosition.Analytics ?: rejectInvalidCursorBinding() + val cursor = DecodedAnalyticsCursor( + prepared.normalized.target, + InternalPlanFingerprint(envelope.planFingerprint.value), + requireNotNull( + NonEmptyList.from(position.dimensionAliases.map { alias -> InternalAnalyticsAlias(alias.value) }), + ), + requireNotNull(NonEmptyList.from(position.afterKey)), + ) + val continuedInvocation = prepared.normalized.copy( + input = NormalizedQueryInput.Analytics( + currentInput.query.copy( + bucketWindow = AnalyticsBucketWindow.After(limit, cursor), + cursorToken = null, + ), + ), + ) + return prepared.copy( + normalized = continuedInvocation, + decision = planner.plan(continuedInvocation, prepared.schema, prepared.constraints), + ) + } + + private fun executeAnalyticsPage( + prepared: PreparedPlanning, + route: QueryExecutionRoute.Planned, + plan: AnalyticsQueryPlan, + mappingDigest: QueryCursorMappingDigest, + securityDigest: me.ahoo.wow.query.internal.cursor.QueryCursorSecurityContextDigest, + currentPage: Int, + session: CursorExecutionSession, + ): Mono = executor.analyze( + route, + QueryExecutionOptions.from(prepared.context), + session.cursorState(), + ).flatMap { page -> + issueNextCursor(prepared, plan, route, page, mappingDigest, securityDigest, currentPage, session) + } + + private fun issueNextCursor( + prepared: PreparedPlanning, + plan: AnalyticsQueryPlan, + route: QueryExecutionRoute.Planned, + page: BackendAnalyticsPage, + mappingDigest: QueryCursorMappingDigest, + securityDigest: me.ahoo.wow.query.internal.cursor.QueryCursorSecurityContextDigest, + currentPage: Int, + session: CursorExecutionSession, + ): Mono { + val registration = route.registry.resolve(plan) + val backendState = page.cursorState()?.let { payload -> + QueryCursorBackendState(registration.descriptor.key.backendId, payload) + } + val afterKey = page.afterKey + if (afterKey == null) { + return session.closeTerminal(backendState).thenReturn(AnalyticsExecutionPage(page, null)) + } + enforceCursorPageBudget(prepared.context.budget, currentPage + 1) + val grouping = plan.grouping as? PlannedAnalyticsGrouping.By ?: rejectInvalidCursorBinding() + val runtime = analyticsCursorRuntime ?: rejectCursorStoreRequired() + val envelope = QueryCursorEnvelope( + target = plan.target, + planFingerprint = BackendPlanFingerprint(plan.fingerprint.value), + mappingGenerationDigest = mappingDigest, + securityContextDigest = securityDigest, + position = QueryCursorPosition.Analytics( + grouping.dimensions.values.map { dimension -> BackendAnalyticsAlias(dimension.alias.value) }, + afterKey, + ), + expiresAt = cursorExpiry(runtime), + backendState = backendState, + pageNumber = currentPage + 1, + budgetCeiling = session.nextBudgetCeiling(prepared.context.budget), + backendId = registration.descriptor.key.backendId, + ) + return session.issue(envelope).map { token -> AnalyticsExecutionPage(page, token) } + } +} + +private fun cursorExpiry(runtime: AnalyticsCursorRuntime): java.time.Instant = try { + runtime.clock.instant().plus(runtime.leaseTtl) +} catch (error: DateTimeException) { + throw IllegalStateException("Cursor lease expiry cannot be represented.", error) +} catch (error: ArithmeticException) { + throw IllegalStateException("Cursor lease expiry cannot be represented.", error) +} + +private fun withCursorSession( + session: CursorExecutionSession, + execute: (CursorExecutionSession) -> Mono, +): Mono = Mono.usingWhen( + Mono.just(session), + execute, + CursorExecutionSession::cleanupTerminal, + { current, _ -> current.cleanupTerminal() }, + CursorExecutionSession::cleanupTerminal, +) + +private class CursorExecutionSession( + private val coordinator: PersistentQueryCursorLeaseCoordinator?, + current: QueryCursorEnvelope?, + private val descriptor: QueryCursorLeaseDescriptor, +) { + private var active: QueryCursorEnvelope? = current + private var settled = false + + fun cursorState(): ByteArray? = active?.backendState?.payload() + + fun nextBudgetCeiling(requested: QueryExecutionBudget): QueryCursorBudgetCeiling = + active?.budgetCeiling?.tighten(requested) ?: requested.toCursorBudgetCeiling() + + fun issue(envelope: QueryCursorEnvelope): Mono { + val owner = coordinator ?: rejectCursorStoreRequired() + active = envelope + return owner.issue(envelope).doOnNext { settled = true } + } + + fun closeTerminal(state: QueryCursorBackendState?): Mono { + val current = active + active = current?.copy(backendState = state) + val envelope = active + if (envelope == null) { + if (state != null) { + val owner = coordinator ?: rejectCursorStoreRequired() + return owner.close(state, descriptor, QueryCursorCleanupReason.TERMINAL).doOnSuccess { settled = true } + } + settled = true + return Mono.empty() + } + val owner = coordinator ?: rejectCursorStoreRequired() + return owner.close(envelope, QueryCursorCleanupReason.TERMINAL).doOnSuccess { settled = true } + } + + fun cleanupTerminal(): Mono { + if (settled) return Mono.empty() + settled = true + val envelope = active ?: return Mono.empty() + return coordinator?.close(envelope, QueryCursorCleanupReason.TERMINAL) ?: Mono.empty() + } +} + +private fun enforceCursorPageBudget(budget: QueryExecutionBudget, pageNumber: Int) { + if (budget.maxCursorPages?.let { maximum -> pageNumber > maximum } == true) { + rejectQuery( + QueryRejectionCategory.BUDGET_EXCEEDED, + CURSOR_PATH, + QueryRejectionCode.CURSOR_PAGE_LIMIT_EXCEEDED, + ) + } +} + +private fun enforceCursorContinuationBudget( + envelope: QueryCursorEnvelope, + budget: QueryExecutionBudget, + pageNumber: Int, +) { + if (!envelope.budgetCeiling.allows(budget)) { + rejectQuery( + QueryRejectionCategory.BUDGET_EXCEEDED, + CURSOR_PATH, + QueryRejectionCode.CURSOR_BUDGET_RELAXATION_NOT_ALLOWED, + ) + } + enforceCursorPageBudget(budget, pageNumber) +} + +private fun QueryExecutionBudget.toCursorBudgetCeiling(): QueryCursorBudgetCeiling = QueryCursorBudgetCeiling( + maxScannedRecords, + maxReturnedRecords, + maxPageWindow, + maxCandidateBuckets, + maxReturnedBuckets, + maxCursorPages, + allowDiskUse, +) + +private fun QueryCursorBudgetCeiling.allows(requested: QueryExecutionBudget): Boolean = + allows(maxScannedRecords, requested.maxScannedRecords) && + allows(maxReturnedRecords, requested.maxReturnedRecords) && + allows(maxPageWindow, requested.maxPageWindow) && + allows(maxCandidateBuckets, requested.maxCandidateBuckets) && + allows(maxReturnedBuckets, requested.maxReturnedBuckets) && + allows(maxCursorPages, requested.maxCursorPages) && + (allowDiskUse || !requested.allowDiskUse) + +private fun QueryCursorBudgetCeiling.tighten(requested: QueryExecutionBudget): QueryCursorBudgetCeiling = + QueryCursorBudgetCeiling( + tighten(maxScannedRecords, requested.maxScannedRecords), + tighten(maxReturnedRecords, requested.maxReturnedRecords), + tighten(maxPageWindow, requested.maxPageWindow), + tighten(maxCandidateBuckets, requested.maxCandidateBuckets), + tighten(maxReturnedBuckets, requested.maxReturnedBuckets), + tighten(maxCursorPages, requested.maxCursorPages), + allowDiskUse && requested.allowDiskUse, + ) + +private fun > allows(initial: T?, requested: T?): Boolean = + initial == null || requested != null && requested <= initial + +private fun > tighten(initial: T?, requested: T?): T? = when { + initial == null -> requested + requested == null -> initial + else -> minOf(initial, requested) +} + +private fun rejectCursorStoreRequired(): Nothing = rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + CURSOR_PATH, + QueryRejectionCode.CURSOR_STORE_REQUIRED, +) + +private fun rejectInvalidCursorBinding(): Nothing = rejectQuery( + QueryRejectionCategory.INVALID_CURSOR, + CURSOR_PATH, + QueryRejectionCode.INVALID_CURSOR_BINDING, +) + +private fun rejectInvalidExecutionDecision(): Nothing = rejectQuery( + QueryRejectionCategory.INTERNAL_FAILURE, + QueryRejectionPath.ROOT.property("execution"), + QueryRejectionCode.EXECUTION_DECISION_INVALID, +) + +private fun rejectInvalidInvocation(): Nothing = rejectQuery( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionPath.ROOT.property("input"), + QueryRejectionCode.INVALID_INVOCATION, +) + +private val CURSOR_PATH = QueryRejectionPath.ROOT.property("cursor") + +private fun PlanningConstraints.constrain(budget: QueryExecutionBudget): PlanningConstraints { + val result = budget.maxReturnedRecords?.let { requestedMaximum -> + val policyMaximum = (resultConstraint as? ResultPlanningConstraint.MaximumRecords)?.value + val effectiveMaximum = policyMaximum?.coerceAtMost(requestedMaximum) ?: requestedMaximum + ResultPlanningConstraint.MaximumRecords(effectiveMaximum) + } ?: resultConstraint + val page = budget.maxPageWindow?.let { requestedMaximum -> + val policyMaximum = (pageConstraint as? me.ahoo.wow.query.internal.planning.PagePlanningConstraint.MaximumWindow) + ?.value + val effectiveMaximum = policyMaximum?.coerceAtMost(requestedMaximum) ?: requestedMaximum + me.ahoo.wow.query.internal.planning.PagePlanningConstraint.MaximumWindow(effectiveMaximum) + } ?: pageConstraint + return copy(resultConstraint = result, pageConstraint = page) +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/DefaultQueryGateway.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/DefaultQueryGateway.kt new file mode 100644 index 00000000000..24e9f902d36 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/DefaultQueryGateway.kt @@ -0,0 +1,558 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.internal.gateway + +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DynamicDocument +import me.ahoo.wow.api.query.IListQuery +import me.ahoo.wow.api.query.IPagedQuery +import me.ahoo.wow.api.query.ISingleQuery +import me.ahoo.wow.api.query.PagedList +import me.ahoo.wow.api.query.SimpleDynamicDocument +import me.ahoo.wow.api.query.analytics.AnalyticsBucket +import me.ahoo.wow.api.query.analytics.AnalyticsCompleteness +import me.ahoo.wow.api.query.analytics.AnalyticsConsistency +import me.ahoo.wow.api.query.analytics.AnalyticsCursor +import me.ahoo.wow.api.query.analytics.AnalyticsPage +import me.ahoo.wow.api.query.analytics.AnalyticsQuery +import me.ahoo.wow.api.query.analytics.AnalyticsValue +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.gateway.AnalyticsQueryGateway +import me.ahoo.wow.query.gateway.QueryAuthorityRequest +import me.ahoo.wow.query.gateway.QueryAuthorityResolver +import me.ahoo.wow.query.gateway.QueryCall +import me.ahoo.wow.query.gateway.QueryErrorCategory +import me.ahoo.wow.query.gateway.QueryExecutionException +import me.ahoo.wow.query.gateway.QueryExecutionProfile +import me.ahoo.wow.query.gateway.QueryExecutionProfiles +import me.ahoo.wow.query.gateway.QueryGateway +import me.ahoo.wow.query.gateway.QueryResultMaterializer +import me.ahoo.wow.query.internal.execution.BackendPage +import me.ahoo.wow.query.internal.execution.BackendRecord +import me.ahoo.wow.query.internal.execution.QueryBackendException +import me.ahoo.wow.query.internal.execution.QueryBackendFailureKind +import me.ahoo.wow.query.internal.model.QueryInput +import me.ahoo.wow.query.internal.model.QueryInvocation +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.policy.QueryAuthorityProvider +import me.ahoo.wow.query.internal.policy.QueryExecutionRequest +import me.ahoo.wow.query.internal.policy.TrustedAuthorityRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.util.LinkedHashMap +import me.ahoo.wow.query.gateway.QueryAuthority as PublicQueryAuthority +import me.ahoo.wow.query.gateway.QueryDocumentKind as PublicDocumentKind +import me.ahoo.wow.query.gateway.QueryExecutionMode as PublicExecutionMode +import me.ahoo.wow.query.gateway.QueryOwnerGrant as PublicOwnerGrant +import me.ahoo.wow.query.gateway.QuerySpaceGrant as PublicSpaceGrant +import me.ahoo.wow.query.gateway.QueryValidationMode as PublicValidationMode +import me.ahoo.wow.query.internal.execution.QueryGateway as InternalQueryGateway +import me.ahoo.wow.query.internal.model.QueryDocumentKind as InternalDocumentKind +import me.ahoo.wow.query.internal.model.QueryExecutionMode as InternalExecutionMode +import me.ahoo.wow.query.internal.model.QueryTarget as InternalQueryTarget +import me.ahoo.wow.query.internal.model.QueryValidationMode as InternalValidationMode +import me.ahoo.wow.query.internal.policy.QueryAuthority as InternalQueryAuthority +import me.ahoo.wow.query.internal.policy.QueryExecutionBudget as InternalExecutionBudget +import me.ahoo.wow.query.internal.policy.QueryOwnerGrant as InternalOwnerGrant +import me.ahoo.wow.query.internal.policy.QueryPurpose as InternalQueryPurpose +import me.ahoo.wow.query.internal.policy.QueryResourceScope as InternalResourceScope +import me.ahoo.wow.query.internal.policy.QuerySpaceGrant as InternalSpaceGrant + +internal class DefaultQueryGateway( + private val delegate: InternalQueryGateway, + private val executionProfiles: QueryExecutionProfiles, + resultMaterializers: Iterable>, +) : QueryGateway { + private val materializers = TargetMaterializerRegistry(resultMaterializers) + + override fun single(call: QueryCall, query: ISingleQuery): Mono = + delegate.singleResult( + call.toRequest(executionProfiles.resolve(call.target, me.ahoo.wow.query.gateway.QueryOperation.SINGLE)), + { + QueryInvocation( + call.target.toInternal(), + QueryOperation.SINGLE, + QueryResultShape.DYNAMIC, + QueryInput.Single(query), + ) + }, + BackendRecord::toDynamicDocument, + ).mapError() + + override fun single( + call: QueryCall, + query: ISingleQuery, + resultType: Class, + ): Mono = delegate.singleResult( + call.toRequest(executionProfiles.resolve(call.target, me.ahoo.wow.query.gateway.QueryOperation.SINGLE)), + { + materializers.require(call.target, resultType) + QueryInvocation( + call.target.toInternal(), + QueryOperation.SINGLE, + QueryResultShape.TYPED, + QueryInput.Single(query), + ) + }, + ) { record -> materializers.materialize(call.target, resultType, record) }.mapError() + + override fun stream(call: QueryCall, query: IListQuery): Flux = + delegate.streamResult( + call.toRequest(executionProfiles.resolve(call.target, me.ahoo.wow.query.gateway.QueryOperation.STREAM)), + { + QueryInvocation( + call.target.toInternal(), + QueryOperation.STREAM, + QueryResultShape.DYNAMIC, + QueryInput.Stream(query), + ) + }, + BackendRecord::toDynamicDocument, + ).mapError() + + override fun stream( + call: QueryCall, + query: IListQuery, + resultType: Class, + ): Flux = delegate.streamResult( + call.toRequest(executionProfiles.resolve(call.target, me.ahoo.wow.query.gateway.QueryOperation.STREAM)), + { + materializers.require(call.target, resultType) + QueryInvocation( + call.target.toInternal(), + QueryOperation.STREAM, + QueryResultShape.TYPED, + QueryInput.Stream(query), + ) + }, + ) { record -> materializers.materialize(call.target, resultType, record) }.mapError() + + override fun page(call: QueryCall, query: IPagedQuery): Mono> = + delegate.pageResult( + call.toRequest(executionProfiles.resolve(call.target, me.ahoo.wow.query.gateway.QueryOperation.PAGE)), + { + QueryInvocation( + call.target.toInternal(), + QueryOperation.PAGE, + QueryResultShape.DYNAMIC, + QueryInput.Page(query), + ) + }, + BackendPage::toDynamicPage, + ).mapError() + + override fun page( + call: QueryCall, + query: IPagedQuery, + resultType: Class, + ): Mono> = delegate.pageResult( + call.toRequest(executionProfiles.resolve(call.target, me.ahoo.wow.query.gateway.QueryOperation.PAGE)), + { + materializers.require(call.target, resultType) + QueryInvocation( + call.target.toInternal(), + QueryOperation.PAGE, + QueryResultShape.TYPED, + QueryInput.Page(query), + ) + }, + ) { page -> + PagedList( + page.total, + page.records.map { record -> materializers.materialize(call.target, resultType, record) }, + ) + }.mapError() + + override fun count(call: QueryCall, condition: Condition): Mono = + delegate.count( + call.toRequest(executionProfiles.resolve(call.target, me.ahoo.wow.query.gateway.QueryOperation.COUNT)), + ) { + QueryInvocation( + call.target.toInternal(), + QueryOperation.COUNT, + QueryResultShape.COUNT, + QueryInput.Count(condition), + ) + }.mapError() +} + +internal class DefaultAnalyticsQueryGateway( + private val delegate: InternalQueryGateway, + private val executionProfiles: QueryExecutionProfiles, +) : AnalyticsQueryGateway { + override fun analyze(call: QueryCall, query: AnalyticsQuery): Mono = + delegate.analyzePublic( + call.toRequest(executionProfiles.resolve(call.target, me.ahoo.wow.query.gateway.QueryOperation.ANALYZE)), + ) { + QueryInvocation( + call.target.toInternal(), + QueryOperation.ANALYZE, + QueryResultShape.ANALYTICS, + QueryInput.AnalyticsWire(query), + ) + }.map { result -> + val decimalMetricAliases = query.metrics + .filterNot { metric -> metric.kind == me.ahoo.wow.api.query.analytics.AnalyticsMetricKind.DOCUMENT_COUNT } + .mapTo(linkedSetOf()) { metric -> metric.alias } + val numericScale = query.numericPolicy?.scale + AnalyticsPage( + result.page.buckets.map { bucket -> + AnalyticsBucket( + bucket.keys.mapKeys { entry -> entry.key.value } + .mapValues { entry -> entry.value.toAnalyticsValue() }, + bucket.metrics.mapKeys { entry -> entry.key.value } + .mapValues { entry -> + entry.value.toAnalyticsValue( + numericScale.takeIf { entry.key in decimalMetricAliases }, + ) + }, + ) + }, + result.nextCursor?.let { token -> AnalyticsCursor(token.value) }, + AnalyticsConsistency.valueOf(result.page.consistency.name), + AnalyticsCompleteness.valueOf(result.page.completeness.name), + ) + }.mapError() +} + +internal class GatewayAuthorityProvider( + private val trustedAuthorityChannel: TrustedAuthorityChannel, + private val resolver: QueryAuthorityResolver, +) : QueryAuthorityProvider { + override fun resolve(request: QueryExecutionRequest): Mono = + Mono.deferContextual { context -> + trustedAuthorityChannel.read(context)?.let { authority -> + Mono.just(authority.toInternal()) + } ?: Mono.defer { + resolver.resolve(request.toPublic()).map(PublicQueryAuthority::toInternal) + } + }.onErrorMap(::mapTrustedAuthorityRejection) +} + +private fun mapTrustedAuthorityRejection(error: Throwable): Throwable { + if (error !is QueryExecutionException || error.category != QueryErrorCategory.ACCESS_DENIED) { + return error + } + val (path, code) = when { + error.path == AUTHORITY_PATH && error.code == QueryRejectionCode.AUTHORITY_REQUIRED.name -> + QueryRejectionPath.ROOT.property("executionContext").property("authority") to + QueryRejectionCode.AUTHORITY_REQUIRED + + error.path == LEGACY_GRANT_PATH && error.code == QueryRejectionCode.LEGACY_CALLER_NOT_ALLOWED.name -> + QueryRejectionPath.ROOT.property("executionContext").property("legacyGrant") to + QueryRejectionCode.LEGACY_CALLER_NOT_ALLOWED + + error.path == TRANSPORT_PATH && error.code == QueryRejectionCode.QUERY_TRANSPORT_AUTHORITY_MISMATCH.name -> + QueryRejectionPath.ROOT.property("executionContext").property("transport") to + QueryRejectionCode.QUERY_TRANSPORT_AUTHORITY_MISMATCH + + else -> return error + } + return TrustedAuthorityRejectedException(path, code, error) +} + +private fun QueryCall.toRequest(profile: QueryExecutionProfile): QueryExecutionRequest = + QueryExecutionRequest( + target = target.toInternal(), + purpose = InternalQueryPurpose(purpose.value), + executionMode = profile.executionMode.toInternal(), + validationMode = profile.validationMode.toInternal(), + resourceScope = InternalResourceScope( + resourceScope.tenantId, + resourceScope.ownerId, + resourceScope.spaceId, + ), + deadline = deadline, + budget = InternalExecutionBudget( + maxReturnedRecords = budget.maxReturnedRecords, + maxScannedRecords = budget.maxScannedRecords, + maxPageWindow = budget.maxPageWindow, + maxCandidateBuckets = budget.maxCandidateBuckets, + maxReturnedBuckets = budget.maxReturnedBuckets, + maxCursorPages = budget.maxCursorPages, + allowDiskUse = budget.allowDiskUse, + ), + ) + +private fun QueryExecutionRequest.toPublic(): QueryAuthorityRequest = + QueryAuthorityRequest( + call = QueryCall( + target = me.ahoo.wow.query.gateway.QueryTarget( + target.namedAggregate, + when (target.documentKind) { + InternalDocumentKind.SNAPSHOT -> PublicDocumentKind.SNAPSHOT + InternalDocumentKind.EVENT_STREAM -> PublicDocumentKind.EVENT_STREAM + }, + ), + purpose = me.ahoo.wow.query.gateway.QueryPurpose(purpose.value), + resourceScope = me.ahoo.wow.query.gateway.QueryResourceScope( + resourceScope.tenantId, + resourceScope.ownerId, + resourceScope.spaceId, + ), + deadline = deadline, + budget = me.ahoo.wow.query.gateway.QueryExecutionBudget( + maxReturnedRecords = budget.maxReturnedRecords, + maxScannedRecords = budget.maxScannedRecords, + maxPageWindow = budget.maxPageWindow, + maxCandidateBuckets = budget.maxCandidateBuckets, + maxReturnedBuckets = budget.maxReturnedBuckets, + maxCursorPages = budget.maxCursorPages, + allowDiskUse = budget.allowDiskUse, + ), + ), + executionMode = executionMode.toPublic(), + validationMode = validationMode.toPublic(), + ) + +private fun PublicQueryAuthority.toInternal(): InternalQueryAuthority = + when (this) { + is PublicQueryAuthority.Subject -> InternalQueryAuthority.Subject( + subjectId, + tenantId, + ownerGrant.toInternal(), + spaceGrant.toInternal(), + ) + + is PublicQueryAuthority.Service -> InternalQueryAuthority.Service( + serviceId, + tenantId, + purposes.mapTo(LinkedHashSet()) { InternalQueryPurpose(it.value) }, + ) + + is PublicQueryAuthority.System -> InternalQueryAuthority.System(principalId, justification) + + is PublicQueryAuthority.Legacy -> InternalQueryAuthority.Legacy( + me.ahoo.wow.query.internal.policy.LegacyQueryGrant( + me.ahoo.wow.query.internal.policy.LegacyQueryCallerId(grant.callerId), + grant.target.toInternal(), + me.ahoo.wow.query.internal.policy.QueryPurpose(grant.purpose.value), + grant.executionMode.toInternal(), + me.ahoo.wow.query.internal.policy.QueryResourceScope( + grant.resourceScope.tenantId, + grant.resourceScope.ownerId, + grant.resourceScope.spaceId, + ), + ), + ) + } + +private fun PublicOwnerGrant.toInternal(): InternalOwnerGrant = + when (this) { + PublicOwnerGrant.Unrestricted -> InternalOwnerGrant.Unrestricted + is PublicOwnerGrant.Only -> InternalOwnerGrant.Only(ownerId) + } + +private fun PublicSpaceGrant.toInternal(): InternalSpaceGrant = + when (this) { + PublicSpaceGrant.Unrestricted -> InternalSpaceGrant.Unrestricted + PublicSpaceGrant.DenyAll -> InternalSpaceGrant.DenyAll + is PublicSpaceGrant.AllowList -> InternalSpaceGrant.AllowList(spaceIds) + } + +private fun me.ahoo.wow.query.gateway.QueryTarget.toInternal(): InternalQueryTarget = + InternalQueryTarget( + namedAggregate, + when (documentKind) { + PublicDocumentKind.SNAPSHOT -> InternalDocumentKind.SNAPSHOT + PublicDocumentKind.EVENT_STREAM -> InternalDocumentKind.EVENT_STREAM + }, + ) + +private fun PublicExecutionMode.toInternal(): InternalExecutionMode = + when (this) { + PublicExecutionMode.LEGACY -> InternalExecutionMode.LEGACY + PublicExecutionMode.SHADOW -> InternalExecutionMode.SHADOW + PublicExecutionMode.PLANNED -> InternalExecutionMode.PLANNED + } + +private fun InternalExecutionMode.toPublic(): PublicExecutionMode = + when (this) { + InternalExecutionMode.LEGACY -> PublicExecutionMode.LEGACY + InternalExecutionMode.SHADOW -> PublicExecutionMode.SHADOW + InternalExecutionMode.PLANNED -> PublicExecutionMode.PLANNED + } + +private fun PublicValidationMode.toInternal(): InternalValidationMode = + when (this) { + PublicValidationMode.COMPATIBLE -> InternalValidationMode.COMPATIBLE + PublicValidationMode.STRICT -> InternalValidationMode.STRICT + } + +private fun InternalValidationMode.toPublic(): PublicValidationMode = + when (this) { + InternalValidationMode.COMPATIBLE -> PublicValidationMode.COMPATIBLE + InternalValidationMode.STRICT -> PublicValidationMode.STRICT + } + +private fun BackendRecord.toDynamicDocument(): DynamicDocument = document.toMutableMap().let(::SimpleDynamicDocument) + +private fun BackendPage.toDynamicPage(): PagedList = + PagedList(total, records.map(BackendRecord::toDynamicDocument)) + +private fun NormalizedValue.ObjectValue.toMutableMap(): MutableMap { + val copy = LinkedHashMap(values.size) + values.forEach { (key, value) -> copy[key] = value.toMutableValue() } + return copy +} + +private fun NormalizedValue.toMutableValue(): Any? = + when (this) { + NormalizedValue.Null -> null + is NormalizedValue.BooleanValue -> value + is NormalizedValue.Text -> value + is NormalizedValue.Int64 -> value + is NormalizedValue.Decimal -> value + is NormalizedValue.InstantValue -> value + is NormalizedValue.Bytes -> toByteArray() + is NormalizedValue.ListValue -> values.mapTo(ArrayList(values.size), NormalizedValue::toMutableValue) + is NormalizedValue.ObjectValue -> toMutableMap() + } + +private fun NormalizedValue.toAnalyticsValue(decimalScale: Int? = null): AnalyticsValue = when (this) { + NormalizedValue.Null -> AnalyticsValue.nullValue() + is NormalizedValue.BooleanValue -> AnalyticsValue.of(value) + is NormalizedValue.Text -> AnalyticsValue.of(value) + is NormalizedValue.Int64 -> AnalyticsValue.of(value) + is NormalizedValue.Decimal -> AnalyticsValue.of( + decimalScale?.let { scale -> + try { + value.setScale(scale, java.math.RoundingMode.UNNECESSARY) + } catch (_: ArithmeticException) { + rejectMaterialization() + } + } ?: value, + ) + is NormalizedValue.InstantValue -> AnalyticsValue.of(value) + is NormalizedValue.Bytes, + is NormalizedValue.ListValue, + is NormalizedValue.ObjectValue, + -> rejectMaterialization() +} + +private class TargetMaterializerRegistry(registrations: Iterable>) { + private val registrations: Map> + + init { + val materialized = registrations.toList() + require(materialized.map(QueryResultMaterializer<*>::target).distinct().size == materialized.size) { + "Query result materializer targets must be unique." + } + val copy = LinkedHashMap>(materialized.size) + materialized.sortedWith(MATERIALIZER_COMPARATOR).forEach { registration -> + copy[registration.target] = registration + } + this.registrations = java.util.Collections.unmodifiableMap(copy) + } + + fun require(target: me.ahoo.wow.query.gateway.QueryTarget, resultType: Class) { + val registration = registrations[target] + if (registration == null || registration.resultType != resultType) { + rejectMaterialization() + } + } + + fun materialize( + target: me.ahoo.wow.query.gateway.QueryTarget, + resultType: Class, + record: BackendRecord, + ): R { + val registration = requireRegistration(target, resultType) + return resultType.castMaterialized(registration.materializeSafely(record)) + } + + private fun requireRegistration( + target: me.ahoo.wow.query.gateway.QueryTarget, + resultType: Class, + ): QueryResultMaterializer<*> = registrations[target]?.takeIf { registration -> + registration.resultType == resultType + } ?: rejectMaterialization() +} + +@Suppress("TooGenericExceptionCaught") +private fun QueryResultMaterializer<*>.materializeSafely(record: BackendRecord): Any = + try { + materialize(record.identity, record.toDynamicDocument()) + } catch (error: RuntimeException) { + rejectMaterialization(error) + } + +private fun Class.castMaterialized(value: Any): R = + try { + cast(value) + } catch (error: ClassCastException) { + rejectMaterialization(error) + } + +private fun rejectMaterialization(cause: Throwable? = null): Nothing = + throw QueryBackendException(QueryBackendFailureKind.MAPPING_FAILURE, cause) + +private val MATERIALIZER_COMPARATOR = compareBy>( + { registration -> registration.target.namedAggregate.contextName }, + { registration -> registration.target.namedAggregate.aggregateName }, + { registration -> registration.target.documentKind.name }, +) + +private fun Mono.mapError(): Mono = onErrorMap(::toPublicException) + +private fun Flux.mapError(): Flux = onErrorMap(::toPublicException) + +private fun toPublicException(error: Throwable): Throwable { + if (error is QueryExecutionException) { + return error + } + val rejected = error as? QueryRejectedException + ?: return QueryExecutionException( + QueryErrorCategory.INTERNAL_FAILURE, + "$", + "UNEXPECTED_QUERY_FAILURE", + error, + ) + return QueryExecutionException( + category = rejected.rejection.category.toPublic(), + path = rejected.rejection.path.toString(), + code = rejected.rejection.code.name, + cause = rejected, + ) +} + +private fun me.ahoo.wow.query.internal.rejection.QueryRejectionCategory.toPublic(): QueryErrorCategory = + when (this) { + me.ahoo.wow.query.internal.rejection.QueryRejectionCategory.ACCESS_DENIED -> QueryErrorCategory.ACCESS_DENIED + me.ahoo.wow.query.internal.rejection.QueryRejectionCategory.INVALID_QUERY -> QueryErrorCategory.INVALID_QUERY + me.ahoo.wow.query.internal.rejection.QueryRejectionCategory.INVALID_CURSOR -> QueryErrorCategory.INVALID_CURSOR + me.ahoo.wow.query.internal.rejection.QueryRejectionCategory.BUDGET_EXCEEDED -> QueryErrorCategory.BUDGET_EXCEEDED + me.ahoo.wow.query.internal.rejection.QueryRejectionCategory.UNSUPPORTED_FEATURE -> + QueryErrorCategory.UNSUPPORTED_FEATURE + + me.ahoo.wow.query.internal.rejection.QueryRejectionCategory.BACKEND_UNAVAILABLE -> + QueryErrorCategory.BACKEND_UNAVAILABLE + + me.ahoo.wow.query.internal.rejection.QueryRejectionCategory.BACKEND_TIMEOUT -> QueryErrorCategory.BACKEND_TIMEOUT + me.ahoo.wow.query.internal.rejection.QueryRejectionCategory.INCOMPLETE_RESULT -> + QueryErrorCategory.INCOMPLETE_RESULT + + me.ahoo.wow.query.internal.rejection.QueryRejectionCategory.MAPPING_FAILURE -> QueryErrorCategory.MAPPING_FAILURE + me.ahoo.wow.query.internal.rejection.QueryRejectionCategory.INTERNAL_FAILURE -> QueryErrorCategory.INTERNAL_FAILURE + } + +private const val AUTHORITY_PATH = "$.executionContext.authority" +private const val LEGACY_GRANT_PATH = "$.executionContext.legacyGrant" +private const val TRANSPORT_PATH = "$.executionContext.transport" diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/LegacyConditionLowerer.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/LegacyConditionLowerer.kt new file mode 100644 index 00000000000..f696a3f86f6 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/LegacyConditionLowerer.kt @@ -0,0 +1,400 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.internal.gateway + +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DeletionState +import me.ahoo.wow.api.query.Operator +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.gateway.QueryElementPathMode +import me.ahoo.wow.query.gateway.QueryLegacyDialect +import me.ahoo.wow.query.gateway.QueryMatchScopeMode +import me.ahoo.wow.query.internal.execution.rejectLegacyMandatory +import me.ahoo.wow.query.internal.normalization.CaseSensitivity +import me.ahoo.wow.query.internal.normalization.JunctionOperator +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedDeletionScope +import me.ahoo.wow.query.internal.normalization.NormalizedPredicateOptions +import me.ahoo.wow.query.internal.normalization.PathBasis +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.SearchScope +import me.ahoo.wow.query.internal.normalization.SystemFieldKind +import me.ahoo.wow.query.internal.plan.PlannedCondition +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.serialization.MessageRecords +import java.util.LinkedHashMap + +internal data class LoweredLegacyCondition( + val condition: Condition, + val matchNone: Boolean, +) + +internal class LegacyConditionLowerer( + private val dialect: QueryLegacyDialect, + private val deletionMode: LegacyDeletionMode = LegacyDeletionMode.SNAPSHOT, +) { + fun lower( + user: NormalizedCondition, + deletionScope: NormalizedDeletionScope, + mandatory: PlannedCondition, + ): LoweredLegacyCondition { + val loweredUser = lowerNormalized(user, emptyList()) + val loweredMandatory = lowerMandatory(mandatory) + val matchNone = loweredUser == Lowered.None || loweredMandatory == Lowered.None + val condition = when (deletionMode) { + LegacyDeletionMode.SNAPSHOT -> { + val deletion = when (deletionScope) { + NormalizedDeletionScope.DEFAULT_ACTIVE -> Condition.deleted(DeletionState.ACTIVE) + NormalizedDeletionScope.EXPLICIT -> Condition.deleted(DeletionState.ALL) + } + val rootChildren = buildList { + add(deletion) + loweredUser.conditionOrNull()?.let(::add) + loweredMandatory.conditionOrNull()?.let(::add) + } + Condition.and(rootChildren) + } + + LegacyDeletionMode.NONE -> lowerAnd(listOf(loweredUser, loweredMandatory)).conditionOrNull() ?: Condition.ALL + } + return LoweredLegacyCondition(condition, matchNone) + } + + private fun lowerMandatory(condition: PlannedCondition): Lowered = + try { + lowerPlanned(condition, emptyList()) + } catch (error: QueryRejectedException) { + rejectLegacyMandatory(error) + } + + private fun lowerNormalized(condition: NormalizedCondition, ancestors: List): Lowered = + when (condition) { + NormalizedCondition.All -> Lowered.All + NormalizedCondition.None -> Lowered.None + is NormalizedCondition.Junction -> lowerJunction( + condition.operator, + condition.children.map { child -> lowerNormalized(child, ancestors) }, + ) + + is NormalizedCondition.Predicate -> lowerPredicate( + condition.field, + condition.operator, + condition.value, + condition.options, + ancestors, + ) + + is NormalizedCondition.ElementMatch -> lowerElementMatch( + condition.field.absoluteSegments(ancestors), + ancestors, + ) { childAncestors -> lowerNormalized(condition.condition, childAncestors) } + + is NormalizedCondition.Search -> lowerSearch(condition, ancestors) + is NormalizedCondition.Native -> rejectLegacyLowering() + } + + private fun lowerPlanned(condition: PlannedCondition, ancestors: List): Lowered = + when (condition) { + PlannedCondition.All -> Lowered.All + PlannedCondition.None -> Lowered.None + is PlannedCondition.Junction -> lowerJunction( + condition.operator, + condition.children.values.map { child -> lowerPlanned(child, ancestors) }, + ) + + is PlannedCondition.Predicate -> lowerPredicate( + condition.field, + condition.operator, + condition.value, + condition.options, + ancestors, + ) + + is PlannedCondition.ElementMatch -> lowerElementMatch( + condition.field.segments, + ancestors, + ) { childAncestors -> lowerPlanned(condition.condition, childAncestors) } + + is PlannedCondition.Search, + is PlannedCondition.Native, + -> rejectLegacyLowering() + } + + private fun lowerElementMatch( + absoluteField: List, + ancestors: List, + lowerChild: (List) -> Lowered, + ): Lowered { + val child = lowerChild(absoluteField) + if (child == Lowered.None) { + return Lowered.None + } + val childCondition = child.conditionOrNull() ?: Condition.ALL + val field = renderField(absoluteField, ancestors) + return Lowered.Wire( + Condition( + field = field, + operator = Operator.ELEM_MATCH, + children = listOf(childCondition), + ), + ) + } + + private fun lowerSearch(condition: NormalizedCondition.Search, ancestors: List): Lowered { + val scope = condition.scope as? SearchScope.LegacyField ?: rejectLegacyLowering() + if (dialect.matchScopeMode == QueryMatchScopeMode.DOCUMENT && ancestors.isNotEmpty()) { + rejectLegacyLowering() + } + val field = renderField(scope.field.absoluteSegments(ancestors), ancestors) + return Lowered.Wire(Condition(field, Operator.MATCH, condition.text)) + } + + private fun lowerPredicate( + field: LogicalField, + operator: PredicateOperator, + value: NormalizedValue?, + options: NormalizedPredicateOptions, + ancestors: List, + ): Lowered = + when (field) { + is LogicalField.System -> lowerSystemPredicate(field.kind, operator, value) + is LogicalField.Path -> wirePredicate( + renderField(field.absoluteSegments(ancestors), ancestors), + operator, + value, + options, + ) + } + + private fun lowerPredicate( + field: QueryFieldId, + operator: PredicateOperator, + value: NormalizedValue?, + options: NormalizedPredicateOptions, + ancestors: List, + ): Lowered = + when (field) { + is QueryFieldId.System -> lowerSystemPredicate(field.kind, operator, value) + is QueryFieldId.Path -> wirePredicate( + renderField(field.segments, ancestors), + operator, + value, + options, + ) + } + + private fun lowerSystemPredicate( + kind: SystemFieldKind, + operator: PredicateOperator, + value: NormalizedValue?, + ): Lowered { + val wire = when (kind) { + SystemFieldKind.IDENTITY -> systemIdentity(operator, value, Operator.ID, Operator.IDS) + SystemFieldKind.AGGREGATE_ID -> systemIdentity( + operator, + value, + Operator.AGGREGATE_ID, + Operator.AGGREGATE_IDS, + ) + + SystemFieldKind.TENANT_ID -> systemText( + operator, + value, + Operator.TENANT_ID, + MessageRecords.TENANT_ID, + ) + + SystemFieldKind.OWNER_ID -> systemText(operator, value, Operator.OWNER_ID, MessageRecords.OWNER_ID) + SystemFieldKind.SPACE_ID -> systemText(operator, value, Operator.SPACE_ID, MessageRecords.SPACE_ID) + SystemFieldKind.DELETED -> when (operator) { + PredicateOperator.IS_FALSE -> Condition.deleted(DeletionState.ACTIVE) + PredicateOperator.IS_TRUE -> Condition.deleted(DeletionState.DELETED) + PredicateOperator.EQ -> Condition.deleted(value.requireBoolean()) + else -> rejectLegacyLowering() + } + } + return Lowered.Wire(wire) + } + + private fun systemIdentity( + operator: PredicateOperator, + value: NormalizedValue?, + single: Operator, + multiple: Operator, + ): Condition = + when (operator) { + PredicateOperator.EQ -> Condition(operator = single, value = value.requireText()) + PredicateOperator.IN -> Condition(operator = multiple, value = value.requireTextList()) + else -> rejectLegacyLowering() + } + + private fun systemText( + operator: PredicateOperator, + value: NormalizedValue?, + wireOperator: Operator, + field: String, + ): Condition = + when (operator) { + PredicateOperator.EQ -> Condition(operator = wireOperator, value = value.requireText()) + PredicateOperator.IN -> Condition(field, Operator.IN, value.requireTextList()) + else -> rejectLegacyLowering() + } + + private fun wirePredicate( + field: String, + operator: PredicateOperator, + value: NormalizedValue?, + options: NormalizedPredicateOptions, + ): Lowered.Wire { + val wireOperator = operator.toWireOperator() + val wireValue = if (operator.requiresValue) value?.toWireValue() ?: rejectLegacyLowering() else Condition.EMPTY_VALUE + val wireOptions = if (options.caseSensitivity == CaseSensitivity.INSENSITIVE) { + Condition.ignoreCaseOptions(true) + } else { + emptyMap() + } + return Lowered.Wire(Condition(field, wireOperator, wireValue, options = wireOptions)) + } + + private fun lowerJunction(operator: JunctionOperator, children: List): Lowered = + when (operator) { + JunctionOperator.AND -> lowerAnd(children) + JunctionOperator.OR -> lowerOr(children) + JunctionOperator.NOR -> lowerNor(children) + } + + private fun lowerAnd(children: List): Lowered { + if (Lowered.None in children) { + return Lowered.None + } + val wire = children.mapNotNull(Lowered::conditionOrNull) + return when (wire.size) { + 0 -> Lowered.All + 1 -> Lowered.Wire(wire.single()) + else -> Lowered.Wire(Condition.and(wire)) + } + } + + private fun lowerOr(children: List): Lowered { + if (Lowered.All in children) { + return Lowered.All + } + val wire = children.mapNotNull(Lowered::conditionOrNull) + return when (wire.size) { + 0 -> Lowered.None + 1 -> Lowered.Wire(wire.single()) + else -> Lowered.Wire(Condition.or(wire)) + } + } + + private fun lowerNor(children: List): Lowered { + if (Lowered.All in children) { + return Lowered.None + } + val wire = children.mapNotNull(Lowered::conditionOrNull) + return if (wire.isEmpty()) Lowered.All else Lowered.Wire(Condition.nor(wire)) + } + + private fun renderField(absolute: List, ancestors: List): String { + val rendered = when (dialect.elementPathMode) { + QueryElementPathMode.ROOT_QUALIFIED -> absolute + QueryElementPathMode.CURRENT_ELEMENT_RELATIVE -> + if (ancestors.isNotEmpty() && absolute.startsWith(ancestors)) { + absolute.drop(ancestors.size) + } else { + absolute + } + } + if (rendered.isEmpty()) { + rejectLegacyLowering() + } + return rendered.joinToString(".") + } + + private sealed interface Lowered { + data object All : Lowered + + data object None : Lowered + + data class Wire(val condition: Condition) : Lowered + + fun conditionOrNull(): Condition? = (this as? Wire)?.condition + } +} + +internal enum class LegacyDeletionMode { + SNAPSHOT, + NONE, +} + +private fun LogicalField.Path.absoluteSegments(ancestors: List): List = + when (basis) { + PathBasis.ROOT -> segments + PathBasis.CURRENT_ELEMENT -> ancestors + segments + } + +private val WIRE_PREDICATE_OPERATORS = mapOf( + PredicateOperator.EQ to Operator.EQ, + PredicateOperator.NE to Operator.NE, + PredicateOperator.GT to Operator.GT, + PredicateOperator.LT to Operator.LT, + PredicateOperator.GTE to Operator.GTE, + PredicateOperator.LTE to Operator.LTE, + PredicateOperator.CONTAINS to Operator.CONTAINS, + PredicateOperator.IN to Operator.IN, + PredicateOperator.NOT_IN to Operator.NOT_IN, + PredicateOperator.BETWEEN to Operator.BETWEEN, + PredicateOperator.ALL_IN to Operator.ALL_IN, + PredicateOperator.STARTS_WITH to Operator.STARTS_WITH, + PredicateOperator.ENDS_WITH to Operator.ENDS_WITH, + PredicateOperator.IS_NULL to Operator.NULL, + PredicateOperator.NOT_NULL to Operator.NOT_NULL, + PredicateOperator.IS_TRUE to Operator.TRUE, + PredicateOperator.IS_FALSE to Operator.FALSE, + PredicateOperator.EXISTS to Operator.EXISTS, +) + +private fun PredicateOperator.toWireOperator(): Operator = WIRE_PREDICATE_OPERATORS.getValue(this) + +private fun NormalizedValue?.requireBoolean(): Boolean = + (this as? NormalizedValue.BooleanValue)?.value ?: rejectLegacyLowering() + +private fun NormalizedValue?.requireText(): String = (this as? NormalizedValue.Text)?.value ?: rejectLegacyLowering() + +private fun NormalizedValue?.requireTextList(): List = + (this as? NormalizedValue.ListValue)?.values?.map { it.requireText() } ?: rejectLegacyLowering() + +private fun NormalizedValue.toWireValue(): Any? = + when (this) { + NormalizedValue.Null -> null + is NormalizedValue.BooleanValue -> value + is NormalizedValue.Text -> value + is NormalizedValue.Int64 -> value + is NormalizedValue.Decimal -> value + is NormalizedValue.InstantValue -> value.toEpochMilli() + is NormalizedValue.Bytes -> toByteArray() + is NormalizedValue.ListValue -> values.map(NormalizedValue::toWireValue) + is NormalizedValue.ObjectValue -> { + val copy = LinkedHashMap(values.size) + values.forEach { (key, value) -> copy[key] = value.toWireValue() } + copy + } + } + +private fun List.startsWith(prefix: List): Boolean = + size >= prefix.size && prefix.indices.all { index -> this[index] == prefix[index] } diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/LegacyDynamicQuery.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/LegacyDynamicQuery.kt new file mode 100644 index 00000000000..934f116ce0e --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/LegacyDynamicQuery.kt @@ -0,0 +1,316 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.internal.gateway + +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DynamicDocument +import me.ahoo.wow.api.query.ListQuery +import me.ahoo.wow.api.query.PagedList +import me.ahoo.wow.api.query.PagedQuery +import me.ahoo.wow.api.query.Pagination +import me.ahoo.wow.api.query.Projection +import me.ahoo.wow.api.query.SingleQuery +import me.ahoo.wow.api.query.Sort +import me.ahoo.wow.query.QueryService +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.gateway.QueryLegacyDialect +import me.ahoo.wow.query.internal.admission.QueryAdmissionLimits +import me.ahoo.wow.query.internal.execution.BackendPage +import me.ahoo.wow.query.internal.execution.BackendPageConsistency +import me.ahoo.wow.query.internal.execution.BackendRecord +import me.ahoo.wow.query.internal.execution.BackendTotalRelation +import me.ahoo.wow.query.internal.execution.LegacyCompilationInput +import me.ahoo.wow.query.internal.execution.LegacyCompiledQuery +import me.ahoo.wow.query.internal.execution.LegacyQueryBackend +import me.ahoo.wow.query.internal.execution.LegacyQueryCompiler +import me.ahoo.wow.query.internal.execution.QueryExecutionOptions +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedDeletionScope +import me.ahoo.wow.query.internal.normalization.NormalizedProjection +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedRecordQuery +import me.ahoo.wow.query.internal.normalization.NormalizedSort +import me.ahoo.wow.query.internal.normalization.NormalizedSortDirection +import me.ahoo.wow.query.internal.plan.PlannedSort +import me.ahoo.wow.query.internal.plan.RecordQueryPlan +import me.ahoo.wow.query.internal.planning.PlanningDecision +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono + +internal sealed interface LegacyWireQuery { + val condition: Condition + + data class Single(val query: SingleQuery) : LegacyWireQuery { + override val condition: Condition = query.condition + } + + data class Stream(val query: ListQuery) : LegacyWireQuery { + override val condition: Condition = query.condition + } + + data class Page(val query: PagedQuery) : LegacyWireQuery { + override val condition: Condition = query.condition + } + + data class Count(override val condition: Condition) : LegacyWireQuery +} + +internal class LegacyDynamicCompiledQuery( + override val target: QueryTarget, + override val operation: QueryOperation, + override val schemaContractId: SchemaContractId, + override val loweringAttestation: me.ahoo.wow.query.internal.execution.LegacyLoweringAttestation, + val wireQuery: LegacyWireQuery, + val matchNone: Boolean, + val identityField: String, + val outputProjection: NormalizedProjection, +) : LegacyCompiledQuery + +internal class LegacyDynamicQueryCompiler( + private val target: QueryTarget, + private val dialect: QueryLegacyDialect, + private val identityField: String, +) : LegacyQueryCompiler { + override fun compile(input: LegacyCompilationInput): LegacyDynamicCompiledQuery { + val invocation = input.invocation + if (invocation.target != target) { + rejectLegacyLowering() + } + val lowered = LegacyConditionLowerer( + dialect, + when (target.documentKind) { + QueryDocumentKind.SNAPSHOT -> LegacyDeletionMode.SNAPSHOT + QueryDocumentKind.EVENT_STREAM -> LegacyDeletionMode.NONE + }, + ).lower( + invocation.userCondition(), + invocation.deletionScope(), + input.enforcementRequirements.mandatoryCondition, + ) + val record = invocation.recordQuery() + val sort = input.decision.plannedRecordSort(identityField) ?: record.toSort() + val wireQuery = when (val normalizedInput = invocation.input) { + is NormalizedQueryInput.Single -> LegacyWireQuery.Single( + SingleQuery( + lowered.condition, + record.requireProjectionWithIdentity(identityField), + sort, + ), + ) + + is NormalizedQueryInput.Stream -> LegacyWireQuery.Stream( + ListQuery( + lowered.condition, + record.requireProjectionWithIdentity(identityField), + sort, + normalizedInput.limit, + ), + ) + + is NormalizedQueryInput.Page -> LegacyWireQuery.Page( + PagedQuery( + lowered.condition, + record.requireProjectionWithIdentity(identityField), + sort, + normalizedInput.page.toLegacyPagination(), + ), + ) + + is NormalizedQueryInput.Count -> LegacyWireQuery.Count(lowered.condition) + is NormalizedQueryInput.Analytics -> rejectLegacyLowering() + } + return LegacyDynamicCompiledQuery( + target, + invocation.operation, + input.schema.contractId, + input.attestLowering( + input.enforcementRequirements.deletionScope, + input.enforcementRequirements.mandatoryCondition, + ), + wireQuery, + lowered.matchNone, + identityField, + record?.projection ?: NormalizedProjection.All, + ) + } +} + +internal class LegacyDynamicQueryBackend( + private val queryService: QueryService<*>, + limits: QueryAdmissionLimits = QueryAdmissionLimits.DEFAULT, +) : LegacyQueryBackend { + private val resultSnapshotter = LegacyResultSnapshotter(limits) + + override fun single( + query: LegacyDynamicCompiledQuery, + options: QueryExecutionOptions, + ): Mono { + if (query.matchNone) { + return Mono.empty() + } + val wire = query.wireQuery as? LegacyWireQuery.Single ?: rejectLegacyLowering() + return queryService.dynamicSingle(wire.query).map { result -> resultSnapshotter.snapshot(query, result) } + } + + override fun stream( + query: LegacyDynamicCompiledQuery, + options: QueryExecutionOptions, + ): Flux { + if (query.matchNone) { + return Flux.empty() + } + val wire = query.wireQuery as? LegacyWireQuery.Stream ?: rejectLegacyLowering() + return queryService.dynamicList(wire.query).map { result -> resultSnapshotter.snapshot(query, result) } + } + + override fun page( + query: LegacyDynamicCompiledQuery, + options: QueryExecutionOptions, + ): Mono { + if (query.matchNone) { + return Mono.just(emptyBackendPage()) + } + val wire = query.wireQuery as? LegacyWireQuery.Page ?: rejectLegacyLowering() + return queryService.dynamicPaged(wire.query).map { result -> result.toBackendPage(query) } + } + + override fun count(query: LegacyDynamicCompiledQuery, options: QueryExecutionOptions): Mono { + if (query.matchNone) { + return Mono.just(0) + } + val wire = query.wireQuery as? LegacyWireQuery.Count ?: rejectLegacyLowering() + return queryService.count(wire.condition) + } + + private fun PagedList.toBackendPage(query: LegacyDynamicCompiledQuery): BackendPage = + BackendPage( + list.map { result -> resultSnapshotter.snapshot(query, result) }, + total, + BackendTotalRelation.EXACT, + BackendPageConsistency.INDEPENDENT, + ) + + private fun emptyBackendPage(): BackendPage = + BackendPage( + emptyList(), + 0, + BackendTotalRelation.EXACT, + BackendPageConsistency.INDEPENDENT, + ) +} + +private fun me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation.userCondition() = + when (val normalizedInput = input) { + is NormalizedQueryInput.Single -> normalizedInput.query.userCondition + is NormalizedQueryInput.Stream -> normalizedInput.query.userCondition + is NormalizedQueryInput.Page -> normalizedInput.query.userCondition + is NormalizedQueryInput.Count -> normalizedInput.userCondition + is NormalizedQueryInput.Analytics -> rejectLegacyLowering() + } + +private fun me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation.deletionScope() = + when (val normalizedInput = input) { + is NormalizedQueryInput.Single -> normalizedInput.query.deletionScope + is NormalizedQueryInput.Stream -> normalizedInput.query.deletionScope + is NormalizedQueryInput.Page -> normalizedInput.query.deletionScope + is NormalizedQueryInput.Count -> normalizedInput.deletionScope + is NormalizedQueryInput.Analytics -> NormalizedDeletionScope.EXPLICIT + } + +private fun me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation.recordQuery(): NormalizedRecordQuery? = + when (val normalizedInput = input) { + is NormalizedQueryInput.Single -> normalizedInput.query + is NormalizedQueryInput.Stream -> normalizedInput.query + is NormalizedQueryInput.Page -> normalizedInput.query + is NormalizedQueryInput.Count, + is NormalizedQueryInput.Analytics, + -> null + } + +private fun NormalizedRecordQuery?.requireProjectionWithIdentity(identityField: String): Projection { + val projection = this?.projection ?: return Projection.ALL + return when (projection) { + NormalizedProjection.All -> Projection.ALL + is NormalizedProjection.Include -> Projection( + include = (projection.fields.values.map(LogicalField.Path::toLegacyField) + identityField).distinct(), + ) + + is NormalizedProjection.Exclude -> Projection( + exclude = projection.fields.values.map(LogicalField.Path::toLegacyField).filterNot(identityField::equals), + ) + + is NormalizedProjection.Mixed -> Projection( + include = (projection.include.values.map(LogicalField.Path::toLegacyField) + identityField).distinct(), + exclude = projection.exclude.values.map(LogicalField.Path::toLegacyField).filterNot(identityField::equals), + ) + } +} + +private fun NormalizedRecordQuery?.toSort(): List = this?.sort.orEmpty().map(NormalizedSort::toLegacy) + +private fun PlanningDecision.plannedRecordSort(identityField: String): List? = + ((this as? PlanningDecision.Planned)?.plan as? RecordQueryPlan)?.sort?.map { sort -> + sort.toLegacy(identityField) + } + +private fun PlannedSort.toLegacy(identityField: String): Sort = + Sort( + field = when (val plannedField = field) { + is me.ahoo.wow.query.backend.QueryFieldId.Path -> plannedField.segments.joinToString(".") + is me.ahoo.wow.query.backend.QueryFieldId.System -> + if (plannedField.kind == me.ahoo.wow.query.internal.normalization.SystemFieldKind.IDENTITY) { + identityField + } else { + rejectLegacyLowering() + } + }, + direction = direction.toLegacy(), + ) + +private fun NormalizedSort.toLegacy(): Sort = + Sort( + field = (field as? LogicalField.Path)?.toLegacyField() ?: rejectLegacyLowering(), + direction = direction.toLegacy(), + ) + +private fun NormalizedSortDirection.toLegacy(): Sort.Direction = + when (this) { + NormalizedSortDirection.ASC -> Sort.Direction.ASC + NormalizedSortDirection.DESC -> Sort.Direction.DESC + } + +private fun LogicalField.Path.toLegacyField(): String = segments.joinToString(".") + +private fun me.ahoo.wow.query.internal.normalization.NormalizedPage.toLegacyPagination(): Pagination { + val legacyOffset = runCatching { Pagination.offset(index, size) }.getOrElse { rejectLegacyLowering() } + if (offset > Int.MAX_VALUE || legacyOffset.toLong() != offset) { + rejectLegacyLowering() + } + return Pagination(index, size) +} + +internal fun rejectLegacyLowering(): Nothing = rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionPath.ROOT.property("execution").property("legacy"), + QueryRejectionCode.LEGACY_LOWERING_UNSUPPORTED, +) diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/LegacyQuerySchema.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/LegacyQuerySchema.kt new file mode 100644 index 00000000000..3a7b8e28f5b --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/LegacyQuerySchema.kt @@ -0,0 +1,132 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.gateway + +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.SystemFieldKind +import me.ahoo.wow.serialization.MessageRecords + +/** + * Bootstrap schema for legacy-compatible execution. + * + * It deliberately declares only framework-owned system fields. User paths remain unknown and therefore produce an + * explicit COMPATIBLE fallback instead of inventing field types or backend capabilities. + */ +internal fun legacyQuerySchema(target: QueryTarget): QueryDocumentSchema = + QueryDocumentSchema( + target, + fields = buildList { + val identityAlias = when (target.documentKind) { + QueryDocumentKind.SNAPSHOT -> MessageRecords.AGGREGATE_ID + QueryDocumentKind.EVENT_STREAM -> MessageRecords.ID + } + add( + textSystemField( + SystemFieldKind.IDENTITY, + Presence.REQUIRED, + sortable = true, + logicalAliases = listOf(path(identityAlias)), + ), + ) + add( + textSystemField( + SystemFieldKind.AGGREGATE_ID, + Presence.REQUIRED, + logicalAliases = if (target.documentKind == QueryDocumentKind.EVENT_STREAM) { + listOf(path(MessageRecords.AGGREGATE_ID)) + } else { + emptyList() + }, + ), + ) + add( + textSystemField( + SystemFieldKind.TENANT_ID, + Presence.OPTIONAL, + logicalAliases = listOf(path(MessageRecords.TENANT_ID)), + ), + ) + add( + textSystemField( + SystemFieldKind.OWNER_ID, + Presence.OPTIONAL, + logicalAliases = listOf(path(MessageRecords.OWNER_ID)), + ), + ) + add( + textSystemField( + SystemFieldKind.SPACE_ID, + Presence.OPTIONAL, + logicalAliases = listOf(path(MessageRecords.SPACE_ID)), + ), + ) + if (target.documentKind == QueryDocumentKind.SNAPSHOT) { + add(booleanSystemField(SystemFieldKind.DELETED, listOf(path(DELETED_FIELD)))) + } + }, + searchScopes = emptyList(), + ) + +private fun textSystemField( + kind: SystemFieldKind, + presence: Presence, + sortable: Boolean = false, + logicalAliases: Iterable = emptyList(), +): QueryFieldSchema = + QueryFieldSchema( + id = QueryFieldId.System(kind), + type = LogicalFieldType.Text, + presence = presence, + nullability = Nullability.NON_NULL, + allowedOperators = setOf(PredicateOperator.EQ, PredicateOperator.IN), + capabilities = buildSet { + add(FieldCapability.EXACT) + add(FieldCapability.PROJECTABLE) + if (sortable) { + add(FieldCapability.SORTABLE) + } + }, + logicalAliases = logicalAliases, + ) + +private fun booleanSystemField( + kind: SystemFieldKind, + logicalAliases: Iterable = emptyList(), +): QueryFieldSchema = + QueryFieldSchema( + id = QueryFieldId.System(kind), + type = LogicalFieldType.Boolean, + presence = Presence.REQUIRED, + nullability = Nullability.NON_NULL, + allowedOperators = setOf( + PredicateOperator.EQ, + PredicateOperator.IS_TRUE, + PredicateOperator.IS_FALSE, + ), + capabilities = setOf(FieldCapability.EXACT, FieldCapability.PROJECTABLE), + logicalAliases = logicalAliases, + ) + +private fun path(field: String): QueryFieldId.Path = QueryFieldId.Path(listOf(field)) + +private const val DELETED_FIELD = "deleted" diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/LegacyResultSnapshotter.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/LegacyResultSnapshotter.kt new file mode 100644 index 00000000000..e54629a4b7e --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/LegacyResultSnapshotter.kt @@ -0,0 +1,123 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.gateway + +import me.ahoo.wow.api.query.DynamicDocument +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.admission.AdmissionBudget +import me.ahoo.wow.query.internal.admission.QueryAdmissionLimits +import me.ahoo.wow.query.internal.admission.RawValueSnapshotter +import me.ahoo.wow.query.internal.execution.BackendRecord +import me.ahoo.wow.query.internal.execution.BackendRecordCompleteness +import me.ahoo.wow.query.internal.execution.QueryBackendException +import me.ahoo.wow.query.internal.execution.QueryBackendFailureKind +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedProjection +import me.ahoo.wow.query.internal.normalization.PathBasis +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import java.util.LinkedHashMap + +internal class LegacyResultSnapshotter( + private val limits: QueryAdmissionLimits, +) { + private val valueSnapshotter = RawValueSnapshotter(limits) + + @Suppress("TooGenericExceptionCaught") + fun snapshot(query: LegacyDynamicCompiledQuery, source: DynamicDocument): BackendRecord { + try { + return snapshotResult(query, source) + } catch (error: RuntimeException) { + throw QueryBackendException(QueryBackendFailureKind.MAPPING_FAILURE, error) + } + } + + private fun snapshotResult(query: LegacyDynamicCompiledQuery, source: DynamicDocument): BackendRecord { + val frozen = requireNotNull( + valueSnapshotter.snapshot( + source, + QueryRejectionPath.ROOT.property("result"), + AdmissionBudget(limits), + ) as? NormalizedValue.ObjectValue, + ) { + "Legacy query result must be an object." + } + val identity = requireNotNull((frozen.values[query.identityField] as? NormalizedValue.Text)?.value) { + "Legacy query result is missing a string identity." + } + return BackendRecord( + identity, + query.outputProjection.applyTo(frozen), + BackendRecordCompleteness.UNKNOWN, + ) + } +} + +private fun NormalizedProjection.applyTo(source: NormalizedValue.ObjectValue): NormalizedValue.ObjectValue = + when (this) { + NormalizedProjection.All -> source + is NormalizedProjection.Include -> source.include(fields.values.map(LogicalField.Path::rootSegments)) + is NormalizedProjection.Exclude -> source.exclude(fields.values.map(LogicalField.Path::rootSegments)) + is NormalizedProjection.Mixed -> { + source + .include(include.values.map(LogicalField.Path::rootSegments)) + .exclude(exclude.values.map(LogicalField.Path::rootSegments)) + } + } + +private fun LogicalField.Path.rootSegments(): List { + if (basis != PathBasis.ROOT) { + rejectLegacyLowering() + } + return segments +} + +private fun NormalizedValue.ObjectValue.include(paths: List>): NormalizedValue.ObjectValue { + val result = LinkedHashMap() + values.forEach { (key, value) -> + val matching = paths.filter { path -> path.firstOrNull() == key } + if (matching.any { path -> path.size == 1 }) { + result[key] = value + } else if (matching.isNotEmpty()) { + result[key] = value.includeNested(matching.map { path -> path.drop(1) }) + } + } + return NormalizedValue.ObjectValue(result) +} + +private fun NormalizedValue.includeNested(paths: List>): NormalizedValue = + when (this) { + NormalizedValue.Null -> NormalizedValue.Null + is NormalizedValue.ObjectValue -> include(paths) + is NormalizedValue.ListValue -> NormalizedValue.ListValue(values.map { value -> value.includeNested(paths) }) + else -> throw IllegalArgumentException("Legacy result does not match the requested projection path.") + } + +private fun NormalizedValue.ObjectValue.exclude(paths: List>): NormalizedValue.ObjectValue { + val result = LinkedHashMap() + values.forEach { (key, value) -> + val matching = paths.filter { path -> path.firstOrNull() == key } + if (matching.none { path -> path.size == 1 }) { + val nested = matching.filter { path -> path.size > 1 }.map { path -> path.drop(1) } + result[key] = if (nested.isEmpty()) value else value.excludeNested(nested) + } + } + return NormalizedValue.ObjectValue(result) +} + +private fun NormalizedValue.excludeNested(paths: List>): NormalizedValue = + when (this) { + is NormalizedValue.ObjectValue -> exclude(paths) + is NormalizedValue.ListValue -> NormalizedValue.ListValue(values.map { value -> value.excludeNested(paths) }) + else -> this + } diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/QueryGatewayRuntimeBuilder.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/QueryGatewayRuntimeBuilder.kt new file mode 100644 index 00000000000..cc1879156aa --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/QueryGatewayRuntimeBuilder.kt @@ -0,0 +1,441 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.query.internal.gateway + +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.modeling.materialize +import me.ahoo.wow.query.QueryService +import me.ahoo.wow.query.backend.AnalyticsQueryCursorLifecycle +import me.ahoo.wow.query.backend.BackendAnalyticsCursorState +import me.ahoo.wow.query.backend.BackendStreamSupport +import me.ahoo.wow.query.backend.QueryBackendComposition +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.cursor.QueryCursorLeaseConfiguration +import me.ahoo.wow.query.gateway.QueryAuthorityResolver +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryExecutionMode +import me.ahoo.wow.query.gateway.QueryExecutionProfiles +import me.ahoo.wow.query.gateway.QueryGateway +import me.ahoo.wow.query.gateway.QueryLegacyDialectResolver +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryRawServiceSource +import me.ahoo.wow.query.gateway.QueryResultMaterializer +import me.ahoo.wow.query.gateway.QueryRuntimeHealthKind +import me.ahoo.wow.query.gateway.QueryRuntimeHealthObservation +import me.ahoo.wow.query.gateway.QueryRuntimeHealthObserver +import me.ahoo.wow.query.gateway.QueryShadowConfiguration +import me.ahoo.wow.query.gateway.QueryShadowObserver +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.internal.admission.QueryAdmissionLimits +import me.ahoo.wow.query.internal.admission.RawAdmissionGuard +import me.ahoo.wow.query.internal.cursor.PersistentQueryCursorBackendLeaseRegistration +import me.ahoo.wow.query.internal.cursor.PersistentQueryCursorLeaseCoordinator +import me.ahoo.wow.query.internal.cursor.PersistentQueryCursorLeaseManager +import me.ahoo.wow.query.internal.cursor.QueryCursorLeaseLimits +import me.ahoo.wow.query.internal.cursor.QueryCursorLeaseObserver +import me.ahoo.wow.query.internal.cursor.QueryCursorSigningKey +import me.ahoo.wow.query.internal.cursor.QueryCursorSigningKeyRing +import me.ahoo.wow.query.internal.execution.AnalyticsCursorRuntime +import me.ahoo.wow.query.internal.execution.BoundedQueryShadowSupervisor +import me.ahoo.wow.query.internal.execution.ExperimentalAnalyticsBackendAdapter +import me.ahoo.wow.query.internal.execution.LegacyBackendRegistry +import me.ahoo.wow.query.internal.execution.LegacyExecutionBinding +import me.ahoo.wow.query.internal.execution.QueryBackendDescriptor +import me.ahoo.wow.query.internal.execution.QueryBackendKey +import me.ahoo.wow.query.internal.execution.QueryBackendRegistration +import me.ahoo.wow.query.internal.execution.QueryBackendRegistry +import me.ahoo.wow.query.internal.execution.QueryBackendStreamSupport +import me.ahoo.wow.query.internal.execution.QueryDeadlineEnforcer +import me.ahoo.wow.query.internal.execution.QueryDecisionObserver +import me.ahoo.wow.query.internal.execution.QueryExecutionRouteResolver +import me.ahoo.wow.query.internal.execution.QueryExecutor +import me.ahoo.wow.query.internal.execution.QueryFallback +import me.ahoo.wow.query.internal.execution.QueryShadowSupervisorFailure +import me.ahoo.wow.query.internal.normalization.QueryNormalizer +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.planning.QueryPlanner +import me.ahoo.wow.query.internal.policy.QueryExecutionContextFactory +import me.ahoo.wow.query.internal.policy.QueryPolicyEnforcer +import me.ahoo.wow.query.internal.policy.TenantIsolationQueryPolicy +import me.ahoo.wow.query.internal.schema.QuerySchemaRegistry +import me.ahoo.wow.serialization.MessageRecords +import reactor.core.scheduler.Scheduler +import java.time.Clock +import me.ahoo.wow.query.internal.execution.QueryGateway as InternalQueryGateway +import me.ahoo.wow.query.internal.model.QueryDocumentKind as InternalDocumentKind +import me.ahoo.wow.query.internal.model.QueryTarget as InternalQueryTarget + +internal object QueryGatewayRuntimeBuilder { + fun build( + namedAggregates: Iterable, + rawServiceSource: QueryRawServiceSource, + resultMaterializers: Iterable>, + dialectResolver: QueryLegacyDialectResolver, + authorityResolver: QueryAuthorityResolver, + clock: Clock, + scheduler: Scheduler, + backendComposition: QueryBackendComposition = QueryBackendComposition.EMPTY, + executionProfiles: QueryExecutionProfiles, + shadowConfiguration: QueryShadowConfiguration, + shadowObserver: QueryShadowObserver, + runtimeHealthObserver: QueryRuntimeHealthObserver, + cursorLeaseConfiguration: QueryCursorLeaseConfiguration? = null, + ): QueryGatewayRuntimeComponents { + val aggregates = materializeAggregates(namedAggregates) + val targets = createTargetBindings(aggregates, rawServiceSource) + val contributedSchemas = ( + backendComposition.contributions.map { contribution -> contribution.schema } + + backendComposition.notReadyBackends.map { backend -> backend.schema } + ).groupBy(QueryDocumentSchema::target) + validateContributedSchemas(targets, contributedSchemas) + validateExecutionProfiles( + targets.map { binding -> binding.target.toPublic() }, + backendComposition, + executionProfiles, + shadowObserver, + runtimeHealthObserver, + ) + val schemas = createSchemaRegistry(targets, contributedSchemas) + val legacyBindings = createLegacyBindings(targets, dialectResolver) + val deadlineEnforcer = QueryDeadlineEnforcer(clock, scheduler) + val trustedAuthorityChannel = TrustedAuthorityChannel.create() + val plannedRegistry = createPlannedRegistry(backendComposition) + val routeResolver = QueryExecutionRouteResolver( + plannedRegistry, + LegacyBackendRegistry(legacyBindings), + ) + val shadowSupervisor = BoundedQueryShadowSupervisor(shadowConfiguration, shadowObserver) + val delegate = InternalQueryGateway( + admissionGuard = RawAdmissionGuard(QueryAdmissionLimits.DEFAULT), + normalizer = QueryNormalizer(clock), + schemaRegistry = schemas, + contextFactory = QueryExecutionContextFactory( + GatewayAuthorityProvider(trustedAuthorityChannel, authorityResolver), + clock, + ), + policyEnforcer = QueryPolicyEnforcer(TenantIsolationQueryPolicy()), + planner = QueryPlanner(), + routeResolver = routeResolver, + executor = QueryExecutor( + deadlineEnforcer = deadlineEnforcer, + shadowSupervisor = shadowSupervisor, + decisionObserver = RuntimeDecisionObserver(runtimeHealthObserver), + shadowProbeTimeout = shadowConfiguration.probeTimeout, + ), + deadlineEnforcer = deadlineEnforcer, + analyticsCursorRuntime = cursorLeaseConfiguration?.toRuntime( + clock, + backendComposition, + runtimeHealthObserver, + ), + ) + return QueryGatewayRuntimeComponents( + DefaultQueryGateway(delegate, executionProfiles, resultMaterializers), + DefaultAnalyticsQueryGateway(delegate, executionProfiles), + trustedAuthorityChannel, + delegate::reapExpiredAnalyticsCursors, + ) + } + + private fun validateContributedSchemas( + targets: List, + contributedSchemas: Map>, + ) { + require(contributedSchemas.keys.all { target -> targets.any { binding -> binding.target == target } }) { + "Query backend composition contains an unknown aggregate target." + } + contributedSchemas.values.forEach { schemas -> + require(schemas.map(QueryDocumentSchema::contractId).distinct().size == 1) { + "Query backend contributions for one target must share a schema contract." + } + } + } + + private fun createSchemaRegistry( + targets: List, + contributedSchemas: Map>, + ): QuerySchemaRegistry = QuerySchemaRegistry( + targets.map { binding -> + contributedSchemas[binding.target]?.first() ?: legacyQuerySchema(binding.target) + }, + ) + + private fun createLegacyBindings( + targets: List, + dialectResolver: QueryLegacyDialectResolver, + ): List = targets.map { binding -> + LegacyExecutionBinding.create( + binding.target, + LegacyDynamicQueryCompiler( + binding.target, + dialectResolver.resolve(binding.target.toPublic()), + binding.identityField, + ), + LegacyDynamicQueryBackend(binding.queryService), + ) + } + + private fun createPlannedRegistry(composition: QueryBackendComposition): QueryBackendRegistry = + QueryBackendRegistry( + composition.contributions.map { contribution -> + QueryBackendRegistration( + descriptor = QueryBackendDescriptor( + key = QueryBackendKey(contribution.schema.target, contribution.backendId), + schemaContractId = contribution.schema.contractId, + supportedOperations = contribution.supportedOperations, + semanticTiers = contribution.semanticTiers.mapTo(linkedSetOf()) { tier -> + SemanticTier.valueOf(tier.name) + }, + fieldCapabilities = contribution.fieldCapabilities, + searchScopes = contribution.searchScopes, + mappingGenerationDigest = contribution.mappingGenerationDigest, + streamSupport = when (contribution.streamSupport) { + BackendStreamSupport.NONE -> QueryBackendStreamSupport.NONE + BackendStreamSupport.BOUNDED_ONLY -> QueryBackendStreamSupport.BOUNDED_ONLY + }, + ), + experimentalRecordBackend = contribution.backend, + analyticsBackend = contribution.analyticsBackend?.let { backend -> + ExperimentalAnalyticsBackendAdapter(backend, contribution.schema) + }, + ) + }, + composition.defaultRoutes, + composition.notReadyBackends.mapTo(linkedSetOf()) { backend -> + QueryBackendKey(backend.schema.target, backend.backendId) + }, + ) + + private fun materializeAggregates(namedAggregates: Iterable) = + namedAggregates.map(NamedAggregate::materialize).also { aggregates -> + require(aggregates.distinct().size == aggregates.size) { + "Query Gateway aggregate targets must be unique." + } + } + + private fun createTargetBindings( + aggregates: List, + rawServiceSource: QueryRawServiceSource, + ): List = aggregates.flatMap { aggregate -> + listOf( + TargetBinding( + InternalQueryTarget(aggregate, InternalDocumentKind.SNAPSHOT), + rawServiceSource.snapshot(aggregate), + MessageRecords.AGGREGATE_ID, + ), + TargetBinding( + InternalQueryTarget(aggregate, InternalDocumentKind.EVENT_STREAM), + rawServiceSource.eventStream(aggregate), + MessageRecords.ID, + ), + ) + } + + private fun validateExecutionProfiles( + targets: List, + backendComposition: QueryBackendComposition, + profiles: QueryExecutionProfiles, + shadowObserver: QueryShadowObserver, + runtimeHealthObserver: QueryRuntimeHealthObserver, + ) { + val knownTargets = targets.toSet() + require(knownTargets.containsAll(profiles.targetProfiles.keys)) { + "Query execution profiles contain an unknown aggregate target." + } + require(knownTargets.containsAll(profiles.operationProfiles.keys.map { key -> key.target })) { + "Query operation profiles contain an unknown aggregate target." + } + val contributionByTargetAndBackend = backendComposition.contributions.associateBy { contribution -> + contribution.schema.target to contribution.backendId + } + val notReadyKeys = backendComposition.notReadyBackends.mapTo(linkedSetOf()) { backend -> + backend.schema.target to backend.backendId + } + targets.forEach { target -> + RUNTIME_OPERATIONS.forEach { operation -> + val profile = profiles.resolve(target, operation) + if (profile.executionMode == QueryExecutionMode.LEGACY) { + return@forEach + } + val backendId = requireNotNull(backendComposition.defaultRoutes[target]) { + "Non-legacy Query execution profile requires a default Backend route for $target." + } + val key = target to backendId + if (key in notReadyKeys) { + require(profile.executionMode == QueryExecutionMode.SHADOW) { + "PLANNED Query execution profile requires a ready Backend contribution for $target/$backendId." + } + return@forEach + } + val contribution = requireNotNull(contributionByTargetAndBackend[key]) { + "Non-legacy Query execution profile requires a Backend contribution for $target/$backendId." + } + val explicitlyScoped = profiles.operationProfiles.containsKey( + me.ahoo.wow.query.gateway.QueryOperationProfileKey(target, operation), + ) + if (profile.executionMode == QueryExecutionMode.PLANNED || explicitlyScoped) { + require(operation in contribution.supportedOperations) { + "Query Backend $backendId does not support configured operation $target/$operation." + } + } + } + } + val shadowConfigured = targets.any { target -> + RUNTIME_OPERATIONS.any { operation -> + profiles.resolve(target, operation).executionMode == QueryExecutionMode.SHADOW + } + } + require(!shadowConfigured || shadowObserver !== QueryShadowObserver.NONE) { + "SHADOW Query execution requires a QueryShadowObserver." + } + require(!shadowConfigured || runtimeHealthObserver !== QueryRuntimeHealthObserver.NONE) { + "SHADOW Query execution requires a QueryRuntimeHealthObserver." + } + } + + private fun InternalQueryTarget.toPublic(): QueryTarget = + QueryTarget( + namedAggregate, + when (documentKind) { + InternalDocumentKind.SNAPSHOT -> QueryDocumentKind.SNAPSHOT + InternalDocumentKind.EVENT_STREAM -> QueryDocumentKind.EVENT_STREAM + }, + ) + + private data class TargetBinding( + val target: InternalQueryTarget, + val queryService: QueryService<*>, + val identityField: String, + ) + + private val RUNTIME_OPERATIONS = listOf( + QueryOperation.SINGLE, + QueryOperation.STREAM, + QueryOperation.PAGE, + QueryOperation.COUNT, + QueryOperation.ANALYZE, + ) + + @OptIn(me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class) + private fun QueryCursorLeaseConfiguration.toRuntime( + clock: Clock, + backendComposition: QueryBackendComposition, + runtimeHealthObserver: QueryRuntimeHealthObserver, + ): AnalyticsCursorRuntime { + val keyRing = QueryCursorSigningKeyRing( + QueryCursorSigningKey(signingKeys.current.id, signingKeys.current.secretCopy()), + signingKeys.previous.map { key -> QueryCursorSigningKey(key.id, key.secretCopy()) }, + ) + val manager = PersistentQueryCursorLeaseManager( + store, + keyRing, + clock, + QueryCursorLeaseLimits( + maxEntries = Int.MAX_VALUE, + maxTtl = maxCursorTtl, + maxBackendStateBytes = maxBackendStateBytes, + ), + ) + val registrations = backendComposition.contributions.mapNotNull { contribution -> + val lifecycle = contribution.analyticsBackend as? AnalyticsQueryCursorLifecycle ?: return@mapNotNull null + PersistentQueryCursorBackendLeaseRegistration( + contribution.schema.target, + contribution.backendId, + ) { state -> + lifecycle.close(BackendAnalyticsCursorState(state.payload())) + } + } + return AnalyticsCursorRuntime( + PersistentQueryCursorLeaseCoordinator( + manager, + registrations, + RuntimeCursorLeaseObserver(runtimeHealthObserver), + ), + leaseTtl, + clock, + ) + } +} + +private class RuntimeDecisionObserver( + private val delegate: QueryRuntimeHealthObserver, +) : QueryDecisionObserver { + override fun onFallback(fallback: QueryFallback) { + delegate.observe( + fallback.target.toPublicTarget(), + fallback.operation.name, + QueryRuntimeHealthKind.FALLBACK, + fallback.issues.values.first().code.name, + ) + } + + override fun onShadowSupervisorFailure(failure: QueryShadowSupervisorFailure) { + delegate.observe( + failure.task.target.toPublicTarget(), + failure.task.operation.name, + QueryRuntimeHealthKind.SHADOW_SUPERVISOR_FAILURE, + failure.issue.code.name, + ) + } +} + +private class RuntimeCursorLeaseObserver( + private val delegate: QueryRuntimeHealthObserver, +) : QueryCursorLeaseObserver { + override fun onCleanupFailure( + descriptor: me.ahoo.wow.query.internal.cursor.QueryCursorLeaseDescriptor, + reason: me.ahoo.wow.query.internal.cursor.QueryCursorCleanupReason, + error: Throwable, + ) { + delegate.observe( + descriptor.target, + QueryOperation.ANALYZE.name, + QueryRuntimeHealthKind.CURSOR_CLEANUP_FAILURE, + "CURSOR_${reason.name}_CLEANUP_FAILED", + ) + } +} + +private fun QueryRuntimeHealthObserver.observe( + target: QueryTarget, + operation: String, + kind: QueryRuntimeHealthKind, + reasonCode: String, +) { + try { + onObservation(QueryRuntimeHealthObservation(target, QueryOperation.valueOf(operation), kind, reasonCode)) + } catch (_: RuntimeException) { + // Runtime observability cannot alter query execution or cleanup. + } +} + +private fun InternalQueryTarget.toPublicTarget(): QueryTarget = QueryTarget( + namedAggregate, + when (documentKind) { + InternalDocumentKind.SNAPSHOT -> QueryDocumentKind.SNAPSHOT + InternalDocumentKind.EVENT_STREAM -> QueryDocumentKind.EVENT_STREAM + }, +) + +internal data class QueryGatewayRuntimeComponents( + val gateway: QueryGateway, + val analyticsGateway: me.ahoo.wow.query.gateway.AnalyticsQueryGateway, + val trustedAuthorityChannel: TrustedAuthorityChannel, + val cursorReaper: (Int) -> reactor.core.publisher.Mono, +) diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/TrustedAuthorityChannel.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/TrustedAuthorityChannel.kt new file mode 100644 index 00000000000..2944d463349 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/gateway/TrustedAuthorityChannel.kt @@ -0,0 +1,36 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.internal.gateway + +import me.ahoo.wow.query.gateway.QueryAuthority +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.util.context.ContextView + +/** Per-runtime object capability used only by the framework-owned compatibility facade. */ +internal class TrustedAuthorityChannel private constructor() { + fun read(context: ContextView): QueryAuthority? = context.getOrDefault(this, null) + + fun bind(source: Mono, authority: QueryAuthority): Mono = + source.contextWrite { context -> context.put(this, authority) } + + fun bind(source: Flux, authority: QueryAuthority): Flux = + source.contextWrite { context -> context.put(this, authority) } + + companion object { + fun create(): TrustedAuthorityChannel = TrustedAuthorityChannel() + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/model/QueryInvocation.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/model/QueryInvocation.kt new file mode 100644 index 00000000000..b1c21fddedb --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/model/QueryInvocation.kt @@ -0,0 +1,97 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.query.internal.model + +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.IListQuery +import me.ahoo.wow.api.query.IPagedQuery +import me.ahoo.wow.api.query.ISingleQuery +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery + +internal typealias QueryDocumentKind = me.ahoo.wow.query.gateway.QueryDocumentKind +internal typealias QueryTarget = me.ahoo.wow.query.gateway.QueryTarget +internal typealias QueryOperation = me.ahoo.wow.query.gateway.QueryOperation +internal typealias QueryResultShape = me.ahoo.wow.query.backend.QueryResultShape +internal typealias RecordResultShape = me.ahoo.wow.query.backend.RecordResultShape +internal typealias QueryExecutionMode = me.ahoo.wow.query.gateway.QueryExecutionMode +internal typealias QueryValidationMode = me.ahoo.wow.query.gateway.QueryValidationMode + +/** + * Invocation input at the application boundary. + * + * Record variants deliberately retain the existing wire DTO so the compatibility adapter can admit and normalize it. + * They are ephemeral references rather than value objects and must never be cached or used as map keys. The gateway + * creates them per subscription and P1-B synchronously materializes an immutable admitted snapshot before normalization. + * AnalyticsWire retains its public wire DTO only until the same per-subscription admission boundary. + */ +internal sealed interface QueryInput { + class Single(val query: ISingleQuery) : QueryInput + + class Stream(val query: IListQuery) : QueryInput + + class Page(val query: IPagedQuery) : QueryInput + + class Count(val condition: Condition) : QueryInput + + data class Analytics(val query: AnalyticsQuery) : QueryInput + + class AnalyticsWire(val query: me.ahoo.wow.api.query.analytics.AnalyticsQuery) : QueryInput +} + +/** + * Per-subscription envelope describing one query operation. + * + * This is intentionally not a value object because legacy record inputs are not deeply immutable. Stable equality, + * fingerprints and cache keys start at the admitted snapshot, normalized query and plan boundaries. + */ +internal class QueryInvocation( + val target: QueryTarget, + val operation: QueryOperation, + val resultShape: QueryResultShape, + val input: QueryInput, +) { + init { + require(operation.accepts(input)) { + "Query input does not match operation $operation." + } + require(operation.accepts(resultShape)) { + "Query result shape $resultShape does not match operation $operation." + } + } + + private fun QueryOperation.accepts(input: QueryInput): Boolean = + when (this) { + QueryOperation.SINGLE -> input is QueryInput.Single + QueryOperation.STREAM -> input is QueryInput.Stream + QueryOperation.PAGE -> input is QueryInput.Page + QueryOperation.COUNT -> input is QueryInput.Count + QueryOperation.ANALYZE -> input is QueryInput.Analytics || input is QueryInput.AnalyticsWire + } + + private fun QueryOperation.accepts(resultShape: QueryResultShape): Boolean = + when (this) { + QueryOperation.SINGLE, + QueryOperation.STREAM, + QueryOperation.PAGE, + -> resultShape == QueryResultShape.TYPED || resultShape == QueryResultShape.DYNAMIC + + QueryOperation.COUNT -> resultShape == QueryResultShape.COUNT + QueryOperation.ANALYZE -> resultShape == QueryResultShape.ANALYTICS + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/normalization/AnalyticsNormalizer.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/normalization/AnalyticsNormalizer.kt new file mode 100644 index 00000000000..a3c6c47012b --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/normalization/AnalyticsNormalizer.kt @@ -0,0 +1,122 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.normalization + +import me.ahoo.wow.api.query.analytics.AnalyticsGroupingKind +import me.ahoo.wow.api.query.analytics.AnalyticsMetricKind +import me.ahoo.wow.query.internal.admission.AdmittedAnalyticsQuery +import me.ahoo.wow.query.internal.admission.AdmittedCondition +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsBucketOrder +import me.ahoo.wow.query.internal.analytics.AnalyticsBucketWindow +import me.ahoo.wow.query.internal.analytics.AnalyticsCompleteness +import me.ahoo.wow.query.internal.analytics.AnalyticsCondition +import me.ahoo.wow.query.internal.analytics.AnalyticsConsistency +import me.ahoo.wow.query.internal.analytics.AnalyticsDimension +import me.ahoo.wow.query.internal.analytics.AnalyticsGrouping +import me.ahoo.wow.query.internal.analytics.AnalyticsMetric +import me.ahoo.wow.query.internal.analytics.AnalyticsMissingPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPromotion +import me.ahoo.wow.query.internal.analytics.AnalyticsOverflowPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.cursor.QueryCursorToken +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.value.NonEmptyList + +internal object AnalyticsNormalizer { + fun normalize( + query: AdmittedAnalyticsQuery, + path: QueryRejectionPath, + conditionNormalizer: (AdmittedCondition, QueryRejectionPath) -> NormalizedCondition, + fieldNormalizer: (String) -> LogicalField.Path, + ): AnalyticsQuery = AnalyticsQuery( + userCondition = conditionNormalizer(query.condition, path.property("condition")), + grouping = normalizeGrouping(query, fieldNormalizer), + metrics = normalizeMetrics(query, fieldNormalizer), + having = AnalyticsCondition.All, + bucketOrder = AnalyticsBucketOrder.Default, + bucketWindow = AnalyticsBucketWindow.First(query.window.limit), + numericPolicy = normalizeNumericPolicy(query), + requiredConsistency = when (query.consistency) { + me.ahoo.wow.api.query.analytics.AnalyticsConsistency.EVENTUAL -> AnalyticsConsistency.EVENTUAL + me.ahoo.wow.api.query.analytics.AnalyticsConsistency.SNAPSHOT -> AnalyticsConsistency.SNAPSHOT + }, + requiredCompleteness = when (query.completeness) { + me.ahoo.wow.api.query.analytics.AnalyticsCompleteness.EXACT -> AnalyticsCompleteness.EXACT + }, + cursorToken = query.window.cursor?.value?.let(::QueryCursorToken), + ) + + private fun normalizeGrouping( + query: AdmittedAnalyticsQuery, + fieldNormalizer: (String) -> LogicalField.Path, + ): AnalyticsGrouping = when (query.grouping.kind) { + AnalyticsGroupingKind.GLOBAL -> AnalyticsGrouping.Global + AnalyticsGroupingKind.BY -> AnalyticsGrouping.By( + checkNotNull( + NonEmptyList.from( + query.grouping.dimensions.map { dimension -> + AnalyticsDimension( + AnalyticsAlias(dimension.alias), + fieldNormalizer(dimension.field), + when (dimension.missingPolicy) { + me.ahoo.wow.api.query.analytics.AnalyticsMissingPolicy.EXCLUDE -> + AnalyticsMissingPolicy.EXCLUDE + + me.ahoo.wow.api.query.analytics.AnalyticsMissingPolicy.AS_NULL_BUCKET -> + AnalyticsMissingPolicy.AS_NULL_BUCKET + }, + ) + }, + ), + ), + ) + } + + private fun normalizeMetrics( + query: AdmittedAnalyticsQuery, + fieldNormalizer: (String) -> LogicalField.Path, + ): NonEmptyList = checkNotNull( + NonEmptyList.from( + query.metrics.map { metric -> + val alias = AnalyticsAlias(metric.alias) + when (metric.kind) { + AnalyticsMetricKind.DOCUMENT_COUNT -> AnalyticsMetric.DocumentCount(alias) + AnalyticsMetricKind.MIN -> AnalyticsMetric.Min(alias, fieldNormalizer(checkNotNull(metric.field))) + AnalyticsMetricKind.MAX -> AnalyticsMetric.Max(alias, fieldNormalizer(checkNotNull(metric.field))) + AnalyticsMetricKind.SUM -> AnalyticsMetric.Sum(alias, fieldNormalizer(checkNotNull(metric.field))) + AnalyticsMetricKind.AVERAGE -> + AnalyticsMetric.Average(alias, fieldNormalizer(checkNotNull(metric.field))) + } + }, + ), + ) + + private fun normalizeNumericPolicy(query: AdmittedAnalyticsQuery): AnalyticsNumericPolicy? = + query.numericPolicy?.let { numericPolicy -> + AnalyticsNumericPolicy( + promotion = when (numericPolicy.promotion) { + me.ahoo.wow.api.query.analytics.AnalyticsNumericPromotion.DECIMAL128 -> + AnalyticsNumericPromotion.DECIMAL128 + }, + precision = numericPolicy.precision, + scale = numericPolicy.scale, + roundingMode = numericPolicy.roundingMode, + overflowPolicy = when (numericPolicy.overflowPolicy) { + me.ahoo.wow.api.query.analytics.AnalyticsOverflowPolicy.REJECT -> AnalyticsOverflowPolicy.REJECT + }, + ) + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/normalization/NormalizedCondition.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/normalization/NormalizedCondition.kt new file mode 100644 index 00000000000..63b495c6908 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/normalization/NormalizedCondition.kt @@ -0,0 +1,128 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + +package me.ahoo.wow.query.internal.normalization + +import me.ahoo.wow.query.backend.NormalizedValue +import java.util.Collections + +internal enum class PathBasis { + ROOT, + CURRENT_ELEMENT, +} + +internal typealias SystemFieldKind = me.ahoo.wow.query.backend.SystemFieldKind + +internal sealed interface LogicalField { + data class System(val kind: SystemFieldKind) : LogicalField + + class Path( + segments: Iterable, + val basis: PathBasis, + ) : LogicalField { + val segments: List = Collections.unmodifiableList(segments.toList()) + + init { + require(this.segments.isNotEmpty()) { + "Logical field path must not be empty." + } + require(this.segments.none { it.isBlank() }) { + "Logical field path segments must not be blank." + } + } + + override fun equals(other: Any?): Boolean = + this === other || other is Path && segments == other.segments && basis == other.basis + + override fun hashCode(): Int = 31 * segments.hashCode() + basis.hashCode() + + override fun toString(): String = "Path(segments=$segments, basis=$basis)" + } +} + +internal typealias JunctionOperator = me.ahoo.wow.query.backend.JunctionOperator +internal typealias PredicateOperator = me.ahoo.wow.query.backend.PredicateOperator +internal typealias CaseSensitivity = me.ahoo.wow.query.backend.CaseSensitivity +internal typealias NormalizedPredicateOptions = me.ahoo.wow.query.backend.NormalizedPredicateOptions +internal typealias SearchScopeId = me.ahoo.wow.query.backend.SearchScopeId + +internal sealed interface SearchScope { + data class Named(val id: SearchScopeId) : SearchScope + + data class LegacyField(val field: LogicalField.Path) : SearchScope +} + +internal typealias BackendId = me.ahoo.wow.query.backend.BackendId +internal typealias Utf8Json = me.ahoo.wow.query.backend.Utf8Json + +internal sealed interface NormalizedCondition { + data object All : NormalizedCondition + + data object None : NormalizedCondition + + class Junction( + val operator: JunctionOperator, + children: Iterable, + ) : NormalizedCondition { + val children: List = Collections.unmodifiableList(children.toList()) + + init { + require(this.children.isNotEmpty()) { + "Junction children must not be empty." + } + } + + override fun equals(other: Any?): Boolean = + this === other || other is Junction && operator == other.operator && children == other.children + + override fun hashCode(): Int = 31 * operator.hashCode() + children.hashCode() + + override fun toString(): String = "Junction(operator=$operator, children=$children)" + } + + data class Predicate( + val field: LogicalField, + val operator: PredicateOperator, + val value: NormalizedValue? = null, + val options: NormalizedPredicateOptions = NormalizedPredicateOptions(), + ) : NormalizedCondition { + init { + require(operator.requiresValue == (value != null)) { + "Predicate value does not match operator $operator." + } + } + } + + data class ElementMatch( + val field: LogicalField.Path, + val condition: NormalizedCondition, + ) : NormalizedCondition + + data class Search( + val scope: SearchScope, + val text: String, + ) : NormalizedCondition { + init { + require(text.isNotBlank()) { + "Search text must not be blank." + } + } + } + + data class Native( + val backendId: BackendId, + val payload: Utf8Json, + ) : NormalizedCondition +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/normalization/NormalizedQuery.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/normalization/NormalizedQuery.kt new file mode 100644 index 00000000000..9522394b492 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/normalization/NormalizedQuery.kt @@ -0,0 +1,109 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + +package me.ahoo.wow.query.internal.normalization + +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.value.NonEmptyList +import java.util.Collections + +internal data class NormalizedQueryInvocation( + val target: QueryTarget, + val operation: QueryOperation, + val resultShape: QueryResultShape, + val input: NormalizedQueryInput, +) + +internal sealed interface NormalizedQueryInput { + data class Single(val query: NormalizedRecordQuery) : NormalizedQueryInput + + data class Stream( + val query: NormalizedRecordQuery, + val limit: Int, + ) : NormalizedQueryInput + + data class Page( + val query: NormalizedRecordQuery, + val page: NormalizedPage, + ) : NormalizedQueryInput + + data class Count( + val userCondition: NormalizedCondition, + val deletionScope: NormalizedDeletionScope, + ) : NormalizedQueryInput + + data class Analytics(val query: AnalyticsQuery) : NormalizedQueryInput +} + +internal class NormalizedRecordQuery( + val userCondition: NormalizedCondition, + val projection: NormalizedProjection, + sort: Iterable, + val deletionScope: NormalizedDeletionScope, +) { + val sort: List = Collections.unmodifiableList(sort.toList()) + + override fun equals(other: Any?): Boolean = + this === other || + other is NormalizedRecordQuery && + userCondition == other.userCondition && + projection == other.projection && + sort == other.sort && + deletionScope == other.deletionScope + + override fun hashCode(): Int { + var result = userCondition.hashCode() + result = 31 * result + projection.hashCode() + result = 31 * result + sort.hashCode() + result = 31 * result + deletionScope.hashCode() + return result + } +} + +/** Captures whether the legacy default-active deletion rule still has to be applied. */ +internal enum class NormalizedDeletionScope { + DEFAULT_ACTIVE, + EXPLICIT, +} + +internal sealed interface NormalizedProjection { + data object All : NormalizedProjection + + data class Include(val fields: NonEmptyList) : NormalizedProjection + + data class Exclude(val fields: NonEmptyList) : NormalizedProjection + + /** Preserved until P1-C applies result-shape and validation-mode policy. */ + data class Mixed( + val include: NonEmptyList, + val exclude: NonEmptyList, + ) : NormalizedProjection +} + +internal typealias NormalizedSortDirection = me.ahoo.wow.query.backend.NormalizedSortDirection + +internal data class NormalizedSort( + val field: LogicalField, + val direction: NormalizedSortDirection, +) + +internal data class NormalizedPage( + val index: Int, + val size: Int, + val offset: Long, +) diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/normalization/QueryNormalizer.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/normalization/QueryNormalizer.kt new file mode 100644 index 00000000000..aa9a3422d2e --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/normalization/QueryNormalizer.kt @@ -0,0 +1,686 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.normalization + +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DeletionState +import me.ahoo.wow.api.query.Operator +import me.ahoo.wow.api.query.Sort +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.admission.AdmittedAnalyticsQuery +import me.ahoo.wow.query.internal.admission.AdmittedCondition +import me.ahoo.wow.query.internal.admission.AdmittedConditionValue +import me.ahoo.wow.query.internal.admission.AdmittedPage +import me.ahoo.wow.query.internal.admission.AdmittedProjection +import me.ahoo.wow.query.internal.admission.AdmittedQueryInput +import me.ahoo.wow.query.internal.admission.AdmittedQueryInvocation +import me.ahoo.wow.query.internal.admission.AdmittedRecordQuery +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import me.ahoo.wow.query.internal.value.NonEmptyList +import java.time.Clock +import java.time.DateTimeException +import java.time.DayOfWeek +import java.time.Instant +import java.time.LocalDate +import java.time.ZoneId +import java.time.format.DateTimeFormatter +import java.time.temporal.TemporalAdjusters +import java.util.LinkedHashSet + +internal class QueryNormalizer( + private val clock: Clock, +) { + fun normalize(invocation: AdmittedQueryInvocation): NormalizedQueryInvocation { + val session = NormalizationSession(clock.instant()) + val inputPath = QueryRejectionPath.ROOT.property("input") + val normalizedInput = + when (val input = invocation.input) { + is AdmittedQueryInput.Single -> NormalizedQueryInput.Single( + normalizeRecordQuery(input.query, inputPath.property("query"), session), + ) + + is AdmittedQueryInput.Stream -> NormalizedQueryInput.Stream( + query = normalizeRecordQuery(input.query, inputPath.property("query"), session), + limit = input.limit, + ) + + is AdmittedQueryInput.Page -> NormalizedQueryInput.Page( + query = normalizeRecordQuery(input.query, inputPath.property("query"), session), + page = input.page.normalize(), + ) + + is AdmittedQueryInput.Count -> NormalizedQueryInput.Count( + userCondition = normalizeCondition( + input.condition, + inputPath.property("condition"), + elementScope = null, + session, + ), + deletionScope = input.condition.deletionScope(), + ) + + is AdmittedQueryInput.Analytics -> NormalizedQueryInput.Analytics(input.query) + is AdmittedQueryInput.AnalyticsWire -> NormalizedQueryInput.Analytics( + normalizeAnalytics(input.query, inputPath.property("query"), session), + ) + } + return NormalizedQueryInvocation( + target = invocation.target, + operation = invocation.operation, + resultShape = invocation.resultShape, + input = normalizedInput, + ) + } + + private fun normalizeAnalytics( + query: AdmittedAnalyticsQuery, + path: QueryRejectionPath, + session: NormalizationSession, + ): AnalyticsQuery = AnalyticsNormalizer.normalize( + query, + path, + conditionNormalizer = { condition, conditionPath -> + normalizeCondition(condition, conditionPath, elementScope = null, session) + }, + fieldNormalizer = { field -> normalizeField(field, elementScope = null) }, + ) + + private fun normalizeRecordQuery( + query: AdmittedRecordQuery, + path: QueryRejectionPath, + session: NormalizationSession, + ): NormalizedRecordQuery = + NormalizedRecordQuery( + userCondition = normalizeCondition( + query.condition, + path.property("condition"), + elementScope = null, + session, + ), + projection = normalizeProjection(query.projection), + sort = query.sort.map { admittedSort -> + NormalizedSort( + field = normalizeField(admittedSort.field, elementScope = null), + direction = + when (admittedSort.direction) { + Sort.Direction.ASC -> NormalizedSortDirection.ASC + Sort.Direction.DESC -> NormalizedSortDirection.DESC + }, + ) + }, + deletionScope = query.condition.deletionScope(), + ) + + private fun AdmittedCondition.deletionScope(): NormalizedDeletionScope = + if (operator == Operator.DELETED || + operator == Operator.AND && children.any { child -> child.operator == Operator.DELETED } + ) { + NormalizedDeletionScope.EXPLICIT + } else { + NormalizedDeletionScope.DEFAULT_ACTIVE + } + + private fun normalizeProjection(projection: AdmittedProjection): NormalizedProjection { + if (projection.include.isNotEmpty() && projection.exclude.isNotEmpty()) { + return NormalizedProjection.Mixed( + include = checkNotNull(NonEmptyList.from(projection.include.map { normalizeField(it, null) })), + exclude = checkNotNull(NonEmptyList.from(projection.exclude.map { normalizeField(it, null) })), + ) + } + if (projection.include.isNotEmpty()) { + return NormalizedProjection.Include( + checkNotNull(NonEmptyList.from(projection.include.map { normalizeField(it, null) })), + ) + } + if (projection.exclude.isNotEmpty()) { + return NormalizedProjection.Exclude( + checkNotNull(NonEmptyList.from(projection.exclude.map { normalizeField(it, null) })), + ) + } + return NormalizedProjection.All + } + + @Suppress("CyclomaticComplexMethod", "LongMethod") + private fun normalizeCondition( + condition: AdmittedCondition, + path: QueryRejectionPath, + elementScope: ElementScope?, + session: NormalizationSession, + ): NormalizedCondition { + validateElementScopedField(condition, path, elementScope) + return when (condition.operator) { + Operator.AND -> normalizeJunction( + JunctionOperator.AND, + normalizeChildren(condition, path, elementScope, session), + ) + + Operator.OR -> normalizeJunction( + JunctionOperator.OR, + normalizeChildren(condition, path, elementScope, session), + ) + + Operator.NOR -> normalizeJunction( + JunctionOperator.NOR, + normalizeChildren(condition, path, elementScope, session), + ) + + Operator.ID -> systemPredicate( + SystemFieldKind.IDENTITY, + PredicateOperator.EQ, + condition.queryValue(), + condition, + path, + elementScope, + ) + + Operator.IDS -> systemCollectionPredicate( + SystemFieldKind.IDENTITY, + PredicateOperator.IN, + condition, + path, + elementScope, + ) + + Operator.AGGREGATE_ID -> systemPredicate( + SystemFieldKind.AGGREGATE_ID, + PredicateOperator.EQ, + condition.queryValue(), + condition, + path, + elementScope, + ) + + Operator.AGGREGATE_IDS -> systemCollectionPredicate( + SystemFieldKind.AGGREGATE_ID, + PredicateOperator.IN, + condition, + path, + elementScope, + ) + + Operator.TENANT_ID -> systemPredicate( + SystemFieldKind.TENANT_ID, + PredicateOperator.EQ, + condition.queryValue(), + condition, + path, + elementScope, + ) + + Operator.OWNER_ID -> systemPredicate( + SystemFieldKind.OWNER_ID, + PredicateOperator.EQ, + condition.queryValue(), + condition, + path, + elementScope, + ) + + Operator.SPACE_ID -> systemPredicate( + SystemFieldKind.SPACE_ID, + PredicateOperator.EQ, + condition.queryValue(), + condition, + path, + elementScope, + ) + + Operator.DELETED -> normalizeDeleted(condition, path, elementScope) + Operator.ALL -> NormalizedCondition.All + Operator.EQ -> fieldPredicate(condition, PredicateOperator.EQ, elementScope) + Operator.NE -> fieldPredicate(condition, PredicateOperator.NE, elementScope) + Operator.GT -> fieldPredicate(condition, PredicateOperator.GT, elementScope) + Operator.LT -> fieldPredicate(condition, PredicateOperator.LT, elementScope) + Operator.GTE -> fieldPredicate(condition, PredicateOperator.GTE, elementScope) + Operator.LTE -> fieldPredicate(condition, PredicateOperator.LTE, elementScope) + Operator.CONTAINS -> fieldPredicate(condition, PredicateOperator.CONTAINS, elementScope) + Operator.IN -> collectionPredicate(condition, PredicateOperator.IN, elementScope) + Operator.NOT_IN -> collectionPredicate(condition, PredicateOperator.NOT_IN, elementScope) + Operator.BETWEEN -> fieldPredicate(condition, PredicateOperator.BETWEEN, elementScope) + Operator.ALL_IN -> collectionPredicate(condition, PredicateOperator.ALL_IN, elementScope) + Operator.STARTS_WITH -> fieldPredicate(condition, PredicateOperator.STARTS_WITH, elementScope) + Operator.ENDS_WITH -> fieldPredicate(condition, PredicateOperator.ENDS_WITH, elementScope) + Operator.ELEM_MATCH -> normalizeElementMatch(condition, path, elementScope, session) + Operator.NULL -> fieldPredicate(condition, PredicateOperator.IS_NULL, elementScope, value = null) + Operator.NOT_NULL -> fieldPredicate(condition, PredicateOperator.NOT_NULL, elementScope, value = null) + Operator.TRUE -> fieldPredicate(condition, PredicateOperator.IS_TRUE, elementScope, value = null) + Operator.FALSE -> fieldPredicate(condition, PredicateOperator.IS_FALSE, elementScope, value = null) + Operator.EXISTS -> fieldPredicate(condition, PredicateOperator.EXISTS, elementScope) + Operator.TODAY -> normalizeTimeRange(condition, path, TimeRangeKind.TODAY, elementScope, session) + Operator.BEFORE_TODAY -> normalizeBeforeToday(condition, path, elementScope, session) + Operator.TOMORROW -> normalizeTimeRange(condition, path, TimeRangeKind.TOMORROW, elementScope, session) + Operator.THIS_WEEK -> normalizeTimeRange(condition, path, TimeRangeKind.THIS_WEEK, elementScope, session) + Operator.NEXT_WEEK -> normalizeTimeRange(condition, path, TimeRangeKind.NEXT_WEEK, elementScope, session) + Operator.LAST_WEEK -> normalizeTimeRange(condition, path, TimeRangeKind.LAST_WEEK, elementScope, session) + Operator.THIS_MONTH -> normalizeTimeRange(condition, path, TimeRangeKind.THIS_MONTH, elementScope, session) + Operator.LAST_MONTH -> normalizeTimeRange(condition, path, TimeRangeKind.LAST_MONTH, elementScope, session) + Operator.RECENT_DAYS -> normalizeRecentDays(condition, path, elementScope, session) + Operator.EARLIER_DAYS -> normalizeEarlierDays(condition, path, elementScope, session) + Operator.MATCH -> NormalizedCondition.Search( + scope = SearchScope.LegacyField(normalizeSearchScope(condition.field, elementScope)), + text = (condition.queryValue() as NormalizedValue.Text).value, + ) + + Operator.RAW -> rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + path, + QueryRejectionCode.NATIVE_BACKEND_UNBOUND, + ) + } + } + + private fun validateElementScopedField( + condition: AdmittedCondition, + path: QueryRejectionPath, + elementScope: ElementScope?, + ) { + if (elementScope == null || condition.field.isEmpty()) { + return + } + val segments = condition.field.split('.') + if (elementScope.qualifiedPrefixes.any { segments == it }) { + rejectInvalid(path.property("field"), QueryRejectionCode.INVALID_FIELD) + } + } + + private fun normalizeChildren( + condition: AdmittedCondition, + path: QueryRejectionPath, + elementScope: ElementScope?, + session: NormalizationSession, + ): List = + condition.children.mapIndexed { index, child -> + normalizeCondition(child, path.property("children").index(index), elementScope, session) + } + + private fun normalizeJunction( + operator: JunctionOperator, + children: List, + ): NormalizedCondition = + when (operator) { + JunctionOperator.AND -> { + if (children.any { it == NormalizedCondition.None }) { + NormalizedCondition.None + } else { + val effective = children.filterNot { it == NormalizedCondition.All } + when (effective.size) { + 0 -> NormalizedCondition.All + 1 -> effective.single() + else -> NormalizedCondition.Junction(operator, effective) + } + } + } + + JunctionOperator.OR -> { + if (children.any { it == NormalizedCondition.All }) { + NormalizedCondition.All + } else { + val effective = children.filterNot { it == NormalizedCondition.None } + when (effective.size) { + 0 -> NormalizedCondition.None + 1 -> effective.single() + else -> NormalizedCondition.Junction(operator, effective) + } + } + } + + JunctionOperator.NOR -> { + if (children.any { it == NormalizedCondition.All }) { + NormalizedCondition.None + } else { + val effective = children.filterNot { it == NormalizedCondition.None } + if (effective.isEmpty()) { + NormalizedCondition.All + } else { + NormalizedCondition.Junction(operator, effective) + } + } + } + } + + private fun normalizeElementMatch( + condition: AdmittedCondition, + path: QueryRejectionPath, + elementScope: ElementScope?, + session: NormalizationSession, + ): NormalizedCondition.ElementMatch { + val normalizedField = normalizeField(condition.field, elementScope) + val nestedScope = elementScope.nest(normalizedField.segments) + return NormalizedCondition.ElementMatch( + field = normalizedField, + condition = normalizeCondition( + condition.children.single(), + path.property("children").index(0), + nestedScope, + session, + ), + ) + } + + private fun fieldPredicate( + condition: AdmittedCondition, + operator: PredicateOperator, + elementScope: ElementScope?, + value: NormalizedValue? = condition.queryValue(), + ): NormalizedCondition = + NormalizedCondition.Predicate( + field = normalizeField(condition.field, elementScope), + operator = operator, + value = value, + options = NormalizedPredicateOptions(condition.options.caseSensitivity), + ) + + private fun collectionPredicate( + condition: AdmittedCondition, + operator: PredicateOperator, + elementScope: ElementScope?, + ): NormalizedCondition { + val values = condition.queryValue() as NormalizedValue.ListValue + val uniqueValues = LinkedHashSet(values.values).toList() + if (uniqueValues.isEmpty()) { + return if (operator == PredicateOperator.NOT_IN) { + NormalizedCondition.All + } else { + NormalizedCondition.None + } + } + return fieldPredicate( + condition, + operator, + elementScope, + NormalizedValue.ListValue(uniqueValues), + ) + } + + private fun systemPredicate( + fieldKind: SystemFieldKind, + operator: PredicateOperator, + value: NormalizedValue?, + condition: AdmittedCondition, + path: QueryRejectionPath, + elementScope: ElementScope?, + ): NormalizedCondition { + ensureRootSystemField(path, elementScope) + return NormalizedCondition.Predicate( + LogicalField.System(fieldKind), + operator, + value, + NormalizedPredicateOptions(condition.options.caseSensitivity), + ) + } + + private fun systemCollectionPredicate( + fieldKind: SystemFieldKind, + operator: PredicateOperator, + condition: AdmittedCondition, + path: QueryRejectionPath, + elementScope: ElementScope?, + ): NormalizedCondition { + ensureRootSystemField(path, elementScope) + val values = (condition.queryValue() as NormalizedValue.ListValue).values + val uniqueValues = LinkedHashSet(values).toList() + if (uniqueValues.isEmpty()) { + return NormalizedCondition.None + } + return systemPredicate( + fieldKind, + operator, + NormalizedValue.ListValue(uniqueValues), + condition, + path, + elementScope, + ) + } + + private fun normalizeDeleted( + condition: AdmittedCondition, + path: QueryRejectionPath, + elementScope: ElementScope?, + ): NormalizedCondition { + ensureRootSystemField(path, elementScope) + return when ((condition.value as AdmittedConditionValue.Deletion).value) { + DeletionState.ACTIVE -> systemPredicate( + SystemFieldKind.DELETED, + PredicateOperator.IS_FALSE, + null, + condition, + path, + elementScope, + ) + + DeletionState.DELETED -> systemPredicate( + SystemFieldKind.DELETED, + PredicateOperator.IS_TRUE, + null, + condition, + path, + elementScope, + ) + + DeletionState.ALL -> NormalizedCondition.All + } + } + + private fun ensureRootSystemField(path: QueryRejectionPath, elementScope: ElementScope?) { + if (elementScope != null) { + rejectInvalid(path, QueryRejectionCode.SYSTEM_FIELD_IN_ELEMENT_SCOPE) + } + } + + private fun normalizeTimeRange( + condition: AdmittedCondition, + path: QueryRejectionPath, + kind: TimeRangeKind, + elementScope: ElementScope?, + session: NormalizationSession, + ): NormalizedCondition { + val zone = condition.options.zoneId ?: clock.zone + val today = session.instant.atZone(zone).toLocalDate() + val (fromDate, toDate) = + when (kind) { + TimeRangeKind.TODAY -> today to today.plusDays(1) + TimeRangeKind.TOMORROW -> today.plusDays(1) to today.plusDays(2) + TimeRangeKind.THIS_WEEK -> weekStart(today) to weekStart(today).plusWeeks(1) + TimeRangeKind.NEXT_WEEK -> weekStart(today).plusWeeks(1) to weekStart(today).plusWeeks(2) + TimeRangeKind.LAST_WEEK -> weekStart(today).minusWeeks(1) to weekStart(today) + TimeRangeKind.THIS_MONTH -> monthStart(today) to monthStart(today).plusMonths(1) + TimeRangeKind.LAST_MONTH -> monthStart(today).minusMonths(1) to monthStart(today) + } + return halfOpenRange( + condition, + elementScope, + fromDate.atStartOfDay(zone).toInstant(), + toDate.atStartOfDay(zone).toInstant(), + zone, + path, + ) + } + + private fun normalizeRecentDays( + condition: AdmittedCondition, + path: QueryRejectionPath, + elementScope: ElementScope?, + session: NormalizationSession, + ): NormalizedCondition { + val zone = condition.options.zoneId ?: clock.zone + val today = session.instant.atZone(zone).toLocalDate() + val days = (condition.queryValue() as NormalizedValue.Int64).value + val from = subtractDays(today, days - 1, path).atStartOfDay(zone).toInstant() + val to = today.plusDays(1).atStartOfDay(zone).toInstant() + return halfOpenRange(condition, elementScope, from, to, zone, path) + } + + private fun normalizeEarlierDays( + condition: AdmittedCondition, + path: QueryRejectionPath, + elementScope: ElementScope?, + session: NormalizationSession, + ): NormalizedCondition { + val zone = condition.options.zoneId ?: clock.zone + val today = session.instant.atZone(zone).toLocalDate() + val days = (condition.queryValue() as NormalizedValue.Int64).value + val cutoff = subtractDays(today, days - 1, path).atStartOfDay(zone).toInstant() + return NormalizedCondition.Predicate( + field = normalizeField(condition.field, elementScope), + operator = PredicateOperator.LT, + value = timeValue(cutoff, condition.options.datePattern?.formatter, zone, path), + ) + } + + private fun normalizeBeforeToday( + condition: AdmittedCondition, + path: QueryRejectionPath, + elementScope: ElementScope?, + session: NormalizationSession, + ): NormalizedCondition { + val zone = condition.options.zoneId ?: clock.zone + val today = session.instant.atZone(zone).toLocalDate() + val time = (condition.value as AdmittedConditionValue.TimeOfDay).value + val cutoff = today.atTime(time).atZone(zone).toInstant() + return NormalizedCondition.Predicate( + field = normalizeField(condition.field, elementScope), + operator = PredicateOperator.LT, + value = timeValue(cutoff, condition.options.datePattern?.formatter, zone, path), + ) + } + + private fun halfOpenRange( + condition: AdmittedCondition, + elementScope: ElementScope?, + from: Instant, + to: Instant, + zone: ZoneId, + path: QueryRejectionPath, + ): NormalizedCondition = + NormalizedCondition.Junction( + JunctionOperator.AND, + listOf( + NormalizedCondition.Predicate( + normalizeField(condition.field, elementScope), + PredicateOperator.GTE, + timeValue(from, condition.options.datePattern?.formatter, zone, path), + ), + NormalizedCondition.Predicate( + normalizeField(condition.field, elementScope), + PredicateOperator.LT, + timeValue(to, condition.options.datePattern?.formatter, zone, path), + ), + ), + ) + + private fun timeValue( + instant: Instant, + datePattern: DateTimeFormatter?, + zone: ZoneId, + path: QueryRejectionPath, + ): NormalizedValue { + if (datePattern == null) { + return NormalizedValue.InstantValue(instant) + } + val formatted = try { + datePattern.format(instant.atZone(zone)) + } catch (error: DateTimeException) { + rejectInvalid( + path.property("options").key(Condition.DATE_PATTERN_OPTION_KEY), + QueryRejectionCode.INVALID_OPTION_VALUE, + error, + ) + } + return NormalizedValue.Text(formatted) + } + + private fun normalizeField(field: String, elementScope: ElementScope?): LogicalField.Path { + val segments = field.split('.') + if (elementScope == null) { + return LogicalField.Path(segments, PathBasis.ROOT) + } + val matchedPrefix = elementScope.qualifiedPrefixes + .filter { prefix -> segments.startsWithPrefix(prefix) } + .maxByOrNull(List::size) + val relative = matchedPrefix?.let { segments.drop(it.size) } ?: segments + return LogicalField.Path(relative, PathBasis.CURRENT_ELEMENT) + } + + private fun normalizeSearchScope(field: String, elementScope: ElementScope?): LogicalField.Path = + normalizeField(field, elementScope) + + private fun ElementScope?.nest(relativeField: List): ElementScope { + if (this == null) { + return ElementScope(relativeField, listOf(relativeField)) + } + val newAbsolute = absoluteSegments + relativeField + val newPrefixes = buildList { + add(newAbsolute) + qualifiedPrefixes.forEach { prefix -> add(prefix + relativeField) } + add(relativeField) + }.distinct() + return ElementScope(newAbsolute, newPrefixes) + } + + private fun AdmittedCondition.queryValue(): NormalizedValue = + (value as AdmittedConditionValue.QueryValue).value + + private fun AdmittedPage.normalize(): NormalizedPage = NormalizedPage(index, size, offset) + + private fun weekStart(date: LocalDate): LocalDate = + date.with(TemporalAdjusters.previousOrSame(DayOfWeek.MONDAY)) + + private fun monthStart(date: LocalDate): LocalDate = date.withDayOfMonth(1) + + private fun subtractDays( + date: LocalDate, + days: Long, + path: QueryRejectionPath, + ): LocalDate = + try { + date.minusDays(days) + } catch (error: DateTimeException) { + rejectQuery( + QueryRejectionCategory.INVALID_QUERY, + path.property("value"), + QueryRejectionCode.INVALID_TIME_VALUE, + error, + ) + } + + private fun List.startsWithPrefix(prefix: List): Boolean = + size >= prefix.size && subList(0, prefix.size) == prefix + + private fun rejectInvalid( + path: QueryRejectionPath, + code: QueryRejectionCode, + cause: Throwable? = null, + ): Nothing = rejectQuery(QueryRejectionCategory.INVALID_QUERY, path, code, cause) + + private data class NormalizationSession(val instant: Instant) + + private data class ElementScope( + val absoluteSegments: List, + val qualifiedPrefixes: List>, + ) + + private enum class TimeRangeKind { + TODAY, + TOMORROW, + THIS_WEEK, + NEXT_WEEK, + LAST_WEEK, + THIS_MONTH, + LAST_MONTH, + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/plan/QueryPlan.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/plan/QueryPlan.kt new file mode 100644 index 00000000000..0c83191d309 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/plan/QueryPlan.kt @@ -0,0 +1,482 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.plan + +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsCompleteness +import me.ahoo.wow.query.internal.analytics.AnalyticsConsistency +import me.ahoo.wow.query.internal.analytics.AnalyticsMissingPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNullPlacement +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsTextCollation +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.model.RecordResultShape +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.JunctionOperator +import me.ahoo.wow.query.internal.normalization.NormalizedPredicateOptions +import me.ahoo.wow.query.internal.normalization.NormalizedSortDirection +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.SearchScopeId +import me.ahoo.wow.query.internal.normalization.Utf8Json +import me.ahoo.wow.query.internal.value.NonEmptyList +import java.util.Collections +import java.util.LinkedHashMap + +@JvmInline +internal value class PlanFingerprint(val value: String) { + init { + require(value.matches(HEX_PATTERN)) { + "Plan fingerprint must be a SHA-256 hex string." + } + } + + private companion object { + val HEX_PATTERN = Regex("[0-9a-f]{64}") + } +} + +internal enum class SemanticTier { + PORTABLE, + SEARCH, + NATIVE, +} + +internal sealed interface PlannedCondition { + data object All : PlannedCondition + + data object None : PlannedCondition + + data class Junction( + val operator: JunctionOperator, + val children: NonEmptyList, + ) : PlannedCondition + + data class Predicate( + val field: QueryFieldId, + val operator: PredicateOperator, + val value: NormalizedValue? = null, + val options: NormalizedPredicateOptions = NormalizedPredicateOptions(), + ) : PlannedCondition { + init { + require(operator.requiresValue == (value != null)) { + "Predicate value does not match operator $operator." + } + } + } + + data class ElementMatch( + val field: QueryFieldId.Path, + val condition: PlannedCondition, + ) : PlannedCondition + + data class Search( + val scope: SearchScopeId, + val text: String, + ) : PlannedCondition + + data class Native( + val backendId: BackendId, + val payload: Utf8Json, + ) : PlannedCondition +} + +internal data class EnforcedFilter( + val user: PlannedCondition, + val mandatory: PlannedCondition, +) { + val condition: PlannedCondition = PlannedCondition.Junction( + JunctionOperator.AND, + NonEmptyList.of(user, mandatory), + ) +} + +internal class RequiredCapabilities( + fieldRequirements: Map> = emptyMap(), + searchRequirements: Set = emptySet(), + val nativeBackend: BackendId? = null, +) { + val fieldRequirements: Map> = immutableRequirements(fieldRequirements) + val searchRequirements: Set = Collections.unmodifiableSet( + LinkedHashSet(searchRequirements.sortedBy(SearchScopeId::value)), + ) + + override fun equals(other: Any?): Boolean = + this === other || + other is RequiredCapabilities && + fieldRequirements == other.fieldRequirements && + searchRequirements == other.searchRequirements && + nativeBackend == other.nativeBackend + + override fun hashCode(): Int { + var result = fieldRequirements.hashCode() + result = 31 * result + searchRequirements.hashCode() + result = 31 * result + (nativeBackend?.hashCode() ?: 0) + return result + } + + private fun immutableRequirements( + requirements: Map>, + ): Map> { + val copy = LinkedHashMap>(requirements.size) + requirements.entries.sortedBy { it.key.stableKey() }.forEach { (field, capabilities) -> + copy[field] = Collections.unmodifiableSet(LinkedHashSet(capabilities.sortedBy(FieldCapability::name))) + } + return Collections.unmodifiableMap(copy) + } +} + +internal sealed interface PlannedProjection { + data object All : PlannedProjection + + data class Include(val fields: NonEmptyList) : PlannedProjection + + data class Exclude(val fields: NonEmptyList) : PlannedProjection +} + +internal enum class PlannedSortOrigin { + USER, + STABILITY_TIE_BREAKER, +} + +internal data class PlannedSort( + val field: QueryFieldId, + val direction: NormalizedSortDirection, + val origin: PlannedSortOrigin, +) + +internal sealed interface QueryPlan { + val target: QueryTarget + val operation: QueryOperation + val schemaContractId: SchemaContractId + val filter: EnforcedFilter + val requiredCapabilities: RequiredCapabilities + val semanticTier: SemanticTier + val fingerprint: PlanFingerprint +} + +internal sealed interface RecordQueryPlan : QueryPlan { + val resultShape: RecordResultShape + val projection: PlannedProjection + val sort: List +} + +@ConsistentCopyVisibility +internal data class SingleQueryPlan private constructor( + override val target: QueryTarget, + override val schemaContractId: SchemaContractId, + override val filter: EnforcedFilter, + override val resultShape: RecordResultShape, + override val projection: PlannedProjection, + override val sort: List, + override val requiredCapabilities: RequiredCapabilities, + override val semanticTier: SemanticTier, +) : RecordQueryPlan { + override val operation: QueryOperation = QueryOperation.SINGLE + override val fingerprint: PlanFingerprint by lazy { QueryPlanFingerprint.compute(this) } + + companion object { + fun create( + target: QueryTarget, + schemaContractId: SchemaContractId, + filter: EnforcedFilter, + resultShape: RecordResultShape, + projection: PlannedProjection, + sort: Iterable, + requiredCapabilities: RequiredCapabilities, + semanticTier: SemanticTier, + ): SingleQueryPlan = SingleQueryPlan( + target, + schemaContractId, + filter, + resultShape, + projection, + Collections.unmodifiableList(sort.toList()), + requiredCapabilities, + semanticTier, + ) + } +} + +internal sealed interface StreamLimit { + data object Unbounded : StreamLimit + + data class Bounded(val value: Int) : StreamLimit { + init { + require(value > 0) { + "Bounded stream limit must be positive." + } + } + } +} + +@ConsistentCopyVisibility +internal data class StreamQueryPlan private constructor( + override val target: QueryTarget, + override val schemaContractId: SchemaContractId, + override val filter: EnforcedFilter, + override val resultShape: RecordResultShape, + override val projection: PlannedProjection, + override val sort: List, + val limit: StreamLimit, + override val requiredCapabilities: RequiredCapabilities, + override val semanticTier: SemanticTier, +) : RecordQueryPlan { + override val operation: QueryOperation = QueryOperation.STREAM + override val fingerprint: PlanFingerprint by lazy { QueryPlanFingerprint.compute(this) } + + companion object { + fun create( + target: QueryTarget, + schemaContractId: SchemaContractId, + filter: EnforcedFilter, + resultShape: RecordResultShape, + projection: PlannedProjection, + sort: Iterable, + limit: StreamLimit, + requiredCapabilities: RequiredCapabilities, + semanticTier: SemanticTier, + ): StreamQueryPlan = StreamQueryPlan( + target, + schemaContractId, + filter, + resultShape, + projection, + Collections.unmodifiableList(sort.toList()), + limit, + requiredCapabilities, + semanticTier, + ) + } +} + +internal data class PageWindow( + val offset: Long, + val size: Int, +) { + init { + require(offset >= 0) { + "Page offset must not be negative." + } + require(size > 0) { + "Page size must be positive." + } + } +} + +internal enum class TotalMode { + EXACT, +} + +internal enum class RequiredConsistency { + SAME_INPUT, +} + +@ConsistentCopyVisibility +internal data class PageQueryPlan private constructor( + override val target: QueryTarget, + override val schemaContractId: SchemaContractId, + override val filter: EnforcedFilter, + override val resultShape: RecordResultShape, + override val projection: PlannedProjection, + override val sort: List, + val page: PageWindow, + val totalMode: TotalMode, + val requiredConsistency: RequiredConsistency, + override val requiredCapabilities: RequiredCapabilities, + override val semanticTier: SemanticTier, +) : RecordQueryPlan { + override val operation: QueryOperation = QueryOperation.PAGE + override val fingerprint: PlanFingerprint by lazy { QueryPlanFingerprint.compute(this) } + + companion object { + fun create( + target: QueryTarget, + schemaContractId: SchemaContractId, + filter: EnforcedFilter, + resultShape: RecordResultShape, + projection: PlannedProjection, + sort: Iterable, + page: PageWindow, + totalMode: TotalMode, + requiredConsistency: RequiredConsistency, + requiredCapabilities: RequiredCapabilities, + semanticTier: SemanticTier, + ): PageQueryPlan = PageQueryPlan( + target, + schemaContractId, + filter, + resultShape, + projection, + Collections.unmodifiableList(sort.toList()), + page, + totalMode, + requiredConsistency, + requiredCapabilities, + semanticTier, + ) + } +} + +@ConsistentCopyVisibility +internal data class CountQueryPlan private constructor( + override val target: QueryTarget, + override val schemaContractId: SchemaContractId, + override val filter: EnforcedFilter, + override val requiredCapabilities: RequiredCapabilities, + override val semanticTier: SemanticTier, +) : QueryPlan { + override val operation: QueryOperation = QueryOperation.COUNT + override val fingerprint: PlanFingerprint by lazy { QueryPlanFingerprint.compute(this) } + + companion object { + fun create( + target: QueryTarget, + schemaContractId: SchemaContractId, + filter: EnforcedFilter, + requiredCapabilities: RequiredCapabilities, + semanticTier: SemanticTier, + ): CountQueryPlan = CountQueryPlan( + target, + schemaContractId, + filter, + requiredCapabilities, + semanticTier, + ) + } +} + +internal data class PlannedAnalyticsDimension( + val alias: AnalyticsAlias, + val field: QueryFieldId, + val missingPolicy: AnalyticsMissingPolicy, +) + +internal sealed interface PlannedAnalyticsGrouping { + data object Global : PlannedAnalyticsGrouping + + data class By(val dimensions: NonEmptyList) : PlannedAnalyticsGrouping +} + +internal sealed interface PlannedAnalyticsMetric { + val alias: AnalyticsAlias + + data class DocumentCount(override val alias: AnalyticsAlias) : PlannedAnalyticsMetric + + data class Min( + override val alias: AnalyticsAlias, + val field: QueryFieldId, + ) : PlannedAnalyticsMetric + + data class Max( + override val alias: AnalyticsAlias, + val field: QueryFieldId, + ) : PlannedAnalyticsMetric + + data class Sum( + override val alias: AnalyticsAlias, + val field: QueryFieldId, + ) : PlannedAnalyticsMetric + + data class Average( + override val alias: AnalyticsAlias, + val field: QueryFieldId, + ) : PlannedAnalyticsMetric +} + +internal sealed interface PlannedAnalyticsCondition { + data object All : PlannedAnalyticsCondition +} + +internal data class AnalyticsPageWindow( + val limit: Int, + val afterKey: NonEmptyList? = null, +) { + init { + require(limit > 0) { + "Analytics bucket limit must be positive." + } + } +} + +internal sealed interface PlannedAnalyticsBucketOrder { + data object Global : PlannedAnalyticsBucketOrder + + data class DimensionKeyAscending( + val nullPlacement: AnalyticsNullPlacement, + val textCollation: AnalyticsTextCollation, + ) : PlannedAnalyticsBucketOrder +} + +@ConsistentCopyVisibility +internal data class AnalyticsQueryPlan private constructor( + override val target: QueryTarget, + override val schemaContractId: SchemaContractId, + override val filter: EnforcedFilter, + val grouping: PlannedAnalyticsGrouping, + val metrics: NonEmptyList, + val having: PlannedAnalyticsCondition, + val bucketOrder: PlannedAnalyticsBucketOrder, + val bucketWindow: AnalyticsPageWindow, + val numericPolicy: AnalyticsNumericPolicy?, + val requiredConsistency: AnalyticsConsistency, + val requiredCompleteness: AnalyticsCompleteness, + override val requiredCapabilities: RequiredCapabilities, + override val semanticTier: SemanticTier, +) : QueryPlan { + override val operation: QueryOperation = QueryOperation.ANALYZE + override val fingerprint: PlanFingerprint by lazy { QueryPlanFingerprint.compute(this) } + + companion object { + fun create( + target: QueryTarget, + schemaContractId: SchemaContractId, + filter: EnforcedFilter, + grouping: PlannedAnalyticsGrouping, + metrics: NonEmptyList, + having: PlannedAnalyticsCondition, + bucketOrder: PlannedAnalyticsBucketOrder, + bucketWindow: AnalyticsPageWindow, + numericPolicy: AnalyticsNumericPolicy?, + requiredConsistency: AnalyticsConsistency, + requiredCompleteness: AnalyticsCompleteness, + requiredCapabilities: RequiredCapabilities, + semanticTier: SemanticTier, + ): AnalyticsQueryPlan = AnalyticsQueryPlan( + target, + schemaContractId, + filter, + grouping, + metrics, + having, + bucketOrder, + bucketWindow, + numericPolicy, + requiredConsistency, + requiredCompleteness, + requiredCapabilities, + semanticTier, + ) + } +} + +private fun QueryFieldId.stableKey(): String = + when (this) { + is QueryFieldId.System -> "0:${kind.name}" + is QueryFieldId.Path -> "1:${segments.joinToString("\u0000")}" + } diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/plan/QueryPlanFingerprint.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/plan/QueryPlanFingerprint.kt new file mode 100644 index 00000000000..edd0637954b --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/plan/QueryPlanFingerprint.kt @@ -0,0 +1,290 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.plan + +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryFieldId +import java.io.ByteArrayOutputStream +import java.io.DataOutputStream +import java.security.MessageDigest + +internal object QueryPlanFingerprint { + const val VERSION: Int = 1 + + fun compute(plan: QueryPlan): PlanFingerprint { + val bytes = ByteArrayOutputStream() + DataOutputStream(bytes).use { output -> + output.writeInt(VERSION) + output.writeUtf8(plan.target.namedAggregate.contextName) + output.writeUtf8(plan.target.namedAggregate.aggregateName) + output.writeUtf8(plan.target.documentKind.name) + output.writeUtf8(plan.operation.name) + output.writeUtf8(plan.schemaContractId.value) + output.writeCondition(plan.filter.user) + output.writeCondition(plan.filter.mandatory) + output.writeUtf8(plan.semanticTier.name) + output.writeCapabilities(plan.requiredCapabilities) + when (plan) { + is SingleQueryPlan -> output.writeRecord(plan) + is StreamQueryPlan -> { + output.writeRecord(plan) + when (val limit = plan.limit) { + StreamLimit.Unbounded -> output.writeByte(0) + is StreamLimit.Bounded -> { + output.writeByte(1) + output.writeInt(limit.value) + } + } + } + + is PageQueryPlan -> { + output.writeRecord(plan) + output.writeLong(plan.page.offset) + output.writeInt(plan.page.size) + output.writeUtf8(plan.totalMode.name) + output.writeUtf8(plan.requiredConsistency.name) + } + + is CountQueryPlan -> Unit + is AnalyticsQueryPlan -> output.writeAnalytics(plan) + } + } + return PlanFingerprint(MessageDigest.getInstance("SHA-256").digest(bytes.toByteArray()).toHex()) + } + + private fun DataOutputStream.writeRecord(plan: RecordQueryPlan) { + writeUtf8(plan.resultShape.name) + writeProjection(plan.projection) + writeInt(plan.sort.size) + plan.sort.forEach { sort -> + writeField(sort.field) + writeUtf8(sort.direction.name) + writeUtf8(sort.origin.name) + } + } + + private fun DataOutputStream.writeAnalytics(plan: AnalyticsQueryPlan) { + when (val grouping = plan.grouping) { + PlannedAnalyticsGrouping.Global -> writeByte(0) + is PlannedAnalyticsGrouping.By -> { + writeByte(1) + writeInt(grouping.dimensions.values.size) + grouping.dimensions.values.forEach { dimension -> + writeUtf8(dimension.alias.value) + writeField(dimension.field) + writeUtf8(dimension.missingPolicy.name) + } + } + } + writeInt(plan.metrics.values.size) + plan.metrics.values.forEach { metric -> + when (metric) { + is PlannedAnalyticsMetric.DocumentCount -> writeByte(0) + is PlannedAnalyticsMetric.Min -> { + writeByte(1) + writeField(metric.field) + } + + is PlannedAnalyticsMetric.Max -> { + writeByte(2) + writeField(metric.field) + } + + is PlannedAnalyticsMetric.Sum -> { + writeByte(3) + writeField(metric.field) + } + + is PlannedAnalyticsMetric.Average -> { + writeByte(4) + writeField(metric.field) + } + } + writeUtf8(metric.alias.value) + } + when (plan.having) { + PlannedAnalyticsCondition.All -> writeByte(0) + } + when (val order = plan.bucketOrder) { + PlannedAnalyticsBucketOrder.Global -> writeByte(0) + is PlannedAnalyticsBucketOrder.DimensionKeyAscending -> { + writeByte(1) + writeUtf8(order.nullPlacement.name) + writeUtf8(order.textCollation.name) + } + } + writeInt(plan.bucketWindow.limit) + writeBoolean(plan.numericPolicy != null) + plan.numericPolicy?.let { policy -> + writeUtf8(policy.promotion.name) + writeInt(policy.precision) + writeInt(policy.scale) + writeUtf8(policy.roundingMode.name) + writeUtf8(policy.overflowPolicy.name) + } + writeUtf8(plan.requiredConsistency.name) + writeUtf8(plan.requiredCompleteness.name) + } + + private fun DataOutputStream.writeProjection(projection: PlannedProjection) { + when (projection) { + PlannedProjection.All -> writeByte(0) + is PlannedProjection.Include -> { + writeByte(1) + writeFields(projection.fields.values) + } + + is PlannedProjection.Exclude -> { + writeByte(2) + writeFields(projection.fields.values) + } + } + } + + private fun DataOutputStream.writeCondition(condition: PlannedCondition) { + when (condition) { + PlannedCondition.All -> writeByte(0) + PlannedCondition.None -> writeByte(1) + is PlannedCondition.Junction -> { + writeByte(2) + writeUtf8(condition.operator.name) + writeInt(condition.children.values.size) + condition.children.values.forEach { child -> writeCondition(child) } + } + + is PlannedCondition.Predicate -> { + writeByte(3) + writeField(condition.field) + writeUtf8(condition.operator.name) + writeBoolean(condition.value != null) + condition.value?.let { value -> writeValue(value) } + writeUtf8(condition.options.caseSensitivity.name) + } + + is PlannedCondition.ElementMatch -> { + writeByte(4) + writeField(condition.field) + writeCondition(condition.condition) + } + + is PlannedCondition.Search -> { + writeByte(5) + writeUtf8(condition.scope.value) + writeUtf8(condition.text) + } + + is PlannedCondition.Native -> { + writeByte(6) + writeUtf8(condition.backendId.value) + writeUtf8(condition.payload.value) + } + } + } + + private fun DataOutputStream.writeValue(value: NormalizedValue) { + when (value) { + NormalizedValue.Null -> writeByte(0) + is NormalizedValue.BooleanValue -> { + writeByte(1) + writeBoolean(value.value) + } + + is NormalizedValue.Text -> { + writeByte(2) + writeUtf8(value.value) + } + + is NormalizedValue.Int64 -> { + writeByte(3) + writeLong(value.value) + } + + is NormalizedValue.Decimal -> { + writeByte(4) + writeByteArray(value.value.unscaledValue().toByteArray()) + writeInt(value.value.scale()) + } + + is NormalizedValue.InstantValue -> { + writeByte(5) + writeLong(value.value.epochSecond) + writeInt(value.value.nano) + } + + is NormalizedValue.Bytes -> { + writeByte(6) + writeByteArray(value.toByteArray()) + } + + is NormalizedValue.ListValue -> { + writeByte(7) + writeInt(value.values.size) + value.values.forEach { item -> writeValue(item) } + } + + is NormalizedValue.ObjectValue -> { + writeByte(8) + writeInt(value.values.size) + value.values.forEach { (key, item) -> + writeUtf8(key) + writeValue(item) + } + } + } + } + + private fun DataOutputStream.writeCapabilities(capabilities: RequiredCapabilities) { + writeInt(capabilities.fieldRequirements.size) + capabilities.fieldRequirements.forEach { (field, requirements) -> + writeField(field) + writeStrings(requirements.map { it.name }.sorted()) + } + writeStrings(capabilities.searchRequirements.map { it.value }.sorted()) + writeBoolean(capabilities.nativeBackend != null) + capabilities.nativeBackend?.let { writeUtf8(it.value) } + } + + private fun DataOutputStream.writeFields(fields: List) { + writeInt(fields.size) + fields.forEach { field -> writeField(field) } + } + + private fun DataOutputStream.writeField(field: QueryFieldId) { + when (field) { + is QueryFieldId.System -> { + writeByte(0) + writeUtf8(field.kind.name) + } + + is QueryFieldId.Path -> { + writeByte(1) + writeStrings(field.segments) + } + } + } + + private fun DataOutputStream.writeStrings(values: List) { + writeInt(values.size) + values.forEach { value -> writeUtf8(value) } + } + + private fun DataOutputStream.writeByteArray(value: ByteArray) { + writeInt(value.size) + write(value) + } + + private fun DataOutputStream.writeUtf8(value: String) = writeByteArray(value.toByteArray(Charsets.UTF_8)) + + private fun ByteArray.toHex(): String = joinToString("") { byte -> "%02x".format(byte) } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/AnalyticsQueryPlanner.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/AnalyticsQueryPlanner.kt new file mode 100644 index 00000000000..28b9edf22ac --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/AnalyticsQueryPlanner.kt @@ -0,0 +1,421 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsBucketOrder +import me.ahoo.wow.query.internal.analytics.AnalyticsBucketWindow +import me.ahoo.wow.query.internal.analytics.AnalyticsCompleteness +import me.ahoo.wow.query.internal.analytics.AnalyticsCondition +import me.ahoo.wow.query.internal.analytics.AnalyticsGrouping +import me.ahoo.wow.query.internal.analytics.AnalyticsMetric +import me.ahoo.wow.query.internal.analytics.AnalyticsMissingPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNullPlacement +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPromotion +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.analytics.AnalyticsTextCollation +import me.ahoo.wow.query.internal.analytics.DecodedAnalyticsCursor +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.plan.AnalyticsPageWindow +import me.ahoo.wow.query.internal.plan.AnalyticsQueryPlan +import me.ahoo.wow.query.internal.plan.EnforcedFilter +import me.ahoo.wow.query.internal.plan.PlanFingerprint +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsBucketOrder +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsCondition +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsDimension +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsGrouping +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsMetric +import me.ahoo.wow.query.internal.plan.RequiredCapabilities +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import me.ahoo.wow.query.internal.value.NonEmptyList + +internal class AnalyticsQueryPlanner( + private val invocation: NormalizedQueryInvocation, + private val schema: QueryDocumentSchema, + private val conditionPlanner: QueryConditionPlanner, + private val constraints: PlanningConstraints, + private val mandatory: ValidatedMandatory, +) { + private val queryPath = QueryRejectionPath.ROOT.property("input").property("query") + + fun plan(query: AnalyticsQuery): AnalyticsQueryPlan { + validateContract(query) + validateAliases(query) + val user = conditionPlanner.plan( + query.userCondition, + queryPath.property("userCondition"), + mandatory = false, + fieldConstraint = constraints.fieldConstraint, + ) + if (user.semanticTier != SemanticTier.PORTABLE) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + queryPath.property("userCondition"), + QueryRejectionCode.CAPABILITY_UNAVAILABLE, + ) + } + if (mandatory.semanticTier != SemanticTier.PORTABLE) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionPath.ROOT.property("constraints").property("mandatoryCondition"), + QueryRejectionCode.CAPABILITY_UNAVAILABLE, + ) + } + val requirements = linkedMapOf>() + val grouping = planGrouping(query.grouping, requirements) + val metrics = planMetrics(query, requirements) + val capabilities = mergeCapabilities( + user.requiredCapabilities, + mandatory.requiredCapabilities, + RequiredCapabilities(requirements.mapValues { it.value.toSet() }), + ) + val afterKey = (query.bucketWindow as? AnalyticsBucketWindow.After)?.cursor?.afterKey + val plan = AnalyticsQueryPlan.create( + invocation.target, + schema.contractId, + EnforcedFilter(user.condition, mandatory.condition), + grouping, + metrics.values, + PlannedAnalyticsCondition.All, + query.grouping.plannedOrder(), + AnalyticsPageWindow(query.effectiveBucketLimit(), afterKey), + metrics.numericPolicy, + query.requiredConsistency, + query.requiredCompleteness, + capabilities, + SemanticTier.PORTABLE, + ) + (query.bucketWindow as? AnalyticsBucketWindow.After)?.cursor?.let { cursor -> + validateCursor(cursor, grouping, plan.fingerprint) + } + return plan + } + + private fun validateContract(query: AnalyticsQuery) { + validateBudget(query) + if (invocation.target.documentKind != QueryDocumentKind.SNAPSHOT) { + rejectUnsupported( + "target.documentKind", + QueryRejectionCode.ANALYTICS_DOCUMENT_KIND_UNSUPPORTED, + root = true, + ) + } + if (query.having != AnalyticsCondition.All) { + rejectUnsupported("having", QueryRejectionCode.ANALYTICS_HAVING_UNSUPPORTED) + } + if (!query.hasPortableOrder()) { + rejectUnsupported("bucketOrder", QueryRejectionCode.ANALYTICS_ORDER_UNSUPPORTED) + } + if (query.requiredCompleteness != AnalyticsCompleteness.EXACT) { + rejectUnsupported("requiredCompleteness", QueryRejectionCode.ANALYTICS_COMPLETENESS_UNSUPPORTED) + } + if (query.grouping == AnalyticsGrouping.Global && query.bucketWindow is AnalyticsBucketWindow.After) { + rejectInvalidCursor(queryPath.property("bucketWindow").property("cursor")) + } + if (query.grouping == AnalyticsGrouping.Global && query.cursorToken != null) { + rejectInvalidCursor(queryPath.property("bucketWindow").property("cursor")) + } + if (query.cursorToken != null && query.bucketWindow is AnalyticsBucketWindow.After) { + rejectInvalidCursor(queryPath.property("bucketWindow").property("cursor")) + } + } + + private fun validateBudget(query: AnalyticsQuery) { + val limits = constraints.analyticsConstraint as? AnalyticsPlanningConstraint.Limits ?: return + val dimensions = (query.grouping as? AnalyticsGrouping.By)?.dimensions?.values?.size ?: 0 + if (dimensions > limits.maxDimensions) { + rejectBudget("grouping.dimensions", QueryRejectionCode.ANALYTICS_DIMENSION_LIMIT_EXCEEDED) + } + if (query.metrics.values.size > limits.maxMetrics) { + rejectBudget("metrics", QueryRejectionCode.ANALYTICS_METRIC_LIMIT_EXCEEDED) + } + if (query.effectiveBucketLimit() > limits.maxBucketLimit) { + rejectBudget("bucketWindow.limit", QueryRejectionCode.ANALYTICS_BUCKET_LIMIT_EXCEEDED) + } + } + + private fun validateAliases(query: AnalyticsQuery) { + val seen = mutableSetOf() + val dimensions = (query.grouping as? AnalyticsGrouping.By)?.dimensions?.values.orEmpty() + dimensions.forEachIndexed { index, dimension -> + if (!seen.add(dimension.alias)) { + rejectDuplicateAlias(queryPath.property("grouping").property("dimensions").index(index)) + } + } + query.metrics.values.forEachIndexed { index, metric -> + if (!seen.add(metric.alias)) { + rejectDuplicateAlias(queryPath.property("metrics").index(index)) + } + } + } + + private fun planGrouping( + grouping: AnalyticsGrouping, + requirements: MutableMap>, + ): PlannedAnalyticsGrouping = + when (grouping) { + AnalyticsGrouping.Global -> PlannedAnalyticsGrouping.Global + is AnalyticsGrouping.By -> PlannedAnalyticsGrouping.By( + checkNotNull( + NonEmptyList.from( + grouping.dimensions.values.mapIndexed { index, dimension -> + val path = queryPath.property("grouping").property("dimensions").index(index) + val field = conditionPlanner.resolveAccessibleField( + dimension.field, + path.property("field"), + constraints.fieldConstraint.analyticsDimensionFields, + QueryRejectionCode.ANALYTICS_DIMENSION_FIELD_NOT_ALLOWED, + ) + if (!schema.fields.getValue(field).type.isPortableDimension()) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + path.property("field"), + QueryRejectionCode.ANALYTICS_DIMENSION_TYPE_UNSUPPORTED, + ) + } + if (field is QueryFieldId.Path && schema.elementOwner(field) != null) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + path.property("field"), + QueryRejectionCode.ANALYTICS_DIMENSION_TYPE_UNSUPPORTED, + ) + } + requireFieldCapability(schema, field, FieldCapability.AGGREGATABLE, path.property("field")) + requirements.require(field, FieldCapability.AGGREGATABLE) + PlannedAnalyticsDimension(dimension.alias, field, dimension.missingPolicy) + }, + ), + ), + ) + } + + private fun planMetrics( + query: AnalyticsQuery, + requirements: MutableMap>, + ): PlannedMetrics { + var needsNumericPolicy = false + val metrics = query.metrics.values.mapIndexed { index, metric -> + val path = queryPath.property("metrics").index(index) + when (metric) { + is AnalyticsMetric.DocumentCount -> PlannedAnalyticsMetric.DocumentCount(metric.alias) + is AnalyticsMetric.Min -> { + val field = planMetricField(metric.field, path, allowInstant = true) + needsNumericPolicy = needsNumericPolicy || schema.fields.getValue(field).type.isNumeric() + requirements.require(field, FieldCapability.AGGREGATABLE) + PlannedAnalyticsMetric.Min(metric.alias, field) + } + + is AnalyticsMetric.Max -> { + val field = planMetricField(metric.field, path, allowInstant = true) + needsNumericPolicy = needsNumericPolicy || schema.fields.getValue(field).type.isNumeric() + requirements.require(field, FieldCapability.AGGREGATABLE) + PlannedAnalyticsMetric.Max(metric.alias, field) + } + + is AnalyticsMetric.Sum -> { + val field = planMetricField(metric.field, path, allowInstant = false) + needsNumericPolicy = true + requirements.require(field, FieldCapability.AGGREGATABLE) + PlannedAnalyticsMetric.Sum(metric.alias, field) + } + + is AnalyticsMetric.Average -> { + val field = planMetricField(metric.field, path, allowInstant = false) + needsNumericPolicy = true + requirements.require(field, FieldCapability.AGGREGATABLE) + PlannedAnalyticsMetric.Average(metric.alias, field) + } + } + } + if (needsNumericPolicy && query.numericPolicy == null) { + rejectUnsupported("numericPolicy", QueryRejectionCode.ANALYTICS_NUMERIC_POLICY_REQUIRED) + } + val numericPolicy = query.numericPolicy.takeIf { needsNumericPolicy } + numericPolicy?.let { policy -> + if (policy.promotion != AnalyticsNumericPromotion.DECIMAL128 || + policy.precision > DECIMAL128_MAX_PRECISION + ) { + rejectUnsupported("numericPolicy", QueryRejectionCode.ANALYTICS_NUMERIC_POLICY_UNSUPPORTED) + } + } + return PlannedMetrics(checkNotNull(NonEmptyList.from(metrics)), numericPolicy) + } + + private fun planMetricField( + logicalField: LogicalField, + path: QueryRejectionPath, + allowInstant: Boolean, + ): QueryFieldId { + val field = conditionPlanner.resolveAccessibleField( + logicalField, + path.property("field"), + constraints.fieldConstraint.analyticsMetricFields, + QueryRejectionCode.ANALYTICS_METRIC_FIELD_NOT_ALLOWED, + ) + val type = schema.fields.getValue(field).type + if (field is QueryFieldId.Path && schema.elementOwner(field) != null) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + path.property("field"), + QueryRejectionCode.ANALYTICS_METRIC_TYPE_UNSUPPORTED, + ) + } + if (!type.isNumeric() && !(allowInstant && type == LogicalFieldType.Instant)) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + path.property("field"), + QueryRejectionCode.ANALYTICS_METRIC_TYPE_UNSUPPORTED, + ) + } + requireFieldCapability(schema, field, FieldCapability.AGGREGATABLE, path.property("field")) + return field + } + + private fun validateCursor( + cursor: DecodedAnalyticsCursor, + grouping: PlannedAnalyticsGrouping, + fingerprint: PlanFingerprint, + ) { + val path = queryPath.property("bucketWindow").property("cursor") + if (cursor.target != invocation.target) { + rejectInvalidCursor(path.property("target")) + } + if (cursor.planFingerprint != fingerprint) { + rejectInvalidCursor(path.property("planFingerprint")) + } + val dimensions = (grouping as? PlannedAnalyticsGrouping.By)?.dimensions?.values + ?: rejectInvalidCursor(path.property("dimensionAliases")) + if (cursor.dimensionAliases.values != dimensions.map { it.alias }) { + rejectInvalidCursor(path.property("dimensionAliases")) + } + if (cursor.afterKey.values.size != dimensions.size) { + rejectInvalidCursor(path.property("afterKey")) + } + cursor.afterKey.values.forEachIndexed { index, value -> + val dimension = dimensions[index] + if (!acceptsCursorKey(dimension, value)) { + rejectInvalidCursor(path.property("afterKey").index(index)) + } + } + } + + private fun acceptsCursorKey( + dimension: PlannedAnalyticsDimension, + value: NormalizedValue, + ): Boolean { + val fieldSchema = schema.fields.getValue(dimension.field) + if (value == NormalizedValue.Null) { + return dimension.missingPolicy == AnalyticsMissingPolicy.AS_NULL_BUCKET && + (fieldSchema.presence == Presence.OPTIONAL || fieldSchema.nullability == Nullability.NULLABLE) + } + return when (fieldSchema.type) { + LogicalFieldType.Text -> value is NormalizedValue.Text + LogicalFieldType.Boolean -> value is NormalizedValue.BooleanValue + LogicalFieldType.Int64 -> value is NormalizedValue.Int64 + LogicalFieldType.Decimal -> value is NormalizedValue.Decimal + LogicalFieldType.Instant -> value is NormalizedValue.InstantValue + LogicalFieldType.Bytes, + LogicalFieldType.Object, + is LogicalFieldType.Array, + -> false + } + } + + private fun rejectUnsupported( + property: String, + code: QueryRejectionCode, + root: Boolean = false, + ): Nothing { + val path = if (root) { + property.split('.').fold(QueryRejectionPath.ROOT) { current, segment -> current.property(segment) } + } else { + queryPath.property(property) + } + rejectQuery(QueryRejectionCategory.UNSUPPORTED_FEATURE, path, code) + } + + private fun rejectDuplicateAlias(path: QueryRejectionPath): Nothing = rejectQuery( + QueryRejectionCategory.INVALID_QUERY, + path.property("alias"), + QueryRejectionCode.DUPLICATE_ANALYTICS_ALIAS, + ) + + private fun rejectBudget(property: String, code: QueryRejectionCode): Nothing { + val path = property.split('.').fold(queryPath) { current, segment -> current.property(segment) } + rejectQuery(QueryRejectionCategory.BUDGET_EXCEEDED, path, code) + } + + private fun rejectInvalidCursor(path: QueryRejectionPath): Nothing = rejectQuery( + QueryRejectionCategory.INVALID_CURSOR, + path, + QueryRejectionCode.INVALID_CURSOR_BINDING, + ) + + private data class PlannedMetrics( + val values: NonEmptyList, + val numericPolicy: AnalyticsNumericPolicy?, + ) +} + +private fun MutableMap>.require( + field: QueryFieldId, + capability: FieldCapability, +) { + getOrPut(field, ::linkedSetOf) += capability +} + +private fun LogicalFieldType.isNumeric(): Boolean = this == LogicalFieldType.Int64 || this == LogicalFieldType.Decimal + +private fun LogicalFieldType.isPortableDimension(): Boolean = + this == LogicalFieldType.Text || + this == LogicalFieldType.Boolean || + this == LogicalFieldType.Int64 || + this == LogicalFieldType.Decimal || + this == LogicalFieldType.Instant + +private fun AnalyticsQuery.hasPortableOrder(): Boolean = + when (grouping) { + AnalyticsGrouping.Global -> bucketOrder == AnalyticsBucketOrder.Default + is AnalyticsGrouping.By -> { + bucketOrder == AnalyticsBucketOrder.Default || bucketOrder == AnalyticsBucketOrder.DimensionKeyAscending + } + } + +private fun AnalyticsGrouping.plannedOrder(): PlannedAnalyticsBucketOrder = + when (this) { + AnalyticsGrouping.Global -> PlannedAnalyticsBucketOrder.Global + is AnalyticsGrouping.By -> PlannedAnalyticsBucketOrder.DimensionKeyAscending( + AnalyticsNullPlacement.FIRST, + AnalyticsTextCollation.BINARY, + ) + } + +private fun AnalyticsQuery.effectiveBucketLimit(): Int = + if (grouping == AnalyticsGrouping.Global) 1 else bucketWindow.limit + +private const val DECIMAL128_MAX_PRECISION = 34 diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/PlanningAccessConstraint.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/PlanningAccessConstraint.kt new file mode 100644 index 00000000000..6d5fa95568f --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/PlanningAccessConstraint.kt @@ -0,0 +1,114 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.wow.query.backend.QUERY_FIELD_ID_COMPARATOR +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.SearchScopeId +import java.util.Collections + +internal sealed interface FieldAccess { + fun permits(field: QueryFieldId): Boolean + + data object Unrestricted : FieldAccess { + override fun permits(field: QueryFieldId): Boolean = true + } + + data object DenyAll : FieldAccess { + override fun permits(field: QueryFieldId): Boolean = false + } + + class AllowList(fields: Iterable) : FieldAccess { + val fields: Set = Collections.unmodifiableSet( + LinkedHashSet(fields.sortedWith(QUERY_FIELD_ID_COMPARATOR)), + ) + + override fun permits(field: QueryFieldId): Boolean = field in fields + + override fun equals(other: Any?): Boolean = this === other || other is AllowList && fields == other.fields + + override fun hashCode(): Int = fields.hashCode() + } +} + +internal sealed interface SearchScopeAccess { + fun permits(scope: SearchScopeId): Boolean + + data object Unrestricted : SearchScopeAccess { + override fun permits(scope: SearchScopeId): Boolean = true + } + + data object DenyAll : SearchScopeAccess { + override fun permits(scope: SearchScopeId): Boolean = false + } + + class AllowList(scopes: Iterable) : SearchScopeAccess { + val scopes: Set = Collections.unmodifiableSet( + LinkedHashSet(scopes.sortedBy(SearchScopeId::value)), + ) + + override fun permits(scope: SearchScopeId): Boolean = scope in scopes + + override fun equals(other: Any?): Boolean = this === other || other is AllowList && scopes == other.scopes + + override fun hashCode(): Int = scopes.hashCode() + } +} + +internal sealed interface NativeBackendAccess { + fun permits(backendId: BackendId): Boolean + + data object Unrestricted : NativeBackendAccess { + override fun permits(backendId: BackendId): Boolean = true + } + + data object DenyAll : NativeBackendAccess { + override fun permits(backendId: BackendId): Boolean = false + } + + class AllowList(backends: Iterable) : NativeBackendAccess { + val backends: Set = Collections.unmodifiableSet( + LinkedHashSet(backends.sortedBy(BackendId::value)), + ) + + override fun permits(backendId: BackendId): Boolean = backendId in backends + + override fun equals(other: Any?): Boolean = this === other || other is AllowList && backends == other.backends + + override fun hashCode(): Int = backends.hashCode() + } +} + +internal data class QueryFieldConstraint( + val filterFields: FieldAccess = FieldAccess.Unrestricted, + val searchScopes: SearchScopeAccess = SearchScopeAccess.Unrestricted, + val nativeBackends: NativeBackendAccess = NativeBackendAccess.Unrestricted, + val projectionFields: FieldAccess = FieldAccess.Unrestricted, + val sortFields: FieldAccess = FieldAccess.Unrestricted, + val analyticsDimensionFields: FieldAccess = FieldAccess.Unrestricted, + val analyticsMetricFields: FieldAccess = FieldAccess.Unrestricted, +) { + companion object { + val DenyAll: QueryFieldConstraint = QueryFieldConstraint( + filterFields = FieldAccess.DenyAll, + searchScopes = SearchScopeAccess.DenyAll, + nativeBackends = NativeBackendAccess.DenyAll, + projectionFields = FieldAccess.DenyAll, + sortFields = FieldAccess.DenyAll, + analyticsDimensionFields = FieldAccess.DenyAll, + analyticsMetricFields = FieldAccess.DenyAll, + ) + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/PlanningModel.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/PlanningModel.kt new file mode 100644 index 00000000000..47cc0cf4c26 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/PlanningModel.kt @@ -0,0 +1,104 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.plan.PlannedCondition +import me.ahoo.wow.query.internal.plan.QueryPlan +import me.ahoo.wow.query.internal.plan.RequiredCapabilities +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.rejection.QueryRejection +import me.ahoo.wow.query.internal.value.NonEmptyList + +internal data class PlanningConstraints( + val validationMode: QueryValidationMode, + val mandatoryCondition: NormalizedCondition = NormalizedCondition.All, + val fieldConstraint: QueryFieldConstraint = QueryFieldConstraint(), + val resultConstraint: ResultPlanningConstraint = ResultPlanningConstraint.Unrestricted, + val streamConstraint: StreamPlanningConstraint = StreamPlanningConstraint.Unrestricted, + val pageConstraint: PagePlanningConstraint = PagePlanningConstraint.Unrestricted, + val analyticsConstraint: AnalyticsPlanningConstraint = AnalyticsPlanningConstraint.Unrestricted, +) + +internal sealed interface ResultPlanningConstraint { + data object Unrestricted : ResultPlanningConstraint + + data class MaximumRecords(val value: Long) : ResultPlanningConstraint { + init { + require(value > 0) { + "Maximum returned records must be positive." + } + } + } +} + +internal sealed interface StreamPlanningConstraint { + data object Unrestricted : StreamPlanningConstraint + + data object BoundedOnly : StreamPlanningConstraint +} + +internal sealed interface PagePlanningConstraint { + data object Unrestricted : PagePlanningConstraint + + data class MaximumWindow(val value: Long) : PagePlanningConstraint { + init { + require(value > 0) { + "Maximum page window must be positive." + } + } + } +} + +internal sealed interface AnalyticsPlanningConstraint { + data object Unrestricted : AnalyticsPlanningConstraint + + data class Limits( + val maxDimensions: Int, + val maxMetrics: Int, + val maxBucketLimit: Int, + ) : AnalyticsPlanningConstraint { + init { + require(maxDimensions > 0 && maxMetrics > 0 && maxBucketLimit > 0) { + "Analytics planning limits must be positive." + } + } + } +} + +internal data class ValidatedMandatory( + val target: QueryTarget, + val schemaContractId: SchemaContractId, + val condition: PlannedCondition, + val requiredCapabilities: RequiredCapabilities, + val semanticTier: SemanticTier, +) + +internal sealed interface PlanningDecision { + data class Planned(val plan: QueryPlan) : PlanningDecision + + data class LegacyFallback( + val issues: NonEmptyList, + val validatedMandatory: ValidatedMandatory, + ) : PlanningDecision +} + +internal data class PlannedConditionResult( + val condition: PlannedCondition, + val requiredCapabilities: RequiredCapabilities, + val semanticTier: SemanticTier, +) diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/PlanningSupport.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/PlanningSupport.kt new file mode 100644 index 00000000000..e7643bc6552 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/PlanningSupport.kt @@ -0,0 +1,73 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.SearchScopeId +import me.ahoo.wow.query.internal.plan.RequiredCapabilities +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery + +internal fun mergeCapabilities(vararg sources: RequiredCapabilities): RequiredCapabilities { + val fields = linkedMapOf>() + val searches = linkedSetOf() + var native: BackendId? = null + sources.forEach { source -> + source.fieldRequirements.forEach { (field, requirements) -> + fields.getOrPut(field, ::linkedSetOf).addAll(requirements) + } + searches.addAll(source.searchRequirements) + source.nativeBackend?.let { backend -> + if (native != null && native != backend) { + rejectQuery( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionPath.ROOT.property("input"), + QueryRejectionCode.NATIVE_BACKEND_CONFLICT, + ) + } + native = backend + } + } + return RequiredCapabilities(fields.mapValues { it.value.toSet() }, searches, native) +} + +internal fun requireFieldCapability( + schema: QueryDocumentSchema, + field: QueryFieldId, + capability: FieldCapability, + path: QueryRejectionPath, +) { + val fieldSchema = schema.fields[field] ?: rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + path, + QueryRejectionCode.FIELD_NOT_FOUND, + ) + if (capability !in fieldSchema.capabilities) { + rejectQuery(QueryRejectionCategory.UNSUPPORTED_FEATURE, path, QueryRejectionCode.CAPABILITY_UNAVAILABLE) + } +} + +internal fun QueryFieldId.stableKey(): String = + when (this) { + is QueryFieldId.System -> "0:${kind.name}" + is QueryFieldId.Path -> "1:${segments.joinToString("\u0000")}" + } + +internal fun SemanticTier.max(other: SemanticTier): SemanticTier = if (ordinal >= other.ordinal) this else other diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/QueryConditionPlanner.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/QueryConditionPlanner.kt new file mode 100644 index 00000000000..2591f1535d7 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/QueryConditionPlanner.kt @@ -0,0 +1,480 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.acceptsOperand +import me.ahoo.wow.query.backend.hasOperandType +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.CaseSensitivity +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.PathBasis +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.SearchScope +import me.ahoo.wow.query.internal.plan.PlannedCondition +import me.ahoo.wow.query.internal.plan.RequiredCapabilities +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import me.ahoo.wow.query.internal.value.NonEmptyList + +internal class QueryConditionPlanner( + internal val schema: QueryDocumentSchema, +) { + fun plan( + condition: NormalizedCondition, + path: QueryRejectionPath, + mandatory: Boolean, + fieldConstraint: QueryFieldConstraint = QueryFieldConstraint(), + ): PlannedConditionResult { + val state = ConditionPlanningState() + val planned = planCondition(condition, path, elementScope = null, mandatory, fieldConstraint, state) + return PlannedConditionResult(planned, state.capabilities(), state.semanticTier) + } + + fun resolveField( + field: LogicalField, + path: QueryRejectionPath, + elementScope: QueryFieldId.Path? = null, + ): QueryFieldId = + when (field) { + is LogicalField.System -> { + if (elementScope != null) { + rejectQuery( + QueryRejectionCategory.INVALID_QUERY, + path, + QueryRejectionCode.SYSTEM_FIELD_IN_ELEMENT_SCOPE, + ) + } + QueryFieldId.System(field.kind) + } + + is LogicalField.Path -> resolvePath(field, path, elementScope) + }.let { unresolved -> + schema.resolveField(unresolved) ?: rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + path, + QueryRejectionCode.FIELD_NOT_FOUND, + ) + }.also { resolved -> + if (elementScope != null && resolved is QueryFieldId.System) { + rejectQuery( + QueryRejectionCategory.INVALID_QUERY, + path, + QueryRejectionCode.SYSTEM_FIELD_IN_ELEMENT_SCOPE, + ) + } + } + + fun resolveAccessibleField( + field: LogicalField, + path: QueryRejectionPath, + access: FieldAccess, + deniedCode: QueryRejectionCode, + elementScope: QueryFieldId.Path? = null, + ): QueryFieldId { + if (access == FieldAccess.DenyAll) { + rejectAccess(path, deniedCode) + } + val resolved = + try { + resolveField(field, path, elementScope) + } catch (error: QueryRejectedException) { + if (access is FieldAccess.AllowList && error.rejection.code == QueryRejectionCode.FIELD_NOT_FOUND) { + rejectQuery(QueryRejectionCategory.ACCESS_DENIED, path, deniedCode, error) + } + throw error + } + if (!access.permits(resolved)) { + rejectAccess(path, deniedCode) + } + return resolved + } + + private fun planCondition( + condition: NormalizedCondition, + path: QueryRejectionPath, + elementScope: QueryFieldId.Path?, + mandatory: Boolean, + fieldConstraint: QueryFieldConstraint, + state: ConditionPlanningState, + ): PlannedCondition = + when (condition) { + NormalizedCondition.All -> PlannedCondition.All + NormalizedCondition.None -> PlannedCondition.None + is NormalizedCondition.Junction -> PlannedCondition.Junction( + condition.operator, + checkNotNull( + NonEmptyList.from( + condition.children.mapIndexed { index, child -> + planCondition( + child, + path.property("children").index(index), + elementScope, + mandatory, + fieldConstraint, + state, + ) + }, + ), + ), + ) + + is NormalizedCondition.Predicate -> planPredicate(condition, path, elementScope, fieldConstraint, state) + is NormalizedCondition.ElementMatch -> + planElementMatch(condition, path, elementScope, mandatory, fieldConstraint, state) + is NormalizedCondition.Search -> planSearch(condition, path, elementScope, fieldConstraint, state) + is NormalizedCondition.Native -> planNative(condition, path, mandatory, fieldConstraint, state) + } + + private fun planPredicate( + condition: NormalizedCondition.Predicate, + path: QueryRejectionPath, + elementScope: QueryFieldId.Path?, + fieldConstraint: QueryFieldConstraint, + state: ConditionPlanningState, + ): PlannedCondition.Predicate { + val field = resolveAccessibleField( + condition.field, + path.property("field"), + fieldConstraint.filterFields, + QueryRejectionCode.FILTER_FIELD_NOT_ALLOWED, + elementScope, + ) + val fieldSchema = schema.fields.getValue(field) + if (condition.operator !in fieldSchema.allowedOperators) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + path.property("operator"), + QueryRejectionCode.OPERATOR_NOT_ALLOWED, + ) + } + if (condition.options.caseSensitivity == CaseSensitivity.INSENSITIVE) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + path.property("options").property("caseSensitivity"), + QueryRejectionCode.CASE_INSENSITIVE_UNSUPPORTED, + ) + } + val capability = condition.operator.requiredCapability() + requireCapability(field, capability, path.property("operator"), state) + validatePredicateValue(condition, fieldSchema, path) + requireNullOperandCapability(condition, field, path, state) + return PlannedCondition.Predicate(field, condition.operator, condition.value, condition.options) + } + + private fun requireNullOperandCapability( + condition: NormalizedCondition.Predicate, + field: QueryFieldId, + path: QueryRejectionPath, + state: ConditionPlanningState, + ) { + val containsNull = condition.value == NormalizedValue.Null || + (condition.value as? NormalizedValue.ListValue)?.values?.contains(NormalizedValue.Null) == true + if (!containsNull) return + val required = when (condition.operator) { + PredicateOperator.ALL_IN -> FieldCapability.ELEMENT_NULL + PredicateOperator.EQ, + PredicateOperator.NE, + PredicateOperator.IN, + PredicateOperator.NOT_IN, + -> FieldCapability.PRESENCE + + else -> return + } + requireCapability(field, required, path.property("value"), state) + } + + private fun validatePredicateValue( + condition: NormalizedCondition.Predicate, + fieldSchema: me.ahoo.wow.query.backend.QueryFieldSchema, + path: QueryRejectionPath, + ) { + if (!acceptsPredicateValue(condition, fieldSchema)) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + path.property("value"), + QueryRejectionCode.VALUE_TYPE_MISMATCH, + ) + } + } + + private fun acceptsPredicateValue( + condition: NormalizedCondition.Predicate, + fieldSchema: me.ahoo.wow.query.backend.QueryFieldSchema, + ): Boolean { + if (condition.operator == PredicateOperator.IS_NULL || condition.operator == PredicateOperator.NOT_NULL) { + return true + } + if (condition.operator.isCollectionOperator()) { + return acceptsCollectionValue(condition, fieldSchema) + } + return when (condition.operator) { + PredicateOperator.IS_TRUE, + PredicateOperator.IS_FALSE, + -> fieldSchema.hasOperandType(LogicalFieldType.Boolean) + + PredicateOperator.EXISTS -> condition.value is NormalizedValue.BooleanValue + PredicateOperator.CONTAINS, + PredicateOperator.STARTS_WITH, + PredicateOperator.ENDS_WITH, + -> fieldSchema.hasOperandType(LogicalFieldType.Text) && condition.value is NormalizedValue.Text + + else -> condition.value?.let { value -> fieldSchema.acceptsOperand(condition.operator, value) } == true + } + } + + private fun acceptsCollectionValue( + condition: NormalizedCondition.Predicate, + fieldSchema: me.ahoo.wow.query.backend.QueryFieldSchema, + ): Boolean { + val values = condition.value as? NormalizedValue.ListValue ?: return false + val validSize = + if (condition.operator == PredicateOperator.BETWEEN) { + values.values.size == 2 + } else { + values.values.isNotEmpty() + } + return validSize && values.values.all { value -> fieldSchema.acceptsOperand(condition.operator, value) } + } + + private fun PredicateOperator.isCollectionOperator(): Boolean = + this == PredicateOperator.IN || + this == PredicateOperator.NOT_IN || + this == PredicateOperator.ALL_IN || + this == PredicateOperator.BETWEEN + + private fun planElementMatch( + condition: NormalizedCondition.ElementMatch, + path: QueryRejectionPath, + elementScope: QueryFieldId.Path?, + mandatory: Boolean, + fieldConstraint: QueryFieldConstraint, + state: ConditionPlanningState, + ): PlannedCondition.ElementMatch { + val field = resolveAccessibleField( + condition.field, + path.property("field"), + fieldConstraint.filterFields, + QueryRejectionCode.FILTER_FIELD_NOT_ALLOWED, + elementScope, + ) + if (field !is QueryFieldId.Path) { + rejectQuery(QueryRejectionCategory.INVALID_QUERY, path.property("field"), QueryRejectionCode.INVALID_FIELD) + } + requireCapability(field, FieldCapability.ELEMENT_MATCH, path.property("field"), state) + return PlannedCondition.ElementMatch( + field, + planCondition( + condition.condition, + path.property("condition"), + field, + mandatory, + fieldConstraint, + state, + ), + ) + } + + private fun planSearch( + condition: NormalizedCondition.Search, + path: QueryRejectionPath, + elementScope: QueryFieldId.Path?, + fieldConstraint: QueryFieldConstraint, + state: ConditionPlanningState, + ): PlannedCondition.Search { + if (fieldConstraint.searchScopes == SearchScopeAccess.DenyAll) { + rejectAccess(path.property("scope"), QueryRejectionCode.SEARCH_SCOPE_NOT_ALLOWED) + } + val definition = + when (val scope = condition.scope) { + is SearchScope.Named -> schema.searchScopes[scope.id]?.takeIf { it.owner == elementScope } + is SearchScope.LegacyField -> { + val alias = resolveSearchAlias(scope.field, path.property("scope"), elementScope, fieldConstraint) + schema.resolveLegacySearchScope(elementScope, alias) + } + } ?: rejectQuery( + if (fieldConstraint.searchScopes is SearchScopeAccess.AllowList) { + QueryRejectionCategory.ACCESS_DENIED + } else { + QueryRejectionCategory.UNSUPPORTED_FEATURE + }, + path.property("scope"), + if (fieldConstraint.searchScopes is SearchScopeAccess.AllowList) { + QueryRejectionCode.SEARCH_SCOPE_NOT_ALLOWED + } else { + QueryRejectionCode.SEARCH_SCOPE_NOT_FOUND + }, + ) + if (!fieldConstraint.searchScopes.permits(definition.id)) { + rejectQuery( + QueryRejectionCategory.ACCESS_DENIED, + path.property("scope"), + QueryRejectionCode.SEARCH_SCOPE_NOT_ALLOWED, + ) + } + state.searchRequirements += definition.id + state.semanticTier = state.semanticTier.max(SemanticTier.SEARCH) + return PlannedCondition.Search(definition.id, condition.text) + } + + private fun planNative( + condition: NormalizedCondition.Native, + path: QueryRejectionPath, + mandatory: Boolean, + fieldConstraint: QueryFieldConstraint, + state: ConditionPlanningState, + ): PlannedCondition.Native { + if (mandatory) { + rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + path, + QueryRejectionCode.MANDATORY_NATIVE_NOT_ALLOWED, + ) + } + if (!fieldConstraint.nativeBackends.permits(condition.backendId)) { + rejectQuery( + QueryRejectionCategory.ACCESS_DENIED, + path.property("backendId"), + QueryRejectionCode.NATIVE_BACKEND_NOT_ALLOWED, + ) + } + state.requireNativeBackend(condition.backendId, path) + state.semanticTier = SemanticTier.NATIVE + return PlannedCondition.Native(condition.backendId, condition.payload) + } + + private fun resolvePath( + field: LogicalField.Path, + path: QueryRejectionPath, + elementScope: QueryFieldId.Path?, + ): QueryFieldId.Path = + when (field.basis) { + PathBasis.ROOT -> { + if (elementScope != null) { + rejectQuery(QueryRejectionCategory.INVALID_QUERY, path, QueryRejectionCode.INVALID_FIELD) + } + QueryFieldId.Path(field.segments) + } + PathBasis.CURRENT_ELEMENT -> { + val scope = elementScope ?: rejectQuery( + QueryRejectionCategory.INVALID_QUERY, + path, + QueryRejectionCode.INVALID_FIELD, + ) + QueryFieldId.Path(scope.segments + field.segments) + } + } + + private fun requireCapability( + field: QueryFieldId, + capability: FieldCapability, + path: QueryRejectionPath, + state: ConditionPlanningState, + ) { + if (capability !in schema.fields.getValue(field).capabilities) { + rejectQuery(QueryRejectionCategory.UNSUPPORTED_FEATURE, path, QueryRejectionCode.CAPABILITY_UNAVAILABLE) + } + state.fieldRequirements.getOrPut(field, ::linkedSetOf) += capability + } + + private fun resolveSearchAlias( + field: LogicalField.Path, + path: QueryRejectionPath, + elementScope: QueryFieldId.Path?, + fieldConstraint: QueryFieldConstraint, + ): QueryFieldId.Path = + try { + resolveField(field, path, elementScope) as? QueryFieldId.Path + ?: rejectQuery( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + path, + QueryRejectionCode.SEARCH_SCOPE_NOT_FOUND, + ) + } catch (error: QueryRejectedException) { + if (fieldConstraint.searchScopes is SearchScopeAccess.AllowList && + error.rejection.category == QueryRejectionCategory.UNSUPPORTED_FEATURE + ) { + rejectQuery( + QueryRejectionCategory.ACCESS_DENIED, + path, + QueryRejectionCode.SEARCH_SCOPE_NOT_ALLOWED, + error, + ) + } + throw error + } + + private fun rejectAccess(path: QueryRejectionPath, code: QueryRejectionCode): Nothing = + rejectQuery( + QueryRejectionCategory.ACCESS_DENIED, + path, + code, + ) + + private fun PredicateOperator.requiredCapability(): FieldCapability = + when (this) { + PredicateOperator.IS_NULL, + PredicateOperator.NOT_NULL, + PredicateOperator.EXISTS, + -> FieldCapability.PRESENCE + + PredicateOperator.GT, + PredicateOperator.LT, + PredicateOperator.GTE, + PredicateOperator.LTE, + PredicateOperator.BETWEEN, + -> FieldCapability.RANGE + + PredicateOperator.CONTAINS, + PredicateOperator.STARTS_WITH, + PredicateOperator.ENDS_WITH, + -> FieldCapability.LITERAL_PATTERN + + else -> FieldCapability.EXACT + } + + private class ConditionPlanningState { + val fieldRequirements: MutableMap> = linkedMapOf() + val searchRequirements: MutableSet = linkedSetOf() + var nativeBackend: BackendId? = null + var semanticTier: SemanticTier = SemanticTier.PORTABLE + + fun requireNativeBackend(backendId: BackendId, path: QueryRejectionPath) { + val current = nativeBackend + if (current != null && current != backendId) { + rejectQuery( + QueryRejectionCategory.INVALID_QUERY, + path.property("backendId"), + QueryRejectionCode.NATIVE_BACKEND_CONFLICT, + ) + } + nativeBackend = backendId + } + + fun capabilities(): RequiredCapabilities = RequiredCapabilities( + fieldRequirements.mapValues { it.value.toSet() }, + searchRequirements, + nativeBackend, + ) + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/QueryPlanner.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/QueryPlanner.kt new file mode 100644 index 00000000000..b967a4c30c6 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/QueryPlanner.kt @@ -0,0 +1,93 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery + +internal class QueryPlanner { + fun plan( + invocation: NormalizedQueryInvocation, + schema: QueryDocumentSchema, + constraints: PlanningConstraints, + ): PlanningDecision { + validateInvocation(invocation, schema) + val conditionPlanner = QueryConditionPlanner(schema) + val mandatory = conditionPlanner.plan( + constraints.mandatoryCondition, + QueryRejectionPath.ROOT.property("constraints").property("mandatoryCondition"), + mandatory = true, + ).let { result -> + ValidatedMandatory( + invocation.target, + schema.contractId, + result.condition, + result.requiredCapabilities, + result.semanticTier, + ) + } + return when (val input = invocation.input) { + is NormalizedQueryInput.Analytics -> PlanningDecision.Planned( + AnalyticsQueryPlanner(invocation, schema, conditionPlanner, constraints, mandatory).plan(input.query), + ) + + else -> RecordQueryPlanner( + invocation, + schema, + conditionPlanner, + constraints, + mandatory, + ).plan(input) + } + } + + private fun validateInvocation( + invocation: NormalizedQueryInvocation, + schema: QueryDocumentSchema, + ) { + if (invocation.target != schema.target) { + rejectQuery( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionPath.ROOT.property("target"), + QueryRejectionCode.TARGET_SCHEMA_MISMATCH, + ) + } + if (!invocation.hasValidMatrix()) { + rejectQuery( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionPath.ROOT.property("input"), + QueryRejectionCode.INVALID_INVOCATION, + ) + } + } + + private fun NormalizedQueryInvocation.hasValidMatrix(): Boolean = + when (operation) { + QueryOperation.SINGLE -> input is NormalizedQueryInput.Single && hasRecordShape() + QueryOperation.STREAM -> input is NormalizedQueryInput.Stream && hasRecordShape() + QueryOperation.PAGE -> input is NormalizedQueryInput.Page && hasRecordShape() + QueryOperation.COUNT -> input is NormalizedQueryInput.Count && resultShape == QueryResultShape.COUNT + QueryOperation.ANALYZE -> input is NormalizedQueryInput.Analytics && resultShape == QueryResultShape.ANALYTICS + } + + private fun NormalizedQueryInvocation.hasRecordShape(): Boolean = + resultShape == QueryResultShape.TYPED || resultShape == QueryResultShape.DYNAMIC +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/RecordQueryPlanner.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/RecordQueryPlanner.kt new file mode 100644 index 00000000000..94ef65a8bc8 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/planning/RecordQueryPlanner.kt @@ -0,0 +1,453 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.model.RecordResultShape +import me.ahoo.wow.query.internal.normalization.JunctionOperator +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedDeletionScope +import me.ahoo.wow.query.internal.normalization.NormalizedPredicateOptions +import me.ahoo.wow.query.internal.normalization.NormalizedProjection +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.normalization.NormalizedRecordQuery +import me.ahoo.wow.query.internal.normalization.NormalizedSort +import me.ahoo.wow.query.internal.normalization.NormalizedSortDirection +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.SystemFieldKind +import me.ahoo.wow.query.internal.plan.CountQueryPlan +import me.ahoo.wow.query.internal.plan.EnforcedFilter +import me.ahoo.wow.query.internal.plan.PageQueryPlan +import me.ahoo.wow.query.internal.plan.PageWindow +import me.ahoo.wow.query.internal.plan.PlannedCondition +import me.ahoo.wow.query.internal.plan.PlannedProjection +import me.ahoo.wow.query.internal.plan.PlannedSort +import me.ahoo.wow.query.internal.plan.PlannedSortOrigin +import me.ahoo.wow.query.internal.plan.QueryPlan +import me.ahoo.wow.query.internal.plan.RecordQueryPlan +import me.ahoo.wow.query.internal.plan.RequiredCapabilities +import me.ahoo.wow.query.internal.plan.RequiredConsistency +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.plan.SingleQueryPlan +import me.ahoo.wow.query.internal.plan.StreamLimit +import me.ahoo.wow.query.internal.plan.StreamQueryPlan +import me.ahoo.wow.query.internal.plan.TotalMode +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejection +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.rejection.rejectQuery +import me.ahoo.wow.query.internal.value.NonEmptyList +import java.util.Collections + +internal class RecordQueryPlanner( + private val invocation: NormalizedQueryInvocation, + private val schema: QueryDocumentSchema, + private val conditionPlanner: QueryConditionPlanner, + private val constraints: PlanningConstraints, + private val mandatory: ValidatedMandatory, +) { + private val issues = mutableListOf() + + fun plan(input: NormalizedQueryInput): PlanningDecision { + val queryPlan = + when (input) { + is NormalizedQueryInput.Single -> planRecord(input.query) { common -> + SingleQueryPlan.create( + invocation.target, + schema.contractId, + common.filter, + invocation.resultShape.toRecordShape(), + common.projection, + common.sort, + common.capabilities, + common.semanticTier, + ) + } + + is NormalizedQueryInput.Stream -> planStream(input) + is NormalizedQueryInput.Page -> planPage(input) + is NormalizedQueryInput.Count -> planCount(input) + is NormalizedQueryInput.Analytics -> error("Analytics input must use AnalyticsQueryPlanner.") + } + if (issues.isNotEmpty()) { + return PlanningDecision.LegacyFallback( + checkNotNull(NonEmptyList.from(issues.sortedWith(REJECTION_ORDER))), + mandatory, + ) + } + return PlanningDecision.Planned(checkNotNull(queryPlan)) + } + + private fun planStream(input: NormalizedQueryInput.Stream): QueryPlan? { + val path = QueryRejectionPath.ROOT.property("input").property("limit") + if (input.limit < 0) { + rejectQuery(QueryRejectionCategory.INVALID_QUERY, path, QueryRejectionCode.INVALID_LIMIT) + } + if (input.limit == 0 && constraints.streamConstraint == StreamPlanningConstraint.BoundedOnly) { + rejectQuery( + QueryRejectionCategory.BUDGET_EXCEEDED, + path, + QueryRejectionCode.UNBOUNDED_STREAM_DISALLOWED, + ) + } + val maximumRecords = (constraints.resultConstraint as? ResultPlanningConstraint.MaximumRecords)?.value + if (maximumRecords != null && (input.limit == 0 || input.limit > maximumRecords)) { + rejectQuery( + QueryRejectionCategory.BUDGET_EXCEEDED, + path, + QueryRejectionCode.RESULT_LIMIT_EXCEEDED, + ) + } + return planRecord(input.query) { common -> + StreamQueryPlan.create( + invocation.target, + schema.contractId, + common.filter, + invocation.resultShape.toRecordShape(), + common.projection, + common.sort, + if (input.limit == 0) StreamLimit.Unbounded else StreamLimit.Bounded(input.limit), + common.capabilities, + common.semanticTier, + ) + } + } + + private fun planPage(input: NormalizedQueryInput.Page): QueryPlan? { + validatePage(input) + return planRecord(input.query) { common -> + PageQueryPlan.create( + invocation.target, + schema.contractId, + common.filter, + invocation.resultShape.toRecordShape(), + common.projection, + common.sort, + PageWindow(input.page.offset, input.page.size), + TotalMode.EXACT, + RequiredConsistency.SAME_INPUT, + common.capabilities, + common.semanticTier, + ) + } + } + + private fun validatePage(input: NormalizedQueryInput.Page) { + val path = QueryRejectionPath.ROOT.property("input").property("page") + val expectedOffset = + if (input.page.index < 1 || input.page.size <= 0) { + null + } else { + Math.multiplyExact(input.page.index.toLong() - 1, input.page.size.toLong()) + } + if (input.page.offset < 0 || expectedOffset == null || input.page.offset != expectedOffset) { + rejectQuery(QueryRejectionCategory.INVALID_QUERY, path, QueryRejectionCode.INVALID_PAGE) + } + val maximumRecords = (constraints.resultConstraint as? ResultPlanningConstraint.MaximumRecords)?.value + if (maximumRecords != null && input.page.size.toLong() > maximumRecords) { + rejectQuery(QueryRejectionCategory.BUDGET_EXCEEDED, path, QueryRejectionCode.RESULT_LIMIT_EXCEEDED) + } + val maximumWindow = (constraints.pageConstraint as? PagePlanningConstraint.MaximumWindow)?.value ?: return + val endExclusive = + try { + Math.addExact(input.page.offset, input.page.size.toLong()) + } catch (error: ArithmeticException) { + rejectQuery( + QueryRejectionCategory.BUDGET_EXCEEDED, + path, + QueryRejectionCode.PAGE_WINDOW_EXCEEDED, + error, + ) + } + if (endExclusive > maximumWindow) { + rejectQuery(QueryRejectionCategory.BUDGET_EXCEEDED, path, QueryRejectionCode.PAGE_WINDOW_EXCEEDED) + } + } + + private fun planCount(input: NormalizedQueryInput.Count): QueryPlan? { + val user = planUserCondition( + input.userCondition, + input.deletionScope, + QueryRejectionPath.ROOT.property("input").property("userCondition"), + ) ?: return null + return CountQueryPlan.create( + invocation.target, + schema.contractId, + EnforcedFilter(user.condition, mandatory.condition), + mergeCapabilities(user.requiredCapabilities, mandatory.requiredCapabilities), + user.semanticTier.max(mandatory.semanticTier), + ) + } + + private fun planRecord( + query: NormalizedRecordQuery, + factory: (RecordCommon) -> RecordQueryPlan, + ): QueryPlan? { + val queryPath = QueryRejectionPath.ROOT.property("input").property("query") + val user = planUserCondition(query.userCondition, query.deletionScope, queryPath.property("condition")) + val projection = planProjection(query.projection, queryPath.property("projection")) + val sort = planSort(query.sort, queryPath.property("sort")) + if (user == null || projection == null || sort == null) { + return null + } + return factory( + RecordCommon( + EnforcedFilter(user.condition, mandatory.condition), + projection.projection, + sort.sort, + mergeCapabilities( + user.requiredCapabilities, + mandatory.requiredCapabilities, + projection.capabilities, + sort.capabilities, + ), + user.semanticTier.max(mandatory.semanticTier), + ), + ) + } + + private fun planUserCondition( + condition: NormalizedCondition, + deletionScope: NormalizedDeletionScope, + path: QueryRejectionPath, + ): PlannedConditionResult? { + val user = planCompatible { + conditionPlanner.plan( + condition, + path, + mandatory = false, + fieldConstraint = constraints.fieldConstraint, + ) + } ?: return null + if (schema.target.documentKind == QueryDocumentKind.EVENT_STREAM || + deletionScope == NormalizedDeletionScope.EXPLICIT || + user.condition == PlannedCondition.None + ) { + return user + } + val defaultActive = conditionPlanner.plan( + DEFAULT_ACTIVE_CONDITION, + path.property("defaultDeletion"), + mandatory = false, + ) + val effectiveCondition = + if (user.condition == PlannedCondition.All) { + defaultActive.condition + } else { + PlannedCondition.Junction( + JunctionOperator.AND, + NonEmptyList.of(defaultActive.condition, user.condition), + ) + } + return PlannedConditionResult( + effectiveCondition, + mergeCapabilities(defaultActive.requiredCapabilities, user.requiredCapabilities), + defaultActive.semanticTier.max(user.semanticTier), + ) + } + + private fun planProjection( + projection: NormalizedProjection, + path: QueryRejectionPath, + ): ProjectionResult? { + val access = constraints.fieldConstraint.projectionFields + if (access != FieldAccess.Unrestricted && + (invocation.resultShape == QueryResultShape.TYPED || projection !is NormalizedProjection.Include) + ) { + rejectQuery( + QueryRejectionCategory.ACCESS_DENIED, + path, + QueryRejectionCode.PROJECTION_FIELD_NOT_ALLOWED, + ) + } + if (projection == NormalizedProjection.All) { + return ProjectionResult(PlannedProjection.All, RequiredCapabilities()) + } + validateProjectionAccess(projection, path) + if (invocation.resultShape == QueryResultShape.TYPED && projection != NormalizedProjection.All) { + return handleTypedProjection(path) + } + if (projection is NormalizedProjection.Mixed) { + rejectQuery(QueryRejectionCategory.INVALID_QUERY, path, QueryRejectionCode.INVALID_PROJECTION) + } + val resolved = planCompatible { + projection.fields().mapIndexed { index, field -> + val fieldPath = path.property("fields").index(index) + conditionPlanner.resolveField(field, fieldPath).also { resolvedField -> + requireFieldCapability(schema, resolvedField, FieldCapability.PROJECTABLE, fieldPath) + } + }.distinct().sortedBy(QueryFieldId::stableKey) + } ?: return null + val nonEmpty = checkNotNull(NonEmptyList.from(resolved)) + val planned = + if (projection is NormalizedProjection.Include) { + PlannedProjection.Include(nonEmpty) + } else { + PlannedProjection.Exclude(nonEmpty) + } + return ProjectionResult( + planned, + RequiredCapabilities(resolved.associateWith { setOf(FieldCapability.PROJECTABLE) }), + ) + } + + private fun validateProjectionAccess( + projection: NormalizedProjection, + path: QueryRejectionPath, + ) { + val access = constraints.fieldConstraint.projectionFields + if (access == FieldAccess.Unrestricted) { + return + } + projection.fields().forEachIndexed { index, field -> + conditionPlanner.resolveAccessibleField( + field, + path.property("fields").index(index), + access, + QueryRejectionCode.PROJECTION_FIELD_NOT_ALLOWED, + ) + } + } + + private fun handleTypedProjection(path: QueryRejectionPath): Nothing { + throw QueryRejectedException( + QueryRejection( + QueryRejectionCategory.INVALID_QUERY, + path, + QueryRejectionCode.TYPED_PROJECTION_NOT_ALLOWED, + ), + ) + } + + private fun planSort( + sort: List, + path: QueryRejectionPath, + ): SortResult? = planCompatible { + val planned = mutableListOf() + val seen = mutableSetOf() + sort.forEachIndexed { index, item -> + val itemPath = path.index(index) + val field = conditionPlanner.resolveAccessibleField( + item.field, + itemPath.property("field"), + constraints.fieldConstraint.sortFields, + QueryRejectionCode.SORT_FIELD_NOT_ALLOWED, + ) + if (!seen.add(field)) { + return@planCompatible handleDuplicateSort(itemPath.property("field")) + } + requireFieldCapability(schema, field, FieldCapability.SORTABLE, itemPath.property("field")) + planned += PlannedSort(field, item.direction, PlannedSortOrigin.USER) + } + appendIdentityTieBreaker(planned, seen, path) + val requirements = planned.associate { it.field to setOf(FieldCapability.SORTABLE) } + SortResult(Collections.unmodifiableList(planned), RequiredCapabilities(requirements)) + } + + private fun handleDuplicateSort(path: QueryRejectionPath): SortResult? { + val rejection = QueryRejection( + QueryRejectionCategory.INVALID_QUERY, + path, + QueryRejectionCode.DUPLICATE_SORT, + ) + if (constraints.validationMode == QueryValidationMode.STRICT) { + throw QueryRejectedException(rejection) + } + issues += rejection + return null + } + + private fun appendIdentityTieBreaker( + sort: MutableList, + seen: Set, + path: QueryRejectionPath, + ) { + if (invocation.operation != QueryOperation.PAGE || constraints.validationMode != QueryValidationMode.STRICT) { + return + } + val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + if (identity in seen) { + return + } + requireFieldCapability(schema, identity, FieldCapability.SORTABLE, path) + sort += PlannedSort(identity, NormalizedSortDirection.ASC, PlannedSortOrigin.STABILITY_TIE_BREAKER) + } + + private fun planCompatible(block: () -> T): T? = + try { + block() + } catch (error: QueryRejectedException) { + if (constraints.validationMode == QueryValidationMode.STRICT || + error.rejection.category != QueryRejectionCategory.UNSUPPORTED_FEATURE + ) { + throw error + } + issues += error.rejection + null + } + + private fun NormalizedProjection.fields(): List = + when (this) { + is NormalizedProjection.Include -> fields.values + is NormalizedProjection.Exclude -> fields.values + is NormalizedProjection.Mixed -> include.values + exclude.values + NormalizedProjection.All -> error("Projection branch was already handled.") + } + + private fun QueryResultShape.toRecordShape(): RecordResultShape = + when (this) { + QueryResultShape.TYPED -> RecordResultShape.TYPED + QueryResultShape.DYNAMIC -> RecordResultShape.DYNAMIC + QueryResultShape.COUNT, + QueryResultShape.ANALYTICS, + -> error("Not a record result shape: $this") + } + + private data class RecordCommon( + val filter: EnforcedFilter, + val projection: PlannedProjection, + val sort: List, + val capabilities: RequiredCapabilities, + val semanticTier: SemanticTier, + ) + + private data class ProjectionResult( + val projection: PlannedProjection, + val capabilities: RequiredCapabilities, + ) + + private data class SortResult( + val sort: List, + val capabilities: RequiredCapabilities, + ) + + private companion object { + val DEFAULT_ACTIVE_CONDITION = NormalizedCondition.Predicate( + LogicalField.System(SystemFieldKind.DELETED), + PredicateOperator.IS_FALSE, + options = NormalizedPredicateOptions(), + ) + val REJECTION_ORDER = compareBy({ it.path.toString() }, { it.code.name }) + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/policy/QueryExecutionContext.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/policy/QueryExecutionContext.kt new file mode 100644 index 00000000000..e1d5a9a1754 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/policy/QueryExecutionContext.kt @@ -0,0 +1,357 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.policy + +import me.ahoo.wow.query.internal.model.QueryExecutionMode +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejection +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.Instant +import java.util.Collections + +internal class QueryPurpose(val value: String) { + init { + requireValidIdentifier(value, "Query purpose") + } + + override fun equals(other: Any?): Boolean = this === other || other is QueryPurpose && value == other.value + + override fun hashCode(): Int = value.hashCode() + + override fun toString(): String = value +} + +internal class LegacyQueryCallerId(val value: String) { + init { + requireValidIdentifier(value, "Legacy query caller id") + } + + override fun equals(other: Any?): Boolean = this === other || other is LegacyQueryCallerId && value == other.value + + override fun hashCode(): Int = value.hashCode() + + override fun toString(): String = value +} + +internal sealed interface QueryOwnerGrant { + data object Unrestricted : QueryOwnerGrant + + data class Only(val ownerId: String) : QueryOwnerGrant { + init { + requireValidIdentifier(ownerId, "Authority owner id") + } + } +} + +internal sealed interface QuerySpaceGrant { + data object Unrestricted : QuerySpaceGrant + + data object DenyAll : QuerySpaceGrant + + class AllowList(spaceIds: Iterable) : QuerySpaceGrant { + val spaceIds: Set = immutableIdentifiers(spaceIds.toSet(), "Authority space id") + + init { + require(this.spaceIds.isNotEmpty()) { + "Authority space allow-list must not be empty." + } + } + + override fun equals(other: Any?): Boolean = this === other || other is AllowList && spaceIds == other.spaceIds + + override fun hashCode(): Int = spaceIds.hashCode() + } +} + +internal sealed interface QueryAuthority { + val principalId: String + + class Subject( + val subjectId: String, + val tenantId: String, + val ownerGrant: QueryOwnerGrant, + val spaceGrant: QuerySpaceGrant, + ) : QueryAuthority { + override val principalId: String = subjectId + + init { + requireValidIdentifier(subjectId, "Authority principal id") + requireValidIdentifier(tenantId, "Authority tenant id") + } + + override fun equals(other: Any?): Boolean = + this === other || + other is Subject && + subjectId == other.subjectId && + tenantId == other.tenantId && + ownerGrant == other.ownerGrant && + spaceGrant == other.spaceGrant + + override fun hashCode(): Int { + var result = subjectId.hashCode() + result = 31 * result + tenantId.hashCode() + result = 31 * result + ownerGrant.hashCode() + result = 31 * result + spaceGrant.hashCode() + return result + } + } + + class Service( + val serviceId: String, + val tenantId: String, + purposes: Set, + ) : QueryAuthority { + override val principalId: String = serviceId + val purposes: Set = Collections.unmodifiableSet( + LinkedHashSet(purposes.sortedBy(QueryPurpose::value)), + ) + + init { + requireValidIdentifier(serviceId, "Authority principal id") + requireValidIdentifier(tenantId, "Authority tenant id") + require(purposes.isNotEmpty()) { + "Service authority purposes must not be empty." + } + } + + override fun equals(other: Any?): Boolean = + this === other || + other is Service && + serviceId == other.serviceId && + tenantId == other.tenantId && + purposes == other.purposes + + override fun hashCode(): Int { + var result = serviceId.hashCode() + result = 31 * result + tenantId.hashCode() + result = 31 * result + purposes.hashCode() + return result + } + } + + data class System( + override val principalId: String, + val justification: String, + ) : QueryAuthority { + init { + requireValidIdentifier(principalId, "Authority principal id") + requireValidIdentifier(justification, "System authority justification") + } + } + + data class Legacy(val grant: LegacyQueryGrant) : QueryAuthority { + override val principalId: String = grant.callerId.value + } +} + +internal data class QueryResourceScope( + val tenantId: String? = null, + val ownerId: String? = null, + val spaceId: String? = null, +) { + init { + tenantId?.let { requireValidIdentifier(it, "Resource tenant id") } + ownerId?.let { requireValidIdentifier(it, "Resource owner id") } + spaceId?.let { requireValidIdentifier(it, "Resource space id") } + } +} + +internal data class QueryExecutionBudget( + val maxScannedRecords: Long? = null, + val maxReturnedRecords: Long? = null, + val maxPageWindow: Long? = null, + val maxCandidateBuckets: Int? = null, + val maxReturnedBuckets: Int? = null, + val maxCursorPages: Int? = null, + val allowDiskUse: Boolean = false, +) { + init { + require(maxScannedRecords == null || maxScannedRecords > 0) + require(maxReturnedRecords == null || maxReturnedRecords > 0) + require(maxPageWindow == null || maxPageWindow > 0) + require(maxCandidateBuckets == null || maxCandidateBuckets > 0) + require(maxReturnedBuckets == null || maxReturnedBuckets > 0) + require(maxCursorPages == null || maxCursorPages > 0) + } +} + +internal data class QueryExecutionRequest( + val target: QueryTarget, + val purpose: QueryPurpose, + val executionMode: QueryExecutionMode, + val validationMode: QueryValidationMode, + val resourceScope: QueryResourceScope = QueryResourceScope(), + val deadline: Instant? = null, + val budget: QueryExecutionBudget = QueryExecutionBudget(), +) + +internal data class LegacyQueryGrant( + val callerId: LegacyQueryCallerId, + val target: QueryTarget, + val purpose: QueryPurpose, + val executionMode: QueryExecutionMode, + val resourceScope: QueryResourceScope, +) + +internal data class QueryExecutionContext( + val target: QueryTarget, + val purpose: QueryPurpose, + val authority: QueryAuthority, + val executionMode: QueryExecutionMode, + val validationMode: QueryValidationMode, + val resourceScope: QueryResourceScope, + val deadline: Instant?, + val budget: QueryExecutionBudget, +) + +internal fun interface QueryAuthorityProvider { + fun resolve(request: QueryExecutionRequest): Mono +} + +internal class LegacyQueryAuthorityProvider( + private val grant: LegacyQueryGrant? = null, +) : QueryAuthorityProvider { + override fun resolve(request: QueryExecutionRequest): Mono = Mono.defer { + val configuredGrant = grant + if (configuredGrant == null || !request.matches(configuredGrant)) { + return@defer Mono.error(LegacyGrantRejectedException()) + } + Mono.just(QueryAuthority.Legacy(configuredGrant)) + } +} + +internal class QueryExecutionContextFactory( + private val authorityProvider: QueryAuthorityProvider, + private val clock: Clock, +) { + fun resolve(request: QueryExecutionRequest): Mono = Mono.defer { + if (request.deadline?.isAfter(clock.instant()) == false) { + return@defer Mono.error( + rejectedException( + QueryRejectionCategory.BUDGET_EXCEEDED, + EXECUTION_CONTEXT_PATH.property("deadline"), + QueryRejectionCode.DEADLINE_EXPIRED, + ), + ) + } + resolveAuthority(request) + .switchIfEmpty( + Mono.error( + rejectedException( + QueryRejectionCategory.ACCESS_DENIED, + EXECUTION_CONTEXT_PATH.property("authority"), + QueryRejectionCode.AUTHORITY_REQUIRED, + ), + ), + ) + .map { authority -> request.toExecutionContext(authority) } + } + + private fun resolveAuthority(request: QueryExecutionRequest): Mono = + Mono.defer { authorityProvider.resolve(request) } + .onErrorMap { error -> error.toAuthorityRejection() } + + private fun Throwable.toAuthorityRejection(): QueryRejectedException = + when (this) { + is LegacyGrantRejectedException -> rejectedException( + QueryRejectionCategory.ACCESS_DENIED, + EXECUTION_CONTEXT_PATH.property("legacyGrant"), + QueryRejectionCode.LEGACY_CALLER_NOT_ALLOWED, + this, + ) + + is TrustedAuthorityRejectedException -> rejectedException( + QueryRejectionCategory.ACCESS_DENIED, + path, + code, + this, + ) + + else -> rejectedException( + QueryRejectionCategory.ACCESS_DENIED, + EXECUTION_CONTEXT_PATH.property("authority"), + QueryRejectionCode.AUTHORITY_RESOLUTION_FAILED, + this, + ) + } + + private fun QueryExecutionRequest.toExecutionContext(authority: QueryAuthority): QueryExecutionContext { + if (authority is QueryAuthority.Legacy && !matches(authority.grant)) { + throw rejectedException( + QueryRejectionCategory.ACCESS_DENIED, + EXECUTION_CONTEXT_PATH.property("legacyGrant"), + QueryRejectionCode.LEGACY_CALLER_NOT_ALLOWED, + ) + } + if (deadline?.isAfter(clock.instant()) == false) { + throw rejectedException( + QueryRejectionCategory.BUDGET_EXCEEDED, + EXECUTION_CONTEXT_PATH.property("deadline"), + QueryRejectionCode.DEADLINE_EXPIRED, + ) + } + return QueryExecutionContext( + target, + purpose, + authority, + executionMode, + validationMode, + resourceScope, + deadline, + budget, + ) + } +} + +private val EXECUTION_CONTEXT_PATH = QueryRejectionPath.ROOT.property("executionContext") + +private class LegacyGrantRejectedException : IllegalStateException("Legacy query grant rejected.") + +internal class TrustedAuthorityRejectedException( + val path: QueryRejectionPath, + val code: QueryRejectionCode, + cause: Throwable, +) : IllegalStateException("Trusted authority rejected.", cause) + +private fun QueryExecutionRequest.matches(grant: LegacyQueryGrant): Boolean = + executionMode == grant.executionMode && + target == grant.target && + purpose == grant.purpose && + resourceScope == grant.resourceScope + +internal fun rejectedException( + category: QueryRejectionCategory, + path: QueryRejectionPath, + code: QueryRejectionCode, + cause: Throwable? = null, +): QueryRejectedException = QueryRejectedException(QueryRejection(category, path, code), cause) + +private fun immutableIdentifiers(values: Set, label: String): Set { + values.forEach { requireValidIdentifier(it, label) } + return Collections.unmodifiableSet(LinkedHashSet(values.sorted())) +} + +private fun requireValidIdentifier(value: String, label: String) { + require(value.isNotBlank() && value.length <= MAX_IDENTIFIER_LENGTH && value.none(Char::isISOControl)) { + "$label must not be blank, exceed $MAX_IDENTIFIER_LENGTH characters or contain control characters." + } +} + +private const val MAX_IDENTIFIER_LENGTH = 512 diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/policy/QueryPolicy.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/policy/QueryPolicy.kt new file mode 100644 index 00000000000..7eaf9f4d2ab --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/policy/QueryPolicy.kt @@ -0,0 +1,269 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.policy + +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.planning.AnalyticsPlanningConstraint +import me.ahoo.wow.query.internal.planning.FieldAccess +import me.ahoo.wow.query.internal.planning.PagePlanningConstraint +import me.ahoo.wow.query.internal.planning.PlanningConstraints +import me.ahoo.wow.query.internal.planning.QueryFieldConstraint +import me.ahoo.wow.query.internal.planning.ResultPlanningConstraint +import me.ahoo.wow.query.internal.planning.SearchScopeAccess +import me.ahoo.wow.query.internal.planning.StreamPlanningConstraint +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import reactor.core.publisher.Mono + +internal data class QueryPolicyInput( + val executionContext: QueryExecutionContext, + val invocation: NormalizedQueryInvocation, + val schema: QueryDocumentSchema, +) + +internal enum class QueryPolicyDenial { + TENANT_MISMATCH, + OWNER_MISMATCH, + SPACE_MISMATCH, + PURPOSE_NOT_ALLOWED, + POLICY_RULE, +} + +internal sealed interface QueryPolicyDecision { + data class Allow(val allowance: QueryPolicyAllowance) : QueryPolicyDecision + + data class Deny(val reason: QueryPolicyDenial) : QueryPolicyDecision +} + +internal fun interface QueryPolicy { + fun decide(input: QueryPolicyInput): Mono +} + +internal class QueryPolicyAllowance private constructor( + val mandatoryCondition: NormalizedCondition, + val fieldConstraint: QueryFieldConstraint, + val resultConstraint: ResultPlanningConstraint, + val streamConstraint: StreamPlanningConstraint, + val pageConstraint: PagePlanningConstraint, + val analyticsConstraint: AnalyticsPlanningConstraint, +) { + fun toPlanningConstraints( + validationMode: QueryValidationMode, + ): PlanningConstraints = + PlanningConstraints( + validationMode = validationMode, + mandatoryCondition = mandatoryCondition, + fieldConstraint = fieldConstraint, + resultConstraint = resultConstraint, + streamConstraint = streamConstraint, + pageConstraint = pageConstraint, + analyticsConstraint = analyticsConstraint, + ) + + override fun equals(other: Any?): Boolean = + this === other || + other is QueryPolicyAllowance && + mandatoryCondition == other.mandatoryCondition && + fieldConstraint == other.fieldConstraint && + resultConstraint == other.resultConstraint && + streamConstraint == other.streamConstraint && + pageConstraint == other.pageConstraint && + analyticsConstraint == other.analyticsConstraint + + override fun hashCode(): Int { + var result = mandatoryCondition.hashCode() + result = 31 * result + fieldConstraint.hashCode() + result = 31 * result + resultConstraint.hashCode() + result = 31 * result + streamConstraint.hashCode() + result = 31 * result + pageConstraint.hashCode() + result = 31 * result + analyticsConstraint.hashCode() + return result + } + + internal class Builder { + private var mandatoryCondition: NormalizedCondition = NormalizedCondition.All + private var fieldConstraint: QueryFieldConstraint = QueryFieldConstraint.DenyAll + private var resultConstraint: ResultPlanningConstraint = ResultPlanningConstraint.Unrestricted + private var streamConstraint: StreamPlanningConstraint = StreamPlanningConstraint.Unrestricted + private var pageConstraint: PagePlanningConstraint = PagePlanningConstraint.Unrestricted + private var analyticsConstraint: AnalyticsPlanningConstraint = AnalyticsPlanningConstraint.Unrestricted + + fun mandatoryCondition(condition: NormalizedCondition): Builder = apply { + mandatoryCondition = condition + } + + fun fieldConstraint(constraint: QueryFieldConstraint): Builder = apply { + fieldConstraint = constraint + } + + fun resultConstraint(constraint: ResultPlanningConstraint): Builder = apply { + resultConstraint = constraint + } + + fun streamConstraint(constraint: StreamPlanningConstraint): Builder = apply { + streamConstraint = constraint + } + + fun pageConstraint(constraint: PagePlanningConstraint): Builder = apply { + pageConstraint = constraint + } + + fun analyticsConstraint(constraint: AnalyticsPlanningConstraint): Builder = apply { + analyticsConstraint = constraint + } + + fun build(): QueryPolicyAllowance { + mandatoryCondition.findNativePath(POLICY_PATH.property("mandatoryCondition"))?.let { path -> + throw QueryPolicyConstraintException(path, QueryRejectionCode.MANDATORY_NATIVE_NOT_ALLOWED) + } + return QueryPolicyAllowance( + mandatoryCondition, + fieldConstraint, + resultConstraint, + streamConstraint, + pageConstraint, + analyticsConstraint, + ) + } + } + + companion object { + fun builder(): Builder = Builder() + } +} + +internal class QueryPolicyEnforcer( + private val policy: QueryPolicy, +) { + fun authorize(input: QueryPolicyInput): Mono = Mono.defer { + validateTarget(input) + evaluatePolicy(input) + }.flatMap { decision -> + when (decision) { + is QueryPolicyDecision.Allow -> { + validateAllowance(decision.allowance, input.schema) + Mono.just(decision.allowance.toPlanningConstraints(input.executionContext.validationMode)) + } + + is QueryPolicyDecision.Deny -> Mono.error( + rejectedException( + QueryRejectionCategory.ACCESS_DENIED, + POLICY_PATH, + QueryRejectionCode.POLICY_DENIED, + QueryPolicyDeniedException(decision.reason), + ), + ) + } + } + + private fun evaluatePolicy(input: QueryPolicyInput): Mono = + Mono.defer { policy.decide(input) } + .onErrorMap { error -> + when (error) { + is QueryPolicyConstraintException -> rejectedException( + QueryRejectionCategory.ACCESS_DENIED, + error.path, + error.code, + error, + ) + + else -> rejectedException( + QueryRejectionCategory.ACCESS_DENIED, + POLICY_PATH, + QueryRejectionCode.POLICY_EVALUATION_FAILED, + error, + ) + } + } + .switchIfEmpty( + Mono.error( + rejectedException( + QueryRejectionCategory.ACCESS_DENIED, + POLICY_PATH, + QueryRejectionCode.POLICY_DECISION_MISSING, + ), + ), + ) + + private fun validateTarget(input: QueryPolicyInput) { + if (input.executionContext.target != input.invocation.target || input.schema.target != input.invocation.target) { + throw rejectedException( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionPath.ROOT.property("target"), + QueryRejectionCode.TARGET_SCHEMA_MISMATCH, + ) + } + } + + private fun validateAllowance( + allowance: QueryPolicyAllowance, + schema: QueryDocumentSchema, + ) { + val fieldConstraints = listOf( + "filterFields" to allowance.fieldConstraint.filterFields, + "projectionFields" to allowance.fieldConstraint.projectionFields, + "sortFields" to allowance.fieldConstraint.sortFields, + "analyticsDimensionFields" to allowance.fieldConstraint.analyticsDimensionFields, + "analyticsMetricFields" to allowance.fieldConstraint.analyticsMetricFields, + ) + fieldConstraints.forEach { (name, access) -> + val invalid = (access as? FieldAccess.AllowList)?.fields?.firstOrNull { it !in schema.fields } + if (invalid != null) { + rejectInvalidConstraint(name) + } + } + val invalidScope = (allowance.fieldConstraint.searchScopes as? SearchScopeAccess.AllowList) + ?.scopes?.firstOrNull { it !in schema.searchScopes } + if (invalidScope != null) { + rejectInvalidConstraint("searchScopes") + } + } + + private fun rejectInvalidConstraint(name: String): Nothing = + throw rejectedException( + QueryRejectionCategory.ACCESS_DENIED, + POLICY_PATH.property("fieldConstraint").property(name), + QueryRejectionCode.POLICY_CONSTRAINT_INVALID, + ) +} + +internal class QueryPolicyConstraintException( + val path: QueryRejectionPath, + val code: QueryRejectionCode, +) : IllegalArgumentException("Invalid query policy constraint: $code at $path") + +internal class QueryPolicyDeniedException( + val reason: QueryPolicyDenial, +) : IllegalStateException("Query policy denied the request.") + +private fun NormalizedCondition.findNativePath(path: QueryRejectionPath): QueryRejectionPath? = + when (this) { + NormalizedCondition.All, + NormalizedCondition.None, + is NormalizedCondition.Predicate, + is NormalizedCondition.Search, + -> null + + is NormalizedCondition.Native -> path + is NormalizedCondition.ElementMatch -> condition.findNativePath(path.property("condition")) + is NormalizedCondition.Junction -> children.mapIndexedNotNull { index, child -> + child.findNativePath(path.property("children").index(index)) + }.firstOrNull() + } + +private val POLICY_PATH = QueryRejectionPath.ROOT.property("policy") diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/policy/TenantIsolationQueryPolicy.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/policy/TenantIsolationQueryPolicy.kt new file mode 100644 index 00000000000..09f2447e2da --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/policy/TenantIsolationQueryPolicy.kt @@ -0,0 +1,180 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.policy + +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.normalization.JunctionOperator +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.SystemFieldKind +import me.ahoo.wow.query.internal.planning.NativeBackendAccess +import me.ahoo.wow.query.internal.planning.QueryFieldConstraint +import reactor.core.publisher.Mono + +/** + * Minimal target-independent tenant boundary used by the future Gateway wiring. + * + * Route/header values are selectors only. A subject or tenant-scoped service must agree with them before the selector + * becomes a mandatory condition. System and exactly granted legacy authorities remain separate, auditable + * authority variants instead of being inferred from a missing authentication context. + */ +internal class TenantIsolationQueryPolicy : QueryPolicy { + override fun decide(input: QueryPolicyInput): Mono = Mono.fromSupplier { + when (val authority = input.executionContext.authority) { + is QueryAuthority.Subject -> decideSubject(authority, input.executionContext.resourceScope) + is QueryAuthority.Service -> decideService( + authority, + input.executionContext.purpose, + input.executionContext.resourceScope, + ) + + is QueryAuthority.System, + is QueryAuthority.Legacy, + -> input.executionContext.resourceScope.let { scope -> + allow(mandatoryScope(scope.tenantId, scope.ownerId, scope.spaceId?.let(::spaceEquals))) + } + } + } + + private fun decideSubject( + authority: QueryAuthority.Subject, + scope: QueryResourceScope, + ): QueryPolicyDecision { + if (scope.tenantId != null && scope.tenantId != authority.tenantId) { + return QueryPolicyDecision.Deny(QueryPolicyDenial.TENANT_MISMATCH) + } + val ownerId = + when (val owner = resolveOwner(authority.ownerGrant, scope.ownerId)) { + ScopeResolution.Denied -> return QueryPolicyDecision.Deny(QueryPolicyDenial.OWNER_MISMATCH) + ScopeResolution.Unrestricted -> null + is ScopeResolution.Restricted -> owner.value + } + val spaceCondition = + when (val space = resolveSpace(authority.spaceGrant, scope.spaceId)) { + ScopeResolution.Denied -> return QueryPolicyDecision.Deny(QueryPolicyDenial.SPACE_MISMATCH) + ScopeResolution.Unrestricted -> null + is ScopeResolution.Restricted -> space.value + } + return allow( + mandatoryScope( + authority.tenantId, + ownerId, + spaceCondition, + ), + ) + } + + private fun decideService( + authority: QueryAuthority.Service, + purpose: QueryPurpose, + scope: QueryResourceScope, + ): QueryPolicyDecision { + if (purpose !in authority.purposes) { + return QueryPolicyDecision.Deny(QueryPolicyDenial.PURPOSE_NOT_ALLOWED) + } + if (scope.tenantId != null && scope.tenantId != authority.tenantId) { + return QueryPolicyDecision.Deny(QueryPolicyDenial.TENANT_MISMATCH) + } + return allow(mandatoryScope(authority.tenantId, scope.ownerId, scope.spaceId?.let(::spaceEquals))) + } + + private fun allow(mandatoryCondition: NormalizedCondition): QueryPolicyDecision.Allow = + QueryPolicyDecision.Allow( + QueryPolicyAllowance.builder() + .mandatoryCondition(mandatoryCondition) + .fieldConstraint(QueryFieldConstraint(nativeBackends = NativeBackendAccess.DenyAll)) + .build(), + ) + + private fun mandatoryScope( + tenantId: String?, + ownerId: String?, + spaceCondition: NormalizedCondition?, + ): NormalizedCondition { + val predicates = buildList { + tenantId?.let { add(systemPredicate(SystemFieldKind.TENANT_ID, it)) } + ownerId?.let { add(systemPredicate(SystemFieldKind.OWNER_ID, it)) } + spaceCondition?.let(::add) + } + return when (predicates.size) { + 0 -> NormalizedCondition.All + 1 -> predicates.single() + else -> NormalizedCondition.Junction( + JunctionOperator.AND, + predicates, + ) + } + } + + private fun systemPredicate(kind: SystemFieldKind, value: String): NormalizedCondition.Predicate = + NormalizedCondition.Predicate( + LogicalField.System(kind), + PredicateOperator.EQ, + NormalizedValue.Text(value), + ) + + private fun resolveOwner(grant: QueryOwnerGrant, selector: String?): ScopeResolution = + when (grant) { + QueryOwnerGrant.Unrestricted -> selector?.let { ScopeResolution.Restricted(it) } + ?: ScopeResolution.Unrestricted + + is QueryOwnerGrant.Only -> { + if (selector == null || selector == grant.ownerId) { + ScopeResolution.Restricted(grant.ownerId) + } else { + ScopeResolution.Denied + } + } + } + + private fun resolveSpace( + grant: QuerySpaceGrant, + selector: String?, + ): ScopeResolution = + when (grant) { + QuerySpaceGrant.Unrestricted -> selector?.let { ScopeResolution.Restricted(spaceEquals(it)) } + ?: ScopeResolution.Unrestricted + + QuerySpaceGrant.DenyAll -> ScopeResolution.Denied + is QuerySpaceGrant.AllowList -> { + if (selector != null) { + if (selector in grant.spaceIds) { + ScopeResolution.Restricted(spaceEquals(selector)) + } else { + ScopeResolution.Denied + } + } else { + ScopeResolution.Restricted( + NormalizedCondition.Predicate( + LogicalField.System(SystemFieldKind.SPACE_ID), + PredicateOperator.IN, + NormalizedValue.ListValue(grant.spaceIds.map(NormalizedValue::Text)), + ), + ) + } + } + } + + private fun spaceEquals(spaceId: String): NormalizedCondition = + systemPredicate(SystemFieldKind.SPACE_ID, spaceId) + + private sealed interface ScopeResolution { + data object Denied : ScopeResolution + + data object Unrestricted : ScopeResolution + + data class Restricted(val value: T) : ScopeResolution + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/rejection/QueryRejection.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/rejection/QueryRejection.kt new file mode 100644 index 00000000000..0fa457de4ca --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/rejection/QueryRejection.kt @@ -0,0 +1,210 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.rejection + +internal enum class QueryRejectionCategory { + ACCESS_DENIED, + INVALID_QUERY, + INVALID_CURSOR, + BUDGET_EXCEEDED, + UNSUPPORTED_FEATURE, + BACKEND_UNAVAILABLE, + BACKEND_TIMEOUT, + INCOMPLETE_RESULT, + MAPPING_FAILURE, + INTERNAL_FAILURE, +} + +internal enum class QueryRejectionCode { + CONDITION_DEPTH_LIMIT_EXCEEDED, + CONDITION_NODE_LIMIT_EXCEEDED, + CHILDREN_LIMIT_EXCEEDED, + COLLECTION_LIMIT_EXCEEDED, + OBJECT_LIMIT_EXCEEDED, + VALUE_DEPTH_LIMIT_EXCEEDED, + VALUE_NODE_LIMIT_EXCEEDED, + NUMERIC_PRECISION_LIMIT_EXCEEDED, + BYTE_ARRAY_LIMIT_EXCEEDED, + PAYLOAD_LIMIT_EXCEEDED, + PROJECTION_LIMIT_EXCEEDED, + SORT_LIMIT_EXCEEDED, + OPTIONS_LIMIT_EXCEEDED, + STRING_LIMIT_EXCEEDED, + CYCLIC_INPUT, + INVALID_CHILDREN, + FIELD_REQUIRED, + INVALID_FIELD, + INVALID_VALUE_TYPE, + INVALID_VALUE_ARITY, + DUPLICATE_OBJECT_KEY, + INVALID_OPTION_TYPE, + INVALID_OPTION_VALUE, + UNKNOWN_OPTION, + OPTION_NOT_ALLOWED, + INVALID_TIME_VALUE, + INVALID_LIMIT, + INVALID_PAGE, + INVALID_PROJECTION, + INVALID_SORT, + SYSTEM_FIELD_IN_ELEMENT_SCOPE, + NATIVE_BACKEND_UNBOUND, + INVALID_INVOCATION, + TARGET_SCHEMA_MISMATCH, + FIELD_NOT_FOUND, + SEARCH_SCOPE_NOT_FOUND, + OPERATOR_NOT_ALLOWED, + CAPABILITY_UNAVAILABLE, + CASE_INSENSITIVE_UNSUPPORTED, + TYPED_PROJECTION_NOT_ALLOWED, + DUPLICATE_SORT, + NATIVE_BACKEND_CONFLICT, + MANDATORY_NATIVE_NOT_ALLOWED, + ANALYTICS_DOCUMENT_KIND_UNSUPPORTED, + ANALYTICS_DIMENSION_TYPE_UNSUPPORTED, + ANALYTICS_METRIC_TYPE_UNSUPPORTED, + ANALYTICS_NUMERIC_POLICY_REQUIRED, + ANALYTICS_NUMERIC_POLICY_UNSUPPORTED, + ANALYTICS_HAVING_UNSUPPORTED, + ANALYTICS_ORDER_UNSUPPORTED, + ANALYTICS_CONSISTENCY_UNSUPPORTED, + ANALYTICS_COMPLETENESS_UNSUPPORTED, + DUPLICATE_ANALYTICS_ALIAS, + INVALID_CURSOR_BINDING, + INVALID_CURSOR_TOKEN, + CURSOR_EXPIRED, + CURSOR_CAPACITY_EXCEEDED, + CURSOR_STORE_REQUIRED, + CURSOR_PAGE_LIMIT_EXCEEDED, + CURSOR_BUDGET_RELAXATION_NOT_ALLOWED, + VALUE_TYPE_MISMATCH, + UNBOUNDED_STREAM_DISALLOWED, + PAGE_WINDOW_EXCEEDED, + ANALYTICS_DIMENSION_LIMIT_EXCEEDED, + ANALYTICS_METRIC_LIMIT_EXCEEDED, + ANALYTICS_BUCKET_LIMIT_EXCEEDED, + AUTHORITY_REQUIRED, + AUTHORITY_RESOLUTION_FAILED, + LEGACY_CALLER_NOT_ALLOWED, + QUERY_TRANSPORT_AUTHORITY_MISMATCH, + DEADLINE_EXPIRED, + POLICY_DENIED, + POLICY_DECISION_MISSING, + POLICY_EVALUATION_FAILED, + POLICY_CONSTRAINT_INVALID, + FILTER_FIELD_NOT_ALLOWED, + SEARCH_SCOPE_NOT_ALLOWED, + NATIVE_BACKEND_NOT_ALLOWED, + PROJECTION_FIELD_NOT_ALLOWED, + SORT_FIELD_NOT_ALLOWED, + ANALYTICS_DIMENSION_FIELD_NOT_ALLOWED, + ANALYTICS_METRIC_FIELD_NOT_ALLOWED, + RESULT_LIMIT_EXCEEDED, + EXECUTION_BUDGET_UNSUPPORTED, + SCHEMA_NOT_REGISTERED, + EXECUTION_MODE_UNSUPPORTED, + EXECUTION_DECISION_INVALID, + SHADOW_PROBE_UNBOUNDED_STREAM, + SHADOW_SUPERVISOR_UNAVAILABLE, + SHADOW_SUPERVISOR_SATURATED, + BACKEND_NOT_REGISTERED, + BACKEND_NOT_READY, + BACKEND_SCHEMA_MISMATCH, + BACKEND_OPERATION_UNSUPPORTED, + BACKEND_CAPABILITY_MISMATCH, + LEGACY_BACKEND_NOT_REGISTERED, + LEGACY_LOWERING_UNSUPPORTED, + MANDATORY_CONDITION_UNENFORCEABLE, + BACKEND_EXECUTION_FAILED, + BACKEND_TIMEOUT, + BACKEND_BUDGET_EXCEEDED, + INCOMPLETE_RESULT, + RESULT_MAPPING_FAILED, + UNEXPECTED_QUERY_FAILURE, +} + +internal class QueryRejectionPath private constructor( + private val segments: List, +) { + fun property(name: String): QueryRejectionPath = QueryRejectionPath(segments + Segment.Property(name)) + + fun index(index: Int): QueryRejectionPath = QueryRejectionPath(segments + Segment.Index(index)) + + fun key(key: String): QueryRejectionPath = QueryRejectionPath(segments + Segment.Key(key)) + + override fun equals(other: Any?): Boolean = + this === other || other is QueryRejectionPath && segments == other.segments + + override fun hashCode(): Int = segments.hashCode() + + override fun toString(): String = buildString { + append('$') + segments.forEach { segment -> + when (segment) { + is Segment.Property -> append('.').append(segment.name) + is Segment.Index -> append('[').append(segment.value).append(']') + is Segment.Key -> append("['").append(segment.value.escapeKey()).append("']") + } + } + } + + private sealed interface Segment { + data class Property(val name: String) : Segment + + data class Index(val value: Int) : Segment + + data class Key(val value: String) : Segment + } + + private fun String.escapeKey(): String = buildString { + this@escapeKey.forEach { character -> + when (character) { + '\\' -> append("\\\\") + '\'' -> append("\\'") + '\n' -> append("\\n") + '\r' -> append("\\r") + '\t' -> append("\\t") + else -> if (character.isISOControl()) { + append("\\u").append(character.code.toString(16).padStart(4, '0')) + } else { + append(character) + } + } + } + } + + companion object { + val ROOT: QueryRejectionPath = QueryRejectionPath(emptyList()) + } +} + +internal data class QueryRejection( + val category: QueryRejectionCategory, + val path: QueryRejectionPath, + val code: QueryRejectionCode, +) + +internal class QueryRejectedException( + val rejection: QueryRejection, + cause: Throwable? = null, +) : IllegalArgumentException( + "${rejection.category}/${rejection.code} at ${rejection.path}", + cause, +) + +internal fun rejectQuery( + category: QueryRejectionCategory, + path: QueryRejectionPath, + code: QueryRejectionCode, + cause: Throwable? = null, +): Nothing = throw QueryRejectedException(QueryRejection(category, path, code), cause) diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/schema/QuerySchemaRegistry.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/schema/QuerySchemaRegistry.kt new file mode 100644 index 00000000000..cfed959b55e --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/schema/QuerySchemaRegistry.kt @@ -0,0 +1,41 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.schema + +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.internal.model.QueryTarget +import java.util.Collections +import java.util.LinkedHashMap + +internal class QuerySchemaRegistry(schemas: Iterable) { + val schemas: Map + + init { + val materialized = schemas.toList() + require(materialized.map(QueryDocumentSchema::target).distinct().size == materialized.size) { + "Query schema targets must be unique." + } + val copy = LinkedHashMap(materialized.size) + materialized.sortedWith(compareBy(QUERY_TARGET_COMPARATOR, QueryDocumentSchema::target)) + .forEach { schema -> copy[schema.target] = schema } + this.schemas = Collections.unmodifiableMap(copy) + } + + operator fun get(target: QueryTarget): QueryDocumentSchema? = schemas[target] +} + +private val QUERY_TARGET_COMPARATOR: Comparator = + compareBy { it.namedAggregate.contextName } + .thenBy { it.namedAggregate.aggregateName } + .thenBy { it.documentKind.name } diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/value/NonEmptyList.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/value/NonEmptyList.kt new file mode 100644 index 00000000000..3f841c32b31 --- /dev/null +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/internal/value/NonEmptyList.kt @@ -0,0 +1,45 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.value + +import java.util.Collections + +internal class NonEmptyList private constructor(values: List) { + val values: List = Collections.unmodifiableList(values) + + val first: T + get() = values.first() + + override fun equals(other: Any?): Boolean = + this === other || other is NonEmptyList<*> && values == other.values + + override fun hashCode(): Int = values.hashCode() + + override fun toString(): String = values.toString() + + companion object { + fun of( + first: T, + vararg rest: T, + ): NonEmptyList = NonEmptyList(listOf(first, *rest)) + + fun from(values: Iterable): NonEmptyList? { + val materialized = values.toList() + if (materialized.isEmpty()) { + return null + } + return NonEmptyList(materialized) + } + } +} diff --git a/wow-query/src/main/kotlin/me/ahoo/wow/query/snapshot/filter/AbacQueryFilter.kt b/wow-query/src/main/kotlin/me/ahoo/wow/query/snapshot/filter/AbacQueryFilter.kt index 77fc4383c85..408d0a8d624 100644 --- a/wow-query/src/main/kotlin/me/ahoo/wow/query/snapshot/filter/AbacQueryFilter.kt +++ b/wow-query/src/main/kotlin/me/ahoo/wow/query/snapshot/filter/AbacQueryFilter.kt @@ -24,6 +24,7 @@ import me.ahoo.wow.api.query.Operator import me.ahoo.wow.filter.FilterChain import me.ahoo.wow.filter.FilterType import me.ahoo.wow.query.dsl.condition +import me.ahoo.wow.query.filter.PreAdmissionQueryFilter import me.ahoo.wow.query.filter.QueryContext import me.ahoo.wow.serialization.state.StateAggregateRecords.TAGS import reactor.core.publisher.Mono @@ -31,10 +32,12 @@ import reactor.kotlin.core.publisher.toMono import reactor.util.context.ContextView /** - * Filters snapshot queries using attribute-based access control (ABAC). + * Compatibility filter that appends legacy ABAC tag conditions before Gateway admission. * * Principal tags from the current context are converted into query conditions and - * appended to snapshot queries. + * appended to snapshot queries. The appended condition remains a user condition; it is not a mandatory policy + * constraint and must not be used as the authorization boundary. New applications should implement + * the Query Gateway policy boundary instead. * * ## Matching rules * @@ -49,7 +52,8 @@ import reactor.util.context.ContextView */ @Order(ORDER_FIRST) @FilterType(SnapshotQueryHandler::class) -abstract class AbacQueryFilter : SnapshotQueryFilter { +@Deprecated("Move ABAC authorization to Query Gateway policy constraints so it is mandatory and fail closed.") +abstract class AbacQueryFilter : SnapshotQueryFilter, PreAdmissionQueryFilter { companion object { /** * Converts one principal tag into a nested query condition. diff --git a/wow-query/src/test/java/me/ahoo/wow/query/JavaQueryServiceCompatibilityTest.java b/wow-query/src/test/java/me/ahoo/wow/query/JavaQueryServiceCompatibilityTest.java new file mode 100644 index 00000000000..070ee50c728 --- /dev/null +++ b/wow-query/src/test/java/me/ahoo/wow/query/JavaQueryServiceCompatibilityTest.java @@ -0,0 +1,84 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query; + +import me.ahoo.wow.api.modeling.NamedAggregate; +import me.ahoo.wow.api.query.Condition; +import me.ahoo.wow.api.query.DynamicDocument; +import me.ahoo.wow.api.query.IListQuery; +import me.ahoo.wow.api.query.IPagedQuery; +import me.ahoo.wow.api.query.ISingleQuery; +import me.ahoo.wow.api.query.PagedList; +import me.ahoo.wow.modeling.MaterializedNamedAggregate; +import me.ahoo.wow.query.filter.QueryType; +import org.junit.jupiter.api.Test; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +import static org.junit.jupiter.api.Assertions.assertFalse; + +class JavaQueryServiceCompatibilityTest { + + @Test + void shouldCompileAndCallThePublicJavaContract() { + QueryService service = new JavaQueryService(); + + assertFalse(QueryType.COUNT.isDynamic()); + assertFalse(service.getAggregateName().isBlank()); + } + + private static final class JavaQueryService implements QueryService { + private final NamedAggregate namedAggregate = new MaterializedNamedAggregate("sales", "order"); + + @Override + public NamedAggregate getNamedAggregate() { + return namedAggregate; + } + + @Override + public Mono single(ISingleQuery singleQuery) { + return Mono.empty(); + } + + @Override + public Mono dynamicSingle(ISingleQuery singleQuery) { + return Mono.empty(); + } + + @Override + public Flux list(IListQuery listQuery) { + return Flux.empty(); + } + + @Override + public Flux dynamicList(IListQuery listQuery) { + return Flux.empty(); + } + + @Override + public Mono> paged(IPagedQuery pagedQuery) { + return Mono.empty(); + } + + @Override + public Mono> dynamicPaged(IPagedQuery pagedQuery) { + return Mono.empty(); + } + + @Override + public Mono count(Condition condition) { + return Mono.empty(); + } + } +} diff --git a/wow-query/src/test/java/me/ahoo/wow/query/PublicAnalyticsJavaCompatibilityTest.java b/wow-query/src/test/java/me/ahoo/wow/query/PublicAnalyticsJavaCompatibilityTest.java new file mode 100644 index 00000000000..01bdea7bfdc --- /dev/null +++ b/wow-query/src/test/java/me/ahoo/wow/query/PublicAnalyticsJavaCompatibilityTest.java @@ -0,0 +1,121 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query; + +import me.ahoo.wow.api.modeling.NamedAggregate; +import me.ahoo.wow.api.query.Condition; +import me.ahoo.wow.api.query.analytics.AnalyticsBucketWindow; +import me.ahoo.wow.api.query.analytics.AnalyticsCompleteness; +import me.ahoo.wow.api.query.analytics.AnalyticsConsistency; +import me.ahoo.wow.api.query.analytics.AnalyticsGrouping; +import me.ahoo.wow.api.query.analytics.AnalyticsMetric; +import me.ahoo.wow.api.query.analytics.AnalyticsMetricKind; +import me.ahoo.wow.api.query.analytics.AnalyticsPage; +import me.ahoo.wow.api.query.analytics.AnalyticsQuery; +import me.ahoo.wow.modeling.MaterializedNamedAggregate; +import me.ahoo.wow.query.analytics.AnalyticsQueryService; +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness; +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency; +import me.ahoo.wow.query.backend.BackendAnalyticsPage; +import me.ahoo.wow.query.cursor.QueryCursorLeaseCreateResult; +import me.ahoo.wow.query.cursor.QueryCursorLeaseEntry; +import me.ahoo.wow.query.cursor.QueryCursorLeaseId; +import me.ahoo.wow.query.cursor.QueryCursorLeaseStore; +import me.ahoo.wow.query.cursor.StoredQueryCursorLease; +import me.ahoo.wow.query.gateway.AnalyticsQueryGateway; +import org.junit.jupiter.api.Test; +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; + +import java.time.Instant; +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +class PublicAnalyticsJavaCompatibilityTest { + + @Test + void shouldCompileAndCallThePublicJavaContracts() { + AnalyticsQuery query = new AnalyticsQuery( + new Condition(), + AnalyticsGrouping.global(), + List.of(new AnalyticsMetric("count", AnalyticsMetricKind.DOCUMENT_COUNT, null)), + new AnalyticsBucketWindow(1, null), + null, + AnalyticsConsistency.EVENTUAL, + AnalyticsCompleteness.EXACT + ); + AnalyticsPage expected = new AnalyticsPage( + List.of(), + null, + AnalyticsConsistency.EVENTUAL, + AnalyticsCompleteness.EXACT + ); + AnalyticsQueryService service = new JavaAnalyticsQueryService(expected); + AnalyticsQueryGateway gateway = (call, request) -> Mono.just(expected); + QueryCursorLeaseStore store = new JavaCursorLeaseStore(); + BackendAnalyticsPage backendPage = new BackendAnalyticsPage( + List.of(), + null, + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT + ); + + assertEquals(expected, service.analyze(query).block()); + assertEquals(expected, gateway.analyze(null, query).block()); + assertEquals(0L, store.scanExpired(Instant.EPOCH, null, 10).count().block()); + assertEquals(BackendAnalyticsConsistency.EVENTUAL, backendPage.getConsistency()); + } + + private static final class JavaAnalyticsQueryService implements AnalyticsQueryService { + private final NamedAggregate namedAggregate = new MaterializedNamedAggregate("sales", "order"); + private final AnalyticsPage result; + + private JavaAnalyticsQueryService(AnalyticsPage result) { + this.result = result; + } + + @Override + public NamedAggregate getNamedAggregate() { + return namedAggregate; + } + + @Override + public Mono analyze(AnalyticsQuery query) { + return Mono.just(result); + } + } + + private static final class JavaCursorLeaseStore implements QueryCursorLeaseStore { + @Override + public Mono create(QueryCursorLeaseEntry entry) { + return Mono.just(QueryCursorLeaseCreateResult.CREATED); + } + + @Override + public Mono load(QueryCursorLeaseId id) { + return Mono.empty(); + } + + @Override + public Mono compareAndDelete(StoredQueryCursorLease expected) { + return Mono.just(false); + } + + @Override + public Flux scanExpired(Instant before, QueryCursorLeaseId afterId, int limit) { + return Flux.empty(); + } + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/PublicAnalyticsQueryContractTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/PublicAnalyticsQueryContractTest.kt new file mode 100644 index 00000000000..fcdfb8ab304 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/PublicAnalyticsQueryContractTest.kt @@ -0,0 +1,78 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.query + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.query.analytics.AnalyticsQueryService +import me.ahoo.wow.query.analytics.AnalyticsQueryServiceFactory +import me.ahoo.wow.query.cursor.QueryCursorHmacKey +import me.ahoo.wow.query.cursor.QueryCursorLeaseStore +import me.ahoo.wow.query.gateway.AnalyticsQueryGateway +import org.junit.jupiter.api.Test +import java.lang.reflect.Modifier + +class PublicAnalyticsQueryContractTest { + @Test + fun `analytics ports should remain additive and operation specific`() { + AnalyticsQueryGateway::class.java.publicGenericSignatures().assert().containsExactly( + "analyze(me.ahoo.wow.query.gateway.QueryCall,me.ahoo.wow.api.query.analytics.AnalyticsQuery):" + + "reactor.core.publisher.Mono", + ) + AnalyticsQueryService::class.java.publicGenericSignatures().assert().containsExactly( + "analyze(me.ahoo.wow.api.query.analytics.AnalyticsQuery):" + + "reactor.core.publisher.Mono", + ) + AnalyticsQueryServiceFactory::class.java.publicGenericSignatures().assert().containsExactly( + "create(me.ahoo.wow.api.modeling.NamedAggregate):me.ahoo.wow.query.analytics.AnalyticsQueryService", + ) + QueryService::class.java.declaredMethods.map { it.name }.assert().doesNotContain("analyze") + } + + @Test + fun `persistent cursor store should expose create load cas delete and bounded expiry scan`() { + QueryCursorLeaseStore::class.java.publicGenericSignatures().assert().containsExactly( + "compareAndDelete(me.ahoo.wow.query.cursor.StoredQueryCursorLease):" + + "reactor.core.publisher.Mono", + "create(me.ahoo.wow.query.cursor.QueryCursorLeaseEntry):" + + "reactor.core.publisher.Mono", + "load(me.ahoo.wow.query.cursor.QueryCursorLeaseId):" + + "reactor.core.publisher.Mono", + "scanExpired(java.time.Instant,me.ahoo.wow.query.cursor.QueryCursorLeaseId,int):" + + "reactor.core.publisher.Flux", + ) + } + + @Test + fun `cursor key should not expose its secret through the supported JVM contract`() { + QueryCursorHmacKey::class.java.declaredMethods + .filter { method -> Modifier.isPublic(method.modifiers) && !method.isSynthetic } + .map { method -> method.name } + .filter { methodName -> methodName.startsWith("secretCopy") } + .assert() + .isEmpty() + } + + private fun Class<*>.publicGenericSignatures(): List = declaredMethods + .filter { method -> Modifier.isPublic(method.modifiers) && !method.isSynthetic } + .map { method -> + val parameters = method.genericParameterTypes.joinToString(",") { type -> type.typeName } + "${method.name}($parameters):${method.genericReturnType.typeName}" + } + .sorted() +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/PublicQueryContractCompatibilityTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/PublicQueryContractCompatibilityTest.kt new file mode 100644 index 00000000000..3ea6a6b7b6c --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/PublicQueryContractCompatibilityTest.kt @@ -0,0 +1,129 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.query.filter.QueryType +import me.ahoo.wow.query.gateway.GatewayEventStreamQueryServiceFactory +import me.ahoo.wow.query.gateway.GatewaySnapshotQueryServiceFactory +import me.ahoo.wow.query.gateway.QueryGatewayRuntime +import org.junit.jupiter.api.Test +import java.lang.reflect.Modifier + +class PublicQueryContractCompatibilityTest { + + @Test + fun `query service should retain seven public generic signatures`() { + QueryService::class.java.declaredMethods + .filter { Modifier.isPublic(it.modifiers) && !it.isSynthetic } + .map { method -> + val parameters = method.genericParameterTypes.joinToString(",") { it.typeName } + "${method.name}($parameters):${method.genericReturnType.typeName}" + } + .sorted() + .assert() + .containsExactly( + "count(me.ahoo.wow.api.query.Condition):reactor.core.publisher.Mono", + "dynamicList(me.ahoo.wow.api.query.IListQuery):reactor.core.publisher.Flux", + "dynamicPaged(me.ahoo.wow.api.query.IPagedQuery):reactor.core.publisher.Mono>", + "dynamicSingle(me.ahoo.wow.api.query.ISingleQuery):reactor.core.publisher.Mono", + "list(me.ahoo.wow.api.query.IListQuery):reactor.core.publisher.Flux", + "paged(me.ahoo.wow.api.query.IPagedQuery):reactor.core.publisher.Mono>", + "single(me.ahoo.wow.api.query.ISingleQuery):reactor.core.publisher.Mono", + ) + + QueryService::class.java.genericInterfaces.map { it.typeName }.assert().containsExactly( + "me.ahoo.wow.api.modeling.NamedAggregateDecorator", + ) + } + + @Test + fun `query type should retain seven values`() { + QueryType.entries.map { it.name }.assert().containsExactly( + "SINGLE", + "DYNAMIC_SINGLE", + "LIST", + "DYNAMIC_LIST", + "PAGED", + "DYNAMIC_PAGED", + "COUNT", + ) + } + + @Test + fun `query gateway runtime should retain legacy composition factory`() { + QueryGatewayRuntime.Companion::class.java.declaredMethods + .filter { method -> method.name == "create" && Modifier.isPublic(method.modifiers) } + .map { method -> + val parameters = method.parameterTypes.joinToString(",") { type -> type.name } + "${method.name}($parameters):${method.returnType.name}" + }.assert() + .contains( + "create(java.lang.Iterable," + + "me.ahoo.wow.query.gateway.QueryRawServiceSource," + + "me.ahoo.wow.query.gateway.QueryLegacyDialectResolver," + + "me.ahoo.wow.query.gateway.QueryAuthorityResolver," + + "me.ahoo.wow.query.gateway.QueryTrustedContextResolver," + + "java.lang.Iterable," + + "me.ahoo.wow.query.gateway.QueryGatewayConfiguration," + + "java.time.Clock," + + "reactor.core.scheduler.Scheduler):" + + "me.ahoo.wow.query.gateway.QueryGatewayRuntime", + ) + } + + @Test + fun `analytics facade factory should use only its construction-time frozen resolver`() { + QueryGatewayRuntime::class.java.declaredMethods + .filter { method -> + method.name == "analyticsQueryServiceFactory" && + Modifier.isPublic(method.modifiers) && + !method.isSynthetic + } + .map { method -> method.parameterCount } + .assert() + .containsExactly(0) + } + + @Test + fun `trusted authority writer must not be JVM public`() { + listOf( + "me.ahoo.wow.query.gateway.QueryGatewayKt", + "me.ahoo.wow.query.gateway.QueryServiceFacadeKt", + ).flatMap { className -> Class.forName(className).methods.toList() } + .map { method -> method.name } + .filter { methodName -> methodName.contains("withTrustedQueryAuthority") } + .assert() + .isEmpty() + + listOf( + QueryGatewayRuntime::class.java, + GatewaySnapshotQueryServiceFactory::class.java, + GatewayEventStreamQueryServiceFactory::class.java, + ).flatMap { type -> + val constructorTypes = type.declaredConstructors + .filter { constructor -> Modifier.isPublic(constructor.modifiers) } + .flatMap { constructor -> constructor.parameterTypes.toList() } + val methodTypes = type.declaredMethods + .filter { method -> Modifier.isPublic(method.modifiers) } + .flatMap { method -> method.parameterTypes.toList() + method.returnType } + constructorTypes + methodTypes + }.map(Class<*>::getName) + .filter { typeName -> typeName.startsWith("me.ahoo.wow.query.internal.") } + .assert() + .isEmpty() + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/analytics/AnalyticsPublicJsonContractTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/analytics/AnalyticsPublicJsonContractTest.kt new file mode 100644 index 00000000000..1052de6bf91 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/analytics/AnalyticsPublicJsonContractTest.kt @@ -0,0 +1,72 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.analytics + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.analytics.AnalyticsBucket +import me.ahoo.wow.api.query.analytics.AnalyticsBucketWindow +import me.ahoo.wow.api.query.analytics.AnalyticsCompleteness +import me.ahoo.wow.api.query.analytics.AnalyticsConsistency +import me.ahoo.wow.api.query.analytics.AnalyticsCursor +import me.ahoo.wow.api.query.analytics.AnalyticsDimension +import me.ahoo.wow.api.query.analytics.AnalyticsGrouping +import me.ahoo.wow.api.query.analytics.AnalyticsMetric +import me.ahoo.wow.api.query.analytics.AnalyticsMetricKind +import me.ahoo.wow.api.query.analytics.AnalyticsPage +import me.ahoo.wow.api.query.analytics.AnalyticsQuery +import me.ahoo.wow.api.query.analytics.AnalyticsValue +import me.ahoo.wow.serialization.toJsonString +import me.ahoo.wow.serialization.toObject +import org.junit.jupiter.api.Test + +class AnalyticsPublicJsonContractTest { + @Test + fun `request and response should round trip with an opaque scalar cursor`() { + val request = AnalyticsQuery( + condition = Condition.eq("state.status", "PAID"), + grouping = AnalyticsGrouping.by(listOf(AnalyticsDimension("status", "state.status"))), + metrics = listOf(AnalyticsMetric("count", AnalyticsMetricKind.DOCUMENT_COUNT)), + window = AnalyticsBucketWindow(100, AnalyticsCursor("payload.signature")), + ) + val requestJson = request.toJsonString() + + requestJson.contains("\"cursor\":\"payload.signature\"").assert().isTrue() + requestJson.toObject().assert().isEqualTo(request) + + val response = AnalyticsPage( + listOf( + AnalyticsBucket( + mapOf("status" to AnalyticsValue.of("PAID")), + mapOf("count" to AnalyticsValue.of(Long.MAX_VALUE)), + ), + ), + AnalyticsCursor("next.signature"), + AnalyticsConsistency.EVENTUAL, + AnalyticsCompleteness.EXACT, + ) + val responseJson = response.toJsonString() + + responseJson.contains("\"value\":\"${Long.MAX_VALUE}\"").assert().isTrue() + responseJson.toObject().assert().isEqualTo(response) + + val finalPageJson = AnalyticsPage( + emptyList(), + null, + AnalyticsConsistency.EVENTUAL, + AnalyticsCompleteness.EXACT, + ).toJsonString() + finalPageJson.contains("\"nextCursor\":null").assert().isTrue() + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/backend/AnalyticsQueryBackendTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/backend/AnalyticsQueryBackendTest.kt new file mode 100644 index 00000000000..4afe681d674 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/backend/AnalyticsQueryBackendTest.kt @@ -0,0 +1,255 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.backend + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryTarget +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.math.BigDecimal +import java.math.RoundingMode +import java.util.function.Consumer + +@OptIn( + ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) +class AnalyticsQueryBackendTest { + @Test + fun `analytics plan and result should freeze every collection boundary`() { + val dimensions = mutableListOf( + BackendAnalyticsDimension(alias, amount, BackendAnalyticsMissingPolicy.AS_NULL_BUCKET), + ) + val metrics = mutableListOf( + BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count")), + BackendAnalyticsMetric.Sum(AnalyticsAlias("total"), amount), + ) + val afterKey = mutableListOf(NormalizedValue.Decimal("1.00".toBigDecimal())) + val plan = analyticsPlan(dimensions, metrics, afterKey) + dimensions.clear() + metrics.clear() + afterKey.clear() + + (plan.grouping as BackendAnalyticsGrouping.By).dimensions.assert().hasSize(1) + plan.metrics.assert().hasSize(2) + plan.bucketWindow.afterKey!!.assert().containsExactly(NormalizedValue.Decimal("1".toBigDecimal())) + @Suppress("UNCHECKED_CAST") + assertThrownBy { + (plan.metrics as MutableList).clear() + } + + val keys = linkedMapOf(alias to NormalizedValue.Decimal("1.0".toBigDecimal())) + val values = linkedMapOf(AnalyticsAlias("count") to NormalizedValue.Int64(2)) + val buckets = mutableListOf(BackendAnalyticsBucket(keys, values)) + val page = BackendAnalyticsPage( + buckets, + listOf(NormalizedValue.Decimal("1.0".toBigDecimal())), + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + ) + keys.clear() + values.clear() + buckets.clear() + + page.buckets.assert().hasSize(1) + page.buckets.single().keys.assert().containsEntry(alias, NormalizedValue.Decimal("1".toBigDecimal())) + page.afterKey!!.assert().hasSize(1) + } + + @Test + fun `analytics aliases should be safe Mongo field names`() { + listOf("", "a.b", "\$metric", "a\u0000b").forEach { invalid -> + assertThrownBy { AnalyticsAlias(invalid) } + } + } + + @Test + fun `analytics plan should make global and grouped paging invariants explicit`() { + val valid = analyticsPlan( + listOf(BackendAnalyticsDimension(alias, amount, BackendAnalyticsMissingPolicy.EXCLUDE)), + listOf(BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + listOf(NormalizedValue.Decimal(BigDecimal.ONE)), + ) + + assertThrownBy { + rebuildPlan( + valid, + BackendAnalyticsGrouping.Global, + BackendAnalyticsBucketOrder.Global, + BackendAnalyticsPageWindow(2), + ) + } + assertThrownBy { + rebuildPlan( + valid, + valid.grouping, + BackendAnalyticsBucketOrder.Global, + BackendAnalyticsPageWindow(1), + ) + } + } + + @Test + fun `analytics contribution should require an explicit analytics backend`() { + assertThrownBy { + contribution(null) + } + contribution(AnalyticsQueryBackend { _, _ -> reactor.core.publisher.Mono.empty() }) + .supportedOperations.assert().containsExactly(QueryOperation.ANALYZE, QueryOperation.COUNT) + } + + @Test + fun `analytics cursor state should be opaque immutable and rejected by a stateless backend`() { + val source = byteArrayOf(1, 2, 3) + val state = BackendAnalyticsCursorState(source) + source[0] = 9 + state.payload().contentEquals(byteArrayOf(1, 2, 3)).assert().isTrue() + + val returned = state.payload() + returned[1] = 9 + state.payload().contentEquals(byteArrayOf(1, 2, 3)).assert().isTrue() + BackendAnalyticsCursorState(byteArrayOf(1, 2, 3)).assert().isEqualTo(state) + + val backend = AnalyticsQueryBackend { _, _ -> Mono.empty() } + assertThrownBy { + backend.analyze( + analyticsPlan( + listOf(BackendAnalyticsDimension(alias, amount, BackendAnalyticsMissingPolicy.EXCLUDE)), + listOf(BackendAnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + listOf(NormalizedValue.Decimal(BigDecimal.ONE)), + ), + QueryBackendExecutionOptions(null, null), + state, + ).block() + }.satisfies( + Consumer { error -> error.kind.assert().isEqualTo(QueryBackendFailureKind.UNSUPPORTED) }, + ) + } + + private fun analyticsPlan( + dimensions: Iterable, + metrics: Iterable, + afterKey: Iterable, + ) = BackendAnalyticsQueryPlan( + target, + schema.contractId, + BackendEnforcedFilter(BackendPlannedCondition.All, BackendPlannedCondition.All), + BackendAnalyticsGrouping.By(dimensions), + metrics, + BackendAnalyticsCondition.All, + BackendAnalyticsBucketOrder.DimensionKeyAscending( + BackendAnalyticsNullPlacement.FIRST, + BackendAnalyticsTextCollation.BINARY, + ), + BackendAnalyticsPageWindow(10, afterKey), + BackendAnalyticsNumericPolicy( + BackendAnalyticsNumericPromotion.DECIMAL128, + 34, + 2, + RoundingMode.HALF_UP, + BackendAnalyticsOverflowPolicy.REJECT, + ), + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + BackendRequiredCapabilities(mapOf(amount to setOf(FieldCapability.AGGREGATABLE))), + SemanticTier.PORTABLE, + PlanFingerprint("1".repeat(64)), + ) + + private fun rebuildPlan( + source: BackendAnalyticsQueryPlan, + grouping: BackendAnalyticsGrouping, + order: BackendAnalyticsBucketOrder, + window: BackendAnalyticsPageWindow, + ) = BackendAnalyticsQueryPlan( + source.target, + source.schemaContractId, + source.filter, + grouping, + source.metrics, + source.having, + order, + window, + source.numericPolicy, + source.requiredConsistency, + source.requiredCompleteness, + source.requiredCapabilities, + source.semanticTier, + source.fingerprint, + ) + + private fun contribution(analytics: AnalyticsQueryBackend?): RecordQueryBackendContribution = + RecordQueryBackendContribution( + schema, + BackendId("test"), + setOf(QueryOperation.COUNT, QueryOperation.ANALYZE), + BackendStreamSupport.NONE, + setOf(SemanticTier.PORTABLE), + mapOf(amount to setOf(FieldCapability.AGGREGATABLE)), + backend = NO_OP_BACKEND, + analyticsBackend = analytics, + ) + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val state = QueryFieldId.Path(listOf("state")) + private val amount = QueryFieldId.Path(listOf("state", "amount")) + private val alias = AnalyticsAlias("amount") + private val schema = QueryDocumentSchema( + target, + listOf( + QueryFieldSchema( + state, + LogicalFieldType.Object, + Presence.OPTIONAL, + Nullability.NULLABLE, + emptyList(), + emptyList(), + ), + QueryFieldSchema( + amount, + LogicalFieldType.Decimal, + Presence.OPTIONAL, + Nullability.NULLABLE, + emptyList(), + listOf(FieldCapability.AGGREGATABLE), + ), + ), + emptyList(), + ) + + private companion object { + val NO_OP_BACKEND = object : RecordQueryBackend { + override fun single( + plan: BackendSingleQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = Mono.empty() + + override fun stream( + plan: BackendStreamQueryPlan, + options: QueryBackendExecutionOptions, + ): Flux = Flux.empty() + + override fun count(plan: BackendCountQueryPlan, options: QueryBackendExecutionOptions): Mono = + Mono.just(0) + } + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/backend/QueryBackendCompositionTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/backend/QueryBackendCompositionTest.kt new file mode 100644 index 00000000000..cc7e4f94820 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/backend/QueryBackendCompositionTest.kt @@ -0,0 +1,189 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.backend + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryTarget +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono + +@OptIn( + ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) +class QueryBackendCompositionTest { + @Test + fun `composition should defensively copy every registration boundary`() { + val capabilities = linkedSetOf(FieldCapability.EXACT) + val fieldCapabilities = linkedMapOf>(identity to capabilities) + val operations = linkedSetOf(QueryOperation.SINGLE) + val tiers = linkedSetOf(SemanticTier.PORTABLE) + val contribution = RecordQueryBackendContribution( + schema, + backendId, + operations, + BackendStreamSupport.NONE, + tiers, + fieldCapabilities, + backend = NO_OP_BACKEND, + ) + val contributions = mutableListOf(contribution) + val routes = linkedMapOf(target to backendId) + val composition = QueryBackendComposition(contributions, routes) + + capabilities += FieldCapability.SORTABLE + fieldCapabilities.clear() + operations += QueryOperation.COUNT + tiers.clear() + contributions.clear() + routes.clear() + + composition.contributions.assert().hasSize(1) + composition.defaultRoutes.assert().containsEntry(target, backendId) + contribution.supportedOperations.assert().containsExactly(QueryOperation.SINGLE) + contribution.semanticTiers.assert().containsExactly(SemanticTier.PORTABLE) + contribution.fieldCapabilities[identity]!!.assert().containsExactly(FieldCapability.EXACT) + + @Suppress("UNCHECKED_CAST") + assertThrownBy { + (composition.contributions as MutableList).clear() + } + @Suppress("UNCHECKED_CAST") + assertThrownBy { + (composition.defaultRoutes as MutableMap).clear() + } + @Suppress("UNCHECKED_CAST") + assertThrownBy { + (contribution.fieldCapabilities[identity] as MutableSet).clear() + } + } + + @Test + fun `composition should reject duplicate and dangling routes`() { + val contribution = contribution() + + assertThrownBy { + QueryBackendComposition(listOf(contribution, contribution), mapOf(target to backendId)) + } + assertThrownBy { + QueryBackendComposition(listOf(contribution), mapOf(target to BackendId("missing"))) + } + assertThrownBy { + QueryBackendComposition( + listOf(contribution), + listOf(RecordQueryBackendNotReady(schema, backendId)), + mapOf(target to backendId), + ) + } + } + + @Test + fun `composition should retain a configured but not ready route without claiming a backend capability`() { + val notReady = mutableListOf(RecordQueryBackendNotReady(schema, backendId)) + val routes = linkedMapOf(target to backendId) + + val composition = QueryBackendComposition(emptyList(), notReady, routes) + notReady.clear() + routes.clear() + + composition.contributions.assert().isEmpty() + composition.notReadyBackends.assert().hasSize(1) + composition.notReadyBackends.single().schema.assert().isSameAs(schema) + composition.defaultRoutes.assert().containsEntry(target, backendId) + @Suppress("UNCHECKED_CAST") + assertThrownBy { + (composition.notReadyBackends as MutableList).clear() + } + } + + @Test + fun `contribution should reject capability overclaim and ambiguous stream support`() { + assertThrownBy { + contribution(fieldCapabilities = mapOf(identity to setOf(FieldCapability.SORTABLE))) + } + assertThrownBy { + contribution( + operations = setOf(QueryOperation.SINGLE), + streamSupport = BackendStreamSupport.BOUNDED_ONLY, + ) + } + assertThrownBy { + contribution( + operations = setOf(QueryOperation.SINGLE, QueryOperation.STREAM), + streamSupport = BackendStreamSupport.NONE, + ) + } + contribution(operations = setOf(QueryOperation.PAGE)).supportedOperations.assert() + .containsExactly(QueryOperation.PAGE) + } + + private fun contribution( + operations: Set = setOf(QueryOperation.SINGLE), + streamSupport: BackendStreamSupport = BackendStreamSupport.NONE, + fieldCapabilities: Map> = mapOf( + identity to setOf(FieldCapability.EXACT), + ), + ): RecordQueryBackendContribution = RecordQueryBackendContribution( + schema, + backendId, + operations, + streamSupport, + setOf(SemanticTier.PORTABLE), + fieldCapabilities, + backend = NO_OP_BACKEND, + ) + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + private val backendId = BackendId("test") + private val schema = QueryDocumentSchema( + target, + listOf( + QueryFieldSchema( + identity, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + listOf(PredicateOperator.EQ), + listOf(FieldCapability.EXACT), + ), + ), + emptyList(), + ) + + private companion object { + val NO_OP_BACKEND = object : RecordQueryBackend { + override fun single( + plan: BackendSingleQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = Mono.empty() + + override fun stream( + plan: BackendStreamQueryPlan, + options: QueryBackendExecutionOptions, + ): Flux = Flux.empty() + + override fun count(plan: BackendCountQueryPlan, options: QueryBackendExecutionOptions): Mono = + Mono.just(0) + } + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/backend/QueryDocumentSchemaTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/backend/QueryDocumentSchemaTest.kt new file mode 100644 index 00000000000..875417f22a2 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/backend/QueryDocumentSchemaTest.kt @@ -0,0 +1,253 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.backend + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.SearchScopeId +import me.ahoo.wow.query.internal.planning.PlanningFixtures +import me.ahoo.wow.query.internal.schema.QuerySchemaRegistry +import org.junit.jupiter.api.Test + +@OptIn(ExperimentalQueryBackendApi::class) +class QueryDocumentSchemaTest { + + @Test + fun `schema digest should be independent of registration order`() { + val original = PlanningFixtures.schema + val reordered = QueryDocumentSchema( + target = original.target, + fields = original.fields.values.reversed(), + searchScopes = original.searchScopes.values.reversed(), + ) + + reordered.contractId.assert().isEqualTo(original.contractId) + reordered.fields.assert().isEqualTo(original.fields) + } + + @Test + fun `schema should reject missing prefixes and duplicate ids`() { + val leaf = QueryFieldSchema( + id = QueryFieldId.Path(listOf("missing", "leaf")), + type = LogicalFieldType.Text, + presence = Presence.OPTIONAL, + nullability = Nullability.NULLABLE, + allowedOperators = setOf(PredicateOperator.EQ), + capabilities = setOf(FieldCapability.EXACT), + ) + + assertThrownBy { + QueryDocumentSchema(PlanningFixtures.target, listOf(leaf), emptyList()) + } + assertThrownBy { + QueryDocumentSchema( + PlanningFixtures.target, + listOf( + PlanningFixtures.schema.fields.getValue(PlanningFixtures.identity), + PlanningFixtures.schema.fields.getValue(PlanningFixtures.identity), + ), + emptyList(), + ) + } + } + + @Test + fun `search scope should resolve a legacy logical alias without physical expansion`() { + val definition = PlanningFixtures.schema.resolveLegacySearchScope( + owner = null, + alias = PlanningFixtures.description, + ) + + val resolved = checkNotNull(definition) + resolved.id.assert().isEqualTo(PlanningFixtures.searchScopeId) + resolved.fields.assert().containsExactly(PlanningFixtures.description) + PlanningFixtures.schema.resolveField(QueryFieldId.Path(listOf("aggregateId"))).assert() + .isEqualTo(PlanningFixtures.identity) + } + + @Test + fun `search scope should enforce its nearest element owner and unambiguous alias`() { + val itemName = PlanningFixtures.schema.fields.getValue(PlanningFixtures.itemName).let { field -> + QueryFieldSchema( + field.id, + field.type, + field.presence, + field.nullability, + field.allowedOperators, + field.capabilities + FieldCapability.FULL_TEXT, + field.logicalAliases, + ) + } + val nestedScope = QuerySearchScopeDefinition( + SearchScopeId("item-name"), + PlanningFixtures.items, + listOf(PlanningFixtures.itemName), + setOf(PlanningFixtures.itemName), + ) + QueryDocumentSchema( + PlanningFixtures.target, + PlanningFixtures.schema.fields.values.map { field -> + if (field.id == PlanningFixtures.itemName) itemName else field + }, + PlanningFixtures.schema.searchScopes.values + nestedScope, + ) + + val crossOwner = QuerySearchScopeDefinition( + SearchScopeId("invalid-nested"), + PlanningFixtures.items, + listOf(PlanningFixtures.description), + setOf(PlanningFixtures.itemName), + ) + assertThrownBy { + QueryDocumentSchema( + PlanningFixtures.target, + PlanningFixtures.schema.fields.values.map { field -> + if (field.id == PlanningFixtures.itemName) itemName else field + }, + listOf(crossOwner), + ) + } + + assertThrownBy { + QueryDocumentSchema( + PlanningFixtures.target, + PlanningFixtures.schema.fields.values, + listOf( + PlanningFixtures.schema.searchScopes.values.single(), + QuerySearchScopeDefinition( + SearchScopeId("duplicate-alias"), + null, + listOf(PlanningFixtures.description), + setOf(PlanningFixtures.description), + ), + ), + ) + } + } + + @Test + fun `search scope should reject duplicate fields`() { + assertThrownBy { + QuerySearchScopeDefinition( + SearchScopeId("duplicate-field"), + null, + listOf(PlanningFixtures.description, PlanningFixtures.description), + setOf(PlanningFixtures.description), + ) + } + } + + @Test + fun `array value validation should preserve element nullability`() { + val nullable = LogicalFieldType.Array( + LogicalFieldType.Text, + Nullability.NULLABLE, + EmptyArraySemantics.DISTINCT, + ) + val nonNull = nullable.copy(elementNullability = Nullability.NON_NULL) + val value = NormalizedValue.ListValue(listOf(NormalizedValue.Text("tag"), NormalizedValue.Null)) + + nullable.accepts(value).assert().isTrue() + nonNull.accepts(value).assert().isFalse() + LogicalFieldType.Array(nullable, Nullability.NON_NULL, EmptyArraySemantics.DISTINCT) + .accepts(NormalizedValue.ListValue(listOf(value))).assert().isTrue() + } + + @Test + fun `schema should reject object exact and ambiguous canonical path encodings`() { + assertThrownBy { + QueryFieldSchema( + PlanningFixtures.state, + LogicalFieldType.Object, + Presence.OPTIONAL, + Nullability.NULLABLE, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT), + ) + } + assertThrownBy { + QueryFieldSchema( + PlanningFixtures.tags, + LogicalFieldType.Array( + LogicalFieldType.Array( + LogicalFieldType.Object, + Nullability.NON_NULL, + EmptyArraySemantics.DISTINCT, + ), + Nullability.NON_NULL, + EmptyArraySemantics.DISTINCT, + ), + Presence.OPTIONAL, + Nullability.NULLABLE, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT), + ) + } + assertThrownBy { + QueryFieldId.Path(listOf("a\u0000b")) + } + + val identity = PlanningFixtures.schema.fields.getValue(PlanningFixtures.identity).let { field -> + QueryFieldSchema( + field.id, + field.type, + field.presence, + field.nullability, + field.allowedOperators, + field.capabilities, + setOf(QueryFieldId.Path(listOf("state", "items", "id"))), + ) + } + assertThrownBy { + QueryDocumentSchema( + PlanningFixtures.target, + PlanningFixtures.schema.fields.values.map { field -> + if (field.id == PlanningFixtures.identity) identity else field + }, + PlanningFixtures.schema.searchScopes.values, + ) + } + } + + @Test + fun `logical field collections should be defensively immutable`() { + val schema = PlanningFixtures.schema + + assertThrownBy { + @Suppress("UNCHECKED_CAST") + (schema.fields as MutableMap).clear() + } + assertThrownBy { + @Suppress("UNCHECKED_CAST") + (schema.fields.getValue(PlanningFixtures.name).capabilities as MutableSet).clear() + } + } + + @Test + fun `registry should resolve the complete query target and reject duplicates`() { + val snapshot = PlanningFixtures.schema + val eventTarget = QueryTarget(snapshot.target.namedAggregate, QueryDocumentKind.EVENT_STREAM) + val event = QueryDocumentSchema(eventTarget, snapshot.fields.values, snapshot.searchScopes.values) + val registry = QuerySchemaRegistry(listOf(event, snapshot)) + + registry[snapshot.target].assert().isEqualTo(snapshot) + registry[eventTarget].assert().isEqualTo(event) + assertThrownBy { + QuerySchemaRegistry(listOf(snapshot, snapshot)) + } + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/backend/RecordQueryBackendTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/backend/RecordQueryBackendTest.kt new file mode 100644 index 00000000000..8bbb5f3c91a --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/backend/RecordQueryBackendTest.kt @@ -0,0 +1,79 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.backend + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import org.junit.jupiter.api.Test +import java.time.Instant + +@OptIn(ExperimentalQueryBackendApi::class) +class RecordQueryBackendTest { + @Test + fun `page result should be an immutable exact envelope`() { + val record = BackendRecord( + "order-1", + NormalizedValue.ObjectValue(mapOf("value" to NormalizedValue.Text("one"))), + BackendRecordCompleteness.COMPLETE, + ) + val source = mutableListOf(record) + val page = BackendPage(source, 1, BackendTotalRelation.EXACT, BackendPageConsistency.SAME_INPUT) + source.clear() + + page.records.assert().containsExactly(record) + page.assert().isEqualTo( + BackendPage(listOf(record), 1, BackendTotalRelation.EXACT, BackendPageConsistency.SAME_INPUT), + ) + @Suppress("UNCHECKED_CAST") + assertThrownBy { + (page.records as MutableList).clear() + } + } + + @Test + fun `page window should reject invalid offset and size`() { + assertThrownBy { BackendPageWindow(-1, 1) } + assertThrownBy { BackendPageWindow(0, 0) } + } + + @Test + fun `execution options should retain every explicit budget without weakening the legacy constructor`() { + val deadline = Instant.parse("2024-01-01T00:00:00Z") + QueryBackendExecutionOptions( + deadline = deadline, + maxReturnedRecords = 10, + maxScannedRecords = 100, + maxPageWindow = 1_000, + maxCandidateBuckets = 20, + maxReturnedBuckets = 5, + maxCursorPages = 3, + allowDiskUse = true, + ).assert().isEqualTo( + QueryBackendExecutionOptions(deadline, 10, 100, 1_000, 20, 5, 3, true), + ) + QueryBackendExecutionOptions(deadline, 10).assert().isEqualTo( + QueryBackendExecutionOptions(deadline = deadline, maxReturnedRecords = 10), + ) + + assertThrownBy { + QueryBackendExecutionOptions(deadline = null, maxReturnedRecords = 0) + } + assertThrownBy { + QueryBackendExecutionOptions(deadline = null, maxReturnedRecords = null, maxScannedRecords = 0) + } + assertThrownBy { + QueryBackendExecutionOptions(deadline = null, maxReturnedRecords = null, maxPageWindow = 0) + } + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/cursor/QueryCursorLeaseStoreTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/cursor/QueryCursorLeaseStoreTest.kt new file mode 100644 index 00000000000..85956400e88 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/cursor/QueryCursorLeaseStoreTest.kt @@ -0,0 +1,77 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(ExperimentalQueryCursorApi::class) + +package me.ahoo.wow.query.cursor + +import io.mockk.mockk +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import org.junit.jupiter.api.Test +import java.time.Instant + +class QueryCursorLeaseStoreTest { + @Test + fun `entry should defensively copy opaque payload and preserve value semantics`() { + val payload = byteArrayOf(1, 2, 3) + val entry = QueryCursorLeaseEntry( + QueryCursorLeaseId("lease_1"), + Instant.parse("2026-08-09T00:05:00Z"), + QueryCursorPayloadFormat.WOW_QUERY_CURSOR_V1, + payload, + ) + payload[0] = 9 + val returned = entry.payload() + returned[1] = 9 + + entry.payload().toList().assert().containsExactly(1.toByte(), 2.toByte(), 3.toByte()) + entry.assert().isEqualTo( + QueryCursorLeaseEntry( + QueryCursorLeaseId("lease_1"), + Instant.parse("2026-08-09T00:05:00Z"), + QueryCursorPayloadFormat.WOW_QUERY_CURSOR_V1, + byteArrayOf(1, 2, 3), + ), + ) + entry.hashCode().assert().isEqualTo( + QueryCursorLeaseEntry( + QueryCursorLeaseId("lease_1"), + Instant.parse("2026-08-09T00:05:00Z"), + QueryCursorPayloadFormat.WOW_QUERY_CURSOR_V1, + byteArrayOf(1, 2, 3), + ).hashCode(), + ) + } + + @Test + fun `lease identifiers and payload should be bounded`() { + assertThrownBy { QueryCursorLeaseId("not+padded=") } + assertThrownBy { QueryCursorStoreRevision("\u0000") } + assertThrownBy { + QueryCursorLeaseEntry( + QueryCursorLeaseId("lease"), + Instant.parse("2026-08-09T00:05:00Z"), + QueryCursorPayloadFormat.WOW_QUERY_CURSOR_V1, + byteArrayOf(), + ) + } + assertThrownBy { + QueryCursorLeaseConfiguration( + mockk(), + QueryCursorSigningKeys(QueryCursorHmacKey(1, ByteArray(32) { 7 })), + maxBackendStateBytes = QueryCursorLeaseConfiguration.MAX_BACKEND_STATE_BYTES + 1, + ) + } + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/event/EventStreamQueryServiceFactoryTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/event/EventStreamQueryServiceFactoryTest.kt new file mode 100644 index 00000000000..03e7fb01f89 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/event/EventStreamQueryServiceFactoryTest.kt @@ -0,0 +1,73 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.event + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.modeling.NamedAggregateDecorator +import me.ahoo.wow.tck.mock.MOCK_AGGREGATE_METADATA +import org.junit.jupiter.api.Test +import java.util.concurrent.CountDownLatch +import java.util.concurrent.Executors +import java.util.concurrent.TimeUnit +import java.util.concurrent.atomic.AtomicInteger + +class EventStreamQueryServiceFactoryTest { + @Test + fun `should share service for equivalent aggregate decorators`() { + val factory = RecordingEventStreamQueryServiceFactory() + + val first = factory.create(TestNamedAggregateDecorator(MOCK_AGGREGATE_METADATA)) + val second = factory.create(TestNamedAggregateDecorator(MOCK_AGGREGATE_METADATA)) + + first.assert().isSameAs(second) + factory.createCount.get().assert().isEqualTo(1) + } + + @Test + fun `should create service once under concurrent access`() { + val factory = RecordingEventStreamQueryServiceFactory() + val executor = Executors.newFixedThreadPool(8) + val start = CountDownLatch(1) + try { + val futures = (1..32).map { + executor.submit { + start.await() + factory.create(TestNamedAggregateDecorator(MOCK_AGGREGATE_METADATA)) + } + } + + start.countDown() + val services = futures.map { it.get(10, TimeUnit.SECONDS) } + + services.all { it === services.first() }.assert().isTrue() + factory.createCount.get().assert().isEqualTo(1) + } finally { + executor.shutdownNow() + } + } +} + +private class RecordingEventStreamQueryServiceFactory : AbstractEventStreamQueryServiceFactory() { + val createCount = AtomicInteger() + + override fun createQueryService(namedAggregate: NamedAggregate): EventStreamQueryService { + createCount.incrementAndGet() + return NoOpEventStreamQueryService(namedAggregate) + } +} + +private class TestNamedAggregateDecorator( + override val namedAggregate: NamedAggregate, +) : NamedAggregateDecorator diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/event/filter/DefaultEventStreamQueryHandlerTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/event/filter/DefaultEventStreamQueryHandlerTest.kt index 6810cd12d42..1130d9b4344 100644 --- a/wow-query/src/test/kotlin/me/ahoo/wow/query/event/filter/DefaultEventStreamQueryHandlerTest.kt +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/event/filter/DefaultEventStreamQueryHandlerTest.kt @@ -13,17 +13,27 @@ package me.ahoo.wow.query.event.filter +import io.mockk.every +import io.mockk.mockk +import io.mockk.verify import me.ahoo.test.asserts.assert +import me.ahoo.wow.event.DomainEventStream +import me.ahoo.wow.filter.ErrorHandler import me.ahoo.wow.filter.FilterChainBuilder import me.ahoo.wow.filter.LogErrorHandler import me.ahoo.wow.query.dsl.condition import me.ahoo.wow.query.dsl.listQuery import me.ahoo.wow.query.dsl.singleQuery +import me.ahoo.wow.query.event.EventStreamQueryService +import me.ahoo.wow.query.event.EventStreamQueryServiceFactory import me.ahoo.wow.query.event.NoOpEventStreamQueryServiceFactory import me.ahoo.wow.query.filter.QueryContext import me.ahoo.wow.tck.mock.MOCK_AGGREGATE_METADATA import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono import reactor.kotlin.test.test +import reactor.test.publisher.PublisherProbe class DefaultEventStreamQueryHandlerTest { private val tailSnapshotQueryFilter = TailEventStreamQueryFilter(NoOpEventStreamQueryServiceFactory) @@ -102,4 +112,120 @@ class DefaultEventStreamQueryHandlerTest { } .verifyComplete() } + + @Test + fun `should handle asynchronous event stream query error`() { + val failure = IllegalStateException("query failed") + val query = listQuery { } + val queryService = mockk { + every { list(query) } returns Flux.error(failure) + } + val queryServiceFactory = mockk { + every { create(any()) } returns queryService + } + val errorHandler = mockk>> { + every { handle(any(), failure) } returns Mono.error(failure) + } + val chain = FilterChainBuilder>() + .addFilters(listOf(TailEventStreamQueryFilter(queryServiceFactory))) + .filterCondition(EventStreamQueryHandler::class) + .build() + val handler = DefaultEventStreamQueryHandler(chain, errorHandler) + + handler.list(MOCK_AGGREGATE_METADATA, query) + .test() + .expectErrorSatisfies { + it.assert().isSameAs(failure) + } + .verify() + + verify(exactly = 1) { errorHandler.handle(any(), failure) } + } + + @Test + fun `should observe synchronous event stream query error once`() { + val failure = IllegalStateException("query failed") + val query = listQuery { } + val queryService = mockk { + every { list(query) } throws failure + } + val queryServiceFactory = mockk { + every { create(any()) } returns queryService + } + val resumeErrorHandler = mockk>> { + every { handle(any(), failure) } returns Mono.empty() + } + val chain = FilterChainBuilder>() + .addFilters(listOf(TailEventStreamQueryFilter(queryServiceFactory))) + .filterCondition(EventStreamQueryHandler::class) + .build() + val handler = DefaultEventStreamQueryHandler(chain, resumeErrorHandler) + + handler.list(MOCK_AGGREGATE_METADATA, query) + .test() + .expectErrorSatisfies { + it.assert().isSameAs(failure) + } + .verify() + + verify(exactly = 1) { resumeErrorHandler.handle(any(), failure) } + } + + @Test + fun `should not recover partial event stream query error`() { + val failure = IllegalStateException("query failed") + val query = listQuery { } + val event = mockk() + val queryService = mockk { + every { list(query) } returns Flux.concat(Flux.just(event), Flux.error(failure)) + } + val queryServiceFactory = mockk { + every { create(any()) } returns queryService + } + val resumeErrorHandler = mockk>> { + every { handle(any(), failure) } returns Mono.empty() + } + val chain = FilterChainBuilder>() + .addFilters(listOf(TailEventStreamQueryFilter(queryServiceFactory))) + .filterCondition(EventStreamQueryHandler::class) + .build() + val handler = DefaultEventStreamQueryHandler(chain, resumeErrorHandler) + + handler.list(MOCK_AGGREGATE_METADATA, query) + .test() + .expectNext(event) + .expectErrorSatisfies { + it.assert().isSameAs(failure) + } + .verify() + + verify(exactly = 1) { resumeErrorHandler.handle(any(), failure) } + } + + @Test + fun `should propagate cancellation without handling it as an error`() { + val query = listQuery { } + val backendPublisher = PublisherProbe.of(Flux.never()) + val queryService = mockk { + every { list(query) } returns backendPublisher.flux() + } + val queryServiceFactory = mockk { + every { create(any()) } returns queryService + } + val errorHandler = mockk>>(relaxed = true) + val chain = FilterChainBuilder>() + .addFilters(listOf(TailEventStreamQueryFilter(queryServiceFactory))) + .filterCondition(EventStreamQueryHandler::class) + .build() + val handler = DefaultEventStreamQueryHandler(chain, errorHandler) + + handler.list(MOCK_AGGREGATE_METADATA, query) + .test() + .then { backendPublisher.assertWasSubscribed() } + .thenCancel() + .verify() + + backendPublisher.assertWasCancelled() + verify(exactly = 0) { errorHandler.handle(any(), any()) } + } } diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/filter/MaskingDynamicDocumentQueryFilterTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/filter/MaskingDynamicDocumentQueryFilterTest.kt index 1c916294500..147aa935a4f 100644 --- a/wow-query/src/test/kotlin/me/ahoo/wow/query/filter/MaskingDynamicDocumentQueryFilterTest.kt +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/filter/MaskingDynamicDocumentQueryFilterTest.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.query.filter import io.mockk.every @@ -21,9 +23,11 @@ import me.ahoo.wow.api.query.DynamicDocument import me.ahoo.wow.api.query.PagedList import me.ahoo.wow.api.query.SimpleDynamicDocument.Companion.toDynamicDocument import me.ahoo.wow.filter.FilterChain +import me.ahoo.wow.query.gateway.QueryExecutionException import me.ahoo.wow.query.mask.AggregateDataMasker import me.ahoo.wow.query.mask.AggregateDynamicDocumentMasker import me.ahoo.wow.query.mask.DataMaskerRegistry +import me.ahoo.wow.serialization.MessageRecords import me.ahoo.wow.tck.mock.MOCK_AGGREGATE_METADATA import org.junit.jupiter.api.Test import reactor.core.publisher.Flux @@ -107,6 +111,27 @@ class MaskingDynamicDocumentQueryFilterTest { result.block().assert().isEqualTo(maskedDoc) } + @Test + fun `should reject masking that injects an authoritative identity`() { + assertSystemFieldMaskingRejected( + original = mutableMapOf(MessageRecords.TENANT_ID to "tenant-1"), + masked = mutableMapOf( + MessageRecords.TENANT_ID to "tenant-1", + MessageRecords.AGGREGATE_ID to "forged-id", + ), + expectedPath = "$.result.aggregateId", + ) + } + + @Test + fun `should reject masking that changes tenant scope`() { + assertSystemFieldMaskingRejected( + original = mutableMapOf(MessageRecords.TENANT_ID to "tenant-1"), + masked = mutableMapOf(MessageRecords.TENANT_ID to "tenant-2"), + expectedPath = "$.result.tenantId", + ) + } + @Test fun `should mask dynamic list result`() { val maskedDoc = mutableMapOf("field" to "masked").toDynamicDocument() @@ -148,4 +173,36 @@ class MaskingDynamicDocumentQueryFilterTest { } filter.filter(context, chain).test().verifyComplete() } + + private fun assertSystemFieldMaskingRejected( + original: MutableMap, + masked: MutableMap, + expectedPath: String, + ) { + val mockAggregateMasker = mockk> { + every { isEmpty() } returns false + every { mask(any()) } returns masked.toDynamicDocument() + } + val filter = MockMaskingFilter(MockMaskerRegistry(mockAggregateMasker)) + val context = DefaultQueryContext( + queryType = QueryType.DYNAMIC_SINGLE, + namedAggregate = MOCK_AGGREGATE_METADATA, + ) + context.setResult(Mono.just(original.toDynamicDocument())) + val chain = mockk>> { + every { filter(context) } returns Mono.empty() + } + + filter.filter(context, chain).test().verifyComplete() + @Suppress("UNCHECKED_CAST") + val result = context.getRequiredResult() as Mono + result.test() + .expectErrorSatisfies { error -> + error.assert().isInstanceOf(QueryExecutionException::class.java) + (error as QueryExecutionException).path.assert().isEqualTo(expectedPath) + error.category.assert().isEqualTo(me.ahoo.wow.query.gateway.QueryErrorCategory.INTERNAL_FAILURE) + error.code.assert().isEqualTo("RESULT_MASKING_SYSTEM_FIELD_VIOLATION") + } + .verify() + } } diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/gateway/AnalyticsQueryGatewayRuntimeTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/gateway/AnalyticsQueryGatewayRuntimeTest.kt new file mode 100644 index 00000000000..fbad1e027e0 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/gateway/AnalyticsQueryGatewayRuntimeTest.kt @@ -0,0 +1,645 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.gateway + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.query.analytics.AnalyticsBucketWindow +import me.ahoo.wow.api.query.analytics.AnalyticsDimension +import me.ahoo.wow.api.query.analytics.AnalyticsGrouping +import me.ahoo.wow.api.query.analytics.AnalyticsMetric +import me.ahoo.wow.api.query.analytics.AnalyticsMetricKind +import me.ahoo.wow.api.query.analytics.AnalyticsNumericPolicy +import me.ahoo.wow.api.query.analytics.AnalyticsQuery +import me.ahoo.wow.api.query.analytics.AnalyticsValue +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.AnalyticsQueryBackend +import me.ahoo.wow.query.backend.AnalyticsQueryCursorLifecycle +import me.ahoo.wow.query.backend.BackendAnalyticsBucket +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsCursorState +import me.ahoo.wow.query.backend.BackendAnalyticsPage +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.BackendRecord +import me.ahoo.wow.query.backend.BackendSingleQueryPlan +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.BackendStreamSupport +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendComposition +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.RecordQueryBackend +import me.ahoo.wow.query.backend.RecordQueryBackendContribution +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.cursor.QueryCursorHmacKey +import me.ahoo.wow.query.cursor.QueryCursorLeaseConfiguration +import me.ahoo.wow.query.cursor.QueryCursorLeaseCreateResult +import me.ahoo.wow.query.cursor.QueryCursorLeaseEntry +import me.ahoo.wow.query.cursor.QueryCursorLeaseId +import me.ahoo.wow.query.cursor.QueryCursorLeaseStore +import me.ahoo.wow.query.cursor.QueryCursorSigningKeys +import me.ahoo.wow.query.cursor.QueryCursorStoreRevision +import me.ahoo.wow.query.cursor.StoredQueryCursorLease +import me.ahoo.wow.query.event.NoOpEventStreamQueryServiceFactory +import me.ahoo.wow.query.snapshot.NoOpSnapshotQueryServiceFactory +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.test.StepVerifier +import java.math.BigDecimal +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneId +import java.time.ZoneOffset +import java.util.concurrent.ConcurrentHashMap +import java.util.concurrent.atomic.AtomicInteger +import java.util.concurrent.atomic.AtomicLong +import java.util.function.Consumer +import me.ahoo.wow.api.query.analytics.AnalyticsConsistency as PublicAnalyticsConsistency + +@OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class, + ExperimentalQueryGatewayApi::class, +) +class AnalyticsQueryGatewayRuntimeTest { + @Test + fun `persistent cursor should resume exactly once across runtime nodes`() { + val store = InMemoryCursorStore() + val backend = PagingAnalyticsBackend() + val firstNode = runtime(store, backend, authority = AUTHORITY).analyticsGateway + val secondNode = runtime(store, backend, authority = AUTHORITY).analyticsGateway + + val first = firstNode.analyze(CALL, query()).block()!! + first.buckets.single().keys.assert().containsEntry("status", AnalyticsValue.of("A")) + first.nextCursor.assert().isNotNull() + store.size.assert().isEqualTo(1) + + val second = secondNode.analyze(CALL, query(first.nextCursor)).block()!! + second.buckets.single().keys.assert().containsEntry("status", AnalyticsValue.of("B")) + second.nextCursor.assert().isNull() + store.size.assert().isZero() + backend.calls.get().assert().isEqualTo(2) + + assertThrownBy { + firstNode.analyze(CALL, query(first.nextCursor)).block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.INVALID_CURSOR) + error.path.assert().isEqualTo("$.cursor") + error.code.assert().isEqualTo("INVALID_CURSOR_TOKEN") + } + ) + backend.calls.get().assert().isEqualTo(2) + } + + @Test + fun `wrong security or mapping binding must not consume a cursor`() { + val store = InMemoryCursorStore() + val backend = PagingAnalyticsBackend() + val first = runtime(store, backend, authority = AUTHORITY).analyticsGateway + .analyze(CALL, query()).block()!! + val token = checkNotNull(first.nextCursor) + + val wrongAuthority = QueryAuthority.System("other", "different-security-context") + assertInvalidBinding { + runtime(store, backend, authority = wrongAuthority).analyticsGateway + .analyze(CALL, query(token)).block() + } + store.size.assert().isEqualTo(1) + + assertInvalidBinding { + runtime(store, backend, authority = AUTHORITY, mappingDigest = "b".repeat(64)).analyticsGateway + .analyze(CALL, query(token)).block() + } + store.size.assert().isEqualTo(1) + + runtime(store, backend, authority = AUTHORITY).analyticsGateway + .analyze(CALL, query(token)).block()!!.nextCursor.assert().isNull() + store.size.assert().isZero() + } + + @Test + fun `cursor continuation must not remove or relax the initial execution budget`() { + val store = InMemoryCursorStore() + val backend = PagingAnalyticsBackend() + val gateway = runtime(store, backend, authority = AUTHORITY).analyticsGateway + val boundedCall = CALL.copy( + budget = QueryExecutionBudget( + maxScannedRecords = 100, + maxCandidateBuckets = 10, + maxCursorPages = 2, + ), + ) + val cursor = gateway.analyze(boundedCall, query()).block()!!.nextCursor + + listOf( + CALL, + boundedCall.copy( + budget = boundedCall.budget.copy(maxScannedRecords = 101), + ), + boundedCall.copy( + budget = boundedCall.budget.copy(allowDiskUse = true), + ), + ).forEach { relaxed -> + assertThrownBy { + gateway.analyze(relaxed, query(cursor)).block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.BUDGET_EXCEEDED) + error.path.assert().isEqualTo("$.cursor") + error.code.assert().isEqualTo("CURSOR_BUDGET_RELAXATION_NOT_ALLOWED") + }, + ) + store.size.assert().isEqualTo(1) + } + + gateway.analyze(boundedCall, query(cursor)).block()!!.nextCursor.assert().isNull() + store.size.assert().isZero() + } + + @Test + fun `numeric policy scale should be preserved in the public decimal representation`() { + val gateway = runtime(null, NumericAnalyticsBackend, authority = AUTHORITY).analyticsGateway + + val page = gateway.analyze(CALL, numericQuery()).block()!! + + page.buckets.single().metrics.getValue("total").value.assert().isEqualTo("120.50") + } + + @Test + fun `grouped analytics without persistent cursor store should reject before backend`() { + val backend = PagingAnalyticsBackend() + val runtime = runtime(null, backend, authority = AUTHORITY) + + assertThrownBy { + runtime.analyticsGateway.analyze(CALL, query()).block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.UNSUPPORTED_FEATURE) + error.path.assert().isEqualTo("$.cursor") + error.code.assert().isEqualTo("CURSOR_STORE_REQUIRED") + } + ) + backend.calls.get().assert().isZero() + } + + @Test + fun `snapshot analytics without a backend cursor lifecycle should reject before backend`() { + val backend = PagingAnalyticsBackend() + val runtime = runtime(InMemoryCursorStore(), backend, authority = AUTHORITY) + + assertThrownBy { + runtime.analyticsGateway.analyze(CALL, query(consistency = PublicAnalyticsConsistency.SNAPSHOT)).block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.BACKEND_UNAVAILABLE) + error.path.assert().isEqualTo("$.backend") + error.code.assert().isEqualTo("BACKEND_OPERATION_UNSUPPORTED") + }, + ) + backend.calls.get().assert().isZero() + } + + @Test + fun `snapshot cursor should transfer opaque backend state and close the terminal lease`() { + val store = InMemoryCursorStore() + val backend = SnapshotPagingAnalyticsBackend() + val runtime = runtime(store, backend, authority = AUTHORITY) + + val first = runtime.analyticsGateway.analyze(CALL, query(consistency = PublicAnalyticsConsistency.SNAPSHOT)) + .block()!! + first.nextCursor.assert().isNotNull() + backend.continuations.assert().containsExactly(null) + + val second = runtime.analyticsGateway.analyze( + CALL, + query(first.nextCursor, PublicAnalyticsConsistency.SNAPSHOT), + ).block()!! + second.nextCursor.assert().isNull() + backend.continuations.assert().containsExactly(null, "pit-1") + backend.closed.assert().containsExactly("pit-2") + store.size.assert().isZero() + } + + @Test + fun `expired snapshot cursor reaper should acquire once and close its backend state`() { + val store = InMemoryCursorStore() + val backend = SnapshotPagingAnalyticsBackend() + val clock = MutableClock(CLOCK.instant()) + val runtime = runtime(store, backend, authority = AUTHORITY, clock = clock) + + runtime.analyticsGateway.analyze(CALL, query(consistency = PublicAnalyticsConsistency.SNAPSHOT)).block()!! + store.size.assert().isEqualTo(1) + clock.advance(Duration.ofMinutes(3)) + + runtime.reapExpiredQueryCursors(10).block()!!.assert().isEqualTo(1) + runtime.reapExpiredQueryCursors(10).block()!!.assert().isZero() + backend.closed.assert().containsExactly("pit-1") + store.size.assert().isZero() + } + + @Test + fun `cursor store rejection should close newly returned snapshot state`() { + val backend = SnapshotPagingAnalyticsBackend() + val runtime = runtime(CapacityExceededCursorStore, backend, authority = AUTHORITY) + + assertThrownBy { + runtime.analyticsGateway.analyze(CALL, query(consistency = PublicAnalyticsConsistency.SNAPSHOT)).block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.BUDGET_EXCEEDED) + error.path.assert().isEqualTo("$.cursor") + error.code.assert().isEqualTo("CURSOR_CAPACITY_EXCEEDED") + }, + ) + backend.closed.assert().containsExactly("pit-1") + } + + @Test + fun `continuation error and cancellation should consume the lease and close its snapshot state`() { + val errorStore = InMemoryCursorStore() + val failing = SnapshotPagingAnalyticsBackend(failContinuation = true) + val errorRuntime = runtime(errorStore, failing, authority = AUTHORITY) + val errorCursor = errorRuntime.analyticsGateway + .analyze(CALL, query(consistency = PublicAnalyticsConsistency.SNAPSHOT)).block()!!.nextCursor + + assertThrownBy { + errorRuntime.analyticsGateway.analyze( + CALL, + query(errorCursor, PublicAnalyticsConsistency.SNAPSHOT), + ).block() + } + failing.closed.assert().containsExactly("pit-1") + errorStore.size.assert().isZero() + + val cancelStore = InMemoryCursorStore() + val never = SnapshotPagingAnalyticsBackend(neverContinuation = true) + val cancelRuntime = runtime(cancelStore, never, authority = AUTHORITY) + val cancelCursor = cancelRuntime.analyticsGateway + .analyze(CALL, query(consistency = PublicAnalyticsConsistency.SNAPSHOT)).block()!!.nextCursor + StepVerifier.create( + cancelRuntime.analyticsGateway.analyze( + CALL, + query(cancelCursor, PublicAnalyticsConsistency.SNAPSHOT), + ), + ).thenAwait(Duration.ofMillis(10)).thenCancel().verify() + never.closed.assert().containsExactly("pit-1") + cancelStore.size.assert().isZero() + } + + private fun assertInvalidBinding(action: () -> Unit) { + assertThrownBy(action).satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.INVALID_CURSOR) + error.path.assert().isEqualTo("$.cursor") + error.code.assert().isEqualTo("INVALID_CURSOR_BINDING") + } + ) + } + + private fun runtime( + store: QueryCursorLeaseStore?, + backend: AnalyticsQueryBackend, + authority: QueryAuthority, + mappingDigest: String = "a".repeat(64), + clock: Clock = CLOCK, + ): QueryGatewayRuntime { + val composition = composition(backend, mappingDigest) + val arguments = RuntimeArguments(composition, authority) + return if (store == null) { + QueryGatewayRuntime.create( + arguments.aggregates, + composition, + arguments.raw, + arguments.dialect, + arguments.authority, + executionProfiles = PROFILES, + clock = clock, + ) + } else { + QueryGatewayRuntime.create( + arguments.aggregates, + composition, + QueryCursorLeaseConfiguration( + store, + QueryCursorSigningKeys(QueryCursorHmacKey(1, ByteArray(32) { 7 })), + ), + arguments.raw, + arguments.dialect, + arguments.authority, + executionProfiles = PROFILES, + clock = clock, + ) + } + } + + private fun composition(analytics: AnalyticsQueryBackend, mappingDigest: String): QueryBackendComposition { + val fields = listOf( + QueryFieldSchema( + DELETED, + LogicalFieldType.Boolean, + Presence.REQUIRED, + Nullability.NON_NULL, + listOf(PredicateOperator.IS_TRUE, PredicateOperator.IS_FALSE), + listOf(FieldCapability.EXACT), + ), + QueryFieldSchema( + STATE, + LogicalFieldType.Object, + Presence.REQUIRED, + Nullability.NON_NULL, + emptyList(), + emptyList(), + ), + QueryFieldSchema( + STATUS, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + emptyList(), + listOf(FieldCapability.AGGREGATABLE), + ), + QueryFieldSchema( + AMOUNT, + LogicalFieldType.Decimal, + Presence.REQUIRED, + Nullability.NON_NULL, + emptyList(), + listOf(FieldCapability.AGGREGATABLE), + ), + ) + val schema = QueryDocumentSchema(CALL.target, fields, emptyList()) + val backendId = BackendId("probe") + return QueryBackendComposition( + listOf( + RecordQueryBackendContribution( + schema, + backendId, + setOf(QueryOperation.ANALYZE), + BackendStreamSupport.NONE, + setOf(SemanticTier.PORTABLE), + mapOf( + DELETED to setOf(FieldCapability.EXACT), + STATUS to setOf(FieldCapability.AGGREGATABLE), + AMOUNT to setOf(FieldCapability.AGGREGATABLE), + ), + backend = NO_OP_RECORD_BACKEND, + analyticsBackend = analytics, + mappingGenerationDigest = mappingDigest, + ), + ), + defaultRoutes = mapOf(CALL.target to backendId), + ) + } + + private fun query( + cursor: me.ahoo.wow.api.query.analytics.AnalyticsCursor? = null, + consistency: PublicAnalyticsConsistency = PublicAnalyticsConsistency.EVENTUAL, + ) = AnalyticsQuery( + grouping = AnalyticsGrouping.by(listOf(AnalyticsDimension("status", "state.status"))), + metrics = listOf(AnalyticsMetric("count", AnalyticsMetricKind.DOCUMENT_COUNT)), + window = AnalyticsBucketWindow(1, cursor), + consistency = consistency, + ) + + private fun numericQuery() = AnalyticsQuery( + grouping = AnalyticsGrouping.global(), + metrics = listOf(AnalyticsMetric("total", AnalyticsMetricKind.SUM, "state.amount")), + window = AnalyticsBucketWindow(1), + numericPolicy = AnalyticsNumericPolicy(scale = 2), + ) + + private data class RuntimeArguments( + val composition: QueryBackendComposition, + val publicAuthority: QueryAuthority, + ) { + val aggregates: List = listOf(NAMED_AGGREGATE) + val raw = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate) = + NoOpSnapshotQueryServiceFactory.create(namedAggregate) + + override fun eventStream(namedAggregate: NamedAggregate) = + NoOpEventStreamQueryServiceFactory.create(namedAggregate) + } + val dialect = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.DOCUMENT) + } + val authority = QueryAuthorityResolver { Mono.just(publicAuthority) } + } + + private class PagingAnalyticsBackend : AnalyticsQueryBackend { + val calls = AtomicInteger() + + override fun analyze( + plan: me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono { + calls.incrementAndGet() + val continued = plan.bucketWindow.afterKey != null + val status = if (continued) "B" else "A" + return Mono.just( + BackendAnalyticsPage( + listOf( + BackendAnalyticsBucket( + mapOf(AnalyticsAlias("status") to NormalizedValue.Text(status)), + mapOf(AnalyticsAlias("count") to NormalizedValue.Int64(1)), + ), + ), + if (continued) null else listOf(NormalizedValue.Text("A")), + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + ), + ) + } + } + + private object NumericAnalyticsBackend : AnalyticsQueryBackend { + override fun analyze( + plan: me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = Mono.just( + BackendAnalyticsPage( + listOf( + BackendAnalyticsBucket( + emptyMap(), + mapOf(AnalyticsAlias("total") to NormalizedValue.Decimal(BigDecimal("120.5"))), + ), + ), + null, + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + ), + ) + } + + private class SnapshotPagingAnalyticsBackend( + private val failContinuation: Boolean = false, + private val neverContinuation: Boolean = false, + ) : AnalyticsQueryBackend, AnalyticsQueryCursorLifecycle { + val continuations = mutableListOf() + val closed = mutableListOf() + + override fun analyze( + plan: me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = analyze(plan, options, null) + + override fun analyze( + plan: me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan, + options: QueryBackendExecutionOptions, + cursorState: BackendAnalyticsCursorState?, + ): Mono = Mono.fromSupplier { + val current = cursorState?.payload()?.decodeToString() + continuations += current + val continued = plan.bucketWindow.afterKey != null + if (continued && failContinuation) { + throw QueryBackendException(QueryBackendFailureKind.UNAVAILABLE) + } + BackendAnalyticsPage( + listOf( + BackendAnalyticsBucket( + mapOf(AnalyticsAlias("status") to NormalizedValue.Text(if (continued) "B" else "A")), + mapOf(AnalyticsAlias("count") to NormalizedValue.Int64(1)), + ), + ), + if (continued) null else listOf(NormalizedValue.Text("A")), + BackendAnalyticsConsistency.SNAPSHOT, + BackendAnalyticsCompleteness.EXACT, + BackendAnalyticsCursorState( + if (continued) "pit-2".encodeToByteArray() else "pit-1".encodeToByteArray(), + ), + ) + }.let { result -> if (neverContinuation && plan.bucketWindow.afterKey != null) Mono.never() else result } + + override fun close(cursorState: BackendAnalyticsCursorState): Mono = Mono.fromRunnable { + closed += cursorState.payload().decodeToString() + } + } + + private class MutableClock(initial: Instant) : Clock() { + private var current: Instant = initial + + fun advance(duration: Duration) { + current = current.plus(duration) + } + + override fun getZone(): ZoneId = ZoneOffset.UTC + + override fun withZone(zone: ZoneId): Clock = this + + override fun instant(): Instant = current + } + + private class InMemoryCursorStore : QueryCursorLeaseStore { + private val revisions = AtomicLong() + private val entries = ConcurrentHashMap() + val size: Int + get() = entries.size + + override fun create(entry: QueryCursorLeaseEntry): Mono = Mono.fromSupplier { + val stored = StoredQueryCursorLease(entry, QueryCursorStoreRevision(revisions.incrementAndGet().toString())) + if (entries.putIfAbsent(entry.id, stored) == null) { + QueryCursorLeaseCreateResult.CREATED + } else { + QueryCursorLeaseCreateResult.COLLISION + } + } + + override fun load(id: QueryCursorLeaseId): Mono = Mono.justOrEmpty(entries[id]) + + override fun compareAndDelete(expected: StoredQueryCursorLease): Mono = Mono.fromSupplier { + entries.remove(expected.entry.id, expected) + } + + override fun scanExpired( + before: Instant, + afterId: QueryCursorLeaseId?, + limit: Int, + ): Flux = Flux.fromIterable( + entries.values.filter { stored -> !stored.entry.expiresAt.isAfter(before) } + .sortedBy { stored -> stored.entry.id.value } + .filter { stored -> afterId == null || stored.entry.id.value > afterId.value } + .take(limit), + ) + } + + private object CapacityExceededCursorStore : QueryCursorLeaseStore { + override fun create(entry: QueryCursorLeaseEntry): Mono = + Mono.just(QueryCursorLeaseCreateResult.CAPACITY_EXCEEDED) + + override fun load(id: QueryCursorLeaseId): Mono = Mono.empty() + + override fun compareAndDelete(expected: StoredQueryCursorLease): Mono = Mono.just(false) + + override fun scanExpired( + before: Instant, + afterId: QueryCursorLeaseId?, + limit: Int, + ): Flux = Flux.empty() + } + + private companion object { + val NAMED_AGGREGATE = MaterializedNamedAggregate("sales", "order") + val CALL = QueryCall( + QueryTarget(NAMED_AGGREGATE, QueryDocumentKind.SNAPSHOT), + QueryPurpose("analytics-test"), + ) + val AUTHORITY = QueryAuthority.System("test", "analytics-runtime") + val CLOCK: Clock = Clock.fixed(Instant.parse("2026-08-09T00:00:00Z"), ZoneOffset.UTC) + val PROFILES = QueryExecutionProfiles( + operationProfiles = mapOf( + QueryOperationProfileKey(CALL.target, QueryOperation.ANALYZE) to + QueryExecutionProfile(QueryExecutionMode.PLANNED, QueryValidationMode.STRICT), + ), + ) + val DELETED = QueryFieldId.System(SystemFieldKind.DELETED) + val STATE = QueryFieldId.Path(listOf("state")) + val STATUS = QueryFieldId.Path(listOf("state", "status")) + val AMOUNT = QueryFieldId.Path(listOf("state", "amount")) + val NO_OP_RECORD_BACKEND = object : RecordQueryBackend { + override fun single( + plan: BackendSingleQueryPlan, + options: QueryBackendExecutionOptions + ): Mono = + Mono.empty() + + override fun stream(plan: BackendStreamQueryPlan, options: QueryBackendExecutionOptions): Flux = + Flux.empty() + + override fun page( + plan: me.ahoo.wow.query.backend.BackendPageQueryPlan, + options: QueryBackendExecutionOptions, + ) = Mono.empty() + + override fun count(plan: BackendCountQueryPlan, options: QueryBackendExecutionOptions): Mono = + Mono.empty() + } + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/gateway/QueryGatewayRuntimeTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/gateway/QueryGatewayRuntimeTest.kt new file mode 100644 index 00000000000..f1857bad5dc --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/gateway/QueryGatewayRuntimeTest.kt @@ -0,0 +1,890 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.gateway + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DynamicDocument +import me.ahoo.wow.api.query.IListQuery +import me.ahoo.wow.api.query.IPagedQuery +import me.ahoo.wow.api.query.ISingleQuery +import me.ahoo.wow.api.query.ListQuery +import me.ahoo.wow.api.query.MaterializedSnapshot +import me.ahoo.wow.api.query.Operator +import me.ahoo.wow.api.query.PagedList +import me.ahoo.wow.api.query.PagedQuery +import me.ahoo.wow.api.query.Pagination +import me.ahoo.wow.api.query.Projection +import me.ahoo.wow.api.query.SimpleDynamicDocument +import me.ahoo.wow.api.query.SingleQuery +import me.ahoo.wow.api.query.Sort +import me.ahoo.wow.event.DomainEventStream +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.BackendRecord +import me.ahoo.wow.query.backend.BackendRecordCompleteness +import me.ahoo.wow.query.backend.BackendSingleQueryPlan +import me.ahoo.wow.query.backend.BackendStreamQueryPlan +import me.ahoo.wow.query.backend.BackendStreamSupport +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendComposition +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.RecordQueryBackend +import me.ahoo.wow.query.backend.RecordQueryBackendContribution +import me.ahoo.wow.query.backend.RecordQueryBackendNotReady +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.event.EventStreamQueryService +import me.ahoo.wow.query.event.NoOpEventStreamQueryServiceFactory +import me.ahoo.wow.query.internal.gateway.TrustedAuthorityChannel +import me.ahoo.wow.query.snapshot.SnapshotQueryService +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.Instant +import java.time.ZoneOffset +import java.util.LinkedHashMap +import java.util.concurrent.CopyOnWriteArrayList +import java.util.concurrent.atomic.AtomicInteger +import java.util.function.Consumer + +@OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + ExperimentalQueryGatewayApi::class, +) +class QueryGatewayRuntimeTest { + @Test + fun `gateway should remain cold and materialize an independent projected document`() { + val authorityCalls = AtomicInteger() + val raw = ProbeSnapshotQueryService(namedAggregate) + val sourceState = linkedMapOf( + "status" to "PAID", + "bytes" to byteArrayOf(1, 2), + "internal" to "must-not-leak", + ) + raw.singleResult = SimpleDynamicDocument( + linkedMapOf( + "aggregateId" to "order-1", + "state" to sourceState, + "backendOnly" to "must-not-leak", + ), + ) + val gateway = gateway(raw) { + authorityCalls.incrementAndGet() + Mono.just(QueryAuthority.System("test", "gateway-test")) + } + val publisher = gateway.single( + snapshotCall, + SingleQuery( + Condition.eq("state.status", "PAID"), + Projection(include = listOf("state.status", "state.bytes")), + ), + ) + + authorityCalls.get().assert().isZero() + raw.singleCalls.get().assert().isZero() + + val first = checkNotNull(publisher.block()) + authorityCalls.get().assert().isEqualTo(1) + raw.singleCalls.get().assert().isEqualTo(1) + raw.lastSingleQuery!!.projection.include.assert().containsExactly("state.status", "state.bytes", "aggregateId") + first.containsKey("aggregateId").assert().isFalse() + first.containsKey("backendOnly").assert().isFalse() + first.getNestedDocument("state").getValue("status").assert().isEqualTo("PAID") + first.getNestedDocument("state").containsKey("internal").assert().isFalse() + + sourceState["status"] = "SHIPPED" + (sourceState["bytes"] as ByteArray)[0] = 9 + first.getNestedDocument("state").getValue("status").assert().isEqualTo("PAID") + first.getNestedDocument("state").getValue("bytes").contentEquals(byteArrayOf(1, 2)).assert().isTrue() + + val second = checkNotNull(publisher.block()) + second.assert().isNotSameAs(first) + authorityCalls.get().assert().isEqualTo(2) + raw.singleCalls.get().assert().isEqualTo(2) + } + + @Test + fun `subject scope should become a mandatory condition before raw storage`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + raw.countResult = 3 + val gateway = gateway(raw) { + Mono.just( + QueryAuthority.Subject( + subjectId = "subject-1", + tenantId = "tenant-1", + ownerGrant = QueryOwnerGrant.Only("owner-1"), + spaceGrant = QuerySpaceGrant.AllowList(listOf("space-1", "space-2")), + ), + ) + } + + gateway.count(snapshotCall, Condition.eq("state.status", "PAID")).block().assert().isEqualTo(3) + + val condition = checkNotNull(raw.lastCountCondition) + condition.operator.assert().isEqualTo(Operator.AND) + condition.children.first().operator.assert().isEqualTo(Operator.DELETED) + condition.operators().assert().contains(Operator.TENANT_ID) + .contains(Operator.OWNER_ID) + .contains(Operator.IN) + condition.flatten().single { it.operator == Operator.IN }.field.assert().isEqualTo("spaceId") + raw.countCalls.get().assert().isEqualTo(1) + } + + @Test + fun `missing authority should map to a public typed error without touching storage`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + val gateway = gateway(raw) { Mono.empty() } + + assertThrownBy { + gateway.count(snapshotCall, Condition.ALL).block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.ACCESS_DENIED) + error.path.assert().isEqualTo("$.executionContext.authority") + error.code.assert().isEqualTo("AUTHORITY_REQUIRED") + }, + ) + raw.countCalls.get().assert().isZero() + } + + @Test + fun `lookalike Reactor context key must not forge trusted authority`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + val gateway = gateway(raw) { Mono.empty() } + + assertThrownBy { + gateway.count(snapshotCall, Condition.ALL) + .contextWrite { context -> + context.put( + "me.ahoo.wow.query.trusted.authority", + QueryAuthority.System("forged", "lookalike string key"), + ) + } + .block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.ACCESS_DENIED) + error.path.assert().isEqualTo("$.executionContext.authority") + error.code.assert().isEqualTo("AUTHORITY_REQUIRED") + }, + ) + raw.countCalls.get().assert().isZero() + } + + @Test + fun `foreign authority capability must not forge trusted authority`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + val gateway = gateway(raw) { Mono.empty() } + val foreignChannel = TrustedAuthorityChannel.create() + + assertThrownBy { + foreignChannel.bind( + gateway.count(snapshotCall, Condition.ALL), + QueryAuthority.System("forged", "foreign authority channel"), + ).block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.ACCESS_DENIED) + error.path.assert().isEqualTo("$.executionContext.authority") + error.code.assert().isEqualTo("AUTHORITY_REQUIRED") + }, + ) + raw.countCalls.get().assert().isZero() + } + + @Test + fun `trusted authority denial should preserve its stable public tuple through the gateway`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + val gateway = gateway(raw) { + Mono.error( + QueryExecutionException( + QueryErrorCategory.ACCESS_DENIED, + "$.executionContext.authority", + "AUTHORITY_REQUIRED", + ), + ) + } + + assertThrownBy { + gateway.count(snapshotCall, Condition.ALL).block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.ACCESS_DENIED) + error.path.assert().isEqualTo("$.executionContext.authority") + error.code.assert().isEqualTo("AUTHORITY_REQUIRED") + }, + ) + raw.countCalls.get().assert().isZero() + } + + @Test + fun `normalized match-none should short-circuit every legacy operation`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + val gateway = gateway(raw) { Mono.just(QueryAuthority.System("test", "match-none")) } + + gateway.count(snapshotCall, Condition.ids(emptyList())).block().assert().isZero() + raw.countCalls.get().assert().isZero() + } + + @Test + fun `planned runtime should execute the contributed backend without touching legacy storage`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + val planned = ProbeRecordQueryBackend() + val gateway = QueryGatewayRuntime.create( + namedAggregates = listOf(namedAggregate), + backendComposition = plannedComposition(planned), + rawServiceSource = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate): SnapshotQueryService<*> = raw + + override fun eventStream(namedAggregate: NamedAggregate) = + NoOpEventStreamQueryServiceFactory.create(namedAggregate) + }, + dialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.DOCUMENT) + }, + authorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.System("test", "planned-runtime")) + }, + executionProfiles = QueryExecutionProfiles( + operationProfiles = listOf(QueryOperation.SINGLE, QueryOperation.STREAM, QueryOperation.COUNT) + .associate { operation -> + QueryOperationProfileKey(snapshotCall.target, operation) to + QueryExecutionProfile(QueryExecutionMode.PLANNED, QueryValidationMode.STRICT) + }, + ), + clock = Clock.fixed(Instant.parse("2024-01-01T00:00:00Z"), ZoneOffset.UTC), + ).gateway + + gateway.count(snapshotCall.copy(budget = fullBudget), Condition.ALL).block().assert().isEqualTo(7) + val document = gateway.single(snapshotCall, SingleQuery(Condition.ALL)).block()!! + document.getValue("aggregateId").assert().isEqualTo("order-1") + document.getNestedDocument("state").getValue("status").assert().isEqualTo("PAID") + gateway.stream(snapshotCall, ListQuery(condition = Condition.ALL, limit = 1)).collectList().block().assert() + .hasSize(1) + gateway.count(eventCall, Condition.ALL).block().assert().isZero() + + planned.countCalls.get().assert().isEqualTo(1) + planned.lastCountOptions.assert().isEqualTo(fullBackendOptions) + planned.singleCalls.get().assert().isEqualTo(1) + planned.streamCalls.get().assert().isEqualTo(1) + raw.countCalls.get().assert().isZero() + raw.singleCalls.get().assert().isZero() + raw.streamCalls.get().assert().isZero() + + assertThrownBy { + gateway.page( + snapshotCall.copy(budget = QueryExecutionBudget(maxPageWindow = 1)), + PagedQuery(Condition.ALL, pagination = Pagination(2, 1)), + ).block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.BUDGET_EXCEEDED) + error.path.assert().isEqualTo("$.input.page") + error.code.assert().isEqualTo("PAGE_WINDOW_EXCEEDED") + }, + ) + raw.lastPagedQuery.assert().isNull() + } + + @Test + fun `shadow runtime should return legacy and compare the planned probe`() { + val raw = ProbeSnapshotQueryService(namedAggregate).apply { countResult = 7 } + val planned = ProbeRecordQueryBackend() + val observations = CopyOnWriteArrayList() + val gateway = QueryGatewayRuntime.create( + namedAggregates = listOf(namedAggregate), + backendComposition = plannedComposition(planned), + rawServiceSource = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate): SnapshotQueryService<*> = raw + + override fun eventStream(namedAggregate: NamedAggregate) = + NoOpEventStreamQueryServiceFactory.create(namedAggregate) + }, + dialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.DOCUMENT) + }, + authorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.System("test", "shadow-runtime")) + }, + executionProfiles = QueryExecutionProfiles( + operationProfiles = mapOf( + QueryOperationProfileKey(snapshotCall.target, QueryOperation.COUNT) to + QueryExecutionProfile(QueryExecutionMode.SHADOW, QueryValidationMode.STRICT), + ), + ), + shadowObserver = QueryShadowObserver(observations::add), + runtimeHealthObserver = QueryRuntimeHealthObserver { }, + clock = Clock.fixed(Instant.parse("2024-01-01T00:00:00Z"), ZoneOffset.UTC), + ).gateway + + gateway.count(snapshotCall, Condition.ALL).block().assert().isEqualTo(7) + observations.single().outcome.assert().isEqualTo(QueryShadowOutcome.MATCH) + + planned.countResult = 9 + gateway.count(snapshotCall, Condition.ALL).block().assert().isEqualTo(7) + observations.last().outcome.assert().isEqualTo(QueryShadowOutcome.VALUE_MISMATCH) + observations.assert().hasSize(2) + raw.countCalls.get().assert().isEqualTo(2) + planned.countCalls.get().assert().isEqualTo(2) + } + + @Test + fun `shadow runtime should keep legacy available and report a configured backend that is not ready`() { + val raw = ProbeSnapshotQueryService(namedAggregate).apply { countResult = 7 } + val observations = CopyOnWriteArrayList() + val gateway = QueryGatewayRuntime.create( + namedAggregates = listOf(namedAggregate), + backendComposition = notReadyComposition(), + rawServiceSource = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate): SnapshotQueryService<*> = raw + + override fun eventStream(namedAggregate: NamedAggregate) = + NoOpEventStreamQueryServiceFactory.create(namedAggregate) + }, + dialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.DOCUMENT) + }, + authorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.System("test", "shadow-not-ready")) + }, + executionProfiles = QueryExecutionProfiles( + operationProfiles = mapOf( + QueryOperationProfileKey(snapshotCall.target, QueryOperation.COUNT) to + QueryExecutionProfile(QueryExecutionMode.SHADOW, QueryValidationMode.STRICT), + ), + ), + shadowObserver = QueryShadowObserver(observations::add), + runtimeHealthObserver = QueryRuntimeHealthObserver { }, + clock = Clock.fixed(Instant.parse("2024-01-01T00:00:00Z"), ZoneOffset.UTC), + ).gateway + + gateway.count(snapshotCall, Condition.ALL).block().assert().isEqualTo(7) + + observations.single().outcome.assert().isEqualTo(QueryShadowOutcome.PROBE_ERROR) + observations.single().reasonCode.assert().isEqualTo("BACKEND_NOT_READY") + raw.countCalls.get().assert().isEqualTo(1) + } + + @Test + fun `compatible fallback should emit a stable descriptor only runtime health observation`() { + val raw = ProbeSnapshotQueryService(namedAggregate).apply { countResult = 3 } + val observations = CopyOnWriteArrayList() + val gateway = gateway( + raw, + runtimeHealthObserver = QueryRuntimeHealthObserver(observations::add), + ) { Mono.just(QueryAuthority.System("test", "fallback-observation")) } + + gateway.count(snapshotCall, Condition.eq("state.unregistered", "value")).block().assert().isEqualTo(3) + + observations.single().let { observation -> + observation.target.assert().isEqualTo(snapshotCall.target) + observation.operation.assert().isEqualTo(QueryOperation.COUNT) + observation.kind.assert().isEqualTo(QueryRuntimeHealthKind.FALLBACK) + observation.reasonCode.assert().isEqualTo("FIELD_NOT_FOUND") + } + } + + @Test + fun `planned runtime should reject a configured backend that is not ready at startup`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + + assertThrownBy { + QueryGatewayRuntime.create( + namedAggregates = listOf(namedAggregate), + backendComposition = notReadyComposition(), + rawServiceSource = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate): SnapshotQueryService<*> = raw + + override fun eventStream(namedAggregate: NamedAggregate) = + NoOpEventStreamQueryServiceFactory.create(namedAggregate) + }, + dialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.DOCUMENT) + }, + authorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.System("test", "planned-not-ready")) + }, + executionProfiles = QueryExecutionProfiles( + operationProfiles = mapOf( + QueryOperationProfileKey(snapshotCall.target, QueryOperation.COUNT) to + QueryExecutionProfile(QueryExecutionMode.PLANNED, QueryValidationMode.STRICT), + ), + ), + ) + } + raw.countCalls.get().assert().isZero() + } + + @Test + fun `target wide planned profile should reject a partial backend matrix at startup`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + assertThrownBy { + QueryGatewayRuntime.create( + namedAggregates = listOf(namedAggregate), + backendComposition = plannedComposition(ProbeRecordQueryBackend()), + rawServiceSource = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate): SnapshotQueryService<*> = raw + + override fun eventStream(namedAggregate: NamedAggregate) = + NoOpEventStreamQueryServiceFactory.create(namedAggregate) + }, + dialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.DOCUMENT) + }, + authorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.System("test", "invalid-planned-profile")) + }, + executionProfiles = QueryExecutionProfiles( + targetProfiles = mapOf( + snapshotCall.target to QueryExecutionProfile( + QueryExecutionMode.PLANNED, + QueryValidationMode.STRICT, + ), + ), + ), + ) + } + raw.countCalls.get().assert().isZero() + } + + @Test + fun `event stream queries should not receive snapshot deletion semantics`() { + val snapshot = ProbeSnapshotQueryService(namedAggregate) + val event = ProbeEventStreamQueryService(namedAggregate) + event.countResult = 2 + val gateway = gateway(snapshot, event) { Mono.just(QueryAuthority.System("test", "event-test")) } + + gateway.count(eventCall, Condition.ALL).block().assert().isEqualTo(2) + + event.lastCountCondition!!.operator.assert().isEqualTo(Operator.ALL) + event.countCalls.get().assert().isEqualTo(1) + } + + @Test + fun `strict page should lower the planner identity tie-breaker`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + val gateway = gateway( + raw = raw, + configuration = QueryGatewayConfiguration( + executionMode = QueryExecutionMode.LEGACY, + validationMode = QueryValidationMode.STRICT, + ), + ) { Mono.just(QueryAuthority.System("test", "strict-page")) } + + gateway.page( + snapshotCall, + PagedQuery(condition = Condition.ALL, pagination = Pagination(1, 10)), + ).block() + + raw.lastPagedQuery!!.sort.assert().hasSize(1) + raw.lastPagedQuery!!.sort.single().field.assert().isEqualTo("aggregateId") + } + + @Test + fun `strict page should resolve the snapshot identity alias without adding a duplicate sort`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + val gateway = gateway( + raw = raw, + configuration = QueryGatewayConfiguration( + executionMode = QueryExecutionMode.LEGACY, + validationMode = QueryValidationMode.STRICT, + ), + ) { Mono.just(QueryAuthority.System("test", "strict-identity-alias")) } + + gateway.page( + snapshotCall, + PagedQuery( + condition = Condition.ALL, + sort = listOf(Sort("aggregateId", Sort.Direction.DESC)), + pagination = Pagination(1, 10), + ), + ).block() + + raw.lastPagedQuery!!.sort.assert().hasSize(1) + raw.lastPagedQuery!!.sort.single().assert().isEqualTo(Sort("aggregateId", Sort.Direction.DESC)) + } + + @Test + fun `typed materializer should be target-bound and run inside the gateway`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + raw.singleResult = SimpleDynamicDocument( + linkedMapOf( + "aggregateId" to "order-1", + "state" to linkedMapOf("status" to "PAID"), + ), + ) + val materializer = QueryResultMaterializer(snapshotCall.target, TypedResult::class.java) { identity, document -> + TypedResult(identity, document.getNestedDocument("state").getValue("status")) + } + val gateway = gateway(raw, resultMaterializers = listOf(materializer)) { + Mono.just(QueryAuthority.System("test", "typed")) + } + + gateway.single(snapshotCall, SingleQuery(Condition.ALL), TypedResult::class.java).block().assert() + .isEqualTo(TypedResult("order-1", "PAID")) + } + + @Test + fun `typed projection should reject before legacy storage in compatible mode`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + val materializer = QueryResultMaterializer(snapshotCall.target, TypedResult::class.java) { identity, document -> + TypedResult(identity, document.getNestedDocument("state").getValue("status")) + } + val gateway = gateway(raw, resultMaterializers = listOf(materializer)) { + Mono.just(QueryAuthority.System("test", "typed-projection")) + } + + assertThrownBy { + gateway.single( + snapshotCall, + SingleQuery(Condition.ALL, Projection(include = listOf("state.status"))), + TypedResult::class.java, + ).block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.INVALID_QUERY) + error.path.assert().isEqualTo("$.input.query.projection") + error.code.assert().isEqualTo("TYPED_PROJECTION_NOT_ALLOWED") + }, + ) + raw.singleCalls.get().assert().isZero() + } + + @Test + fun `legacy include projection should preserve a nullable nested parent`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + raw.singleResult = SimpleDynamicDocument( + linkedMapOf( + "aggregateId" to "order-1", + "profile" to null, + ), + ) + val gateway = gateway(raw) { Mono.just(QueryAuthority.System("test", "nullable-projection")) } + + val result = gateway.single( + snapshotCall, + SingleQuery(Condition.ALL, Projection(include = listOf("profile.name"))), + ).block()!! + + result.containsKey("profile").assert().isTrue() + result["profile"].assert().isNull() + } + + @Test + fun `missing typed materializer and returned-record budget should fail before storage`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + val gateway = gateway(raw) { Mono.just(QueryAuthority.System("test", "fail-closed")) } + + assertThrownBy { + gateway.single(snapshotCall, SingleQuery(Condition.ALL), TypedResult::class.java).block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.MAPPING_FAILURE) + error.path.assert().isEqualTo("$.result") + error.code.assert().isEqualTo("RESULT_MAPPING_FAILED") + }, + ) + raw.singleCalls.get().assert().isZero() + + assertThrownBy { + gateway.stream( + snapshotCall.copy(budget = QueryExecutionBudget(maxReturnedRecords = 1)), + ListQuery(condition = Condition.ALL, limit = 2), + ).collectList().block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.BUDGET_EXCEEDED) + error.code.assert().isEqualTo("RESULT_LIMIT_EXCEEDED") + }, + ) + raw.streamCalls.get().assert().isZero() + } + + @Test + fun `raw result should be observed once through the bounded snapshot`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + val source = linkedMapOf( + "aggregateId" to "order-1", + "state" to linkedMapOf("status" to "PAID"), + ) + raw.singleResult = GetRejectingDynamicDocument(source) + val gateway = gateway(raw) { Mono.just(QueryAuthority.System("test", "single-observation")) } + + gateway.single(snapshotCall, SingleQuery(Condition.ALL)).block()!!.getValue("aggregateId").assert() + .isEqualTo("order-1") + } + + @Test + fun `oversized raw result should fail mapping without escaping the admission budget`() { + val raw = ProbeSnapshotQueryService(namedAggregate) + val source = LinkedHashMap() + source["aggregateId"] = "order-1" + repeat(1024) { index -> source["field-$index"] = index } + raw.singleResult = SimpleDynamicDocument(source) + val gateway = gateway(raw) { Mono.just(QueryAuthority.System("test", "bounded-result")) } + + assertThrownBy { + gateway.single(snapshotCall, SingleQuery(Condition.ALL)).block() + }.satisfies( + Consumer { error -> + error.category.assert().isEqualTo(QueryErrorCategory.MAPPING_FAILURE) + error.path.assert().isEqualTo("$.result") + error.code.assert().isEqualTo("RESULT_MAPPING_FAILED") + }, + ) + } + + private fun gateway( + raw: ProbeSnapshotQueryService, + eventRaw: EventStreamQueryService = NoOpEventStreamQueryServiceFactory.create(namedAggregate), + configuration: QueryGatewayConfiguration = QueryGatewayConfiguration(), + resultMaterializers: Iterable> = emptyList(), + runtimeHealthObserver: QueryRuntimeHealthObserver = QueryRuntimeHealthObserver.NONE, + authority: QueryAuthorityResolver, + ): QueryGateway = QueryGatewayRuntime.create( + namedAggregates = listOf(namedAggregate), + backendComposition = QueryBackendComposition.EMPTY, + rawServiceSource = object : QueryRawServiceSource { + override fun snapshot(namedAggregate: NamedAggregate): SnapshotQueryService<*> = raw + + override fun eventStream(namedAggregate: NamedAggregate) = eventRaw + }, + dialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.DOCUMENT) + }, + authorityResolver = authority, + resultMaterializers = resultMaterializers, + runtimeHealthObserver = runtimeHealthObserver, + configuration = configuration, + clock = Clock.fixed(Instant.parse("2024-01-01T00:00:00Z"), ZoneOffset.UTC), + ).gateway + + private fun plannedComposition(backend: RecordQueryBackend): QueryBackendComposition { + val identity = QueryFieldId.System(me.ahoo.wow.query.backend.SystemFieldKind.IDENTITY) + val deleted = QueryFieldId.System(me.ahoo.wow.query.backend.SystemFieldKind.DELETED) + val schema = QueryDocumentSchema( + target = snapshotCall.target, + fields = listOf( + QueryFieldSchema( + id = identity, + type = LogicalFieldType.Text, + presence = Presence.REQUIRED, + nullability = Nullability.NON_NULL, + allowedOperators = listOf(PredicateOperator.EQ, PredicateOperator.IN), + capabilities = listOf(FieldCapability.EXACT, FieldCapability.SORTABLE), + logicalAliases = listOf(QueryFieldId.Path(listOf("aggregateId"))), + ), + QueryFieldSchema( + id = deleted, + type = LogicalFieldType.Boolean, + presence = Presence.REQUIRED, + nullability = Nullability.NON_NULL, + allowedOperators = listOf(PredicateOperator.IS_TRUE, PredicateOperator.IS_FALSE), + capabilities = listOf(FieldCapability.EXACT), + ), + ), + searchScopes = emptyList(), + ) + val backendId = BackendId("probe") + return QueryBackendComposition( + contributions = listOf( + RecordQueryBackendContribution( + schema = schema, + backendId = backendId, + supportedOperations = setOf(QueryOperation.SINGLE, QueryOperation.STREAM, QueryOperation.COUNT), + streamSupport = BackendStreamSupport.BOUNDED_ONLY, + semanticTiers = setOf(SemanticTier.PORTABLE), + fieldCapabilities = mapOf( + identity to setOf(FieldCapability.EXACT, FieldCapability.SORTABLE), + deleted to setOf(FieldCapability.EXACT), + ), + backend = backend, + ), + ), + defaultRoutes = mapOf(snapshotCall.target to backendId), + ) + } + + private fun notReadyComposition(): QueryBackendComposition { + val ready = plannedComposition(ProbeRecordQueryBackend()) + val contribution = ready.contributions.single() + return QueryBackendComposition( + contributions = emptyList(), + notReadyBackends = listOf(RecordQueryBackendNotReady(contribution.schema, contribution.backendId)), + defaultRoutes = ready.defaultRoutes, + ) + } + + private val namedAggregate = MaterializedNamedAggregate("sales", "order") + private val snapshotCall = QueryCall( + QueryTarget(namedAggregate, QueryDocumentKind.SNAPSHOT), + QueryPurpose("query-test"), + ) + private val eventCall = QueryCall( + QueryTarget(namedAggregate, QueryDocumentKind.EVENT_STREAM), + QueryPurpose("query-test"), + ) + private val fullBudget = QueryExecutionBudget( + maxReturnedRecords = 10, + maxScannedRecords = 100, + maxPageWindow = 1_000, + maxCandidateBuckets = 20, + maxReturnedBuckets = 5, + maxCursorPages = 3, + allowDiskUse = true, + ) + private val fullBackendOptions = QueryBackendExecutionOptions(null, 10, 100, 1_000, 20, 5, 3, true) + + private data class TypedResult(val identity: String, val status: String) + + private class GetRejectingDynamicDocument( + private val delegate: MutableMap, + ) : DynamicDocument, + MutableMap by delegate { + override fun get(key: String): Any? = error("Dynamic result must be traversed once instead of read by key.") + + override fun getNestedDocument(key: String): DynamicDocument = error("Not used by the raw snapshot boundary.") + } + + private fun Condition.operators(): List = + listOf(operator) + children.flatMap { child -> child.operators() } + + private fun Condition.flatten(): List = + listOf(this) + children.flatMap { child -> child.flatten() } + + private class ProbeSnapshotQueryService( + override val namedAggregate: NamedAggregate, + ) : SnapshotQueryService { + override val name: String = "probe" + val singleCalls = AtomicInteger() + val streamCalls = AtomicInteger() + val countCalls = AtomicInteger() + var singleResult: DynamicDocument? = null + var countResult: Long = 0 + var lastSingleQuery: ISingleQuery? = null + var lastCountCondition: Condition? = null + var lastPagedQuery: IPagedQuery? = null + + override fun single(singleQuery: ISingleQuery): Mono> = Mono.empty() + + override fun dynamicSingle(singleQuery: ISingleQuery): Mono { + singleCalls.incrementAndGet() + lastSingleQuery = singleQuery + return Mono.justOrEmpty(singleResult) + } + + override fun list(listQuery: IListQuery): Flux> = Flux.empty() + + override fun dynamicList(listQuery: IListQuery): Flux { + streamCalls.incrementAndGet() + return Flux.empty() + } + + override fun paged(pagedQuery: IPagedQuery): Mono>> = + Mono.just(PagedList.empty()) + + override fun dynamicPaged(pagedQuery: IPagedQuery): Mono> { + lastPagedQuery = pagedQuery + return Mono.just(PagedList.empty()) + } + + override fun count(condition: Condition): Mono { + countCalls.incrementAndGet() + lastCountCondition = condition + return Mono.just(countResult) + } + } + + private class ProbeEventStreamQueryService( + override val namedAggregate: NamedAggregate, + ) : EventStreamQueryService { + val countCalls = AtomicInteger() + var countResult: Long = 0 + var lastCountCondition: Condition? = null + + override fun single(singleQuery: ISingleQuery): Mono = Mono.empty() + + override fun dynamicSingle(singleQuery: ISingleQuery): Mono = Mono.empty() + + override fun list(listQuery: IListQuery): Flux = Flux.empty() + + override fun dynamicList(listQuery: IListQuery): Flux = Flux.empty() + + override fun paged(pagedQuery: IPagedQuery): Mono> = Mono.just(PagedList.empty()) + + override fun dynamicPaged(pagedQuery: IPagedQuery): Mono> = + Mono.just(PagedList.empty()) + + override fun count(condition: Condition): Mono { + countCalls.incrementAndGet() + lastCountCondition = condition + return Mono.just(countResult) + } + } + + private class ProbeRecordQueryBackend : RecordQueryBackend { + val singleCalls = AtomicInteger() + val streamCalls = AtomicInteger() + val countCalls = AtomicInteger() + var countResult: Long = 7 + var lastCountOptions: QueryBackendExecutionOptions? = null + + override fun single( + plan: BackendSingleQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono { + singleCalls.incrementAndGet() + return Mono.just(record()) + } + + override fun stream( + plan: BackendStreamQueryPlan, + options: QueryBackendExecutionOptions, + ): Flux { + streamCalls.incrementAndGet() + return Flux.just(record()) + } + + override fun count(plan: BackendCountQueryPlan, options: QueryBackendExecutionOptions): Mono { + countCalls.incrementAndGet() + lastCountOptions = options + return Mono.just(countResult) + } + + private fun record(): BackendRecord = BackendRecord( + identity = "order-1", + document = NormalizedValue.ObjectValue( + linkedMapOf( + "aggregateId" to NormalizedValue.Text("order-1"), + "state" to NormalizedValue.ObjectValue( + linkedMapOf("status" to NormalizedValue.Text("PAID")), + ), + ), + ), + completeness = BackendRecordCompleteness.COMPLETE, + ) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/gateway/QueryLegacyContextResolverTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/gateway/QueryLegacyContextResolverTest.kt new file mode 100644 index 00000000000..0da995bb2aa --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/gateway/QueryLegacyContextResolverTest.kt @@ -0,0 +1,100 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.gateway + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.filter.QueryType +import org.junit.jupiter.api.Test +import reactor.core.publisher.Mono +import reactor.kotlin.test.test + +class QueryLegacyContextResolverTest { + private val target = QueryTarget( + MaterializedNamedAggregate("example", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val grant = QueryLegacyGrant( + callerId = "compensation-retry", + target = target, + purpose = QueryPurpose("compensation-retry"), + executionMode = QueryExecutionMode.LEGACY, + resourceScope = QueryResourceScope(tenantId = "tenant-1"), + ) + private val resolver = QueryLegacyContextResolver(listOf(grant)) + + @Test + fun `should resolve an exact pre-registered legacy grant`() { + Mono.defer { resolver.resolve(request()) } + .withLegacyQueryCaller("compensation-retry") + .test() + .consumeNextWith { context -> + context.call.target.assert().isEqualTo(target) + context.call.purpose.assert().isEqualTo(grant.purpose) + context.call.resourceScope.assert().isEqualTo(grant.resourceScope) + context.authority.assert().isEqualTo(QueryAuthority.Legacy(grant)) + } + .verifyComplete() + } + + @Test + fun `should reject target and execution mode mismatch`() { + val anotherTarget = QueryTarget( + MaterializedNamedAggregate("example", "cart"), + QueryDocumentKind.SNAPSHOT, + ) + resolver.resolve(request(anotherTarget)) + .withLegacyQueryCaller("compensation-retry") + .test() + .expectErrorSatisfies(::assertLegacyRejected) + .verify() + + resolver.resolve(request(executionMode = QueryExecutionMode.SHADOW)) + .withLegacyQueryCaller("compensation-retry") + .test() + .expectErrorSatisfies(::assertLegacyRejected) + .verify() + } + + @Test + fun `should not grant legacy authority without its trusted context marker`() { + resolver.resolve(request()) + .test() + .verifyComplete() + } + + @Test + fun `should reject ambiguous grants`() { + runCatching { QueryLegacyContextResolver(listOf(grant, grant.copy(purpose = QueryPurpose("other")))) } + .exceptionOrNull().assert().isInstanceOf(IllegalArgumentException::class.java) + } + + private fun assertLegacyRejected(error: Throwable) { + error.assert().isInstanceOf(QueryExecutionException::class.java) + (error as QueryExecutionException).category.assert().isEqualTo(QueryErrorCategory.ACCESS_DENIED) + error.path.assert().isEqualTo("$.executionContext.legacyGrant") + error.code.assert().isEqualTo("LEGACY_CALLER_NOT_ALLOWED") + } + + private fun request( + target: QueryTarget = this.target, + executionMode: QueryExecutionMode = QueryExecutionMode.LEGACY, + ): QueryTrustedContextRequest = QueryTrustedContextRequest( + QueryCallResolutionRequest(target, QueryType.COUNT), + executionMode, + QueryValidationMode.COMPATIBLE, + ) +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/gateway/QueryServiceFacadeTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/gateway/QueryServiceFacadeTest.kt new file mode 100644 index 00000000000..a635c8e028d --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/gateway/QueryServiceFacadeTest.kt @@ -0,0 +1,153 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.query.gateway + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DynamicDocument +import me.ahoo.wow.api.query.IListQuery +import me.ahoo.wow.api.query.IPagedQuery +import me.ahoo.wow.api.query.ISingleQuery +import me.ahoo.wow.api.query.ListQuery +import me.ahoo.wow.api.query.PagedList +import me.ahoo.wow.api.query.PagedQuery +import me.ahoo.wow.api.query.SingleQuery +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.filter.QueryType +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.test.StepVerifier + +class QueryServiceFacadeTest { + private val requests = mutableListOf() + private val gateway = EmptyQueryGateway() + private val resolver = QueryCallResolver { request -> + requests += request + Mono.just(QueryCall(request.target, PURPOSE)) + } + + @Test + fun `all compatibility methods should remain cold and resolve exact calls per subscription`() { + val snapshot = GatewaySnapshotQueryServiceFactory(gateway, resolver).create(AGGREGATE) + val eventStream = GatewayEventStreamQueryServiceFactory(gateway, resolver).create(AGGREGATE) + val snapshotPublishers = listOf( + snapshot.single(SINGLE_QUERY), + snapshot.dynamicSingle(SINGLE_QUERY), + snapshot.list(LIST_QUERY), + snapshot.dynamicList(LIST_QUERY), + snapshot.paged(PAGED_QUERY), + snapshot.dynamicPaged(PAGED_QUERY), + snapshot.count(CONDITION), + ) + val eventPublishers = listOf( + eventStream.single(SINGLE_QUERY), + eventStream.dynamicSingle(SINGLE_QUERY), + eventStream.list(LIST_QUERY), + eventStream.dynamicList(LIST_QUERY), + eventStream.paged(PAGED_QUERY), + eventStream.dynamicPaged(PAGED_QUERY), + eventStream.count(CONDITION), + ) + + requests.assert().isEmpty() + snapshotPublishers.forEach(::verifyPublisher) + eventPublishers.forEach(::verifyPublisher) + + requests.map(QueryCallResolutionRequest::queryType).assert().containsExactly( + QueryType.SINGLE, + QueryType.DYNAMIC_SINGLE, + QueryType.LIST, + QueryType.DYNAMIC_LIST, + QueryType.PAGED, + QueryType.DYNAMIC_PAGED, + QueryType.COUNT, + QueryType.SINGLE, + QueryType.DYNAMIC_SINGLE, + QueryType.LIST, + QueryType.DYNAMIC_LIST, + QueryType.PAGED, + QueryType.DYNAMIC_PAGED, + QueryType.COUNT, + ) + requests.take(7).map { request -> request.target.documentKind }.assert() + .allMatch { documentKind -> documentKind == QueryDocumentKind.SNAPSHOT } + requests.drop(7).map { request -> request.target.documentKind }.assert() + .allMatch { documentKind -> documentKind == QueryDocumentKind.EVENT_STREAM } + requests.map { request -> request.target.namedAggregate }.assert() + .allMatch { namedAggregate -> namedAggregate == AGGREGATE } + } + + @Test + fun `mismatched call target should fail before gateway execution`() { + val mismatchedResolver = QueryCallResolver { + Mono.just( + QueryCall( + QueryTarget(MaterializedNamedAggregate("test", "other"), QueryDocumentKind.SNAPSHOT), + PURPOSE, + ), + ) + } + val service = GatewaySnapshotQueryServiceFactory(gateway, mismatchedResolver).create(AGGREGATE) + + StepVerifier.create(service.count(CONDITION)) + .expectErrorSatisfies { error -> + error.assert().isInstanceOf(QueryExecutionException::class.java) + (error as QueryExecutionException).code.assert().isEqualTo("QUERY_CALL_TARGET_MISMATCH") + error.path.assert().isEqualTo("$.executionContext.call") + } + .verify() + gateway.countCalls.assert().isZero() + } + + private fun verifyPublisher(publisher: org.reactivestreams.Publisher<*>) { + StepVerifier.create(publisher).thenConsumeWhile { true }.verifyComplete() + } + + private class EmptyQueryGateway : QueryGateway { + var countCalls: Int = 0 + + override fun single(call: QueryCall, query: ISingleQuery): Mono = Mono.empty() + + override fun single(call: QueryCall, query: ISingleQuery, resultType: Class): Mono = Mono.empty() + + override fun stream(call: QueryCall, query: IListQuery): Flux = Flux.empty() + + override fun stream(call: QueryCall, query: IListQuery, resultType: Class): Flux = Flux.empty() + + override fun page(call: QueryCall, query: IPagedQuery): Mono> = Mono.empty() + + override fun page( + call: QueryCall, + query: IPagedQuery, + resultType: Class, + ): Mono> = Mono.empty() + + override fun count(call: QueryCall, condition: Condition): Mono { + countCalls++ + return Mono.empty() + } + } + + private companion object { + val AGGREGATE = MaterializedNamedAggregate("test", "order") + val CONDITION = Condition.all() + val SINGLE_QUERY = SingleQuery(CONDITION) + val LIST_QUERY = ListQuery(CONDITION) + val PAGED_QUERY = PagedQuery(CONDITION) + val PURPOSE = QueryPurpose("facade-test") + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/admission/RawAdmissionGuardBoundaryTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/admission/RawAdmissionGuardBoundaryTest.kt new file mode 100644 index 00000000000..2ce7ce207a0 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/admission/RawAdmissionGuardBoundaryTest.kt @@ -0,0 +1,544 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.admission + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.Operator +import me.ahoo.wow.api.query.Projection +import me.ahoo.wow.api.query.SingleQuery +import me.ahoo.wow.api.query.Sort +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryInput +import me.ahoo.wow.query.internal.model.QueryInvocation +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import java.math.BigDecimal +import java.math.BigInteger +import java.time.format.DateTimeFormatterBuilder +import java.util.AbstractMap.SimpleImmutableEntry +import java.util.UUID +import java.util.function.Consumer + +class RawAdmissionGuardBoundaryTest { + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val baseLimits = QueryAdmissionLimits( + maxConditionDepth = 8, + maxConditionNodes = 32, + maxChildrenPerNode = 8, + maxFieldLength = 24, + maxStringLength = 16, + maxCollectionSize = 4, + maxObjectFields = 4, + maxValueDepth = 4, + maxValueNodes = 32, + maxByteArrayLength = 4, + maxValuePayloadBytes = 128, + maxProjectionFields = 4, + maxSortFields = 4, + maxOptions = 4, + ) + + @Test + fun `should enforce cumulative value node budget across sibling conditions`() { + val guard = RawAdmissionGuard(baseLimits.copy(maxValueNodes = 4)) + guard.admit( + countInvocation( + Condition.and( + Condition.eq("first", listOf(1, 2)), + Condition.eq("second", 3), + ), + ), + ) + + assertRejected( + guard, + Condition.and( + Condition.eq("first", listOf(1, 2)), + Condition.eq("second", 3), + Condition.eq("third", 4), + ), + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.VALUE_NODE_LIMIT_EXCEEDED, + "$.input.condition.children[2].value", + ) + } + + @Test + fun `should enforce cumulative UTF-8 payload budget without rejecting the exact boundary`() { + val guard = RawAdmissionGuard(baseLimits.copy(maxValuePayloadBytes = 19)) + guard.admit( + countInvocation( + Condition.and( + Condition.eq("first", "éé"), + Condition.eq("second", "éé"), + ), + ), + ) + + assertRejected( + guard, + Condition.and( + Condition.eq("first", "éé"), + Condition.eq("second", "ééa"), + ), + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.PAYLOAD_LIMIT_EXCEEDED, + "$.input.condition.children[1].value", + ) + } + + @Test + fun `should include condition projection and sort fields in the cumulative payload budget`() { + val exactGuard = RawAdmissionGuard(baseLimits.copy(maxValuePayloadBytes = 8)) + exactGuard.admit( + singleInvocation( + SingleQuery( + condition = Condition("abc", Operator.NULL), + projection = Projection(include = listOf("de")), + sort = listOf(Sort("fgh", Sort.Direction.ASC)), + ), + ), + ) + + assertInvocationRejected( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.PAYLOAD_LIMIT_EXCEEDED, + "$.input.query.sort[0].field", + ) { + exactGuard.admit( + singleInvocation( + SingleQuery( + condition = Condition("abc", Operator.NULL), + projection = Projection(include = listOf("de")), + sort = listOf(Sort("fghi", Sort.Direction.ASC)), + ), + ), + ) + } + } + + @Test + fun `should reject excessive numeric precision before materialization`() { + assertRejected( + RawAdmissionGuard(baseLimits.copy(maxNumericPrecision = 4)), + Condition.eq("field", BigInteger("12345")), + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.NUMERIC_PRECISION_LIMIT_EXCEEDED, + "$.input.condition.value", + ) + } + + @Test + fun `should reject decimal canonicalization overflow with a typed rejection`() { + assertRejected( + RawAdmissionGuard(baseLimits), + Condition.eq("field", BigDecimal(BigInteger.TEN, Int.MIN_VALUE)), + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.INVALID_VALUE_TYPE, + "$.input.condition.value", + ) + } + + @Test + fun `should accept exact local value boundaries and reject the next item`() { + val guard = RawAdmissionGuard(baseLimits) + guard.admit(countInvocation(Condition.eq("field", byteArrayOf(1, 2, 3, 4)))) + guard.admit(countInvocation(Condition.eq("field", listOf(1, 2, 3, 4)))) + guard.admit( + countInvocation( + Condition.eq("field", linkedMapOf("a" to 1, "b" to 2, "c" to 3, "d" to 4)), + ), + ) + guard.admit(countInvocation(Condition.eq("field", listOf(listOf(listOf(1)))))) + + val cases = listOf( + Condition.eq("field", byteArrayOf(1, 2, 3, 4, 5)) to + QueryRejectionCode.BYTE_ARRAY_LIMIT_EXCEEDED, + Condition.eq("field", listOf(1, 2, 3, 4, 5)) to + QueryRejectionCode.COLLECTION_LIMIT_EXCEEDED, + Condition.eq("field", linkedMapOf("a" to 1, "b" to 2, "c" to 3, "d" to 4, "e" to 5)) to + QueryRejectionCode.OBJECT_LIMIT_EXCEEDED, + Condition.eq("field", listOf(listOf(listOf(listOf(1))))) to + QueryRejectionCode.VALUE_DEPTH_LIMIT_EXCEEDED, + ) + cases.forEach { (condition, code) -> + assertThrownBy { + guard.admit(countInvocation(condition)) + }.satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(QueryRejectionCategory.BUDGET_EXCEEDED) + error.rejection.code.assert().isEqualTo(code) + }, + ) + } + } + + @Test + fun `should enforce condition node child and field boundaries`() { + RawAdmissionGuard(baseLimits.copy(maxConditionNodes = 3)).admit( + countInvocation(Condition.and(Condition.eq("a", 1), Condition.eq("b", 2))), + ) + assertRejected( + RawAdmissionGuard(baseLimits.copy(maxConditionNodes = 3)), + Condition.and(Condition.eq("a", 1), Condition.eq("b", 2), Condition.eq("c", 3)), + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.CONDITION_NODE_LIMIT_EXCEEDED, + "$.input.condition.children[2]", + ) + + RawAdmissionGuard(baseLimits.copy(maxChildrenPerNode = 2)).admit( + countInvocation(Condition.and(Condition.eq("a", 1), Condition.eq("b", 2))), + ) + assertRejected( + RawAdmissionGuard(baseLimits.copy(maxChildrenPerNode = 2)), + Condition.and(Condition.eq("a", 1), Condition.eq("b", 2), Condition.eq("c", 3)), + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.CHILDREN_LIMIT_EXCEEDED, + "$.input.condition.children", + ) + + RawAdmissionGuard(baseLimits).admit( + countInvocation(Condition.eq("f".repeat(baseLimits.maxFieldLength), "x")), + ) + assertRejected( + RawAdmissionGuard(baseLimits), + Condition.eq("f".repeat(baseLimits.maxFieldLength + 1), "x"), + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.STRING_LIMIT_EXCEEDED, + "$.input.condition.field", + ) + } + + @Test + fun `should enforce projection and sort boundaries`() { + val fields = listOf("a", "b") + val sorts = listOf( + Sort("a", Sort.Direction.ASC), + Sort("b", Sort.Direction.DESC), + ) + val exactGuard = RawAdmissionGuard(baseLimits.copy(maxProjectionFields = 2, maxSortFields = 2)) + exactGuard.admit(singleInvocation(SingleQuery(Condition.ALL, Projection(include = fields), sorts))) + exactGuard.admit( + singleInvocation( + SingleQuery(Condition.ALL, Projection(include = listOf("a"), exclude = listOf("b")), sorts), + ), + ) + + assertInvocationRejected( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.PROJECTION_LIMIT_EXCEEDED, + "$.input.query.projection.include", + ) { + exactGuard.admit( + singleInvocation( + SingleQuery(Condition.ALL, Projection(include = fields + "c"), sorts), + ), + ) + } + assertInvocationRejected( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.SORT_LIMIT_EXCEEDED, + "$.input.query.sort", + ) { + exactGuard.admit( + singleInvocation( + SingleQuery( + Condition.ALL, + Projection(include = fields), + sorts + Sort("c", Sort.Direction.ASC), + ), + ), + ) + } + assertInvocationRejected( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.PROJECTION_LIMIT_EXCEEDED, + "$.input.query.projection.exclude", + ) { + exactGuard.admit( + singleInvocation( + SingleQuery( + Condition.ALL, + Projection(include = fields, exclude = listOf("c")), + sorts, + ), + ), + ) + } + } + + @Test + fun `should enforce option count boundary`() { + val optionGuard = RawAdmissionGuard(baseLimits.copy(maxOptions = 1)) + optionGuard.admit( + countInvocation( + Condition("field", Operator.TODAY, options = mapOf(Condition.ZONE_ID_OPTION_KEY to "UTC")), + ), + ) + assertRejected( + optionGuard, + Condition( + "field", + Operator.TODAY, + options = linkedMapOf( + Condition.ZONE_ID_OPTION_KEY to "UTC", + Condition.DATE_PATTERN_OPTION_KEY to "yyyy-MM-dd", + ), + ), + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.OPTIONS_LIMIT_EXCEEDED, + "$.input.condition.options", + ) + } + + @Test + fun `should reject pre-normalized values at the raw boundary`() { + val values = listOf( + NormalizedValue.Text("text"), + NormalizedValue.Bytes(byteArrayOf(1)), + NormalizedValue.ListValue(listOf(NormalizedValue.Int64(1))), + NormalizedValue.ObjectValue(mapOf("key" to NormalizedValue.Int64(1))), + ) + + values.forEach { value -> + assertRejected( + RawAdmissionGuard(baseLimits), + Condition.eq("field", value), + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.INVALID_VALUE_TYPE, + "$.input.condition.value", + ) + } + } + + @Test + fun `should validate original IDS element types before canonicalization`() { + listOf(UUID.randomUUID(), 'x', TestValue.VALUE).forEach { invalidId -> + assertRejected( + RawAdmissionGuard(baseLimits), + Condition(operator = Operator.IDS, value = listOf(invalidId)), + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.INVALID_VALUE_TYPE, + "$.input.condition.value[0]", + ) + } + } + + @Test + fun `should type reject null collection members injected through Java compatible lists`() { + assertRejected( + RawAdmissionGuard(baseLimits), + Condition(operator = Operator.AND, children = unsafeList(null)), + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.INVALID_CHILDREN, + "$.input.condition.children[0]", + ) + assertInvocationRejected( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.INVALID_PROJECTION, + "$.input.query.projection.include[0]", + ) { + RawAdmissionGuard(baseLimits).admit( + singleInvocation( + SingleQuery(Condition.ALL, Projection(include = unsafeList(null))), + ), + ) + } + assertInvocationRejected( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.INVALID_SORT, + "$.input.query.sort[0]", + ) { + RawAdmissionGuard(baseLimits).admit( + singleInvocation(SingleQuery(Condition.ALL, sort = unsafeList(null))), + ) + } + } + + @Test + fun `should apply string budget before parsing time deletion and options`() { + val overLimit = "x".repeat(baseLimits.maxStringLength + 1) + val oversizedFormatter = DateTimeFormatterBuilder().appendLiteral(overLimit).toFormatter() + val cases = listOf( + Condition(operator = Operator.DELETED, value = overLimit) to "$.input.condition.value", + Condition("field", Operator.BEFORE_TODAY, overLimit) to "$.input.condition.value", + Condition("field", Operator.TODAY, options = mapOf(Condition.ZONE_ID_OPTION_KEY to overLimit)) to + "$.input.condition.options['zoneId']", + Condition("field", Operator.TODAY, options = mapOf(Condition.DATE_PATTERN_OPTION_KEY to overLimit)) to + "$.input.condition.options['datePattern']", + Condition("field", Operator.TODAY, options = mapOf(Condition.DATE_PATTERN_OPTION_KEY to oversizedFormatter)) to + "$.input.condition.options['datePattern']", + ) + + cases.forEach { (condition, path) -> + assertRejected( + RawAdmissionGuard(baseLimits), + condition, + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.STRING_LIMIT_EXCEEDED, + path, + ) + } + } + + @Test + fun `should count a string date pattern exactly once`() { + val exactPayloadSize = "field".length + Condition.DATE_PATTERN_OPTION_KEY.length + "yyyy".length + RawAdmissionGuard(baseLimits.copy(maxValuePayloadBytes = exactPayloadSize.toLong())).admit( + countInvocation( + Condition( + "field", + Operator.TODAY, + options = mapOf(Condition.DATE_PATTERN_OPTION_KEY to "yyyy"), + ), + ), + ) + } + + @Test + fun `should reject unsupported sql date values with a typed rejection`() { + listOf( + java.sql.Date.valueOf("2024-03-10"), + java.sql.Time.valueOf("12:30:00"), + ).forEach { value -> + assertRejected( + RawAdmissionGuard(baseLimits), + Condition.eq("field", value), + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.INVALID_VALUE_TYPE, + "$.input.condition.value", + ) + } + } + + @Test + fun `should reject hostile duplicate map entries after a bounded number of reads`() { + val duplicateMap = DuplicateKeyMap() + + assertRejected( + RawAdmissionGuard(baseLimits), + Condition.eq("field", duplicateMap), + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.DUPLICATE_OBJECT_KEY, + "$.input.condition.value['same']", + ) + duplicateMap.nextReads.assert().isEqualTo(2) + } + + @Test + fun `should keep option rejection path stable across map iteration order`() { + val optionsA = linkedMapOf(Condition.ZONE_ID_OPTION_KEY to "UTC", "unexpected" to true) + val optionsB = linkedMapOf("unexpected" to true, Condition.ZONE_ID_OPTION_KEY to "UTC") + + listOf(optionsA, optionsB).forEach { options -> + assertRejected( + RawAdmissionGuard(baseLimits), + Condition("field", Operator.TODAY, options = options), + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.UNKNOWN_OPTION, + "$.input.condition.options['unexpected']", + ) + } + } + + private fun countInvocation(condition: Condition): QueryInvocation = + QueryInvocation( + target = target, + operation = QueryOperation.COUNT, + resultShape = QueryResultShape.COUNT, + input = QueryInput.Count(condition), + ) + + @Suppress("UNCHECKED_CAST") + private fun unsafeList(value: Any?): List = listOf(value) as List + + private fun singleInvocation(query: SingleQuery): QueryInvocation = + QueryInvocation( + target = target, + operation = QueryOperation.SINGLE, + resultShape = QueryResultShape.TYPED, + input = QueryInput.Single(query), + ) + + private fun assertInvocationRejected( + category: QueryRejectionCategory, + code: QueryRejectionCode, + path: String, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo(path) + }, + ) + } + + private fun assertRejected( + guard: RawAdmissionGuard, + condition: Condition, + category: QueryRejectionCategory, + code: QueryRejectionCode, + path: String, + ) { + assertThrownBy { + guard.admit(countInvocation(condition)) + }.satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo(path) + }, + ) + } + + private enum class TestValue { + VALUE, + } + + private class DuplicateKeyMap : AbstractMap() { + var nextReads: Int = 0 + private set + + override val entries: Set> = object : AbstractSet>() { + override val size: Int = Int.MAX_VALUE + + override fun iterator(): Iterator> = object : Iterator> { + override fun hasNext(): Boolean = true + + override fun next(): Map.Entry { + if (!hasNext()) { + throw NoSuchElementException() + } + nextReads++ + return SimpleImmutableEntry("same", nextReads) + } + } + } + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/admission/RawAdmissionGuardTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/admission/RawAdmissionGuardTest.kt new file mode 100644 index 00000000000..4718fc08c3f --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/admission/RawAdmissionGuardTest.kt @@ -0,0 +1,322 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.admission + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.ISingleQuery +import me.ahoo.wow.api.query.Operator +import me.ahoo.wow.api.query.Projection +import me.ahoo.wow.api.query.Sort +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryInput +import me.ahoo.wow.query.internal.model.QueryInvocation +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import java.math.BigDecimal +import java.util.function.Consumer + +class RawAdmissionGuardTest { + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val limits = QueryAdmissionLimits( + maxConditionDepth = 4, + maxConditionNodes = 12, + maxChildrenPerNode = 4, + maxFieldLength = 24, + maxStringLength = 16, + maxCollectionSize = 4, + maxObjectFields = 4, + maxValueDepth = 4, + maxByteArrayLength = 4, + maxProjectionFields = 4, + maxSortFields = 3, + maxOptions = 3, + ) + private val guard = RawAdmissionGuard(limits) + + @Test + fun `should read each query getter once and freeze every mutable boundary`() { + val bytes = byteArrayOf(1, 2) + val objectValue = linkedMapOf("bytes" to bytes) + val oneShot = OneShotIterable(listOf(objectValue)) + val projectionFields = mutableListOf("state.name") + val sorts = mutableListOf(Sort("state.name", Sort.Direction.ASC)) + val query = SingleReadQuery( + conditionValue = Condition("state.items", Operator.IN, oneShot), + projectionValue = Projection(include = projectionFields), + sortValue = sorts, + ) + + val admitted = guard.admit(singleInvocation(query)) + projectionFields += "late" + sorts.clear() + objectValue.clear() + bytes[0] = 9 + + query.conditionReads.assert().isEqualTo(1) + query.projectionReads.assert().isEqualTo(1) + query.sortReads.assert().isEqualTo(1) + oneShot.iteratorReads.assert().isEqualTo(1) + + val admittedQuery = (admitted.input as AdmittedQueryInput.Single).query + admittedQuery.projection.include.assert().containsExactly("state.name") + admittedQuery.sort.map { it.field }.assert().containsExactly("state.name") + admittedQuery.condition.queryValue().assert().isEqualTo( + NormalizedValue.ListValue( + listOf( + NormalizedValue.ObjectValue( + mapOf("bytes" to NormalizedValue.Bytes(byteArrayOf(1, 2))), + ), + ), + ), + ) + } + + @Test + fun `should canonicalize equivalent numeric values during admission`() { + val condition = Condition( + "state.numbers", + Operator.ALL_IN, + OneShotIterable(listOf(1, 1L, 1.0, BigDecimal("1.00"))), + ) + + val admitted = guard.admit(singleInvocation(SingleReadQuery(condition))) + val admittedCondition = (admitted.input as AdmittedQueryInput.Single).query.condition + val values = admittedCondition.queryValue() as NormalizedValue.ListValue + + values.values.assert().containsExactly( + NormalizedValue.Int64(1), + NormalizedValue.Int64(1), + NormalizedValue.Int64(1), + NormalizedValue.Int64(1), + ) + } + + @Test + fun `should reject budgets at the exact offending path`() { + val tooDeep = Condition.and( + Condition.and( + Condition.and( + Condition.and(Condition.eq("field", "value")), + ), + ), + ) + assertRejected( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.CONDITION_DEPTH_LIMIT_EXCEEDED, + "$.input.query.condition.children[0].children[0].children[0].children[0]", + ) { + guard.admit(singleInvocation(SingleReadQuery(tooDeep))) + } + + assertRejected( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.STRING_LIMIT_EXCEEDED, + "$.input.query.condition.value", + ) { + guard.admit( + singleInvocation( + SingleReadQuery(Condition.eq("field", "x".repeat(limits.maxStringLength + 1))), + ), + ) + } + + assertRejected( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.COLLECTION_LIMIT_EXCEEDED, + "$.input.query.condition.value", + ) { + guard.admit( + singleInvocation( + SingleReadQuery( + Condition("field", Operator.IN, OneShotIterable((0..limits.maxCollectionSize).toList())), + ), + ), + ) + } + } + + @Test + fun `should reject condition and value cycles without overflowing the stack`() { + val children = mutableListOf() + val cyclicCondition = Condition(operator = Operator.AND, children = children) + children += cyclicCondition + + assertRejected( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.CYCLIC_INPUT, + "$.input.query.condition.children[0]", + ) { + guard.admit(singleInvocation(SingleReadQuery(cyclicCondition))) + } + + val values = mutableListOf() + values.add(values) + assertRejected( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.CYCLIC_INPUT, + "$.input.query.condition.value[0]", + ) { + guard.admit(singleInvocation(SingleReadQuery(Condition.eq("field", values)))) + } + } + + @Test + fun `should reject malformed operator values with typed errors`() { + val cases = listOf( + Condition(operator = Operator.AND) to QueryRejectionCode.INVALID_CHILDREN, + Condition("field", Operator.BETWEEN, listOf(1)) to QueryRejectionCode.INVALID_VALUE_ARITY, + Condition("field", Operator.ELEM_MATCH) to QueryRejectionCode.INVALID_CHILDREN, + Condition(operator = Operator.ID, value = 1) to QueryRejectionCode.INVALID_VALUE_TYPE, + Condition(operator = Operator.IDS, value = listOf("id", 1)) to QueryRejectionCode.INVALID_VALUE_TYPE, + Condition("field", Operator.IN, "value") to QueryRejectionCode.INVALID_VALUE_TYPE, + Condition("field", Operator.EXISTS, "true") to QueryRejectionCode.INVALID_VALUE_TYPE, + Condition("field", Operator.RECENT_DAYS, 0) to QueryRejectionCode.INVALID_TIME_VALUE, + Condition("field", Operator.BEFORE_TODAY, 86_400) to QueryRejectionCode.INVALID_TIME_VALUE, + ) + + cases.forEach { (condition, code) -> + assertThrownBy { + guard.admit(singleInvocation(SingleReadQuery(condition))) + }.satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(QueryRejectionCategory.INVALID_QUERY) + error.rejection.code.assert().isEqualTo(code) + } + ) + } + } + + @Test + fun `should reject unknown or mistyped options and unbound native input`() { + val cases = listOf( + Condition( + "field", + Operator.CONTAINS, + "value", + options = mapOf(Condition.IGNORE_CASE_OPTION_KEY to "true"), + ) to QueryRejectionCode.INVALID_OPTION_TYPE, + Condition( + "field", + Operator.TODAY, + options = mapOf("unknown" to true), + ) to QueryRejectionCode.UNKNOWN_OPTION, + Condition.eq("field", "value").copy( + options = mapOf(Condition.IGNORE_CASE_OPTION_KEY to true), + ) to QueryRejectionCode.OPTION_NOT_ALLOWED, + ) + + cases.forEach { (condition, code) -> + assertThrownBy { + guard.admit(singleInvocation(SingleReadQuery(condition))) + }.satisfies( + Consumer { error -> + error.rejection.code.assert().isEqualTo(code) + } + ) + } + + val raw = guard.admit(singleInvocation(SingleReadQuery(Condition.raw(HostileRawValue())))) + val rawCondition = (raw.input as AdmittedQueryInput.Single).query.condition + rawCondition.value.assert().isEqualTo(AdmittedConditionValue.NativeUnbound) + } + + private fun singleInvocation(query: ISingleQuery): QueryInvocation = + QueryInvocation( + target = target, + operation = QueryOperation.SINGLE, + resultShape = QueryResultShape.TYPED, + input = QueryInput.Single(query), + ) + + private fun assertRejected( + category: QueryRejectionCategory, + code: QueryRejectionCode, + path: String, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo(path) + } + ) + } + + private class OneShotIterable(private val values: List) : Iterable { + var iteratorReads: Int = 0 + private set + + override fun iterator(): Iterator { + iteratorReads++ + check(iteratorReads == 1) { + "Iterable can only be consumed once." + } + return values.iterator() + } + } + + private class HostileRawValue { + override fun toString(): String = error("RAW driver objects must not be inspected during admission.") + } + + private class SingleReadQuery( + private val conditionValue: Condition, + private val projectionValue: Projection = Projection.ALL, + private val sortValue: List = emptyList(), + ) : ISingleQuery { + var conditionReads: Int = 0 + private set + var projectionReads: Int = 0 + private set + var sortReads: Int = 0 + private set + + override val condition: Condition + get() = conditionValue.also { + conditionReads++ + check(conditionReads == 1) + } + override val projection: Projection + get() = projectionValue.also { + projectionReads++ + check(projectionReads == 1) + } + override val sort: List + get() = sortValue.also { + sortReads++ + check(sortReads == 1) + } + + override fun withCondition(newCondition: Condition): ISingleQuery = this + + override fun withProjection(newProjection: Projection): ISingleQuery = this + } +} + +private fun AdmittedCondition.queryValue(): NormalizedValue = + (value as AdmittedConditionValue.QueryValue).value diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/admission/RawQueryGetterSnapshotTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/admission/RawQueryGetterSnapshotTest.kt new file mode 100644 index 00000000000..ccd1a4271ae --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/admission/RawQueryGetterSnapshotTest.kt @@ -0,0 +1,120 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.admission + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.IListQuery +import me.ahoo.wow.api.query.IPagedQuery +import me.ahoo.wow.api.query.Pagination +import me.ahoo.wow.api.query.Projection +import me.ahoo.wow.api.query.Sort +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryInput +import me.ahoo.wow.query.internal.model.QueryInvocation +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryTarget +import org.junit.jupiter.api.Test + +class RawQueryGetterSnapshotTest { + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val guard = RawAdmissionGuard(QueryAdmissionLimits.DEFAULT) + + @Test + fun `should read every stream query getter exactly once`() { + val query = SingleReadListQuery() + + val admitted = guard.admit( + QueryInvocation( + target, + QueryOperation.STREAM, + QueryResultShape.TYPED, + QueryInput.Stream(query), + ), + ) + + query.reads.assert().containsExactlyInAnyOrderEntriesOf( + mapOf("condition" to 1, "projection" to 1, "sort" to 1, "limit" to 1), + ) + (admitted.input as AdmittedQueryInput.Stream).limit.assert().isEqualTo(10) + } + + @Test + fun `should read every page query getter exactly once`() { + val query = SingleReadPagedQuery() + + val admitted = guard.admit( + QueryInvocation( + target, + QueryOperation.PAGE, + QueryResultShape.DYNAMIC, + QueryInput.Page(query), + ), + ) + + query.reads.assert().containsExactlyInAnyOrderEntriesOf( + mapOf("condition" to 1, "projection" to 1, "sort" to 1, "pagination" to 1), + ) + (admitted.input as AdmittedQueryInput.Page).page.assert().isEqualTo(AdmittedPage(2, 25, 25)) + } + + private abstract class SingleReadQueryable { + val reads: MutableMap = linkedMapOf() + + protected fun readOnce(name: String, value: T): T { + val count = reads.getOrDefault(name, 0) + 1 + reads[name] = count + check(count == 1) { + "$name can only be read once." + } + return value + } + } + + private class SingleReadListQuery : SingleReadQueryable(), IListQuery { + override val condition: Condition + get() = readOnce("condition", Condition.ALL) + override val projection: Projection + get() = readOnce("projection", Projection.ALL) + override val sort: List + get() = readOnce("sort", emptyList()) + override val limit: Int + get() = readOnce("limit", 10) + + override fun withCondition(newCondition: Condition): IListQuery = this + + override fun withProjection(newProjection: Projection): IListQuery = this + } + + private class SingleReadPagedQuery : SingleReadQueryable(), IPagedQuery { + override val condition: Condition + get() = readOnce("condition", Condition.ALL) + override val projection: Projection + get() = readOnce("projection", Projection.ALL) + override val sort: List + get() = readOnce("sort", emptyList()) + override val pagination: Pagination + get() = readOnce("pagination", Pagination(2, 25)) + + override fun withCondition(newCondition: Condition): IPagedQuery = this + + override fun withProjection(newProjection: Projection): IPagedQuery = this + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/cursor/PersistentQueryCursorLeaseCoordinatorTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/cursor/PersistentQueryCursorLeaseCoordinatorTest.kt new file mode 100644 index 00000000000..601cebc042f --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/cursor/PersistentQueryCursorLeaseCoordinatorTest.kt @@ -0,0 +1,68 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.cursor + +import io.mockk.every +import io.mockk.mockk +import me.ahoo.test.asserts.assert +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Instant +import java.util.concurrent.CopyOnWriteArrayList + +@OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) +class PersistentQueryCursorLeaseCoordinatorTest { + @Test + fun `reaper should report only successfully closed leases and expose cleanup failure`() { + val manager = mockk() + every { manager.reapExpired(NOW, limit = 10) } returns Flux.just(envelope()) + val failures = CopyOnWriteArrayList() + val coordinator = PersistentQueryCursorLeaseCoordinator( + manager, + listOf(PersistentQueryCursorBackendLeaseRegistration(TARGET, BACKEND) { Mono.error(FAILURE) }), + QueryCursorLeaseObserver { _, _, error -> failures += error }, + ) + + coordinator.reapExpired(NOW, 10).block().assert().isZero() + failures.assert().containsExactly(FAILURE) + } + + private fun envelope() = QueryCursorEnvelope( + TARGET, + PlanFingerprint("1".repeat(64)), + QueryCursorMappingDigest("a".repeat(64)), + QueryCursorSecurityContextDigest("c".repeat(64)), + QueryCursorPosition.Analytics( + listOf(AnalyticsAlias("status")), + listOf(NormalizedValue.Text("PAID")), + ), + NOW.plusSeconds(60), + QueryCursorBackendState(BACKEND, "pit".toByteArray()), + ) + + private companion object { + val NOW: Instant = Instant.parse("2026-08-09T00:00:00Z") + val TARGET = QueryTarget(MaterializedNamedAggregate("sales", "order"), QueryDocumentKind.SNAPSHOT) + val BACKEND = BackendId("elasticsearch") + val FAILURE = IllegalStateException("close failed") + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/cursor/PersistentQueryCursorLeaseManagerTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/cursor/PersistentQueryCursorLeaseManagerTest.kt new file mode 100644 index 00000000000..2559d553f35 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/cursor/PersistentQueryCursorLeaseManagerTest.kt @@ -0,0 +1,249 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.query.internal.cursor + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.cursor.QueryCursorLeaseCreateResult +import me.ahoo.wow.query.cursor.QueryCursorLeaseEntry +import me.ahoo.wow.query.cursor.QueryCursorLeaseId +import me.ahoo.wow.query.cursor.QueryCursorLeaseStore +import me.ahoo.wow.query.cursor.QueryCursorStoreRevision +import me.ahoo.wow.query.cursor.StoredQueryCursorLease +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.test.StepVerifier +import java.time.Clock +import java.time.Instant +import java.time.ZoneOffset +import java.time.temporal.ChronoUnit +import java.util.concurrent.ConcurrentHashMap +import java.util.concurrent.atomic.AtomicLong + +class PersistentQueryCursorLeaseManagerTest { + @Test + fun `shared store should transfer one-time cursor ownership across nodes`() { + val store = InMemoryStore() + val firstNode = manager(store) + val secondNode = manager(store) + val envelope = envelope() + val token = firstNode.issue(envelope).block()!! + + val loadedBySecond = secondNode.load(token).block()!! + secondNode.acquire(loadedBySecond, envelope.binding()).block().assert().isEqualTo(envelope) + + StepVerifier.create(firstNode.load(token)) + .expectErrorSatisfies { error -> assertRejected(error, QueryRejectionCode.INVALID_CURSOR_TOKEN) } + .verify() + } + + @Test + fun `binding mismatch should not consume the legitimate store revision`() { + val store = InMemoryStore() + val manager = manager(store) + val envelope = envelope() + val token = manager.issue(envelope).block()!! + val loaded = manager.load(token).block()!! + + StepVerifier.create( + manager.acquire( + loaded, + envelope.binding().copy(securityContextDigest = QueryCursorSecurityContextDigest("d".repeat(64))), + ), + ).expectErrorSatisfies { error -> assertRejected(error, QueryRejectionCode.INVALID_CURSOR_BINDING) } + .verify() + + manager.acquire(manager.load(token).block()!!, envelope.binding()).block().assert().isEqualTo(envelope) + } + + @Test + fun `corrupt opaque payload and store capacity should fail with stable categories`() { + val store = InMemoryStore() + val manager = manager(store) + val token = manager.issue(envelope()).block()!! + store.corruptSinglePayload() + + StepVerifier.create(manager.load(token)) + .expectErrorSatisfies { error -> assertRejected(error, QueryRejectionCode.INVALID_CURSOR_TOKEN) } + .verify() + + val full = InMemoryStore(capacity = 0) + StepVerifier.create(manager(full).issue(envelope())) + .expectErrorSatisfies { error -> + val rejected = error as QueryRejectedException + rejected.rejection.category.assert().isEqualTo(QueryRejectionCategory.BUDGET_EXCEEDED) + rejected.rejection.code.assert().isEqualTo(QueryRejectionCode.CURSOR_CAPACITY_EXCEEDED) + }.verify() + } + + @Test + fun `reaper should delete only the exact expired revision`() { + val store = InMemoryStore() + val manager = manager(store) + manager.issue(envelope(expiresAt = NOW.plusSeconds(30))).block() + manager.issue(envelope(expiresAt = NOW.plusSeconds(60), fingerprint = "2".repeat(64))).block() + + StepVerifier.create(manager.reapExpired(NOW.plusSeconds(45), limit = 10)) + .expectNextMatches { it.planFingerprint == PlanFingerprint("1".repeat(64)) } + .verifyComplete() + + store.size.assert().isEqualTo(1) + } + + @Test + fun `configured backend state limit should also configure the signed envelope codec`() { + val store = InMemoryStore() + val manager = PersistentQueryCursorLeaseManager( + store, + QueryCursorSigningKeyRing(QueryCursorSigningKey(1, SECRET)), + Clock.fixed(NOW, ZoneOffset.UTC), + QueryCursorLeaseLimits(maxBackendStateBytes = 5_000), + ) + val envelope = envelope().copy( + backendState = QueryCursorBackendState(BackendId("elasticsearch"), ByteArray(4_097) { 7 }), + ) + + val token = manager.issue(envelope).block()!! + + manager.load(token).block()!!.envelope.assert().isEqualTo(envelope) + } + + @Test + fun `lease expiry should be normalized before signing and persistence`() { + val manager = manager(InMemoryStore()) + val envelope = envelope(expiresAt = NOW.plusSeconds(60).plusNanos(123_456)) + + val token = manager.issue(envelope).block()!! + + manager.load(token).block()!!.envelope.expiresAt.assert().isEqualTo( + envelope.expiresAt.truncatedTo(ChronoUnit.MILLIS), + ) + } + + private fun manager(store: QueryCursorLeaseStore): PersistentQueryCursorLeaseManager = + PersistentQueryCursorLeaseManager( + store, + QueryCursorSigningKeyRing(QueryCursorSigningKey(1, SECRET)), + Clock.fixed(NOW, ZoneOffset.UTC), + ) + + private fun envelope( + expiresAt: Instant = NOW.plusSeconds(60), + fingerprint: String = "1".repeat(64), + ) = QueryCursorEnvelope( + TARGET, + PlanFingerprint(fingerprint), + QueryCursorMappingDigest("a".repeat(64)), + QueryCursorSecurityContextDigest("c".repeat(64)), + QueryCursorPosition.Analytics( + listOf(AnalyticsAlias("status")), + listOf(NormalizedValue.Text("PAID")), + ), + expiresAt, + QueryCursorBackendState(BackendId("elasticsearch"), "pit-id".toByteArray()), + budgetCeiling = QueryCursorBudgetCeiling( + maxScannedRecords = 1_000, + maxReturnedRecords = 100, + maxPageWindow = 10_000, + maxCandidateBuckets = 500, + maxReturnedBuckets = 50, + maxCursorPages = 5, + allowDiskUse = true, + ), + ) + + private fun assertRejected(error: Throwable, code: QueryRejectionCode) { + val rejected = error as QueryRejectedException + rejected.rejection.category.assert().isEqualTo(QueryRejectionCategory.INVALID_CURSOR) + rejected.rejection.code.assert().isEqualTo(code) + rejected.rejection.path.toString().assert().isEqualTo("$.cursor") + } + + private class InMemoryStore( + private val capacity: Int = 100, + ) : QueryCursorLeaseStore { + private val revisions = AtomicLong() + private val entries = ConcurrentHashMap() + + val size: Int + get() = entries.size + + override fun create(entry: QueryCursorLeaseEntry): Mono = Mono.fromSupplier { + if (entries.size >= capacity) return@fromSupplier QueryCursorLeaseCreateResult.CAPACITY_EXCEEDED + val stored = StoredQueryCursorLease( + entry, + QueryCursorStoreRevision(revisions.incrementAndGet().toString()), + ) + if (entries.putIfAbsent(entry.id, stored) == null) { + QueryCursorLeaseCreateResult.CREATED + } else { + QueryCursorLeaseCreateResult.COLLISION + } + } + + override fun load(id: QueryCursorLeaseId): Mono = Mono.justOrEmpty(entries[id]) + + override fun compareAndDelete(expected: StoredQueryCursorLease): Mono = + Mono.fromSupplier { entries.remove(expected.entry.id, expected) } + + override fun scanExpired( + before: Instant, + afterId: QueryCursorLeaseId?, + limit: Int, + ): Flux = Flux.fromIterable( + entries.values.filter { stored -> + !stored.entry.expiresAt.isAfter(before) && + (afterId == null || stored.entry.id.value > afterId.value) + }.sortedBy { stored -> stored.entry.id.value }.take(limit), + ) + + fun corruptSinglePayload() { + val current = entries.values.single() + val payload = current.entry.payload().also { bytes -> bytes[bytes.lastIndex] = (bytes.last() + 1).toByte() } + entries[current.entry.id] = StoredQueryCursorLease( + QueryCursorLeaseEntry( + current.entry.id, + current.entry.expiresAt, + current.entry.payloadFormat, + payload, + ), + current.revision, + ) + } + } + + private companion object { + val NOW: Instant = Instant.parse("2026-08-09T00:00:00Z") + val SECRET = ByteArray(32) { 7 } + val TARGET = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorLeaseCoordinatorTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorLeaseCoordinatorTest.kt new file mode 100644 index 00000000000..be14ce7e883 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorLeaseCoordinatorTest.kt @@ -0,0 +1,192 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.cursor + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import reactor.core.publisher.Mono +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneId +import java.time.ZoneOffset +import java.util.concurrent.atomic.AtomicInteger +import java.util.concurrent.atomic.AtomicReference +import java.util.function.Consumer + +class QueryCursorLeaseCoordinatorTest { + @Test + fun `acquired lease should transfer ownership once and close the terminal backend state once`() { + val closed = AtomicInteger() + val closedPayload = AtomicReference() + val coordinator = coordinator { state -> + closed.incrementAndGet() + closedPayload.set(String(state.payload())) + Mono.empty() + } + val first = coordinator.issue(envelope()) + val acquired = coordinator.acquire(first, envelope().binding()) + val nextPosition = position("bob") + + val next = acquired.transfer(nextPosition, NOW.plusSeconds(90)) + acquired.close().block() + val continued = coordinator.acquire(next, envelope().binding()) + + continued.envelope.position.assert().isEqualTo(nextPosition) + closed.get().assert().isEqualTo(0) + continued.close().block() + continued.close().block() + closed.get().assert().isEqualTo(1) + closedPayload.get().assert().isEqualTo("pit-secret") + } + + @Test + fun `abandoned lease cleanup and observer failures should never escape the reaper`() { + val clock = MutableClock(NOW) + val cleanupCalls = AtomicInteger() + val observed = AtomicInteger() + val manager = InMemoryQueryCursorLeaseManager(SECRET, clock) + val coordinator = QueryCursorLeaseCoordinator( + manager, + listOf( + QueryCursorBackendLeaseRegistration(BACKEND) { + cleanupCalls.incrementAndGet() + Mono.error(IllegalStateException("close failed")) + }, + ), + QueryCursorLeaseObserver { descriptor, reason, _ -> + descriptor.target.assert().isEqualTo(TARGET) + descriptor.backendId.assert().isEqualTo(BACKEND) + reason.assert().isEqualTo(QueryCursorCleanupReason.ABANDONED) + observed.incrementAndGet() + error("observer failed") + }, + ) + coordinator.issue(envelope(expiresAt = NOW.plusSeconds(1))) + clock.advance(Duration.ofSeconds(2)) + + coordinator.reapExpired().block() + + cleanupCalls.get().assert().isEqualTo(1) + observed.get().assert().isEqualTo(1) + manager.size.assert().isEqualTo(0) + } + + @Test + fun `backend state must have a registered owner and cannot transfer across backends`() { + val manager = InMemoryQueryCursorLeaseManager(SECRET, Clock.fixed(NOW, ZoneOffset.UTC)) + val withoutOwner = QueryCursorLeaseCoordinator(manager, emptyList()) + assertRejected(QueryRejectionCategory.BACKEND_UNAVAILABLE, QueryRejectionCode.BACKEND_NOT_REGISTERED) { + withoutOwner.issue(envelope()) + } + + val coordinator = coordinator { Mono.empty() } + val envelope = envelope() + val acquired = coordinator.acquire(coordinator.issue(envelope), envelope.binding()) + assertRejected(QueryRejectionCategory.INVALID_CURSOR, QueryRejectionCode.INVALID_CURSOR_BINDING) { + acquired.transfer( + position("bob"), + NOW.plusSeconds(90), + QueryCursorBackendState(BackendId("mongo"), "cursor".toByteArray()), + ) + } + acquired.close().block() + } + + @Test + fun `duplicate backend owners should fail configuration before serving queries`() { + assertThrownBy { + QueryCursorLeaseCoordinator( + InMemoryQueryCursorLeaseManager(SECRET, Clock.fixed(NOW, ZoneOffset.UTC)), + listOf( + QueryCursorBackendLeaseRegistration(BACKEND) { Mono.empty() }, + QueryCursorBackendLeaseRegistration(BACKEND) { Mono.empty() }, + ), + ) + } + } + + private fun coordinator(closer: QueryCursorBackendLeaseCloser): QueryCursorLeaseCoordinator = + QueryCursorLeaseCoordinator( + InMemoryQueryCursorLeaseManager(SECRET, Clock.fixed(NOW, ZoneOffset.UTC)), + listOf(QueryCursorBackendLeaseRegistration(BACKEND, closer)), + ) + + private fun envelope( + expiresAt: Instant = NOW.plusSeconds(60), + ) = QueryCursorEnvelope( + TARGET, + PlanFingerprint("1".repeat(64)), + QueryCursorMappingDigest("a".repeat(64)), + QueryCursorSecurityContextDigest("c".repeat(64)), + position("alice"), + expiresAt, + QueryCursorBackendState(BACKEND, "pit-secret".toByteArray()), + ) + + private fun position(value: String) = QueryCursorPosition.Analytics( + listOf(AnalyticsAlias("name")), + listOf(NormalizedValue.Text(value)), + ) + + private fun assertRejected( + category: QueryRejectionCategory, + code: QueryRejectionCode, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo("$.cursor") + }, + ) + } + + private class MutableClock( + private var current: Instant, + private val currentZone: ZoneId = ZoneOffset.UTC, + ) : Clock() { + override fun getZone(): ZoneId = currentZone + + override fun withZone(zone: ZoneId): Clock = MutableClock(current, zone) + + override fun instant(): Instant = current + + fun advance(duration: Duration) { + current = current.plus(duration) + } + } + + private companion object { + val NOW: Instant = Instant.parse("2026-08-08T00:00:00Z") + val SECRET = ByteArray(32) { 1 } + val BACKEND = BackendId("elasticsearch") + val TARGET = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorLeaseManagerTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorLeaseManagerTest.kt new file mode 100644 index 00000000000..6aa273dd0a1 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/cursor/QueryCursorLeaseManagerTest.kt @@ -0,0 +1,282 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.cursor + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.PlanFingerprint +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import java.math.BigDecimal +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneId +import java.time.ZoneOffset +import java.util.Base64 +import java.util.function.Consumer + +class QueryCursorLeaseManagerTest { + @Test + fun `token should be opaque tamper evident immutable and one time`() { + val stateSource = "pit-secret-value".toByteArray() + val manager = manager() + val envelope = envelope(stateSource) + + val token = manager.issue(envelope) + stateSource.fill(0) + + token.value.contains("pit-secret-value").assert().isFalse() + String(Base64.getUrlDecoder().decode(token.value.substringBefore('.'))) + .contains("pit-secret-value") + .assert().isFalse() + manager.size.assert().isEqualTo(1) + + val acquired = manager.acquire(token, envelope.binding()) + acquired.assert().isEqualTo(envelope) + String(acquired.backendState!!.payload()).assert().isEqualTo("pit-secret-value") + manager.size.assert().isEqualTo(0) + + assertRejected(QueryRejectionCategory.INVALID_CURSOR, QueryRejectionCode.INVALID_CURSOR_TOKEN) { + manager.acquire(token, envelope.binding()) + } + val tampered = QueryCursorToken( + (if (token.value.first() == 'A') 'B' else 'A') + token.value.drop(1), + ) + assertRejected(QueryRejectionCategory.INVALID_CURSOR, QueryRejectionCode.INVALID_CURSOR_TOKEN) { + manager.acquire(tampered, envelope.binding()) + } + } + + @Test + fun `token version and signing key should be bound without leaking registry state`() { + val versionTwo = manager(keyId = 2) + val envelope = envelope() + val token = versionTwo.issue(envelope) + + assertRejected(QueryRejectionCategory.INVALID_CURSOR, QueryRejectionCode.INVALID_CURSOR_TOKEN) { + manager().acquire(token, envelope.binding()) + } + assertRejected(QueryRejectionCategory.INVALID_CURSOR, QueryRejectionCode.INVALID_CURSOR_TOKEN) { + manager(secret = ByteArray(32) { 2 }).acquire(token, envelope.binding()) + } + } + + @Test + fun `signing key rotation should retain previous verification without issuing old key ids`() { + val leaseId = Base64.getUrlEncoder().withoutPadding().encodeToString(ByteArray(32) { 7 }) + val oldSecret = ByteArray(32) { 1 } + val oldCodec = QueryCursorTokenCodec( + QueryCursorSigningKeyRing(QueryCursorSigningKey(1, oldSecret)), + ) + val oldToken = oldCodec.encode(leaseId, NOW.plusSeconds(60)) + oldSecret.fill(0) + val rotated = QueryCursorTokenCodec( + QueryCursorSigningKeyRing( + QueryCursorSigningKey(2, ByteArray(32) { 2 }), + listOf(QueryCursorSigningKey(1, ByteArray(32) { 1 })), + ), + ) + + rotated.decode(oldToken).id.assert().isEqualTo(leaseId) + val currentToken = rotated.encode(leaseId, NOW.plusSeconds(60)) + assertRejected(QueryRejectionCategory.INVALID_CURSOR, QueryRejectionCode.INVALID_CURSOR_TOKEN) { + oldCodec.decode(currentToken) + } + val retired = QueryCursorTokenCodec( + QueryCursorSigningKeyRing(QueryCursorSigningKey(2, ByteArray(32) { 2 })), + ) + assertRejected(QueryRejectionCategory.INVALID_CURSOR, QueryRejectionCode.INVALID_CURSOR_TOKEN) { + retired.decode(oldToken) + } + } + + @Test + fun `signing key ring should reject duplicate ids and unbounded previous keys`() { + assertThrownBy { + QueryCursorSigningKeyRing( + QueryCursorSigningKey(1, SECRET), + listOf(QueryCursorSigningKey(1, ByteArray(32) { 2 })), + ) + } + assertThrownBy { + QueryCursorSigningKeyRing( + QueryCursorSigningKey(1, SECRET), + (2..6).map { id -> QueryCursorSigningKey(id, ByteArray(32) { id.toByte() }) }, + ) + } + } + + @Test + fun `expired leases should reject acquisition and be transferred for cleanup`() { + val clock = MutableClock(NOW) + val manager = manager(clock = clock) + val acquired = envelope(expiresAt = NOW.plusSeconds(1)) + val acquiredToken = manager.issue(acquired) + val abandoned = envelope(expiresAt = NOW.plusSeconds(2), fingerprint = "2".repeat(64)) + manager.issue(abandoned) + + clock.advance(Duration.ofSeconds(3)) + + assertRejected(QueryRejectionCategory.INVALID_CURSOR, QueryRejectionCode.CURSOR_EXPIRED) { + manager.acquire(acquiredToken, acquired.binding()) + } + val expired = manager.reapExpired() + expired.assert().hasSize(2) + expired.contains(acquired).assert().isTrue() + expired.contains(abandoned).assert().isTrue() + manager.size.assert().isEqualTo(0) + } + + @Test + fun `registry should bound ttl entry count and backend state before publishing a token`() { + val limits = QueryCursorLeaseLimits( + maxEntries = 1, + maxTtl = Duration.ofSeconds(10), + maxBackendStateBytes = 4, + ) + val manager = manager(limits = limits) + manager.issue(envelope("pit".toByteArray(), NOW.plusSeconds(10))) + + assertRejected(QueryRejectionCategory.BUDGET_EXCEEDED, QueryRejectionCode.CURSOR_CAPACITY_EXCEEDED) { + manager.issue(envelope("next".toByteArray(), NOW.plusSeconds(10), "2".repeat(64))) + } + + val ttlManager = manager(limits = limits) + assertRejected(QueryRejectionCategory.BUDGET_EXCEEDED, QueryRejectionCode.CURSOR_CAPACITY_EXCEEDED) { + ttlManager.issue(envelope(expiresAt = NOW.plusSeconds(11))) + } + assertRejected(QueryRejectionCategory.BUDGET_EXCEEDED, QueryRejectionCode.CURSOR_CAPACITY_EXCEEDED) { + ttlManager.issue(envelope("oversized".toByteArray(), NOW.plusSeconds(10))) + } + ttlManager.size.assert().isEqualTo(0) + } + + @Test + fun `security target plan and mapping binding mismatch must not consume the legitimate lease`() { + val manager = manager() + val envelope = envelope() + val token = manager.issue(envelope) + val expected = envelope.binding() + listOf( + expected.copy(securityContextDigest = QueryCursorSecurityContextDigest("b".repeat(64))), + expected.copy(planFingerprint = PlanFingerprint("2".repeat(64))), + expected.copy(mappingGenerationDigest = QueryCursorMappingDigest("b".repeat(64))), + expected.copy( + target = QueryTarget(MaterializedNamedAggregate("sales", "cart"), QueryDocumentKind.SNAPSHOT), + ), + ).forEach { mismatched -> + assertRejected(QueryRejectionCategory.INVALID_CURSOR, QueryRejectionCode.INVALID_CURSOR_BINDING) { + manager.acquire(token, mismatched) + } + manager.size.assert().isEqualTo(1) + } + + manager.acquire(token, expected).assert().isEqualTo(envelope) + manager.size.assert().isEqualTo(0) + } + + @Test + fun `cursor position should preserve canonical scalar value semantics`() { + val position = QueryCursorPosition.Analytics( + listOf(AnalyticsAlias("text"), AnalyticsAlias("decimal"), AnalyticsAlias("missing")), + listOf( + NormalizedValue.Text("A"), + NormalizedValue.Decimal(BigDecimal("1.00")), + NormalizedValue.Null, + ), + ) + val independent = QueryCursorPosition.Analytics( + listOf(AnalyticsAlias("text"), AnalyticsAlias("decimal"), AnalyticsAlias("missing")), + listOf( + NormalizedValue.Text("A"), + NormalizedValue.Decimal(BigDecimal.ONE), + NormalizedValue.Null, + ), + ) + + position.assert().isEqualTo(independent) + position.hashCode().assert().isEqualTo(independent.hashCode()) + } + + private fun manager( + secret: ByteArray = SECRET, + clock: Clock = Clock.fixed(NOW, ZoneOffset.UTC), + limits: QueryCursorLeaseLimits = QueryCursorLeaseLimits(), + keyId: Int = 1, + ) = InMemoryQueryCursorLeaseManager(secret, clock, limits, keyId) + + private fun envelope( + state: ByteArray = "pit-secret-value".toByteArray(), + expiresAt: Instant = NOW.plusSeconds(60), + fingerprint: String = "1".repeat(64), + ) = QueryCursorEnvelope( + TARGET, + PlanFingerprint(fingerprint), + QueryCursorMappingDigest("a".repeat(64)), + QueryCursorSecurityContextDigest("c".repeat(64)), + QueryCursorPosition.Analytics( + listOf(AnalyticsAlias("name")), + listOf(NormalizedValue.Text("alice")), + ), + expiresAt, + QueryCursorBackendState(BackendId("elasticsearch"), state), + ) + + private fun assertRejected( + category: QueryRejectionCategory, + code: QueryRejectionCode, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo("$.cursor") + }, + ) + } + + private class MutableClock( + private var current: Instant, + private val currentZone: ZoneId = ZoneOffset.UTC, + ) : Clock() { + override fun getZone(): ZoneId = currentZone + + override fun withZone(zone: ZoneId): Clock = MutableClock(current, zone) + + override fun instant(): Instant = current + + fun advance(duration: Duration) { + current = current.plus(duration) + } + } + + private companion object { + val NOW: Instant = Instant.parse("2026-08-08T00:00:00Z") + val SECRET = ByteArray(32) { 1 } + val TARGET = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/BoundedQueryShadowSupervisorTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/BoundedQueryShadowSupervisorTest.kt new file mode 100644 index 00000000000..57139bc25b0 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/BoundedQueryShadowSupervisorTest.kt @@ -0,0 +1,117 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.query.gateway.QueryShadowConfiguration +import me.ahoo.wow.query.gateway.QueryShadowObservation +import me.ahoo.wow.query.gateway.QueryShadowObserver +import me.ahoo.wow.query.gateway.QueryShadowOutcome +import me.ahoo.wow.query.internal.plan.PlanFingerprint +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.planning.PlanningFixtures +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.time.Duration +import java.util.concurrent.CopyOnWriteArrayList + +@OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) +class BoundedQueryShadowSupervisorTest { + @Test + fun `saturated probes should be rejected and capacity should be released after cancellation`() { + val observations = CopyOnWriteArrayList() + val supervisor = BoundedQueryShadowSupervisor( + QueryShadowConfiguration( + maxConcurrentProbes = 1, + maxComparedRecords = 10, + probeTimeout = Duration.ofSeconds(1), + ), + QueryShadowObserver(observations::add), + ) + val first = supervisor.submit(countTask(Mono.never())) as QueryShadowSubmission.Accepted + + val saturated = supervisor.submit(countTask(Mono.just(1))) as QueryShadowSubmission.Rejected + saturated.issue.code.assert().isEqualTo(QueryRejectionCode.SHADOW_SUPERVISOR_SATURATED) + observations.single().outcome.assert().isEqualTo(QueryShadowOutcome.SATURATED) + + first.handle.onPrimary(QueryShadowPrimarySignal.Cancelled) + first.handle.cancelProbe() + val accepted = supervisor.submit(countTask(Mono.just(3))) as QueryShadowSubmission.Accepted + accepted.handle.onPrimary(QueryShadowPrimarySignal.CountValue(3)) + accepted.handle.onPrimary(QueryShadowPrimarySignal.Complete) + + observations.map(QueryShadowObservation::outcome).assert().containsExactly( + QueryShadowOutcome.SATURATED, + QueryShadowOutcome.CANCELLED, + QueryShadowOutcome.MATCH, + ) + } + + @Test + fun `observer failure should remain isolated from comparison lifecycle`() { + val supervisor = BoundedQueryShadowSupervisor( + QueryShadowConfiguration(maxConcurrentProbes = 1), + QueryShadowObserver { error("observer unavailable") }, + ) + + val accepted = supervisor.submit(countTask(Mono.just(2))) as QueryShadowSubmission.Accepted + accepted.handle.onPrimary(QueryShadowPrimarySignal.CountValue(2)) + accepted.handle.onPrimary(QueryShadowPrimarySignal.Complete) + + val next = supervisor.submit(countTask(Mono.just(2))) + (next is QueryShadowSubmission.Accepted).assert().isTrue() + } + + @Test + fun `primary comparison should be bounded and cancel a slow probe exactly once`() { + val observations = CopyOnWriteArrayList() + val supervisor = BoundedQueryShadowSupervisor( + QueryShadowConfiguration(maxConcurrentProbes = 1, maxComparedRecords = 2), + QueryShadowObserver(observations::add), + ) + val accepted = supervisor.submit(streamTask(Flux.never())) as QueryShadowSubmission.Accepted + + repeat(100) { index -> + accepted.handle.onPrimary(QueryShadowPrimarySignal.RecordValue(record("order-$index"))) + } + accepted.handle.onPrimary(QueryShadowPrimarySignal.Complete) + + observations.assert().hasSize(1) + observations.single().outcome.assert().isEqualTo(QueryShadowOutcome.PROBE_ERROR) + observations.single().reasonCode.assert().isEqualTo(QueryRejectionCode.RESULT_LIMIT_EXCEEDED.name) + (supervisor.submit(countTask(Mono.just(1))) is QueryShadowSubmission.Accepted).assert().isTrue() + } + + private fun countTask(result: Mono): QueryShadowTask.Count = QueryShadowTask.Count( + target = PlanningFixtures.target, + fingerprint = PlanFingerprint("f".repeat(64)), + semanticTier = SemanticTier.PORTABLE, + publisher = result, + ) + + private fun streamTask(result: Flux): QueryShadowTask.Stream = QueryShadowTask.Stream( + target = PlanningFixtures.target, + fingerprint = PlanFingerprint("f".repeat(64)), + semanticTier = SemanticTier.PORTABLE, + publisher = result, + ) + + private fun record(identity: String): BackendRecord = BackendRecord( + identity, + me.ahoo.wow.query.backend.NormalizedValue.ObjectValue(emptyMap()), + BackendRecordCompleteness.COMPLETE, + ) +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/ExperimentalAnalyticsBackendAdapterTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/ExperimentalAnalyticsBackendAdapterTest.kt new file mode 100644 index 00000000000..8c7a97de715 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/ExperimentalAnalyticsBackendAdapterTest.kt @@ -0,0 +1,191 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.AnalyticsAlias +import me.ahoo.wow.query.backend.BackendAnalyticsBucket +import me.ahoo.wow.query.backend.BackendAnalyticsCompleteness +import me.ahoo.wow.query.backend.BackendAnalyticsConsistency +import me.ahoo.wow.query.backend.BackendAnalyticsGrouping +import me.ahoo.wow.query.backend.BackendAnalyticsMetric +import me.ahoo.wow.query.backend.BackendAnalyticsPage +import me.ahoo.wow.query.backend.BackendAnalyticsQueryPlan +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryBackendException +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryBackendFailureKind +import me.ahoo.wow.query.internal.analytics.AnalyticsDimension +import me.ahoo.wow.query.internal.analytics.AnalyticsGrouping +import me.ahoo.wow.query.internal.analytics.AnalyticsMetric +import me.ahoo.wow.query.internal.analytics.AnalyticsMissingPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPromotion +import me.ahoo.wow.query.internal.analytics.AnalyticsOverflowPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.plan.AnalyticsQueryPlan +import me.ahoo.wow.query.internal.planning.PlanningConstraints +import me.ahoo.wow.query.internal.planning.PlanningDecision +import me.ahoo.wow.query.internal.planning.PlanningFixtures +import me.ahoo.wow.query.internal.planning.QueryPlanner +import me.ahoo.wow.query.internal.policy.QueryExecutionBudget +import me.ahoo.wow.query.internal.value.NonEmptyList +import org.junit.jupiter.api.Test +import reactor.core.publisher.Mono +import java.math.RoundingMode +import java.time.Instant +import java.util.concurrent.atomic.AtomicReference +import me.ahoo.wow.query.backend.AnalyticsQueryBackend as BackendAnalyticsQueryBackend +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias as InternalAnalyticsAlias + +class ExperimentalAnalyticsBackendAdapterTest { + @Test + fun `should translate analytics plan options and result without leaking internal types`() { + val capturedPlan = AtomicReference() + val capturedOptions = AtomicReference() + val backend = BackendAnalyticsQueryBackend { plan, options -> + capturedPlan.set(plan) + capturedOptions.set(options) + Mono.just( + BackendAnalyticsPage( + listOf( + BackendAnalyticsBucket( + mapOf(AnalyticsAlias("amount") to NormalizedValue.Decimal("10.00".toBigDecimal())), + mapOf( + AnalyticsAlias("count") to NormalizedValue.Int64(2), + AnalyticsAlias("total") to NormalizedValue.Decimal("20.00".toBigDecimal()), + ), + ), + ), + listOf(NormalizedValue.Decimal("10.00".toBigDecimal())), + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + ), + ) + } + val deadline = Instant.parse("2026-08-08T00:00:00Z") + val options = QueryExecutionOptions( + deadline, + QueryExecutionBudget( + maxScannedRecords = 100, + maxReturnedRecords = 20, + maxPageWindow = 50, + maxCandidateBuckets = 30, + maxReturnedBuckets = 10, + maxCursorPages = 3, + allowDiskUse = true, + ), + ) + + val result = ExperimentalAnalyticsBackendAdapter( + backend, + PlanningFixtures.schema + ).analyze(plan(), options).block()!! + + val translated = capturedPlan.get() + translated.target.assert().isEqualTo(PlanningFixtures.target) + (translated.grouping as BackendAnalyticsGrouping.By).dimensions.single().alias.value.assert() + .isEqualTo("amount") + translated.metrics.filterIsInstance().single().field.assert() + .isEqualTo(PlanningFixtures.amount) + capturedOptions.get().assert().isEqualTo( + QueryBackendExecutionOptions(deadline, 20, 100, 50, 30, 10, 3, true), + ) + result.buckets.single().keys.keys.single().value.assert().isEqualTo("amount") + result.buckets.single().metrics.keys.map(InternalAnalyticsAlias::value).assert() + .containsExactly("count", "total") + result.afterKey!!.assert().containsExactly(NormalizedValue.Decimal("10".toBigDecimal())) + } + + @Test + fun `should reject malformed backend values as mapping failure before they leave the adapter`() { + val backend = BackendAnalyticsQueryBackend { _, _ -> + Mono.just( + BackendAnalyticsPage( + listOf( + BackendAnalyticsBucket( + mapOf(AnalyticsAlias("amount") to NormalizedValue.Bytes(byteArrayOf(1))), + mapOf(AnalyticsAlias("count") to NormalizedValue.Int64(-1)), + ), + ), + listOf(NormalizedValue.Bytes(byteArrayOf(1))), + BackendAnalyticsConsistency.EVENTUAL, + BackendAnalyticsCompleteness.EXACT, + ), + ) + } + + assertThrownBy { + ExperimentalAnalyticsBackendAdapter(backend, PlanningFixtures.schema) + .analyze(plan(), QueryExecutionOptions(null, QueryExecutionBudget())) + .block() + }.satisfies( + java.util.function.Consumer { error -> + error.kind.assert().isEqualTo(QueryBackendFailureKind.MAPPING_FAILURE) + }, + ) + } + + private fun plan(): AnalyticsQueryPlan { + val invocation = NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.ANALYZE, + QueryResultShape.ANALYTICS, + NormalizedQueryInput.Analytics( + AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.By( + NonEmptyList.of( + AnalyticsDimension( + InternalAnalyticsAlias("amount"), + PlanningFixtures.path("state", "amount"), + AnalyticsMissingPolicy.AS_NULL_BUCKET, + ), + ), + ), + NonEmptyList.of( + AnalyticsMetric.DocumentCount(InternalAnalyticsAlias("count")), + AnalyticsMetric.Sum( + InternalAnalyticsAlias("total"), + PlanningFixtures.path("state", "amount"), + ), + ), + numericPolicy = AnalyticsNumericPolicy( + AnalyticsNumericPromotion.DECIMAL128, + 34, + 2, + RoundingMode.HALF_UP, + AnalyticsOverflowPolicy.REJECT, + ), + ), + ), + ) + return ( + QueryPlanner().plan( + invocation, + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + ).plan as AnalyticsQueryPlan + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/LegacyQueryExecutionTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/LegacyQueryExecutionTest.kt new file mode 100644 index 00000000000..09c745897c8 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/LegacyQueryExecutionTest.kt @@ -0,0 +1,278 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.planning.PlanningConstraints +import me.ahoo.wow.query.internal.planning.PlanningDecision +import me.ahoo.wow.query.internal.planning.PlanningFixtures +import me.ahoo.wow.query.internal.planning.QueryPlanner +import me.ahoo.wow.query.internal.policy.QueryExecutionBudget +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.util.concurrent.atomic.AtomicInteger +import java.util.function.Consumer + +class LegacyQueryExecutionTest { + private val options = QueryExecutionOptions(null, QueryExecutionBudget()) + private val input = LegacyCompilationInput( + PlanningFixtures.single(resultShape = me.ahoo.wow.query.internal.model.QueryResultShape.DYNAMIC), + PlanningFixtures.schema, + QueryPlanner().plan( + PlanningFixtures.single(resultShape = me.ahoo.wow.query.internal.model.QueryResultShape.DYNAMIC), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ), + ) + + @Test + fun `legacy compiler and backend should both remain cold and execute once per subscription`() { + val compilerCalls = AtomicInteger() + val backendCalls = AtomicInteger() + val compiler = LegacyQueryCompiler { compilation -> + compilerCalls.incrementAndGet() + ProbeCompiledQuery( + compilation.invocation.target, + compilation.invocation.operation, + compilation.schema.contractId, + compilation.attestLowering( + compilation.enforcementRequirements.deletionScope, + compilation.enforcementRequirements.mandatoryCondition, + ), + ) + } + val backend = ProbeLegacyBackend { + backendCalls.incrementAndGet() + Mono.just(3) + } + val binding = LegacyExecutionBinding.create(PlanningFixtures.target, compiler, backend) + + val result = binding.count(countInput(), options) + compilerCalls.get().assert().isZero() + backendCalls.get().assert().isZero() + result.block().assert().isEqualTo(3) + result.block().assert().isEqualTo(3) + compilerCalls.get().assert().isEqualTo(2) + backendCalls.get().assert().isEqualTo(2) + } + + @Test + fun `legacy compiler mismatch and mandatory lowering failure should fail before backend`() { + val backendCalls = AtomicInteger() + val backend = ProbeLegacyBackend { + backendCalls.incrementAndGet() + Mono.just(3) + } + val otherTarget = QueryTarget( + me.ahoo.wow.modeling.MaterializedNamedAggregate("sales", "other"), + QueryDocumentKind.SNAPSHOT, + ) + val mismatched = LegacyExecutionBinding.create( + PlanningFixtures.target, + LegacyQueryCompiler { compilation -> + ProbeCompiledQuery( + otherTarget, + QueryOperation.COUNT, + PlanningFixtures.schema.contractId, + compilation.attestLowering( + compilation.enforcementRequirements.deletionScope, + compilation.enforcementRequirements.mandatoryCondition, + ), + ) + }, + backend, + ) + assertRejected(QueryRejectionCode.LEGACY_LOWERING_UNSUPPORTED) { + mismatched.count(countInput(), options).block() + } + backendCalls.get().assert().isZero() + + listOf>( + LegacyQueryCompiler { compilation -> + compilation.attestLowering( + me.ahoo.wow.query.internal.normalization.NormalizedDeletionScope.DEFAULT_ACTIVE, + compilation.enforcementRequirements.mandatoryCondition, + ) + error("unreachable") + }, + LegacyQueryCompiler { compilation -> + compilation.attestLowering( + compilation.enforcementRequirements.deletionScope, + me.ahoo.wow.query.internal.plan.PlannedCondition.None, + ) + error("unreachable") + }, + ).forEach { compiler -> + val mandatoryFailure = LegacyExecutionBinding.create(PlanningFixtures.target, compiler, backend) + assertRejected( + QueryRejectionCode.MANDATORY_CONDITION_UNENFORCEABLE, + QueryRejectionCategory.ACCESS_DENIED, + ) { + mandatoryFailure.count(countInput(), options).block() + }.also { error -> error.rejection.path.toString().assert().isEqualTo("$.constraints.mandatoryCondition") } + } + backendCalls.get().assert().isZero() + } + + @Test + fun `legacy compiled query should be bound to exactly one immutable compilation input`() { + val backendCalls = AtomicInteger() + var cached: ProbeCompiledQuery? = null + val binding = LegacyExecutionBinding.create( + PlanningFixtures.target, + LegacyQueryCompiler { compilation -> + cached ?: ProbeCompiledQuery( + compilation.invocation.target, + compilation.invocation.operation, + compilation.schema.contractId, + compilation.attestLowering( + compilation.enforcementRequirements.deletionScope, + compilation.enforcementRequirements.mandatoryCondition, + ), + ).also { compiled -> cached = compiled } + }, + ProbeLegacyBackend { + backendCalls.incrementAndGet() + Mono.just(3) + }, + ) + + binding.count(countInput(), options).block().assert().isEqualTo(3) + assertRejected(QueryRejectionCode.LEGACY_LOWERING_UNSUPPORTED) { + binding.count(countInput(), options).block() + } + backendCalls.get().assert().isEqualTo(1) + } + + @Test + fun `legacy execution should reject backend-only budgets before compilation and storage`() { + val compilerCalls = AtomicInteger() + val backendCalls = AtomicInteger() + val binding = LegacyExecutionBinding.create( + PlanningFixtures.target, + LegacyQueryCompiler { compilation -> + compilerCalls.incrementAndGet() + ProbeCompiledQuery( + compilation.invocation.target, + compilation.invocation.operation, + compilation.schema.contractId, + compilation.attestLowering( + compilation.enforcementRequirements.deletionScope, + compilation.enforcementRequirements.mandatoryCondition, + ), + ) + }, + ProbeLegacyBackend { + backendCalls.incrementAndGet() + Mono.just(3) + }, + ) + val unsupported = QueryExecutionOptions( + null, + QueryExecutionBudget(maxScannedRecords = 1), + ) + + assertRejected(QueryRejectionCode.EXECUTION_BUDGET_UNSUPPORTED) { + binding.count(countInput(), unsupported).block() + }.rejection.path.toString().assert().isEqualTo("$.executionContext.budget") + compilerCalls.get().assert().isZero() + backendCalls.get().assert().isZero() + } + + @Test + fun `legacy compilation input should expose normalized immutable state instead of raw DTO`() { + LegacyCompilationInput::class.java.declaredFields.map { field -> field.type }.assert() + .doesNotContain(me.ahoo.wow.query.internal.model.QueryInvocation::class.java) + .doesNotContain(me.ahoo.wow.api.query.Condition::class.java) + .doesNotContain(Any::class.java) + input.invocation.input.assert() + .isInstanceOf(me.ahoo.wow.query.internal.normalization.NormalizedQueryInput.Single::class.java) + input.decision.assert().isInstanceOf(PlanningDecision.Planned::class.java) + } + + private fun countInput(): LegacyCompilationInput { + val invocation = me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.COUNT, + me.ahoo.wow.query.internal.model.QueryResultShape.COUNT, + me.ahoo.wow.query.internal.normalization.NormalizedQueryInput.Count( + me.ahoo.wow.query.internal.normalization.NormalizedCondition.All, + me.ahoo.wow.query.internal.normalization.NormalizedDeletionScope.EXPLICIT, + ), + ) + return LegacyCompilationInput( + invocation, + PlanningFixtures.schema, + QueryPlanner().plan( + invocation, + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ), + ) + } + + private fun assertRejected( + code: QueryRejectionCode, + category: QueryRejectionCategory = QueryRejectionCategory.UNSUPPORTED_FEATURE, + action: () -> Any?, + ): QueryRejectedException { + var captured: QueryRejectedException? = null + assertThrownBy { action() }.satisfies( + Consumer { error -> + captured = error + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + }, + ) + return checkNotNull(captured) + } + + private data class ProbeCompiledQuery( + override val target: QueryTarget, + override val operation: QueryOperation, + override val schemaContractId: SchemaContractId, + override val loweringAttestation: LegacyLoweringAttestation, + ) : LegacyCompiledQuery + + private class ProbeLegacyBackend( + private val countAction: () -> Mono, + ) : LegacyQueryBackend { + override fun single( + query: ProbeCompiledQuery, + options: QueryExecutionOptions, + ): Mono = Mono.empty() + + override fun stream( + query: ProbeCompiledQuery, + options: QueryExecutionOptions, + ): Flux = Flux.empty() + + override fun page( + query: ProbeCompiledQuery, + options: QueryExecutionOptions, + ): Mono = Mono.empty() + + override fun count(query: ProbeCompiledQuery, options: QueryExecutionOptions): Mono = countAction() + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryBackendRegistryTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryBackendRegistryTest.kt new file mode 100644 index 00000000000..48c35e34c80 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryBackendRegistryTest.kt @@ -0,0 +1,224 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.RecordResultShape +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.Utf8Json +import me.ahoo.wow.query.internal.plan.EnforcedFilter +import me.ahoo.wow.query.internal.plan.PlannedCondition +import me.ahoo.wow.query.internal.plan.PlannedProjection +import me.ahoo.wow.query.internal.plan.RequiredCapabilities +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.plan.SingleQueryPlan +import me.ahoo.wow.query.internal.planning.PlanningFixtures +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.util.function.Consumer + +class QueryBackendRegistryTest { + private val backendId = BackendId("mongo") + private val backend = EmptyRecordBackend() + + @Test + fun `registry should defensively copy routes and capabilities and resolve exact plan`() { + val operations = mutableSetOf(QueryOperation.SINGLE) + val semanticTiers = mutableSetOf(SemanticTier.PORTABLE) + val innerCapabilities = mutableSetOf(FieldCapability.EXACT) + val searchScopes = mutableSetOf(PlanningFixtures.searchScopeId) + val fieldCapabilities = mutableMapOf>( + PlanningFixtures.name to innerCapabilities, + ) + val routes = mutableMapOf(PlanningFixtures.target to backendId) + val registration = registration( + fieldCapabilities, + operations = operations, + semanticTiers = semanticTiers, + searchScopes = searchScopes, + ) + val registry = QueryBackendRegistry(listOf(registration), routes) + + operations.clear() + semanticTiers.clear() + innerCapabilities.clear() + searchScopes.clear() + fieldCapabilities.clear() + routes.clear() + + registry.resolve(plan()).assert().isSameAs(registration) + registration.descriptor.supportedOperations.assert().containsExactly(QueryOperation.SINGLE) + registration.descriptor.semanticTiers.assert().containsExactly(SemanticTier.PORTABLE) + registration.descriptor.searchScopes.assert().containsExactly(PlanningFixtures.searchScopeId) + registration.descriptor.fieldCapabilities[PlanningFixtures.name].assert() + .containsExactly(FieldCapability.EXACT) + registry.defaultRoutes.assert().hasSize(1) + registry.registrations.assert().hasSize(1) + assertThrownBy { + (registration.descriptor.supportedOperations as MutableSet).clear() + } + assertThrownBy { + (registration.descriptor.fieldCapabilities[PlanningFixtures.name] as MutableSet).clear() + } + } + + @Test + fun `registry should reject duplicate missing schema operation and capability routes`() { + val registration = registration(mapOf(PlanningFixtures.name to setOf(FieldCapability.EXACT))) + assertThrownBy { + QueryBackendRegistry(listOf(registration, registration), mapOf(PlanningFixtures.target to backendId)) + } + + assertRejected(QueryRejectionCode.BACKEND_NOT_REGISTERED) { + QueryBackendRegistry(emptyList(), emptyMap()).resolve(plan()) + } + assertRejected(QueryRejectionCode.BACKEND_NOT_READY) { + QueryBackendRegistry( + emptyList(), + mapOf(PlanningFixtures.target to backendId), + setOf(QueryBackendKey(PlanningFixtures.target, backendId)), + ).resolve(plan()) + } + assertRejected(QueryRejectionCode.BACKEND_SCHEMA_MISMATCH) { + val wrongSchema = registration( + mapOf(PlanningFixtures.name to setOf(FieldCapability.EXACT)), + schemaContractId = me.ahoo.wow.query.backend.SchemaContractId("0".repeat(64)), + ) + QueryBackendRegistry(listOf(wrongSchema), mapOf(PlanningFixtures.target to backendId)).resolve(plan()) + } + assertRejected(QueryRejectionCode.BACKEND_OPERATION_UNSUPPORTED) { + val countOnly = registration( + mapOf(PlanningFixtures.name to setOf(FieldCapability.EXACT)), + operations = setOf(QueryOperation.COUNT), + ) + QueryBackendRegistry(listOf(countOnly), mapOf(PlanningFixtures.target to backendId)).resolve(plan()) + } + assertRejected(QueryRejectionCode.BACKEND_CAPABILITY_MISMATCH) { + val missingCapability = registration(emptyMap()) + QueryBackendRegistry(listOf(missingCapability), mapOf(PlanningFixtures.target to backendId)).resolve(plan()) + } + } + + @Test + fun `native requirement should pin the backend instead of using the default route`() { + val elasticsearchId = BackendId("elasticsearch") + val defaultRegistration = QueryBackendRegistration( + QueryBackendDescriptor( + QueryBackendKey(PlanningFixtures.target, elasticsearchId), + PlanningFixtures.schema.contractId, + setOf(QueryOperation.SINGLE), + setOf(SemanticTier.PORTABLE), + emptyMap(), + ), + recordBackend = backend, + ) + val nativeRegistration = QueryBackendRegistration( + QueryBackendDescriptor( + QueryBackendKey(PlanningFixtures.target, backendId), + PlanningFixtures.schema.contractId, + setOf(QueryOperation.SINGLE), + setOf(SemanticTier.NATIVE), + emptyMap(), + ), + recordBackend = backend, + ) + val nativePlan = SingleQueryPlan.create( + PlanningFixtures.target, + PlanningFixtures.schema.contractId, + EnforcedFilter( + PlannedCondition.Native(backendId, Utf8Json("{}")), + PlannedCondition.All, + ), + RecordResultShape.DYNAMIC, + PlannedProjection.All, + emptyList(), + RequiredCapabilities(nativeBackend = backendId), + SemanticTier.NATIVE, + ) + val registry = QueryBackendRegistry( + listOf(defaultRegistration, nativeRegistration), + mapOf(PlanningFixtures.target to elasticsearchId), + ) + + registry.resolve(nativePlan).assert().isSameAs(nativeRegistration) + } + + private fun registration( + capabilities: Map>, + schemaContractId: me.ahoo.wow.query.backend.SchemaContractId = PlanningFixtures.schema.contractId, + operations: Set = setOf(QueryOperation.SINGLE), + semanticTiers: Set = setOf(SemanticTier.PORTABLE), + searchScopes: Set = emptySet(), + ): QueryBackendRegistration = QueryBackendRegistration( + QueryBackendDescriptor( + QueryBackendKey(PlanningFixtures.target, backendId), + schemaContractId, + operations, + semanticTiers, + capabilities, + searchScopes, + ), + recordBackend = backend, + ) + + private fun plan(): SingleQueryPlan = SingleQueryPlan.create( + PlanningFixtures.target, + PlanningFixtures.schema.contractId, + EnforcedFilter(PlannedCondition.All, PlannedCondition.All), + RecordResultShape.DYNAMIC, + PlannedProjection.All, + emptyList(), + RequiredCapabilities(mapOf(PlanningFixtures.name to setOf(FieldCapability.EXACT))), + SemanticTier.PORTABLE, + ) + + private fun assertRejected(code: QueryRejectionCode, action: () -> Any?) { + assertThrownBy { action() }.satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(QueryRejectionCategory.BACKEND_UNAVAILABLE) + error.rejection.path.toString().assert().isEqualTo("$.backend") + error.rejection.code.assert().isEqualTo(code) + }, + ) + } + + private class EmptyRecordBackend : RecordQueryBackend { + override fun single( + plan: SingleQueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.empty() + + override fun stream( + plan: me.ahoo.wow.query.internal.plan.StreamQueryPlan, + options: QueryExecutionOptions, + ): Flux = Flux.empty() + + override fun page( + plan: me.ahoo.wow.query.internal.plan.PageQueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.empty() + + override fun count( + plan: me.ahoo.wow.query.internal.plan.CountQueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.empty() + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryBackendResultTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryBackendResultTest.kt new file mode 100644 index 00000000000..07c4ff67397 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryBackendResultTest.kt @@ -0,0 +1,99 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsCompleteness +import me.ahoo.wow.query.internal.analytics.AnalyticsConsistency +import org.junit.jupiter.api.Test + +class QueryBackendResultTest { + @Test + fun `record and page must remain immutable with stable value semantics`() { + val documentSource = linkedMapOf( + "payload" to NormalizedValue.Bytes(byteArrayOf(1, 2, 3)), + ) + val record = BackendRecord( + "order-1", + NormalizedValue.ObjectValue(documentSource), + BackendRecordCompleteness.COMPLETE, + ) + val recordSource = mutableListOf(record) + val page = BackendPage( + recordSource, + 1, + BackendTotalRelation.EXACT, + BackendPageConsistency.SAME_INPUT, + ) + val originalHash = page.hashCode() + + documentSource.clear() + recordSource.clear() + + page.records.assert().containsExactly(record) + page.hashCode().assert().isEqualTo(originalHash) + page.assert().isEqualTo( + BackendPage( + listOf(record), + 1, + BackendTotalRelation.EXACT, + BackendPageConsistency.SAME_INPUT, + ), + ) + assertThrownBy { + (page.records as MutableList).add(record) + } + } + + @Test + fun `analytics bucket and page must defensively copy every collection boundary`() { + val keyAlias = AnalyticsAlias("group") + val metricAlias = AnalyticsAlias("count") + val keySource = linkedMapOf(keyAlias to NormalizedValue.Text("A")) + val metricSource = linkedMapOf(metricAlias to NormalizedValue.Int64(1)) + val bucket = BackendAnalyticsBucket(keySource, metricSource) + val bucketSource = mutableListOf(bucket) + val afterKeySource = mutableListOf(NormalizedValue.Text("A")) + val page = BackendAnalyticsPage( + bucketSource, + afterKeySource, + AnalyticsConsistency.SNAPSHOT, + AnalyticsCompleteness.EXACT, + ) + val originalHash = page.hashCode() + + keySource.clear() + metricSource.clear() + bucketSource.clear() + afterKeySource.clear() + + bucket.keys.assert().containsEntry(keyAlias, NormalizedValue.Text("A")) + bucket.metrics.assert().containsEntry(metricAlias, NormalizedValue.Int64(1)) + page.buckets.assert().containsExactly(bucket) + page.afterKey.assert().containsExactly(NormalizedValue.Text("A")) + page.hashCode().assert().isEqualTo(originalHash) + assertThrownBy { + (bucket.keys as MutableMap)[keyAlias] = NormalizedValue.Text("B") + } + assertThrownBy { + (page.buckets as MutableList).clear() + } + assertThrownBy { + (page.afterKey as MutableList).clear() + } + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutionRouteResolverTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutionRouteResolverTest.kt new file mode 100644 index 00000000000..06c02ccace9 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutionRouteResolverTest.kt @@ -0,0 +1,281 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsGrouping +import me.ahoo.wow.query.internal.analytics.AnalyticsMetric +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.model.QueryExecutionMode +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.plan.RequiredCapabilities +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.planning.PlanningConstraints +import me.ahoo.wow.query.internal.planning.PlanningDecision +import me.ahoo.wow.query.internal.planning.PlanningFixtures +import me.ahoo.wow.query.internal.planning.QueryPlanner +import me.ahoo.wow.query.internal.planning.ValidatedMandatory +import me.ahoo.wow.query.internal.policy.QueryAuthority +import me.ahoo.wow.query.internal.policy.QueryExecutionBudget +import me.ahoo.wow.query.internal.policy.QueryExecutionContext +import me.ahoo.wow.query.internal.policy.QueryPurpose +import me.ahoo.wow.query.internal.policy.QueryResourceScope +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejection +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.value.NonEmptyList +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.util.function.Consumer + +class QueryExecutionRouteResolverTest { + private val backendId = BackendId("mongo") + private val planned = QueryPlanner().plan( + PlanningFixtures.single(resultShape = me.ahoo.wow.query.internal.model.QueryResultShape.DYNAMIC), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + private val registration = QueryBackendRegistration( + QueryBackendDescriptor( + QueryBackendKey(PlanningFixtures.target, backendId), + PlanningFixtures.schema.contractId, + setOf(QueryOperation.SINGLE), + setOf(SemanticTier.PORTABLE), + PlanningFixtures.schema.fields.mapValues { entry -> entry.value.capabilities }, + PlanningFixtures.schema.searchScopes.keys, + ), + recordBackend = EmptyRecordBackend(), + ) + private val legacy = emptyLegacyBinding() + private val resolver = QueryExecutionRouteResolver( + QueryBackendRegistry(listOf(registration), mapOf(PlanningFixtures.target to backendId)), + LegacyBackendRegistry(listOf(legacy)), + ) + + @Test + fun `resolver should implement the execution mode and compatibility matrix`() { + resolve(QueryExecutionMode.LEGACY, QueryValidationMode.COMPATIBLE, planned).assert() + .isInstanceOf(QueryExecutionRoute.Legacy::class.java) + resolve(QueryExecutionMode.LEGACY, QueryValidationMode.STRICT, planned).assert() + .isInstanceOf(QueryExecutionRoute.Legacy::class.java) + resolve(QueryExecutionMode.SHADOW, QueryValidationMode.COMPATIBLE, planned).assert() + .isInstanceOf(QueryExecutionRoute.Shadow::class.java) + resolve(QueryExecutionMode.SHADOW, QueryValidationMode.STRICT, planned).assert() + .isInstanceOf(QueryExecutionRoute.Shadow::class.java) + resolve(QueryExecutionMode.PLANNED, QueryValidationMode.COMPATIBLE, planned).assert() + .isInstanceOf(QueryExecutionRoute.Planned::class.java) + resolve(QueryExecutionMode.PLANNED, QueryValidationMode.STRICT, planned).assert() + .isInstanceOf(QueryExecutionRoute.Planned::class.java) + + QueryExecutionMode.entries.forEach { mode -> + val route = resolve(mode, QueryValidationMode.COMPATIBLE, fallback()) as QueryExecutionRoute.Legacy + route.fallback.assert().isNotNull() + val fallback = checkNotNull(route.fallback) + fallback.executionMode.assert().isEqualTo(mode) + fallback.target.assert().isEqualTo(PlanningFixtures.target) + fallback.operation.assert().isEqualTo(QueryOperation.SINGLE) + } + } + + @Test + fun `strict fallback should be an invariant failure in every execution mode`() { + QueryExecutionMode.entries.forEach { mode -> + assertRejected(QueryRejectionCategory.INTERNAL_FAILURE, QueryRejectionCode.EXECUTION_DECISION_INVALID) { + resolve(mode, QueryValidationMode.STRICT, fallback()) + } + } + } + + @Test + fun `planned route failure should never become legacy fallback`() { + val missingPlannedResolver = QueryExecutionRouteResolver( + QueryBackendRegistry(emptyList(), emptyMap()), + LegacyBackendRegistry(listOf(legacy)), + ) + + val missingPlanned = missingPlannedResolver.resolve( + context(QueryExecutionMode.PLANNED, QueryValidationMode.COMPATIBLE), + invocation(), + PlanningFixtures.schema, + planned, + ) as QueryExecutionRoute.Planned + assertRejected(QueryRejectionCategory.BACKEND_UNAVAILABLE, QueryRejectionCode.BACKEND_NOT_REGISTERED) { + missingPlanned.registry.resolve(missingPlanned.plan) + } + missingPlannedResolver.resolve( + context(QueryExecutionMode.SHADOW, QueryValidationMode.COMPATIBLE), + invocation(), + PlanningFixtures.schema, + planned, + ).assert().isInstanceOf(QueryExecutionRoute.Shadow::class.java) + } + + @Test + fun `analytics should never enter legacy or shadow execution`() { + val analytics = NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.ANALYZE, + me.ahoo.wow.query.internal.model.QueryResultShape.ANALYTICS, + NormalizedQueryInput.Analytics( + AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.Global, + NonEmptyList.of(AnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + ), + ), + ) + listOf(QueryExecutionMode.LEGACY, QueryExecutionMode.SHADOW, QueryExecutionMode.PLANNED).forEach { mode -> + assertRejected(QueryRejectionCategory.UNSUPPORTED_FEATURE, QueryRejectionCode.EXECUTION_MODE_UNSUPPORTED) { + resolver.resolve( + context(mode, QueryValidationMode.COMPATIBLE), + analytics, + PlanningFixtures.schema, + fallback(), + ) + } + } + } + + private fun resolve( + mode: QueryExecutionMode, + validationMode: QueryValidationMode, + decision: PlanningDecision, + ): QueryExecutionRoute = resolver.resolve( + context(mode, validationMode), + invocation(), + PlanningFixtures.schema, + decision, + ) + + private fun invocation() = PlanningFixtures.single( + resultShape = me.ahoo.wow.query.internal.model.QueryResultShape.DYNAMIC, + ) + + private fun fallback(): PlanningDecision.LegacyFallback = PlanningDecision.LegacyFallback( + NonEmptyList.of( + QueryRejection( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionPath.ROOT.property("input"), + QueryRejectionCode.CAPABILITY_UNAVAILABLE, + ), + ), + ValidatedMandatory( + PlanningFixtures.target, + PlanningFixtures.schema.contractId, + me.ahoo.wow.query.internal.plan.PlannedCondition.All, + RequiredCapabilities(), + SemanticTier.PORTABLE, + ), + ) + + private fun context( + mode: QueryExecutionMode, + validationMode: QueryValidationMode, + ): QueryExecutionContext = QueryExecutionContext( + PlanningFixtures.target, + QueryPurpose("test"), + QueryAuthority.System("test", "execution-route-test"), + mode, + validationMode, + QueryResourceScope(), + deadline = null, + QueryExecutionBudget(), + ) + + private fun assertRejected( + category: QueryRejectionCategory, + code: QueryRejectionCode, + action: () -> Any?, + ) { + assertThrownBy { action() }.satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + }, + ) + } + + private fun emptyLegacyBinding(): LegacyExecutionBinding = LegacyExecutionBinding.create( + PlanningFixtures.target, + LegacyQueryCompiler { input -> + RouteCompiledQuery( + input.invocation.target, + input.invocation.operation, + input.schema.contractId, + input.attestLowering( + input.enforcementRequirements.deletionScope, + input.enforcementRequirements.mandatoryCondition, + ), + ) + }, + object : LegacyQueryBackend { + override fun single( + query: RouteCompiledQuery, + options: QueryExecutionOptions, + ): Mono = Mono.empty() + + override fun stream( + query: RouteCompiledQuery, + options: QueryExecutionOptions, + ): Flux = Flux.empty() + + override fun page( + query: RouteCompiledQuery, + options: QueryExecutionOptions, + ): Mono = Mono.empty() + + override fun count(query: RouteCompiledQuery, options: QueryExecutionOptions): Mono = Mono.empty() + }, + ) + + private data class RouteCompiledQuery( + override val target: me.ahoo.wow.query.internal.model.QueryTarget, + override val operation: QueryOperation, + override val schemaContractId: SchemaContractId, + override val loweringAttestation: LegacyLoweringAttestation, + ) : LegacyCompiledQuery + + private class EmptyRecordBackend : RecordQueryBackend { + override fun single( + plan: me.ahoo.wow.query.internal.plan.SingleQueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.empty() + + override fun stream( + plan: me.ahoo.wow.query.internal.plan.StreamQueryPlan, + options: QueryExecutionOptions, + ): Flux = Flux.empty() + + override fun page( + plan: me.ahoo.wow.query.internal.plan.PageQueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.empty() + + override fun count( + plan: me.ahoo.wow.query.internal.plan.CountQueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.empty() + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutorInvariantTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutorInvariantTest.kt new file mode 100644 index 00000000000..f6c736527b5 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutorInvariantTest.kt @@ -0,0 +1,606 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsCompleteness +import me.ahoo.wow.query.internal.analytics.AnalyticsConsistency +import me.ahoo.wow.query.internal.analytics.AnalyticsDimension +import me.ahoo.wow.query.internal.analytics.AnalyticsGrouping +import me.ahoo.wow.query.internal.analytics.AnalyticsMetric +import me.ahoo.wow.query.internal.analytics.AnalyticsMissingPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.model.QueryExecutionMode +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.plan.AnalyticsQueryPlan +import me.ahoo.wow.query.internal.plan.CountQueryPlan +import me.ahoo.wow.query.internal.plan.PageQueryPlan +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.plan.SingleQueryPlan +import me.ahoo.wow.query.internal.plan.StreamQueryPlan +import me.ahoo.wow.query.internal.planning.PlanningConstraints +import me.ahoo.wow.query.internal.planning.PlanningDecision +import me.ahoo.wow.query.internal.planning.PlanningFixtures +import me.ahoo.wow.query.internal.planning.QueryPlanner +import me.ahoo.wow.query.internal.policy.QueryExecutionBudget +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejection +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import me.ahoo.wow.query.internal.value.NonEmptyList +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.core.scheduler.Schedulers +import reactor.test.StepVerifier +import java.time.Clock +import java.util.concurrent.atomic.AtomicInteger +import java.util.concurrent.atomic.AtomicReference +import java.util.function.Consumer + +class QueryExecutorInvariantTest { + private val planner = QueryPlanner() + private val options = QueryExecutionOptions(null, QueryExecutionBudget()) + private val executor = QueryExecutor(QueryDeadlineEnforcer(Clock.systemUTC(), Schedulers.immediate())) + private val record = BackendRecord( + "order-1", + NormalizedValue.ObjectValue(emptyMap()), + BackendRecordCompleteness.COMPLETE, + ) + + @Test + fun `page count and analytics must emit exactly one result`() { + val countInput = countInput() + val pageInput = pageInput(size = 1) + val analyticsPlan = analyticsPlan(limit = 1) + + listOf( + { executor.count(plannedRoute(countInput.decision, countBackend = { Mono.empty() }), options).block() }, + { executor.count(legacyRoute(countInput, countResult = Mono.empty()), options).block() }, + { executor.page(plannedRoute(pageInput.decision, pageBackend = { Mono.empty() }), options).block() }, + { executor.page(legacyRoute(pageInput, pageResult = Mono.empty()), options).block() }, + { + executor.analyze( + plannedAnalyticsRoute(analyticsPlan) { Mono.empty() }, + options, + ).block() + }, + ).forEach { action -> assertIncomplete(action) } + } + + @Test + fun `page and stream result envelopes must honor plan cardinality`() { + val pageInput = pageInput(size = 1) + val oversizedPage = BackendPage( + listOf(record, record.copy(identity = "order-2")), + 2, + BackendTotalRelation.EXACT, + BackendPageConsistency.SAME_INPUT, + ) + assertIncomplete { + executor.page(plannedRoute(pageInput.decision, pageBackend = { Mono.just(oversizedPage) }), options).block() + } + assertIncomplete { + executor.page(legacyRoute(pageInput, pageResult = Mono.just(oversizedPage)), options).block() + } + + val inconsistentTotal = BackendPage( + listOf(record), + 0, + BackendTotalRelation.EXACT, + BackendPageConsistency.SAME_INPUT, + ) + assertIncomplete { + executor.page(plannedRoute(pageInput.decision, pageBackend = { Mono.just(inconsistentTotal) }), options) + .block() + } + + val streamDecision = planner.plan( + normalizedStream(limit = 1), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + StepVerifier.create( + executor.stream( + plannedRoute(streamDecision, streamBackend = { Flux.just(record, record.copy(identity = "order-2")) }), + options, + ), + ) + .expectNext(record) + .expectErrorSatisfies(::assertIncompleteError) + .verify() + } + + @Test + fun `analytics result envelope must honor plan consistency completeness and shape`() { + val analyticsPlan = analyticsPlan(limit = 1) + val bucket = BackendAnalyticsBucket( + emptyMap(), + mapOf(AnalyticsAlias("count") to NormalizedValue.Int64(1)), + ) + assertIncomplete { + executor.analyze( + plannedAnalyticsRoute(analyticsPlan) { + Mono.just( + BackendAnalyticsPage( + listOf(bucket, bucket), + null, + AnalyticsConsistency.EVENTUAL, + AnalyticsCompleteness.EXACT, + ), + ) + }, + options, + ).block() + } + + val snapshotPlan = analyticsPlan(limit = 1, consistency = AnalyticsConsistency.SNAPSHOT) + listOf( + BackendAnalyticsPage( + listOf(bucket), + null, + AnalyticsConsistency.EVENTUAL, + AnalyticsCompleteness.EXACT, + ), + BackendAnalyticsPage( + listOf(bucket), + null, + AnalyticsConsistency.SNAPSHOT, + AnalyticsCompleteness.APPROXIMATE, + ), + BackendAnalyticsPage( + listOf(bucket), + listOf(NormalizedValue.Text("unexpected-global-cursor")), + AnalyticsConsistency.SNAPSHOT, + AnalyticsCompleteness.EXACT, + ), + BackendAnalyticsPage( + listOf(bucket), + emptyList(), + AnalyticsConsistency.SNAPSHOT, + AnalyticsCompleteness.EXACT, + ), + ).forEach { invalidPage -> + assertIncomplete { + executor.analyze(plannedAnalyticsRoute(snapshotPlan) { Mono.just(invalidPage) }, options).block() + } + } + } + + @Test + fun `negative count must be rejected in every execution mode`() { + val countInput = countInput() + assertIncomplete { + executor.count(plannedRoute(countInput.decision, countBackend = { Mono.just(-1) }), options).block() + } + assertIncomplete { + executor.count(legacyRoute(countInput, countResult = Mono.just(-1)), options).block() + } + val shadow = QueryExecutionRoute.Shadow( + legacyRoute(countInput, countResult = Mono.just(-1)).binding, + countInput, + registry(countInput.decision, countBackend = { Mono.just(1) }), + (countInput.decision as PlanningDecision.Planned).plan, + ) + assertIncomplete { executor.count(shadow, options).block() } + } + + @Test + fun `grouped analytics must bind exact dimension aliases and cursor arity`() { + val plan = groupedAnalyticsPlan() + val dimensionAlias = AnalyticsAlias("amount") + val metricAlias = AnalyticsAlias("count") + val key = NormalizedValue.Decimal(java.math.BigDecimal.TEN) + val validBucket = BackendAnalyticsBucket( + mapOf(dimensionAlias to key), + mapOf(metricAlias to NormalizedValue.Int64(1)), + ) + val valid = BackendAnalyticsPage( + listOf(validBucket), + listOf(key), + AnalyticsConsistency.EVENTUAL, + AnalyticsCompleteness.EXACT, + ) + executor.analyze(plannedAnalyticsRoute(plan) { Mono.just(valid) }, options).block() + .assert().isEqualTo(valid) + + listOf( + BackendAnalyticsPage( + listOf(BackendAnalyticsBucket(emptyMap(), mapOf(metricAlias to NormalizedValue.Int64(1)))), + listOf(key), + AnalyticsConsistency.EVENTUAL, + AnalyticsCompleteness.EXACT, + ), + BackendAnalyticsPage( + listOf(validBucket), + emptyList(), + AnalyticsConsistency.EVENTUAL, + AnalyticsCompleteness.EXACT, + ), + BackendAnalyticsPage( + listOf(validBucket), + listOf(key, key), + AnalyticsConsistency.EVENTUAL, + AnalyticsCompleteness.EXACT, + ), + ).forEach { invalid -> + assertIncomplete { executor.analyze(plannedAnalyticsRoute(plan) { Mono.just(invalid) }, options).block() } + } + } + + @Test + fun `exact same-input page should return precisely the remaining window`() { + listOf( + PageSuccessCase( + index = 1, + offset = 0, + total = 3, + records = listOf(record, record.copy(identity = "order-2")) + ), + PageSuccessCase(index = 2, offset = 2, total = 3, records = listOf(record)), + PageSuccessCase(index = 3, offset = 4, total = 3, records = emptyList()), + ).forEach { case -> + val pageInput = pageInput(size = 2, index = case.index, offset = case.offset) + val page = BackendPage( + case.records, + case.total, + BackendTotalRelation.EXACT, + BackendPageConsistency.SAME_INPUT, + ) + executor.page(plannedRoute(pageInput.decision, pageBackend = { Mono.just(page) }), options).block() + .assert().isEqualTo(page) + } + } + + @Test + fun `unbounded shadow stream should skip probe without resolving planned registry`() { + val invocation = normalizedStream(limit = 0) + val input = compilationInput(invocation) + val legacy = legacyRoute(input) + val submitted = AtomicInteger() + val skipped = AtomicReference() + val supervisor = object : QueryShadowSupervisor { + override fun submit(task: QueryShadowTask): QueryShadowSubmission { + submitted.incrementAndGet() + return QueryShadowSubmission.Accepted(QueryShadowHandle.NONE) + } + + override fun onSkipped(skip: QueryShadowSkip) { + skipped.set(skip) + } + } + val shadow = QueryExecutionRoute.Shadow( + legacy.binding, + input, + QueryBackendRegistry(emptyList(), emptyMap()), + (input.decision as PlanningDecision.Planned).plan, + ) + + QueryExecutor( + QueryDeadlineEnforcer(Clock.systemUTC(), Schedulers.immediate()), + shadowSupervisor = supervisor, + ).stream(shadow, options).collectList().block().assert().isEmpty() + submitted.get().assert().isZero() + skipped.get().issues.first.code.assert().isEqualTo(QueryRejectionCode.SHADOW_PROBE_UNBOUNDED_STREAM) + } + + @Test + fun `compatible fallback should be observable in every execution mode and only shadow should report a skipped probe`() { + val input = countInput() + val issue = QueryRejection( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionPath.ROOT.property("input"), + QueryRejectionCode.CAPABILITY_UNAVAILABLE, + ) + val submitted = AtomicInteger() + val skipped = mutableListOf() + val fallbacks = mutableListOf() + val supervisor = object : QueryShadowSupervisor { + override fun submit(task: QueryShadowTask): QueryShadowSubmission { + submitted.incrementAndGet() + return QueryShadowSubmission.Accepted(QueryShadowHandle.NONE) + } + + override fun onSkipped(skip: QueryShadowSkip) { + skipped += skip + } + } + val observer = object : QueryDecisionObserver { + override fun onFallback(fallback: QueryFallback) { + fallbacks += fallback + } + } + val executor = QueryExecutor( + QueryDeadlineEnforcer(Clock.systemUTC(), Schedulers.immediate()), + shadowSupervisor = supervisor, + decisionObserver = observer, + ) + + QueryExecutionMode.entries.forEach { executionMode -> + val route = legacyRoute(input, countResult = Mono.just(1L)).copy( + fallback = QueryFallback( + input.invocation.target, + input.invocation.operation, + executionMode, + NonEmptyList.of(issue), + ), + ) + executor.count(route, options).block().assert().isEqualTo(1L) + } + + submitted.get().assert().isZero() + fallbacks.map(QueryFallback::executionMode).assert().containsExactly(*QueryExecutionMode.entries.toTypedArray()) + skipped.assert().hasSize(1) + skipped.single().issues.assert().isEqualTo(NonEmptyList.of(issue)) + } + + private fun plannedRoute( + decision: PlanningDecision, + countBackend: (CountQueryPlan) -> Mono = { Mono.empty() }, + pageBackend: (PageQueryPlan) -> Mono = { Mono.empty() }, + streamBackend: (StreamQueryPlan) -> Flux = { Flux.empty() }, + ): QueryExecutionRoute.Planned { + val planned = decision as PlanningDecision.Planned + return QueryExecutionRoute.Planned( + registry(decision, countBackend, pageBackend, streamBackend), + planned.plan, + ) + } + + private fun registry( + decision: PlanningDecision, + countBackend: (CountQueryPlan) -> Mono = { Mono.empty() }, + pageBackend: (PageQueryPlan) -> Mono = { Mono.empty() }, + streamBackend: (StreamQueryPlan) -> Flux = { Flux.empty() }, + ): QueryBackendRegistry { + val planned = decision as PlanningDecision.Planned + val backendId = BackendId("probe") + val backend = object : RecordQueryBackend { + override fun single( + plan: SingleQueryPlan, + options: QueryExecutionOptions + ): Mono = Mono.empty() + + override fun stream(plan: StreamQueryPlan, options: QueryExecutionOptions): Flux = + streamBackend(plan) + + override fun page(plan: PageQueryPlan, options: QueryExecutionOptions): Mono = pageBackend( + plan + ) + + override fun count(plan: CountQueryPlan, options: QueryExecutionOptions): Mono = countBackend(plan) + } + val descriptor = QueryBackendDescriptor( + QueryBackendKey(PlanningFixtures.target, backendId), + PlanningFixtures.schema.contractId, + setOf(planned.plan.operation), + setOf(SemanticTier.PORTABLE), + PlanningFixtures.schema.fields.mapValues { entry -> entry.value.capabilities }, + ) + return QueryBackendRegistry( + listOf(QueryBackendRegistration(descriptor, recordBackend = backend)), + mapOf(PlanningFixtures.target to backendId), + ) + } + + private fun plannedAnalyticsRoute( + plan: AnalyticsQueryPlan, + result: (AnalyticsQueryPlan) -> Mono, + ): QueryExecutionRoute.Planned { + val backendId = BackendId("analytics") + val descriptor = QueryBackendDescriptor( + QueryBackendKey(PlanningFixtures.target, backendId), + PlanningFixtures.schema.contractId, + setOf(QueryOperation.ANALYZE), + setOf(SemanticTier.PORTABLE), + PlanningFixtures.schema.fields.mapValues { entry -> entry.value.capabilities }, + ) + return QueryExecutionRoute.Planned( + QueryBackendRegistry( + listOf( + QueryBackendRegistration( + descriptor, + analyticsBackend = AnalyticsQueryBackend { planned, _ -> result(planned) }, + ), + ), + mapOf(PlanningFixtures.target to backendId), + ), + plan, + ) + } + + private fun legacyRoute( + input: LegacyCompilationInput, + countResult: Mono = Mono.empty(), + pageResult: Mono = Mono.empty(), + ): QueryExecutionRoute.Legacy = QueryExecutionRoute.Legacy( + LegacyExecutionBinding.create( + PlanningFixtures.target, + LegacyQueryCompiler { compilation -> + InvariantCompiledQuery( + compilation.invocation.target, + compilation.invocation.operation, + compilation.schema.contractId, + compilation.attestLowering( + compilation.enforcementRequirements.deletionScope, + compilation.enforcementRequirements.mandatoryCondition, + ), + ) + }, + object : LegacyQueryBackend { + override fun single( + query: InvariantCompiledQuery, + options: QueryExecutionOptions, + ): Mono = Mono.empty() + + override fun stream( + query: InvariantCompiledQuery, + options: QueryExecutionOptions, + ): Flux = Flux.empty() + + override fun page( + query: InvariantCompiledQuery, + options: QueryExecutionOptions, + ): Mono = pageResult + + override fun count(query: InvariantCompiledQuery, options: QueryExecutionOptions): Mono = + countResult + }, + ), + input, + ) + + private data class InvariantCompiledQuery( + override val target: me.ahoo.wow.query.internal.model.QueryTarget, + override val operation: QueryOperation, + override val schemaContractId: SchemaContractId, + override val loweringAttestation: LegacyLoweringAttestation, + ) : LegacyCompiledQuery + + private fun countInput(): LegacyCompilationInput { + val invocation = NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.COUNT, + QueryResultShape.COUNT, + NormalizedQueryInput.Count( + NormalizedCondition.All, + me.ahoo.wow.query.internal.normalization.NormalizedDeletionScope.EXPLICIT, + ), + ) + return compilationInput(invocation) + } + + private fun pageInput( + size: Int, + index: Int = 1, + offset: Long = 0, + ): LegacyCompilationInput = compilationInput(PlanningFixtures.page(size = size, index = index, offset = offset)) + + private fun compilationInput(invocation: NormalizedQueryInvocation): LegacyCompilationInput = LegacyCompilationInput( + invocation, + PlanningFixtures.schema, + planner.plan(invocation, PlanningFixtures.schema, PlanningConstraints(QueryValidationMode.STRICT)), + ) + + private fun normalizedStream(limit: Int): NormalizedQueryInvocation = NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.STREAM, + QueryResultShape.TYPED, + NormalizedQueryInput.Stream(PlanningFixtures.recordQuery(), limit), + ) + + private fun analyticsPlan( + limit: Int, + consistency: AnalyticsConsistency = AnalyticsConsistency.EVENTUAL, + ): AnalyticsQueryPlan { + val invocation = NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.ANALYZE, + QueryResultShape.ANALYTICS, + NormalizedQueryInput.Analytics( + AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.Global, + NonEmptyList.of(AnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + bucketWindow = me.ahoo.wow.query.internal.analytics.AnalyticsBucketWindow.First(limit), + ), + ), + ) + val planned = ( + planner.plan( + invocation, + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + ).plan as AnalyticsQueryPlan + if (consistency == planned.requiredConsistency) { + return planned + } + return AnalyticsQueryPlan.create( + planned.target, + planned.schemaContractId, + planned.filter, + planned.grouping, + planned.metrics, + planned.having, + planned.bucketOrder, + planned.bucketWindow, + planned.numericPolicy, + consistency, + planned.requiredCompleteness, + planned.requiredCapabilities, + planned.semanticTier, + ) + } + + private fun groupedAnalyticsPlan(): AnalyticsQueryPlan { + val invocation = NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.ANALYZE, + QueryResultShape.ANALYTICS, + NormalizedQueryInput.Analytics( + AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.By( + NonEmptyList.of( + AnalyticsDimension( + AnalyticsAlias("amount"), + PlanningFixtures.path("state", "amount"), + AnalyticsMissingPolicy.EXCLUDE, + ), + ), + ), + NonEmptyList.of(AnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + bucketWindow = me.ahoo.wow.query.internal.analytics.AnalyticsBucketWindow.First(1), + ), + ), + ) + return ( + planner.plan( + invocation, + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + ).plan as AnalyticsQueryPlan + } + + private fun assertIncomplete(action: () -> Any?) { + assertThrownBy { action() }.satisfies(Consumer(::assertIncompleteError)) + } + + private fun assertIncompleteError(error: Throwable) { + val rejected = error as QueryRejectedException + rejected.rejection.category.assert().isEqualTo(QueryRejectionCategory.INCOMPLETE_RESULT) + rejected.rejection.path.toString().assert().isEqualTo("$.backend.result") + rejected.rejection.code.assert().isEqualTo(QueryRejectionCode.INCOMPLETE_RESULT) + } + + private data class PageSuccessCase( + val index: Int, + val offset: Long, + val total: Long, + val records: List, + ) +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutorTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutorTest.kt new file mode 100644 index 00000000000..764b0d60c2a --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryExecutorTest.kt @@ -0,0 +1,407 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.SchemaContractId +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedDeletionScope +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.plan.CountQueryPlan +import me.ahoo.wow.query.internal.plan.PageQueryPlan +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.plan.SingleQueryPlan +import me.ahoo.wow.query.internal.plan.StreamQueryPlan +import me.ahoo.wow.query.internal.planning.PlanningConstraints +import me.ahoo.wow.query.internal.planning.PlanningDecision +import me.ahoo.wow.query.internal.planning.PlanningFixtures +import me.ahoo.wow.query.internal.planning.QueryPlanner +import me.ahoo.wow.query.internal.policy.QueryExecutionBudget +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.core.publisher.SignalType +import reactor.core.scheduler.Schedulers +import reactor.test.scheduler.VirtualTimeScheduler +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset +import java.util.concurrent.atomic.AtomicBoolean +import java.util.concurrent.atomic.AtomicInteger +import java.util.concurrent.atomic.AtomicReference + +class QueryExecutorTest { + private val options = QueryExecutionOptions(null, QueryExecutionBudget()) + private val invocation = NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.COUNT, + QueryResultShape.COUNT, + NormalizedQueryInput.Count(NormalizedCondition.All, NormalizedDeletionScope.EXPLICIT), + ) + private val decision = QueryPlanner().plan( + invocation, + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + private val plan = decision.plan as CountQueryPlan + private val deadlineEnforcer = QueryDeadlineEnforcer(Clock.systemUTC(), Schedulers.parallel()) + + @Test + fun `shadow failure and supervisor failure should never alter legacy result`() { + val probeError = AtomicReference() + val plannedCalls = AtomicInteger() + val supervisor = QueryShadowSupervisor { task -> + Flux.from(task.publisher).subscribe({}, probeError::set) + QueryShadowSubmission.Accepted(QueryShadowHandle.NONE) + } + val route = shadowRoute( + plannedBackend = StubRecordBackend( + countAction = { + plannedCalls.incrementAndGet() + Mono.error(QueryBackendException(QueryBackendFailureKind.UNAVAILABLE)) + }, + ), + legacyCount = { Mono.just(11) }, + ) + + QueryExecutor( + deadlineEnforcer, + shadowSupervisor = supervisor, + ).count(route, options).block().assert().isEqualTo(11) + plannedCalls.get().assert().isEqualTo(1) + val normalizedProbeError = probeError.get() as QueryRejectedException + normalizedProbeError.rejection.category.assert().isEqualTo(QueryRejectionCategory.BACKEND_UNAVAILABLE) + normalizedProbeError.rejection.code.assert().isEqualTo(QueryRejectionCode.BACKEND_EXECUTION_FAILED) + + val throwingSupervisor = QueryShadowSupervisor { error("shadow supervisor unavailable") } + val unavailable = AtomicReference() + val decisionObserver = object : QueryDecisionObserver { + override fun onShadowSupervisorFailure(failure: QueryShadowSupervisorFailure) { + unavailable.set(failure) + } + } + QueryExecutor( + deadlineEnforcer, + shadowSupervisor = throwingSupervisor, + decisionObserver = decisionObserver, + ).count(route, options).block() + .assert().isEqualTo(11) + unavailable.get().task.target.assert().isEqualTo(PlanningFixtures.target) + unavailable.get().task.operation.assert().isEqualTo(QueryOperation.COUNT) + unavailable.get().issue.category.assert().isEqualTo(QueryRejectionCategory.BACKEND_UNAVAILABLE) + unavailable.get().issue.path.toString().assert().isEqualTo("$.shadow.supervisor") + unavailable.get().issue.code.assert().isEqualTo(QueryRejectionCode.SHADOW_SUPERVISOR_UNAVAILABLE) + unavailable.get().cause?.message.assert().isEqualTo("shadow supervisor unavailable") + } + + @Test + fun `disabled or declined shadow submission should emit a typed health event without altering primary`() { + val route = shadowRoute( + plannedBackend = StubRecordBackend(countAction = { Mono.just(9L) }), + legacyCount = { Mono.just(11L) }, + ) + val disabledFailure = AtomicReference() + val disabledObserver = object : QueryDecisionObserver { + override fun onShadowSupervisorFailure(failure: QueryShadowSupervisorFailure) { + disabledFailure.set(failure) + } + } + + QueryExecutor(deadlineEnforcer, decisionObserver = disabledObserver) + .count(route, options).block().assert().isEqualTo(11L) + disabledFailure.get().issue.category.assert().isEqualTo(QueryRejectionCategory.BACKEND_UNAVAILABLE) + disabledFailure.get().issue.path.toString().assert().isEqualTo("$.shadow.supervisor") + disabledFailure.get().issue.code.assert().isEqualTo(QueryRejectionCode.SHADOW_SUPERVISOR_UNAVAILABLE) + disabledFailure.get().cause.assert().isNull() + + val declinedFailures = mutableListOf() + val decliningSupervisor = QueryShadowSupervisor { + QueryShadowSubmission.Rejected(disabledFailure.get().issue) + } + val throwingObserver = object : QueryDecisionObserver { + override fun onShadowSupervisorFailure(failure: QueryShadowSupervisorFailure) { + declinedFailures += failure + error("observer unavailable") + } + } + QueryExecutor( + deadlineEnforcer, + shadowSupervisor = decliningSupervisor, + decisionObserver = throwingObserver, + ).count(route, options).block().assert().isEqualTo(11L) + declinedFailures.assert().hasSize(1) + declinedFailures.single().task.target.assert().isEqualTo(PlanningFixtures.target) + } + + @Test + fun `legacy error should cancel the supervised shadow task`() { + val cancelled = AtomicBoolean() + val supervisor = QueryShadowSupervisor { + QueryShadowSubmission.Accepted(object : QueryShadowHandle { + override fun onPrimary(signal: QueryShadowPrimarySignal) = Unit + + override fun cancelProbe() { + cancelled.set(true) + } + }) + } + val legacyError = IllegalStateException("legacy-primary-error") + val route = shadowRoute( + plannedBackend = StubRecordBackend(countAction = { Mono.never() }), + legacyCount = { Mono.error(legacyError) }, + ) + + try { + QueryExecutor(deadlineEnforcer, shadowSupervisor = supervisor).count(route, options).block() + } catch (error: RuntimeException) { + val rejected = error as QueryRejectedException + rejected.rejection.category.assert().isEqualTo(QueryRejectionCategory.INTERNAL_FAILURE) + rejected.rejection.code.assert().isEqualTo(QueryRejectionCode.BACKEND_EXECUTION_FAILED) + rejected.cause.assert().isSameAs(legacyError) + } + cancelled.get().assert().isTrue() + } + + @Test + fun `successful mono primary should complete without cancelling the shadow probe`() { + val terminal = AtomicReference() + val cancelled = AtomicBoolean() + val signals = mutableListOf() + val supervisor = QueryShadowSupervisor { + QueryShadowSubmission.Accepted(object : QueryShadowHandle { + override fun onPrimary(signal: QueryShadowPrimarySignal) { + signals += signal + } + + override fun cancelProbe() { + cancelled.set(true) + } + }) + } + val route = shadowRoute( + plannedBackend = StubRecordBackend(countAction = { Mono.just(9L) }), + legacyCount = { Mono.just(11L).doFinally(terminal::set) }, + ) + + QueryExecutor(deadlineEnforcer, shadowSupervisor = supervisor).count(route, options).block() + .assert().isEqualTo(11) + terminal.get().assert().isEqualTo(SignalType.ON_COMPLETE) + cancelled.get().assert().isFalse() + signals.assert().containsExactly( + QueryShadowPrimarySignal.CountValue(11L), + QueryShadowPrimarySignal.Complete, + ) + } + + @Test + fun `shadow registry failure and deadline should remain isolated from the legacy primary`() { + val instant = Instant.parse("2026-08-07T00:00:00Z") + val scheduler = VirtualTimeScheduler.create() + val probeError = AtomicReference() + val plannedCancelled = AtomicBoolean() + val supervisor = QueryShadowSupervisor { task -> + Flux.from(task.publisher).subscribe({}, probeError::set) + QueryShadowSubmission.Accepted(QueryShadowHandle.NONE) + } + val emptyRegistryRoute = shadowRoute( + plannedBackend = StubRecordBackend(countAction = { Mono.just(9) }), + legacyCount = { Mono.just(11) }, + ).copy(plannedRegistry = QueryBackendRegistry(emptyList(), emptyMap())) + val executor = QueryExecutor( + QueryDeadlineEnforcer(Clock.fixed(instant, ZoneOffset.UTC), scheduler), + shadowSupervisor = supervisor, + ) + + executor.count(emptyRegistryRoute, options).block().assert().isEqualTo(11) + val missing = probeError.get() as QueryRejectedException + missing.rejection.category.assert().isEqualTo(QueryRejectionCategory.BACKEND_UNAVAILABLE) + missing.rejection.code.assert().isEqualTo(QueryRejectionCode.BACKEND_NOT_REGISTERED) + + probeError.set(null) + val deadlineRoute = shadowRoute( + plannedBackend = StubRecordBackend( + countAction = { Mono.never().doOnCancel { plannedCancelled.set(true) } }, + ), + legacyCount = { Mono.just(11) }, + ) + val deadlineOptions = QueryExecutionOptions(instant.plusSeconds(5), QueryExecutionBudget()) + executor.count(deadlineRoute, deadlineOptions).block().assert().isEqualTo(11) + scheduler.advanceTimeBy(Duration.ofSeconds(5)) + + plannedCancelled.get().assert().isTrue() + val expired = probeError.get() as QueryRejectedException + expired.rejection.category.assert().isEqualTo(QueryRejectionCategory.BUDGET_EXCEEDED) + expired.rejection.path.toString().assert().isEqualTo("$.executionContext.deadline") + expired.rejection.code.assert().isEqualTo(QueryRejectionCode.DEADLINE_EXPIRED) + + probeError.set(null) + val noDeadlineCancelled = AtomicBoolean() + val noDeadlineRoute = shadowRoute( + plannedBackend = StubRecordBackend( + countAction = { Mono.never().doOnCancel { noDeadlineCancelled.set(true) } }, + ), + legacyCount = { Mono.just(11) }, + ) + executor.count(noDeadlineRoute, options).block().assert().isEqualTo(11) + scheduler.advanceTimeBy(Duration.ofSeconds(30)) + + noDeadlineCancelled.get().assert().isTrue() + val capped = probeError.get() as QueryRejectedException + capped.rejection.category.assert().isEqualTo(QueryRejectionCategory.BUDGET_EXCEEDED) + capped.rejection.path.toString().assert().isEqualTo("$.executionContext.deadline") + capped.rejection.code.assert().isEqualTo(QueryRejectionCode.DEADLINE_EXPIRED) + } + + @Test + fun `planned records should require explicit completeness while legacy preserves unknown provenance`() { + val singleInvocation = PlanningFixtures.single(resultShape = QueryResultShape.DYNAMIC) + val singleDecision = QueryPlanner().plan( + singleInvocation, + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + val unknown = BackendRecord( + "order-1", + me.ahoo.wow.query.backend.NormalizedValue.ObjectValue(emptyMap()), + BackendRecordCompleteness.UNKNOWN, + ) + val registration = QueryBackendRegistration( + QueryBackendDescriptor( + QueryBackendKey(PlanningFixtures.target, BackendId("probe")), + PlanningFixtures.schema.contractId, + setOf(QueryOperation.SINGLE), + setOf(SemanticTier.PORTABLE), + PlanningFixtures.schema.fields.mapValues { entry -> entry.value.capabilities }, + ), + recordBackend = object : StubRecordBackend({ Mono.just(1) }) { + override fun single( + plan: SingleQueryPlan, + options: QueryExecutionOptions, + ): Mono = Mono.just(unknown) + }, + ) + assertThrownBy { + QueryExecutor(deadlineEnforcer).single( + QueryExecutionRoute.Planned( + QueryBackendRegistry( + listOf(registration), + mapOf(PlanningFixtures.target to BackendId("probe")), + ), + singleDecision.plan, + ), + options, + ).block() + }.satisfies( + java.util.function.Consumer { error -> + error.rejection.category.assert().isEqualTo(QueryRejectionCategory.INCOMPLETE_RESULT) + error.rejection.code.assert().isEqualTo(QueryRejectionCode.INCOMPLETE_RESULT) + }, + ) + + val legacy = legacyBinding(singleAction = { Mono.just(unknown) }) + QueryExecutor(deadlineEnforcer).single( + QueryExecutionRoute.Legacy( + legacy, + LegacyCompilationInput(singleInvocation, PlanningFixtures.schema, singleDecision), + ), + options, + ).block().assert().isEqualTo(unknown) + } + + private fun shadowRoute( + plannedBackend: RecordQueryBackend, + legacyCount: () -> Mono, + ): QueryExecutionRoute.Shadow { + val registration = QueryBackendRegistration( + QueryBackendDescriptor( + QueryBackendKey(PlanningFixtures.target, BackendId("probe")), + PlanningFixtures.schema.contractId, + setOf(QueryOperation.COUNT), + setOf(SemanticTier.PORTABLE), + PlanningFixtures.schema.fields.mapValues { entry -> entry.value.capabilities }, + ), + recordBackend = plannedBackend, + ) + val legacy = legacyBinding(countAction = { Mono.defer(legacyCount) }) + return QueryExecutionRoute.Shadow( + legacy, + LegacyCompilationInput(invocation, PlanningFixtures.schema, decision), + QueryBackendRegistry( + listOf(registration), + mapOf(PlanningFixtures.target to BackendId("probe")), + ), + plan, + ) + } + + private open class StubRecordBackend( + private val countAction: (CountQueryPlan) -> Mono, + ) : RecordQueryBackend { + override fun single(plan: SingleQueryPlan, options: QueryExecutionOptions): Mono = Mono.empty() + + override fun stream(plan: StreamQueryPlan, options: QueryExecutionOptions): Flux = Flux.empty() + + override fun page(plan: PageQueryPlan, options: QueryExecutionOptions): Mono = Mono.empty() + + override fun count(plan: CountQueryPlan, options: QueryExecutionOptions): Mono = countAction(plan) + } + + private fun legacyBinding( + singleAction: () -> Mono = { Mono.empty() }, + streamAction: () -> Flux = { Flux.empty() }, + pageAction: () -> Mono = { Mono.empty() }, + countAction: () -> Mono = { Mono.empty() }, + ): LegacyExecutionBinding = LegacyExecutionBinding.create( + PlanningFixtures.target, + LegacyQueryCompiler { input -> + TestCompiledQuery( + input.invocation.target, + input.invocation.operation, + input.schema.contractId, + input.attestLowering( + input.enforcementRequirements.deletionScope, + input.enforcementRequirements.mandatoryCondition, + ), + ) + }, + object : LegacyQueryBackend { + override fun single(query: TestCompiledQuery, options: QueryExecutionOptions): Mono = + singleAction() + + override fun stream(query: TestCompiledQuery, options: QueryExecutionOptions): Flux = + streamAction() + + override fun page(query: TestCompiledQuery, options: QueryExecutionOptions): Mono = pageAction() + + override fun count(query: TestCompiledQuery, options: QueryExecutionOptions): Mono = countAction() + }, + ) + + private data class TestCompiledQuery( + override val target: me.ahoo.wow.query.internal.model.QueryTarget, + override val operation: QueryOperation, + override val schemaContractId: SchemaContractId, + override val loweringAttestation: LegacyLoweringAttestation, + ) : LegacyCompiledQuery +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryGatewayLifecycleTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryGatewayLifecycleTest.kt new file mode 100644 index 00000000000..7d476bbe7d8 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/execution/QueryGatewayLifecycleTest.kt @@ -0,0 +1,601 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.execution + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DeletionState +import me.ahoo.wow.api.query.ListQuery +import me.ahoo.wow.api.query.Operator +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.admission.QueryAdmissionLimits +import me.ahoo.wow.query.internal.admission.RawAdmissionGuard +import me.ahoo.wow.query.internal.model.QueryExecutionMode +import me.ahoo.wow.query.internal.model.QueryInput +import me.ahoo.wow.query.internal.model.QueryInvocation +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.QueryNormalizer +import me.ahoo.wow.query.internal.plan.CountQueryPlan +import me.ahoo.wow.query.internal.plan.PageQueryPlan +import me.ahoo.wow.query.internal.plan.PlannedCondition +import me.ahoo.wow.query.internal.plan.SingleQueryPlan +import me.ahoo.wow.query.internal.plan.StreamQueryPlan +import me.ahoo.wow.query.internal.planning.PlanningFixtures +import me.ahoo.wow.query.internal.planning.QueryFieldConstraint +import me.ahoo.wow.query.internal.planning.QueryPlanner +import me.ahoo.wow.query.internal.policy.QueryAuthority +import me.ahoo.wow.query.internal.policy.QueryAuthorityProvider +import me.ahoo.wow.query.internal.policy.QueryExecutionContextFactory +import me.ahoo.wow.query.internal.policy.QueryExecutionRequest +import me.ahoo.wow.query.internal.policy.QueryPolicy +import me.ahoo.wow.query.internal.policy.QueryPolicyAllowance +import me.ahoo.wow.query.internal.policy.QueryPolicyDecision +import me.ahoo.wow.query.internal.policy.QueryPolicyEnforcer +import me.ahoo.wow.query.internal.policy.QueryPurpose +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.schema.QuerySchemaRegistry +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.core.publisher.Sinks +import reactor.core.scheduler.Scheduler +import reactor.core.scheduler.Schedulers +import reactor.test.StepVerifier +import reactor.test.scheduler.VirtualTimeScheduler +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset +import java.util.concurrent.atomic.AtomicBoolean +import java.util.concurrent.atomic.AtomicInteger +import java.util.concurrent.atomic.AtomicReference + +class QueryGatewayLifecycleTest { + private val instant = Instant.parse("2026-08-07T00:00:00Z") + private val clock = Clock.fixed(instant, ZoneOffset.UTC) + private val record = BackendRecord( + "order-1", + NormalizedValue.ObjectValue(mapOf("name" to NormalizedValue.Text("Ada"))), + BackendRecordCompleteness.COMPLETE, + ) + + @Test + fun `gateway should remain cold and recreate the complete pipeline per subscription`() { + val invocationReads = AtomicInteger() + val authorityReads = AtomicInteger() + val backendReads = AtomicInteger() + val observer = RecordingObserver() + val backend = StubRecordBackend( + countAction = { + backendReads.incrementAndGet() + Mono.just(7) + }, + ) + val gateway = gateway( + backend, + authorityProvider = QueryAuthorityProvider { + authorityReads.incrementAndGet() + Mono.just(QueryAuthority.System("test", "gateway-lifecycle")) + }, + observer = observer, + ) + + val result = gateway.count(request()) { + invocationReads.incrementAndGet() + countInvocation() + } + invocationReads.get().assert().isZero() + authorityReads.get().assert().isZero() + backendReads.get().assert().isZero() + + result.block().assert().isEqualTo(7) + result.block().assert().isEqualTo(7) + + invocationReads.get().assert().isEqualTo(2) + authorityReads.get().assert().isEqualTo(2) + backendReads.get().assert().isEqualTo(2) + observer.starts.size.assert().isEqualTo(2) + observer.terminals.map(QueryLifecycleTerminal::kind).assert().containsExactly( + QueryTerminationKind.COMPLETE, + QueryTerminationKind.COMPLETE, + ) + } + + @Test + fun `sync and async backend errors should cross the same safe error boundary`() { + listOf( + StubRecordBackend(countAction = { throw IllegalStateException("sync-secret") }), + StubRecordBackend(countAction = { Mono.error(IllegalStateException("async-secret")) }), + ).forEach { backend -> + StepVerifier.create(gateway(backend).count(request(), ::countInvocation)) + .expectErrorSatisfies { error -> + val rejected = error as QueryRejectedException + rejected.rejection.category.assert().isEqualTo(QueryRejectionCategory.INTERNAL_FAILURE) + rejected.rejection.path.toString().assert().isEqualTo("$.backend") + rejected.rejection.code.assert().isEqualTo(QueryRejectionCode.BACKEND_EXECUTION_FAILED) + rejected.cause.assert().isInstanceOf(IllegalStateException::class.java) + } + .verify() + } + } + + @Test + fun `typed backend failures should map to stable rejection tuples`() { + listOf( + BackendFailureExpectation( + QueryBackendFailureKind.UNAVAILABLE, + QueryRejectionCategory.BACKEND_UNAVAILABLE, + "$.backend", + QueryRejectionCode.BACKEND_EXECUTION_FAILED, + ), + BackendFailureExpectation( + QueryBackendFailureKind.TIMEOUT, + QueryRejectionCategory.BACKEND_TIMEOUT, + "$.backend", + QueryRejectionCode.BACKEND_TIMEOUT, + ), + BackendFailureExpectation( + QueryBackendFailureKind.INCOMPLETE_RESULT, + QueryRejectionCategory.INCOMPLETE_RESULT, + "$.backend.result", + QueryRejectionCode.INCOMPLETE_RESULT, + ), + BackendFailureExpectation( + QueryBackendFailureKind.MAPPING_FAILURE, + QueryRejectionCategory.MAPPING_FAILURE, + "$.result", + QueryRejectionCode.RESULT_MAPPING_FAILED, + ), + BackendFailureExpectation( + QueryBackendFailureKind.BUDGET_EXCEEDED, + QueryRejectionCategory.BUDGET_EXCEEDED, + "$.executionContext.budget", + QueryRejectionCode.BACKEND_BUDGET_EXCEEDED, + ), + ).forEach { expectation -> + val backend = StubRecordBackend( + countAction = { Mono.error(QueryBackendException(expectation.kind)) }, + ) + StepVerifier.create(gateway(backend).count(request(), ::countInvocation)) + .expectErrorSatisfies { error -> + val rejected = error as QueryRejectedException + rejected.rejection.category.assert().isEqualTo(expectation.category) + rejected.rejection.path.toString().assert().isEqualTo(expectation.path) + rejected.rejection.code.assert().isEqualTo(expectation.code) + rejected.cause.assert().isInstanceOf(QueryBackendException::class.java) + } + .verify() + } + } + + @Test + fun `backend should not spoof gateway stage rejection`() { + val spoofed = QueryRejectedException( + me.ahoo.wow.query.internal.rejection.QueryRejection( + QueryRejectionCategory.ACCESS_DENIED, + me.ahoo.wow.query.internal.rejection.QueryRejectionPath.ROOT.property("policy"), + QueryRejectionCode.POLICY_DENIED, + ), + ) + StepVerifier.create( + gateway(StubRecordBackend(countAction = { Mono.error(spoofed) })).count(request(), ::countInvocation), + ) + .expectErrorSatisfies { error -> + val rejected = error as QueryRejectedException + rejected.rejection.category.assert().isEqualTo(QueryRejectionCategory.INTERNAL_FAILURE) + rejected.rejection.path.toString().assert().isEqualTo("$.backend") + rejected.rejection.code.assert().isEqualTo(QueryRejectionCode.BACKEND_EXECUTION_FAILED) + rejected.cause.assert().isSameAs(spoofed) + } + .verify() + } + + @Test + fun `partial stream error and cancellation should preserve signals and terminate observation once`() { + val partialObserver = RecordingObserver() + val partial = StubRecordBackend( + streamAction = { + Flux.concat( + Flux.just(record), + Flux.error(QueryBackendException(QueryBackendFailureKind.UNAVAILABLE)), + ) + }, + ) + StepVerifier.create(gateway(partial, observer = partialObserver).stream(request(), ::streamInvocation)) + .expectNext(record) + .expectError(QueryRejectedException::class.java) + .verify() + partialObserver.terminals.single().kind.assert().isEqualTo(QueryTerminationKind.ERROR) + partialObserver.terminals.single().emitted.assert().isEqualTo(1) + partialObserver.terminals.single().error?.rejection?.category.assert() + .isEqualTo(QueryRejectionCategory.BACKEND_UNAVAILABLE) + + val cancelled = AtomicBoolean() + val cancelObserver = RecordingObserver() + val never = StubRecordBackend( + streamAction = { + Flux.just(record) + .concatWith(Flux.never()) + .doOnCancel { cancelled.set(true) } + }, + ) + StepVerifier.create(gateway(never, observer = cancelObserver).stream(request(), ::streamInvocation)) + .expectNext(record) + .thenCancel() + .verify() + cancelled.get().assert().isTrue() + cancelObserver.terminals.single().kind.assert().isEqualTo(QueryTerminationKind.CANCEL) + cancelObserver.terminals.single().error.assert().isNull() + } + + @Test + fun `absolute deadline should stop a continuously emitting stream and cancel upstream`() { + val scheduler = VirtualTimeScheduler.create() + val cancelled = AtomicBoolean() + val backend = StubRecordBackend( + streamAction = { + Flux.interval(Duration.ofSeconds(1), scheduler) + .map { record } + .doOnCancel { cancelled.set(true) } + }, + ) + val deadlineRequest = request().copy(deadline = instant.plusMillis(4_500)) + val result = gateway(backend, scheduler = scheduler).stream(deadlineRequest, ::streamInvocation) + + StepVerifier.withVirtualTime({ result }, { scheduler }, 0) + .thenRequest(Long.MAX_VALUE) + .thenAwait(Duration.ofSeconds(4)) + .expectNextCount(4) + .thenAwait(Duration.ofMillis(500)) + .expectErrorSatisfies { error -> + val rejected = error as QueryRejectedException + rejected.rejection.category.assert().isEqualTo(QueryRejectionCategory.BUDGET_EXCEEDED) + rejected.rejection.path.toString().assert().isEqualTo("$.executionContext.deadline") + rejected.rejection.code.assert().isEqualTo(QueryRejectionCode.DEADLINE_EXPIRED) + } + .verify() + cancelled.get().assert().isTrue() + } + + @Test + fun `absolute deadline should also cancel authority resolution before backend routing`() { + val scheduler = VirtualTimeScheduler.create() + val authorityCancelled = AtomicBoolean() + val backendCalls = AtomicInteger() + val backend = StubRecordBackend( + countAction = { + backendCalls.incrementAndGet() + Mono.just(1) + }, + ) + val gateway = gateway( + backend, + authorityProvider = QueryAuthorityProvider { + Mono.never().doOnCancel { authorityCancelled.set(true) } + }, + scheduler = scheduler, + ) + val result = gateway.count( + request().copy(deadline = instant.plusSeconds(2)), + ::countInvocation, + ) + + StepVerifier.withVirtualTime({ result }, { scheduler }, 0) + .thenRequest(1) + .thenAwait(Duration.ofSeconds(2)) + .expectErrorSatisfies { error -> + val rejected = error as QueryRejectedException + rejected.rejection.code.assert().isEqualTo(QueryRejectionCode.DEADLINE_EXPIRED) + } + .verify() + authorityCancelled.get().assert().isTrue() + backendCalls.get().assert().isZero() + } + + @Test + fun `mono deadline should preserve backend completion and propagate downstream cancellation`() { + val completedSignal = AtomicReference() + val completeObserver = RecordingObserver() + val completed = StubRecordBackend( + countAction = { Mono.just(7L).doFinally(completedSignal::set) }, + ) + gateway(completed, observer = completeObserver) + .count(request().copy(deadline = instant.plusSeconds(5)), ::countInvocation) + .block() + .assert().isEqualTo(7L) + completedSignal.get().assert().isEqualTo(reactor.core.publisher.SignalType.ON_COMPLETE) + completeObserver.terminals.single().kind.assert().isEqualTo(QueryTerminationKind.COMPLETE) + + val cancelled = AtomicBoolean() + val subscribed = Sinks.one() + val cancelObserver = RecordingObserver() + val never = StubRecordBackend( + countAction = { + Mono.never() + .doOnSubscribe { subscribed.tryEmitValue(Unit) } + .doOnCancel { cancelled.set(true) } + }, + ) + val subscription = gateway(never, observer = cancelObserver) + .count(request().copy(deadline = instant.plusSeconds(5)), ::countInvocation) + .subscribe() + subscribed.asMono().block(Duration.ofSeconds(1)) + subscription.dispose() + cancelled.get().assert().isTrue() + cancelObserver.terminals.assert().hasSize(1) + cancelObserver.terminals.single().kind.assert().isEqualTo(QueryTerminationKind.CANCEL) + cancelObserver.terminals.single().error.assert().isNull() + } + + @Test + fun `observer failures should never replace query success`() { + val observer = object : QueryLifecycleObserver { + override fun onStart(descriptor: QueryLifecycleDescriptor) { + error("start observer failure") + } + + override fun onTerminal(terminal: QueryLifecycleTerminal) { + error("terminal observer failure") + } + } + gateway(StubRecordBackend(countAction = { Mono.just(9) }), observer = observer) + .count(request(), ::countInvocation) + .block() + .assert().isEqualTo(9) + } + + @Test + fun `observer failures should never replace query error or cancellation`() { + val observer = object : QueryLifecycleObserver { + override fun onStart(descriptor: QueryLifecycleDescriptor) { + error("start observer failure") + } + + override fun onTerminal(terminal: QueryLifecycleTerminal) { + error("terminal observer failure") + } + } + val backend = StubRecordBackend( + streamAction = { + Flux.concat( + Flux.just(record), + Flux.error(QueryBackendException(QueryBackendFailureKind.UNAVAILABLE)), + ) + }, + ) + StepVerifier.create(gateway(backend, observer = observer).stream(request(), ::streamInvocation)) + .expectNext(record) + .expectErrorSatisfies { error -> + val rejected = error as QueryRejectedException + rejected.rejection.category.assert().isEqualTo(QueryRejectionCategory.BACKEND_UNAVAILABLE) + } + .verify() + + val cancelled = AtomicBoolean() + val subscribed = Sinks.one() + val never = StubRecordBackend( + streamAction = { + Flux.never() + .doOnSubscribe { subscribed.tryEmitValue(Unit) } + .doOnCancel { cancelled.set(true) } + }, + ) + val subscription = gateway(never, observer = observer).stream(request(), ::streamInvocation).subscribe() + subscribed.asMono().block(Duration.ofSeconds(1)) + subscription.dispose() + cancelled.get().assert().isTrue() + } + + @Test + fun `gateway should freeze mutable wire input before asynchronous authority and fail closed before backend`() { + val authority = Sinks.one() + val sourceChildren = mutableListOf(Condition.eq("state.name", "Ada")) + val capturedPlan = AtomicReference() + val backendCalls = AtomicInteger() + val backend = StubRecordBackend( + countAction = { plan -> + capturedPlan.set(plan) + backendCalls.incrementAndGet() + Mono.just(1) + }, + ) + val gateway = gateway( + backend, + authorityProvider = QueryAuthorityProvider { authority.asMono() }, + ) + val result = gateway.count(request()) { + QueryInvocation( + PlanningFixtures.target, + QueryOperation.COUNT, + QueryResultShape.COUNT, + QueryInput.Count(Condition(operator = Operator.AND, children = sourceChildren)), + ) + } + + StepVerifier.create(result) + .then { + sourceChildren.clear() + authority.tryEmitValue(QueryAuthority.System("test", "freeze-boundary")) + } + .expectNext(1) + .verifyComplete() + backendCalls.get().assert().isEqualTo(1) + val effective = capturedPlan.get().filter.user as PlannedCondition.Junction + effective.children.values.assert().hasSize(2) + + val deniedCalls = AtomicInteger() + val deniedBackend = StubRecordBackend( + countAction = { + deniedCalls.incrementAndGet() + Mono.just(1) + }, + ) + val missingDecision = QueryPolicy { Mono.empty() } + StepVerifier.create(gateway(deniedBackend, policy = missingDecision).count(request(), ::countInvocation)) + .expectErrorSatisfies { error -> + val rejected = error as QueryRejectedException + rejected.rejection.category.assert().isEqualTo(QueryRejectionCategory.ACCESS_DENIED) + rejected.rejection.code.assert().isEqualTo(QueryRejectionCode.POLICY_DECISION_MISSING) + } + .verify() + deniedCalls.get().assert().isZero() + } + + @Test + fun `unbound raw should be rejected before routing without inspecting driver object`() { + val backendCalls = AtomicInteger() + val backend = StubRecordBackend( + countAction = { + backendCalls.incrementAndGet() + Mono.just(1) + }, + ) + val hostileDriver = object { + override fun toString(): String = error("driver object must not be inspected") + } + StepVerifier.create( + gateway(backend).count(request()) { + QueryInvocation( + PlanningFixtures.target, + QueryOperation.COUNT, + QueryResultShape.COUNT, + QueryInput.Count(Condition.raw(hostileDriver)), + ) + }, + ).expectErrorSatisfies { error -> + val rejected = error as QueryRejectedException + rejected.rejection.category.assert().isEqualTo(QueryRejectionCategory.UNSUPPORTED_FEATURE) + rejected.rejection.code.assert().isEqualTo(QueryRejectionCode.NATIVE_BACKEND_UNBOUND) + }.verify() + backendCalls.get().assert().isZero() + } + + private fun gateway( + backend: StubRecordBackend, + authorityProvider: QueryAuthorityProvider = QueryAuthorityProvider { + Mono.just(QueryAuthority.System("test", "gateway-lifecycle")) + }, + observer: QueryLifecycleObserver = QueryLifecycleObserver.NONE, + scheduler: Scheduler = Schedulers.parallel(), + policy: QueryPolicy = allowPolicy(), + ): QueryGateway { + val backendId = BackendId("probe") + val registration = QueryBackendRegistration( + QueryBackendDescriptor( + QueryBackendKey(PlanningFixtures.target, backendId), + PlanningFixtures.schema.contractId, + setOf( + QueryOperation.SINGLE, + QueryOperation.STREAM, + QueryOperation.PAGE, + QueryOperation.COUNT, + ), + me.ahoo.wow.query.internal.plan.SemanticTier.entries.toSet(), + PlanningFixtures.schema.fields.mapValues { entry -> entry.value.capabilities }, + PlanningFixtures.schema.searchScopes.keys, + ), + recordBackend = backend, + ) + val plannedRegistry = QueryBackendRegistry( + listOf(registration), + mapOf(PlanningFixtures.target to backendId), + ) + val deadlineEnforcer = QueryDeadlineEnforcer(clock, scheduler) + return QueryGateway( + RawAdmissionGuard(QueryAdmissionLimits.DEFAULT), + QueryNormalizer(clock), + QuerySchemaRegistry(listOf(PlanningFixtures.schema)), + QueryExecutionContextFactory(authorityProvider, clock), + QueryPolicyEnforcer(policy), + QueryPlanner(), + QueryExecutionRouteResolver(plannedRegistry, LegacyBackendRegistry(emptyList())), + QueryExecutor(deadlineEnforcer), + deadlineEnforcer, + lifecycleMonitor = QueryLifecycleMonitor(observer), + ) + } + + private fun request(): QueryExecutionRequest = QueryExecutionRequest( + PlanningFixtures.target, + QueryPurpose("test"), + QueryExecutionMode.PLANNED, + QueryValidationMode.STRICT, + ) + + private fun countInvocation(): QueryInvocation = QueryInvocation( + PlanningFixtures.target, + QueryOperation.COUNT, + QueryResultShape.COUNT, + QueryInput.Count(Condition.deleted(DeletionState.ALL)), + ) + + private fun streamInvocation(): QueryInvocation = QueryInvocation( + PlanningFixtures.target, + QueryOperation.STREAM, + QueryResultShape.DYNAMIC, + QueryInput.Stream(ListQuery(Condition.deleted(DeletionState.ALL), limit = 10)), + ) + + private fun allowPolicy(): QueryPolicy = QueryPolicy { + Mono.just( + QueryPolicyDecision.Allow( + QueryPolicyAllowance.builder() + .fieldConstraint(QueryFieldConstraint()) + .build(), + ), + ) + } + + private class RecordingObserver : QueryLifecycleObserver { + val starts = mutableListOf() + val terminals = mutableListOf() + + override fun onStart(descriptor: QueryLifecycleDescriptor) { + starts += descriptor + } + + override fun onTerminal(terminal: QueryLifecycleTerminal) { + terminals += terminal + } + } + + private data class BackendFailureExpectation( + val kind: QueryBackendFailureKind, + val category: QueryRejectionCategory, + val path: String, + val code: QueryRejectionCode, + ) + + private class StubRecordBackend( + private val singleAction: (SingleQueryPlan) -> Mono = { Mono.empty() }, + private val streamAction: (StreamQueryPlan) -> Flux = { Flux.empty() }, + private val pageAction: (PageQueryPlan) -> Mono = { Mono.empty() }, + private val countAction: (CountQueryPlan) -> Mono = { Mono.empty() }, + ) : RecordQueryBackend { + override fun single(plan: SingleQueryPlan, options: QueryExecutionOptions): Mono = + singleAction(plan) + + override fun stream(plan: StreamQueryPlan, options: QueryExecutionOptions): Flux = + streamAction(plan) + + override fun page(plan: PageQueryPlan, options: QueryExecutionOptions): Mono = pageAction(plan) + + override fun count(plan: CountQueryPlan, options: QueryExecutionOptions): Mono = countAction(plan) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/gateway/LegacyConditionLowererTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/gateway/LegacyConditionLowererTest.kt new file mode 100644 index 00000000000..1a18dce700f --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/gateway/LegacyConditionLowererTest.kt @@ -0,0 +1,135 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.gateway + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.api.query.DeletionState +import me.ahoo.wow.api.query.Operator +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi +import me.ahoo.wow.query.gateway.QueryElementPathMode +import me.ahoo.wow.query.gateway.QueryLegacyDialect +import me.ahoo.wow.query.gateway.QueryMatchScopeMode +import me.ahoo.wow.query.internal.normalization.JunctionOperator +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedDeletionScope +import me.ahoo.wow.query.internal.normalization.PathBasis +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.SearchScopeId +import me.ahoo.wow.query.internal.plan.PlannedCondition +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import java.util.function.Consumer + +@OptIn(ExperimentalQueryGatewayApi::class) +class LegacyConditionLowererTest { + @Test + fun `should lower deletion as a direct guard-neutralizing clause`() { + val user = predicate(rootPath("state", "status"), "PAID") + + val defaultActive = mongoLowerer.lower(user, NormalizedDeletionScope.DEFAULT_ACTIVE, PlannedCondition.All) + defaultActive.matchNone.assert().isFalse() + defaultActive.condition.operator.assert().isEqualTo(Operator.AND) + defaultActive.condition.children.map { it.operator }.assert().containsExactly(Operator.DELETED, Operator.EQ) + defaultActive.condition.children.first().deletionState().assert().isEqualTo(DeletionState.ACTIVE) + + val explicit = mongoLowerer.lower(user, NormalizedDeletionScope.EXPLICIT, PlannedCondition.All) + explicit.condition.children.first().deletionState().assert().isEqualTo(DeletionState.ALL) + } + + @Test + fun `should render nested fields relative for Mongo and root-qualified for Elasticsearch`() { + val nested = NormalizedCondition.ElementMatch( + rootPath("items"), + NormalizedCondition.Junction( + JunctionOperator.AND, + listOf( + predicate(elementPath("sku"), "sku-1"), + NormalizedCondition.ElementMatch( + elementPath("attributes"), + predicate(elementPath("code"), "size"), + ), + ), + ), + ) + + val mongo = mongoLowerer.lower(nested, NormalizedDeletionScope.DEFAULT_ACTIVE, PlannedCondition.All) + val mongoElement = mongo.condition.children[1] + mongoElement.field.assert().isEqualTo("items") + mongoElement.children.first().children[0].field.assert().isEqualTo("sku") + mongoElement.children.first().children[1].field.assert().isEqualTo("attributes") + mongoElement.children.first().children[1].children.first().field.assert().isEqualTo("code") + + val elasticsearch = elasticsearchLowerer.lower( + nested, + NormalizedDeletionScope.DEFAULT_ACTIVE, + PlannedCondition.All, + ) + val elasticsearchElement = elasticsearch.condition.children[1] + elasticsearchElement.field.assert().isEqualTo("items") + elasticsearchElement.children.first().children[0].field.assert().isEqualTo("items.sku") + elasticsearchElement.children.first().children[1].field.assert().isEqualTo("items.attributes") + elasticsearchElement.children.first().children[1].children.first().field.assert() + .isEqualTo("items.attributes.code") + } + + @Test + fun `should short-circuit normalized none without relying on backend empty-list behavior`() { + val lowered = mongoLowerer.lower( + NormalizedCondition.None, + NormalizedDeletionScope.DEFAULT_ACTIVE, + PlannedCondition.All, + ) + + lowered.matchNone.assert().isTrue() + lowered.condition.children.first().deletionState().assert().isEqualTo(DeletionState.ACTIVE) + } + + @Test + fun `unsupported mandatory lowering should fail closed as access denied`() { + assertThrownBy { + mongoLowerer.lower( + NormalizedCondition.All, + NormalizedDeletionScope.DEFAULT_ACTIVE, + PlannedCondition.Search(SearchScopeId("search"), "text"), + ) + }.satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(QueryRejectionCategory.ACCESS_DENIED) + error.rejection.path.toString().assert().isEqualTo("$.constraints.mandatoryCondition") + error.rejection.code.assert().isEqualTo(QueryRejectionCode.MANDATORY_CONDITION_UNENFORCEABLE) + }, + ) + } + + private fun predicate(field: LogicalField, value: String): NormalizedCondition.Predicate = + NormalizedCondition.Predicate(field, PredicateOperator.EQ, NormalizedValue.Text(value)) + + private fun rootPath(vararg segments: String): LogicalField.Path = + LogicalField.Path(segments.asList(), PathBasis.ROOT) + + private fun elementPath(vararg segments: String): LogicalField.Path = + LogicalField.Path(segments.asList(), PathBasis.CURRENT_ELEMENT) + + private val mongoLowerer = LegacyConditionLowerer( + QueryLegacyDialect(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, QueryMatchScopeMode.DOCUMENT), + ) + private val elasticsearchLowerer = LegacyConditionLowerer( + QueryLegacyDialect(QueryElementPathMode.ROOT_QUALIFIED, QueryMatchScopeMode.FIELD), + ) +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/gateway/LegacyQuerySchemaTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/gateway/LegacyQuerySchemaTest.kt new file mode 100644 index 00000000000..12c4157b628 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/gateway/LegacyQuerySchemaTest.kt @@ -0,0 +1,52 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.gateway + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.normalization.SystemFieldKind +import org.junit.jupiter.api.Test + +class LegacyQuerySchemaTest { + @Test + fun `snapshot aliases should resolve framework-owned logical fields`() { + val schema = legacyQuerySchema(QueryTarget(AGGREGATE, QueryDocumentKind.SNAPSHOT)) + + schema.resolveField(path("aggregateId")).assert().isEqualTo(system(SystemFieldKind.IDENTITY)) + schema.resolveField(path("tenantId")).assert().isEqualTo(system(SystemFieldKind.TENANT_ID)) + schema.resolveField(path("ownerId")).assert().isEqualTo(system(SystemFieldKind.OWNER_ID)) + schema.resolveField(path("spaceId")).assert().isEqualTo(system(SystemFieldKind.SPACE_ID)) + schema.resolveField(path("deleted")).assert().isEqualTo(system(SystemFieldKind.DELETED)) + } + + @Test + fun `event stream aliases should distinguish stream identity and aggregate identity`() { + val schema = legacyQuerySchema(QueryTarget(AGGREGATE, QueryDocumentKind.EVENT_STREAM)) + + schema.resolveField(path("id")).assert().isEqualTo(system(SystemFieldKind.IDENTITY)) + schema.resolveField(path("aggregateId")).assert().isEqualTo(system(SystemFieldKind.AGGREGATE_ID)) + schema.resolveField(path("deleted")).assert().isNull() + } + + private fun path(field: String): QueryFieldId.Path = QueryFieldId.Path(listOf(field)) + + private fun system(kind: SystemFieldKind): QueryFieldId.System = QueryFieldId.System(kind) + + private companion object { + val AGGREGATE = MaterializedNamedAggregate("sales", "order") + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/model/QueryInvocationTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/model/QueryInvocationTest.kt new file mode 100644 index 00000000000..44e73c41bd5 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/model/QueryInvocationTest.kt @@ -0,0 +1,156 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.model + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.modeling.NamedAggregateDecorator +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.ListQuery +import me.ahoo.wow.api.query.PagedQuery +import me.ahoo.wow.api.query.SingleQuery +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsGrouping +import me.ahoo.wow.query.internal.analytics.AnalyticsMetric +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.value.NonEmptyList +import org.junit.jupiter.api.Test + +class QueryInvocationTest { + + private val namedAggregate = MaterializedNamedAggregate("sales", "order") + private val target = QueryTarget(namedAggregate, QueryDocumentKind.SNAPSHOT) + private val analytics = AnalyticsQuery( + userCondition = NormalizedCondition.All, + grouping = AnalyticsGrouping.Global, + metrics = NonEmptyList.of(AnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + ) + + @Test + fun `target should materialize aggregate decorators`() { + val decorator = object : NamedAggregateDecorator { + override val namedAggregate: NamedAggregate = this@QueryInvocationTest.namedAggregate + } + + val actual = QueryTarget(decorator, QueryDocumentKind.EVENT_STREAM) + + actual.namedAggregate.assert().isSameAs(namedAggregate) + actual.documentKind.assert().isEqualTo(QueryDocumentKind.EVENT_STREAM) + } + + @Test + fun `should accept the complete operation result and input matrix`() { + val invocations = listOf( + QueryInvocation( + target = target, + operation = QueryOperation.SINGLE, + resultShape = QueryResultShape.TYPED, + input = QueryInput.Single(SingleQuery(Condition.ALL)), + ), + QueryInvocation( + target = target, + operation = QueryOperation.SINGLE, + resultShape = QueryResultShape.DYNAMIC, + input = QueryInput.Single(SingleQuery(Condition.ALL)), + ), + QueryInvocation( + target = target, + operation = QueryOperation.STREAM, + resultShape = QueryResultShape.TYPED, + input = QueryInput.Stream(ListQuery(Condition.ALL)), + ), + QueryInvocation( + target = target, + operation = QueryOperation.STREAM, + resultShape = QueryResultShape.DYNAMIC, + input = QueryInput.Stream(ListQuery(Condition.ALL)), + ), + QueryInvocation( + target = target, + operation = QueryOperation.PAGE, + resultShape = QueryResultShape.TYPED, + input = QueryInput.Page(PagedQuery(Condition.ALL)), + ), + QueryInvocation( + target = target, + operation = QueryOperation.PAGE, + resultShape = QueryResultShape.DYNAMIC, + input = QueryInput.Page(PagedQuery(Condition.ALL)), + ), + QueryInvocation( + target = target, + operation = QueryOperation.COUNT, + resultShape = QueryResultShape.COUNT, + input = QueryInput.Count(Condition.ALL), + ), + QueryInvocation( + target = target, + operation = QueryOperation.ANALYZE, + resultShape = QueryResultShape.ANALYTICS, + input = QueryInput.Analytics(analytics), + ), + ) + + invocations.assert().hasSize(8) + } + + @Test + fun `analytics invocation should carry the semantic request`() { + val invocation = QueryInvocation( + target = target, + operation = QueryOperation.ANALYZE, + resultShape = QueryResultShape.ANALYTICS, + input = QueryInput.Analytics(analytics), + ) + + (invocation.input as QueryInput.Analytics).query.assert().isSameAs(analytics) + } + + @Test + fun `should reject an input that does not match the operation`() { + assertThrownBy { + QueryInvocation( + target = target, + operation = QueryOperation.SINGLE, + resultShape = QueryResultShape.TYPED, + input = QueryInput.Stream(ListQuery(Condition.ALL)), + ) + } + } + + @Test + fun `should reject a result shape that does not match the operation`() { + assertThrownBy { + QueryInvocation( + target = target, + operation = QueryOperation.COUNT, + resultShape = QueryResultShape.DYNAMIC, + input = QueryInput.Count(Condition.ALL), + ) + } + } + + @Test + fun `execution and validation modes should remain independent`() { + val combinations = QueryExecutionMode.entries.flatMap { executionMode -> + QueryValidationMode.entries.map { validationMode -> executionMode to validationMode } + } + + combinations.assert().hasSize(6) + combinations.assert().contains(QueryExecutionMode.SHADOW to QueryValidationMode.STRICT) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/normalization/NormalizedConditionTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/normalization/NormalizedConditionTest.kt new file mode 100644 index 00000000000..8a10794c747 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/normalization/NormalizedConditionTest.kt @@ -0,0 +1,79 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.normalization + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.NormalizedValue +import org.junit.jupiter.api.Test + +class NormalizedConditionTest { + + @Test + fun `logical path should isolate and expose immutable segments`() { + val source = mutableListOf("state", "name") + val path = LogicalField.Path(source, PathBasis.ROOT) + + source.add("late") + + path.segments.assert().containsExactly("state", "name") + assertThrownBy { + @Suppress("UNCHECKED_CAST") + (path.segments as MutableList).add("other") + } + } + + @Test + fun `junction should isolate mutable children`() { + val source = mutableListOf(NormalizedCondition.All) + val junction = NormalizedCondition.Junction(JunctionOperator.AND, source) + + source.add(NormalizedCondition.All) + + junction.children.assert().containsExactly(NormalizedCondition.All) + } + + @Test + fun `predicate should enforce value arity`() { + val field = LogicalField.Path(listOf("state", "status"), PathBasis.ROOT) + + assertThrownBy { + NormalizedCondition.Predicate(field, PredicateOperator.EQ) + } + assertThrownBy { + NormalizedCondition.Predicate(field, PredicateOperator.IS_NULL, NormalizedValue.Null) + } + } + + @Test + fun `search scope and text should not be blank`() { + assertThrownBy { + SearchScopeId(" ") + } + assertThrownBy { + NormalizedCondition.Search(SearchScope.Named(SearchScopeId("default")), " ") + } + } + + @Test + fun `native condition should retain immutable backend json`() { + val condition = NormalizedCondition.Native( + backendId = BackendId("elasticsearch"), + payload = Utf8Json("{\"term\":{\"state\":\"PAID\"}}"), + ) + + condition.backendId.value.assert().isEqualTo("elasticsearch") + condition.payload.value.assert().isEqualTo("{\"term\":{\"state\":\"PAID\"}}") + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/normalization/NormalizedValueTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/normalization/NormalizedValueTest.kt new file mode 100644 index 00000000000..2f867affdf3 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/normalization/NormalizedValueTest.kt @@ -0,0 +1,101 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.normalization + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.NormalizedValue +import org.junit.jupiter.api.Test + +class NormalizedValueTest { + + @Test + fun `bytes should have content equality and defensive access`() { + val source = byteArrayOf(1, 2, 3) + val value = NormalizedValue.Bytes(source) + val expected = NormalizedValue.Bytes(byteArrayOf(1, 2, 3)) + val initialHashCode = value.hashCode() + + source[0] = 9 + val exposed = value.toByteArray() + exposed[1] = 9 + + value.assert().isEqualTo(expected) + value.hashCode().assert().isEqualTo(initialHashCode) + value.toByteArray().assert().isEqualTo(byteArrayOf(1, 2, 3)) + } + + @Test + fun `list and object should deeply isolate mutable inputs`() { + val sourceBytes = byteArrayOf(1, 2) + val sourceList = mutableListOf(NormalizedValue.Bytes(sourceBytes)) + val sourceMap = linkedMapOf("items" to NormalizedValue.ListValue(sourceList)) + val value = NormalizedValue.ObjectValue(sourceMap) + val initialHashCode = value.hashCode() + + sourceBytes[0] = 9 + sourceList.add(NormalizedValue.Text("late")) + sourceMap.clear() + + value.assert().isEqualTo( + NormalizedValue.ObjectValue( + mapOf("items" to NormalizedValue.ListValue(listOf(NormalizedValue.Bytes(byteArrayOf(1, 2))))), + ), + ) + value.hashCode().assert().isEqualTo(initialHashCode) + } + + @Test + fun `immutable collections should not be mutable through a cast`() { + val list = NormalizedValue.ListValue(listOf(NormalizedValue.Text("value"))) + val map = NormalizedValue.ObjectValue(mapOf("key" to NormalizedValue.Text("value"))) + + assertThrownBy { + @Suppress("UNCHECKED_CAST") + (list.values as MutableList).add(NormalizedValue.Null) + } + assertThrownBy { + @Suppress("UNCHECKED_CAST") + (map.values as MutableMap)["other"] = NormalizedValue.Null + } + } + + @Test + fun `object equality should preserve Mongo document field order`() { + val first = NormalizedValue.ObjectValue( + linkedMapOf("a" to NormalizedValue.Int64(1), "b" to NormalizedValue.Int64(2)), + ) + val reversed = NormalizedValue.ObjectValue( + linkedMapOf("b" to NormalizedValue.Int64(2), "a" to NormalizedValue.Int64(1)), + ) + + first.assert().isNotEqualTo(reversed) + first.hashCode().assert().isNotEqualTo(reversed.hashCode()) + } + + @Test + fun `list should materialize a one-shot iterable exactly once`() { + var iteratorCalls = 0 + val oneShot = Iterable { + iteratorCalls++ + check(iteratorCalls == 1) + listOf(NormalizedValue.Text("value")).iterator() + } + + val value = NormalizedValue.ListValue(oneShot) + + value.values.assert().containsExactly(NormalizedValue.Text("value")) + iteratorCalls.assert().isEqualTo(1) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/normalization/QueryNormalizerTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/normalization/QueryNormalizerTest.kt new file mode 100644 index 00000000000..35491bc4b15 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/normalization/QueryNormalizerTest.kt @@ -0,0 +1,683 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.normalization + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DeletionState +import me.ahoo.wow.api.query.ListQuery +import me.ahoo.wow.api.query.Operator +import me.ahoo.wow.api.query.PagedQuery +import me.ahoo.wow.api.query.Pagination +import me.ahoo.wow.api.query.Projection +import me.ahoo.wow.api.query.SingleQuery +import me.ahoo.wow.api.query.Sort +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.admission.QueryAdmissionLimits +import me.ahoo.wow.query.internal.admission.RawAdmissionGuard +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryInput +import me.ahoo.wow.query.internal.model.QueryInvocation +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import java.time.Clock +import java.time.Instant +import java.time.ZoneId +import java.time.format.DateTimeFormatterBuilder +import java.time.temporal.ChronoField +import java.util.function.Consumer + +class QueryNormalizerTest { + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val guard = RawAdmissionGuard(QueryAdmissionLimits.DEFAULT) + private val fixedInstant = Instant.parse("2024-03-10T06:30:00Z") + private val zoneId = ZoneId.of("America/New_York") + + @Test + fun `should normalize all wire operators or reject native explicitly`() { + Operator.entries.forEach { operator -> + val invocation = countInvocation(validCondition(operator)) + if (operator == Operator.RAW) { + assertThrownBy { + QueryNormalizer(Clock.fixed(fixedInstant, zoneId)).normalize(guard.admit(invocation)) + }.satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(QueryRejectionCategory.UNSUPPORTED_FEATURE) + error.rejection.code.assert().isEqualTo(QueryRejectionCode.NATIVE_BACKEND_UNBOUND) + } + ) + } else { + QueryNormalizer(Clock.fixed(fixedInstant, zoneId)).normalize(guard.admit(invocation)) + } + } + } + + @Test + fun `should normalize system fields without guessing ordinary id paths`() { + val condition = Condition.and( + Condition.id("identity"), + Condition.ids("identity-1", "identity-2"), + Condition.aggregateId("aggregate"), + Condition.aggregateIds("aggregate-1", "aggregate-2"), + Condition.tenantId("tenant"), + Condition.ownerId("owner"), + Condition.spaceId("space"), + Condition.deleted(DeletionState.ACTIVE), + Condition.eq("id", "element-or-state-id"), + ) + + val normalized = normalizeCount(condition) + val children = (normalized as NormalizedCondition.Junction).children + + (children[0] as NormalizedCondition.Predicate).field.assert().isEqualTo( + LogicalField.System(SystemFieldKind.IDENTITY), + ) + (children[2] as NormalizedCondition.Predicate).field.assert().isEqualTo( + LogicalField.System(SystemFieldKind.AGGREGATE_ID), + ) + (children[4] as NormalizedCondition.Predicate).field.assert().isEqualTo( + LogicalField.System(SystemFieldKind.TENANT_ID), + ) + (children[7] as NormalizedCondition.Predicate).operator.assert().isEqualTo(PredicateOperator.IS_FALSE) + (children[8] as NormalizedCondition.Predicate).field.assert().isEqualTo( + LogicalField.Path(listOf("id"), PathBasis.ROOT), + ) + } + + @Test + fun `should normalize element match fields relative to each element scope`() { + val condition = Condition.elemMatch( + "items", + Condition.and( + Condition.eq("items.sku", "sku-a"), + Condition.eq("id", "line-id"), + Condition.elemMatch( + "attributes", + Condition.and( + Condition.eq("name", "size"), + Condition.eq("attributes.code", "size-code"), + Condition.eq("items.attributes.label", "size-label"), + ), + ), + ), + ) + + val normalized = normalizeCount(condition) as NormalizedCondition.ElementMatch + normalized.field.assert().isEqualTo(LogicalField.Path(listOf("items"), PathBasis.ROOT)) + val children = (normalized.condition as NormalizedCondition.Junction).children + (children[0] as NormalizedCondition.Predicate).field.assert().isEqualTo( + LogicalField.Path(listOf("sku"), PathBasis.CURRENT_ELEMENT), + ) + (children[1] as NormalizedCondition.Predicate).field.assert().isEqualTo( + LogicalField.Path(listOf("id"), PathBasis.CURRENT_ELEMENT), + ) + val nested = children[2] as NormalizedCondition.ElementMatch + nested.field.assert().isEqualTo( + LogicalField.Path(listOf("attributes"), PathBasis.CURRENT_ELEMENT), + ) + val nestedChildren = (nested.condition as NormalizedCondition.Junction).children + (nestedChildren[0] as NormalizedCondition.Predicate).field.assert().isEqualTo( + LogicalField.Path(listOf("name"), PathBasis.CURRENT_ELEMENT), + ) + (nestedChildren[1] as NormalizedCondition.Predicate).field.assert().isEqualTo( + LogicalField.Path(listOf("code"), PathBasis.CURRENT_ELEMENT), + ) + (nestedChildren[2] as NormalizedCondition.Predicate).field.assert().isEqualTo( + LogicalField.Path(listOf("label"), PathBasis.CURRENT_ELEMENT), + ) + } + + @Test + fun `should resolve absolute and qualified prefixes through three element levels`() { + val condition = Condition.elemMatch( + "items", + Condition.elemMatch( + "attributes", + Condition.elemMatch( + "attributes.values", + Condition.and( + Condition.eq("value", "large"), + Condition.eq("values.code", "large-code"), + Condition.eq("attributes.values.label", "large-label"), + Condition.eq("items.attributes.values.kind", "size-kind"), + ), + ), + ), + ) + + val firstLevel = normalizeCount(condition) as NormalizedCondition.ElementMatch + val secondLevel = firstLevel.condition as NormalizedCondition.ElementMatch + val thirdLevel = secondLevel.condition as NormalizedCondition.ElementMatch + thirdLevel.field.assert().isEqualTo(LogicalField.Path(listOf("values"), PathBasis.CURRENT_ELEMENT)) + val fields = (thirdLevel.condition as NormalizedCondition.Junction).children.map { child -> + (child as NormalizedCondition.Predicate).field + } + fields.assert().containsExactly( + LogicalField.Path(listOf("value"), PathBasis.CURRENT_ELEMENT), + LogicalField.Path(listOf("code"), PathBasis.CURRENT_ELEMENT), + LogicalField.Path(listOf("label"), PathBasis.CURRENT_ELEMENT), + LogicalField.Path(listOf("kind"), PathBasis.CURRENT_ELEMENT), + ) + } + + @Test + fun `should reject system operators inside element scope`() { + val condition = Condition.elemMatch("items", Condition.id("line-id")) + + assertThrownBy { + normalizeCount(condition) + }.satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(QueryRejectionCategory.INVALID_QUERY) + error.rejection.code.assert().isEqualTo(QueryRejectionCode.SYSTEM_FIELD_IN_ELEMENT_SCOPE) + error.rejection.path.toString().assert().isEqualTo("$.input.condition.children[0]") + } + ) + } + + @Test + fun `should freeze clock once and expand time macros as DST safe half open ranges`() { + val clock = CountingClock(fixedInstant, zoneId) + val condition = Condition.and( + Condition.today("createdAt"), + Condition.tomorrow("updatedAt"), + ) + val admitted = guard.admit(countInvocation(condition)) + + val normalized = QueryNormalizer(clock).normalize(admitted) + val children = ((normalized.input as NormalizedQueryInput.Count).userCondition as NormalizedCondition.Junction) + .children + + clock.instantReads.assert().isEqualTo(1) + children[0].assert().isEqualTo( + halfOpenRange( + "createdAt", + Instant.parse("2024-03-10T05:00:00Z"), + Instant.parse("2024-03-11T04:00:00Z"), + ), + ) + children[1].assert().isEqualTo( + halfOpenRange( + "updatedAt", + Instant.parse("2024-03-11T04:00:00Z"), + Instant.parse("2024-03-12T04:00:00Z"), + ), + ) + } + + @Test + fun `should normalize search and Mongo baseline empty collection constants`() { + normalizeCount(Condition.match("description", "distributed systems")).assert().isEqualTo( + NormalizedCondition.Search( + SearchScope.LegacyField(LogicalField.Path(listOf("description"), PathBasis.ROOT)), + "distributed systems", + ), + ) + normalizeCount(Condition("field", Operator.IN, emptyList())).assert() + .isEqualTo(NormalizedCondition.None) + normalizeCount(Condition("field", Operator.NOT_IN, emptyList())).assert() + .isEqualTo(NormalizedCondition.All) + normalizeCount(Condition("field", Operator.ALL_IN, emptyList())).assert() + .isEqualTo(NormalizedCondition.None) + + val allIn = normalizeCount( + Condition("field", Operator.ALL_IN, listOf(1, 1L, 1.0)), + ) as NormalizedCondition.Predicate + (allIn.value as NormalizedValue.ListValue).values.assert().containsExactly(NormalizedValue.Int64(1)) + + val orderedDocument = linkedMapOf("a" to 1, "b" to 2) + val reversedDocument = linkedMapOf("b" to 2, "a" to 1) + val documentIn = normalizeCount( + Condition("field", Operator.IN, listOf(orderedDocument, reversedDocument)), + ) as NormalizedCondition.Predicate + (documentIn.value as NormalizedValue.ListValue).values.assert().containsExactly( + NormalizedValue.ObjectValue( + linkedMapOf("a" to NormalizedValue.Int64(1), "b" to NormalizedValue.Int64(2)), + ), + NormalizedValue.ObjectValue( + linkedMapOf("b" to NormalizedValue.Int64(2), "a" to NormalizedValue.Int64(1)), + ), + ) + } + + @Test + fun `should map field predicate operators without backend escaping`() { + val comparisonOperators = mapOf( + Operator.EQ to PredicateOperator.EQ, + Operator.NE to PredicateOperator.NE, + Operator.GT to PredicateOperator.GT, + Operator.LT to PredicateOperator.LT, + Operator.GTE to PredicateOperator.GTE, + Operator.LTE to PredicateOperator.LTE, + Operator.IN to PredicateOperator.IN, + Operator.NOT_IN to PredicateOperator.NOT_IN, + Operator.ALL_IN to PredicateOperator.ALL_IN, + Operator.BETWEEN to PredicateOperator.BETWEEN, + ) + comparisonOperators.forEach { (wire, expected) -> + val rawValue = if (wire in COLLECTION_OPERATORS) listOf(1, 2) else 1 + val predicate = normalizeCount(Condition("field", wire, rawValue)) as NormalizedCondition.Predicate + predicate.operator.assert().isEqualTo(expected) + } + + val literal = normalizeCount(Condition.contains("field", "*?\\", ignoreCase = true)) + as NormalizedCondition.Predicate + literal.value.assert().isEqualTo(NormalizedValue.Text("*?\\")) + literal.operator.assert().isEqualTo(PredicateOperator.CONTAINS) + literal.options.caseSensitivity.assert().isEqualTo(CaseSensitivity.INSENSITIVE) + } + + @Test + fun `should normalize constant boolean deletion and junction truth tables`() { + val fieldless = mapOf( + Operator.NULL to PredicateOperator.IS_NULL, + Operator.NOT_NULL to PredicateOperator.NOT_NULL, + Operator.TRUE to PredicateOperator.IS_TRUE, + Operator.FALSE to PredicateOperator.IS_FALSE, + ) + fieldless.forEach { (wire, expected) -> + val predicate = normalizeCount(Condition("field", wire)) as NormalizedCondition.Predicate + predicate.operator.assert().isEqualTo(expected) + predicate.value.assert().isNull() + } + (normalizeCount(Condition.exists("field", false)) as NormalizedCondition.Predicate).value.assert() + .isEqualTo(NormalizedValue.BooleanValue(false)) + (normalizeCount(Condition.deleted(true)) as NormalizedCondition.Predicate).operator.assert() + .isEqualTo(PredicateOperator.IS_TRUE) + (normalizeCount(Condition.deleted(false)) as NormalizedCondition.Predicate).operator.assert() + .isEqualTo(PredicateOperator.IS_FALSE) + normalizeCount(Condition.deleted(DeletionState.ALL)).assert().isEqualTo(NormalizedCondition.All) + normalizeCount(Condition.nor(Condition.ALL)).assert().isEqualTo(NormalizedCondition.None) + normalizeCount(Condition.nor(Condition("field", Operator.IN, emptyList()))).assert() + .isEqualTo(NormalizedCondition.All) + normalizeCount(Condition(operator = Operator.IDS, value = emptyList())).assert() + .isEqualTo(NormalizedCondition.None) + normalizeCount(Condition(operator = Operator.AGGREGATE_IDS, value = emptyList())).assert() + .isEqualTo(NormalizedCondition.None) + } + + @Test + fun `should preserve root deletion intent before normalization removes deleted all`() { + val ordinaryAll = normalizeCountInput(Condition.ALL) + ordinaryAll.deletionScope.assert().isEqualTo(NormalizedDeletionScope.DEFAULT_ACTIVE) + ordinaryAll.userCondition.assert().isEqualTo(NormalizedCondition.All) + + val explicitAll = normalizeCountInput(Condition.deleted(DeletionState.ALL)) + explicitAll.deletionScope.assert().isEqualTo(NormalizedDeletionScope.EXPLICIT) + explicitAll.userCondition.assert().isEqualTo(NormalizedCondition.All) + + normalizeCountInput( + Condition.and( + Condition.eq("state.name", "Ada"), + Condition.deleted(DeletionState.ALL), + ), + ).deletionScope.assert().isEqualTo(NormalizedDeletionScope.EXPLICIT) + + normalizeCountInput( + Condition.or( + Condition.eq("state.name", "Ada"), + Condition.deleted(DeletionState.ALL), + ), + ).deletionScope.assert().isEqualTo(NormalizedDeletionScope.DEFAULT_ACTIVE) + + val single = normalize( + QueryInvocation( + target, + QueryOperation.SINGLE, + QueryResultShape.DYNAMIC, + QueryInput.Single(SingleQuery(Condition.ALL)), + ), + ).input as NormalizedQueryInput.Single + single.query.deletionScope.assert().isEqualTo(NormalizedDeletionScope.DEFAULT_ACTIVE) + } + + @Test + fun `should expand week month and relative time operators from the frozen instant`() { + normalizeCount(Condition.thisWeek("field")).assert().isEqualTo( + halfOpenRange("field", Instant.parse("2024-03-04T05:00:00Z"), Instant.parse("2024-03-11T04:00:00Z")), + ) + normalizeCount(Condition.nextWeek("field")).assert().isEqualTo( + halfOpenRange("field", Instant.parse("2024-03-11T04:00:00Z"), Instant.parse("2024-03-18T04:00:00Z")), + ) + normalizeCount(Condition.lastWeek("field")).assert().isEqualTo( + halfOpenRange("field", Instant.parse("2024-02-26T05:00:00Z"), Instant.parse("2024-03-04T05:00:00Z")), + ) + normalizeCount(Condition.thisMonth("field")).assert().isEqualTo( + halfOpenRange("field", Instant.parse("2024-03-01T05:00:00Z"), Instant.parse("2024-04-01T04:00:00Z")), + ) + normalizeCount(Condition.lastMonth("field")).assert().isEqualTo( + halfOpenRange("field", Instant.parse("2024-02-01T05:00:00Z"), Instant.parse("2024-03-01T05:00:00Z")), + ) + normalizeCount(Condition.recentDays("field", 2)).assert().isEqualTo( + halfOpenRange("field", Instant.parse("2024-03-09T05:00:00Z"), Instant.parse("2024-03-11T04:00:00Z")), + ) + normalizeCount(Condition.earlierDays("field", 2)).assert().isEqualTo( + NormalizedCondition.Predicate( + LogicalField.Path(listOf("field"), PathBasis.ROOT), + PredicateOperator.LT, + NormalizedValue.InstantValue(Instant.parse("2024-03-09T05:00:00Z")), + ), + ) + normalizeCount(Condition.beforeToday("field", "12:30:00")).assert().isEqualTo( + NormalizedCondition.Predicate( + LogicalField.Path(listOf("field"), PathBasis.ROOT), + PredicateOperator.LT, + NormalizedValue.InstantValue(Instant.parse("2024-03-10T16:30:00Z")), + ), + ) + } + + @Test + fun `should apply explicit time zone and date pattern without backend types`() { + val condition = Condition( + field = "field", + operator = Operator.TODAY, + options = mapOf( + Condition.ZONE_ID_OPTION_KEY to "UTC", + Condition.DATE_PATTERN_OPTION_KEY to "yyyy-MM-dd HH:mm", + ), + ) + + normalizeCount(condition).assert().isEqualTo( + NormalizedCondition.Junction( + JunctionOperator.AND, + listOf( + NormalizedCondition.Predicate( + LogicalField.Path(listOf("field"), PathBasis.ROOT), + PredicateOperator.GTE, + NormalizedValue.Text("2024-03-10 00:00"), + ), + NormalizedCondition.Predicate( + LogicalField.Path(listOf("field"), PathBasis.ROOT), + PredicateOperator.LT, + NormalizedValue.Text("2024-03-11 00:00"), + ), + ), + ), + ) + } + + @Test + fun `should reject an admitted formatter that cannot format the frozen instant`() { + val formatter = DateTimeFormatterBuilder() + .appendValue(ChronoField.YEAR, 1) + .toFormatter() + val condition = Condition( + field = "field", + operator = Operator.TODAY, + options = mapOf(Condition.DATE_PATTERN_OPTION_KEY to formatter), + ) + + assertRejected(QueryRejectionCode.INVALID_OPTION_VALUE, "$.input.condition.options['datePattern']") { + normalizeCount(condition) + } + } + + @Test + fun `should return typed rejections for malformed search element scope and temporal overflow`() { + listOf("", " ").forEach { text -> + assertRejected(QueryRejectionCode.INVALID_VALUE_TYPE, "$.input.condition.value") { + normalizeCount(Condition.match("field", text)) + } + } + assertRejected(QueryRejectionCode.INVALID_FIELD, "$.input.condition.children[0].field") { + normalizeCount(Condition.elemMatch("items", Condition.eq("items", "value"))) + } + listOf(Operator.RECENT_DAYS, Operator.EARLIER_DAYS).forEach { operator -> + assertRejected(QueryRejectionCode.INVALID_TIME_VALUE, "$.input.condition.value") { + normalizeCount(Condition("field", operator, Long.MAX_VALUE)) + } + } + assertRejected( + QueryRejectionCode.NATIVE_BACKEND_UNBOUND, + "$.input.condition", + QueryRejectionCategory.UNSUPPORTED_FEATURE, + ) { + normalizeCount(Condition.raw("{}")) + } + } + + @Test + fun `should normalize projection sort list limit and long page offset`() { + val pageInvocation = QueryInvocation( + target = target, + operation = QueryOperation.PAGE, + resultShape = QueryResultShape.DYNAMIC, + input = QueryInput.Page( + PagedQuery( + condition = Condition.ALL, + projection = Projection(include = listOf("state.name", "state.amount")), + sort = listOf( + Sort("state.amount", Sort.Direction.DESC), + Sort("id", Sort.Direction.ASC), + ), + pagination = Pagination(Int.MAX_VALUE, Int.MAX_VALUE), + ), + ), + ) + + val normalized = normalize(pageInvocation) + val page = normalized.input as NormalizedQueryInput.Page + val projection = page.query.projection as NormalizedProjection.Include + + projection.fields.values.assert().containsExactly( + LogicalField.Path(listOf("state", "name"), PathBasis.ROOT), + LogicalField.Path(listOf("state", "amount"), PathBasis.ROOT), + ) + page.query.sort.map { it.direction }.assert().containsExactly( + NormalizedSortDirection.DESC, + NormalizedSortDirection.ASC, + ) + page.page.offset.assert().isEqualTo( + (Int.MAX_VALUE.toLong() - 1) * Int.MAX_VALUE.toLong(), + ) + + val listInvocation = QueryInvocation( + target = target, + operation = QueryOperation.STREAM, + resultShape = QueryResultShape.TYPED, + input = QueryInput.Stream(ListQuery(Condition.ALL, limit = 0)), + ) + (normalize(listInvocation).input as NormalizedQueryInput.Stream).limit.assert().isEqualTo(0) + } + + @Test + fun `should preserve mixed projection for planner policy and reject invalid limit or page`() { + val mixed = SingleQuery( + condition = Condition.ALL, + projection = Projection(include = listOf("state.name"), exclude = listOf("state.secret")), + ) + val normalizedMixed = normalize( + QueryInvocation( + target, + QueryOperation.SINGLE, + QueryResultShape.TYPED, + QueryInput.Single(mixed), + ), + ) + val projection = (normalizedMixed.input as NormalizedQueryInput.Single).query.projection + projection.assert().isInstanceOf(NormalizedProjection.Mixed::class.java) + + assertAdmissionRejected(QueryRejectionCode.INVALID_LIMIT) { + guard.admit( + QueryInvocation( + target, + QueryOperation.STREAM, + QueryResultShape.TYPED, + QueryInput.Stream(ListQuery(Condition.ALL, limit = -1)), + ), + ) + } + listOf(Pagination(index = 0, size = 10), Pagination(index = 1, size = 0)).forEach { page -> + assertAdmissionRejected(QueryRejectionCode.INVALID_PAGE) { + guard.admit( + QueryInvocation( + target, + QueryOperation.PAGE, + QueryResultShape.TYPED, + QueryInput.Page(PagedQuery(Condition.ALL, pagination = page)), + ), + ) + } + } + } + + private fun normalize(invocation: QueryInvocation): NormalizedQueryInvocation = + QueryNormalizer(Clock.fixed(fixedInstant, zoneId)).normalize(guard.admit(invocation)) + + private fun normalizeCount(condition: Condition): NormalizedCondition = + normalizeCountInput(condition).userCondition + + private fun normalizeCountInput(condition: Condition): NormalizedQueryInput.Count = + normalize(countInvocation(condition)).input as NormalizedQueryInput.Count + + private fun countInvocation(condition: Condition): QueryInvocation = + QueryInvocation( + target = target, + operation = QueryOperation.COUNT, + resultShape = QueryResultShape.COUNT, + input = QueryInput.Count(condition), + ) + + private fun assertAdmissionRejected(code: QueryRejectionCode, action: () -> Any?) { + assertThrownBy { + action() + }.satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(QueryRejectionCategory.INVALID_QUERY) + error.rejection.code.assert().isEqualTo(code) + } + ) + } + + private fun assertRejected( + code: QueryRejectionCode, + path: String, + category: QueryRejectionCategory = QueryRejectionCategory.INVALID_QUERY, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo(path) + }, + ) + } + + private fun halfOpenRange(field: String, from: Instant, to: Instant): NormalizedCondition = + NormalizedCondition.Junction( + JunctionOperator.AND, + listOf( + NormalizedCondition.Predicate( + LogicalField.Path(field.split('.'), PathBasis.ROOT), + PredicateOperator.GTE, + NormalizedValue.InstantValue(from), + ), + NormalizedCondition.Predicate( + LogicalField.Path(field.split('.'), PathBasis.ROOT), + PredicateOperator.LT, + NormalizedValue.InstantValue(to), + ), + ), + ) + + @Suppress("CyclomaticComplexMethod") + private fun validCondition(operator: Operator): Condition = + when (operator) { + Operator.AND -> Condition.and(Condition.ALL) + Operator.OR -> Condition.or(Condition.ALL) + Operator.NOR -> Condition.nor(Condition.eq("field", "value")) + Operator.ID -> Condition.id("id") + Operator.IDS -> Condition.ids("id") + Operator.AGGREGATE_ID -> Condition.aggregateId("id") + Operator.AGGREGATE_IDS -> Condition.aggregateIds("id") + Operator.TENANT_ID -> Condition.tenantId("tenant") + Operator.OWNER_ID -> Condition.ownerId("owner") + Operator.SPACE_ID -> Condition.spaceId("space") + Operator.DELETED -> Condition.deleted(false) + Operator.ALL -> Condition.ALL + Operator.EQ, + Operator.NE, + Operator.GT, + Operator.LT, + Operator.GTE, + Operator.LTE, + -> Condition("field", operator, 1) + Operator.CONTAINS, + Operator.STARTS_WITH, + Operator.ENDS_WITH, + Operator.MATCH, + -> Condition("field", operator, "value") + Operator.IN, + Operator.NOT_IN, + Operator.ALL_IN, + -> Condition("field", operator, listOf(1, 2)) + Operator.BETWEEN -> Condition.between("field", 1, 2) + Operator.ELEM_MATCH -> Condition.elemMatch("items", Condition.eq("name", "value")) + Operator.NULL, + Operator.NOT_NULL, + Operator.TRUE, + Operator.FALSE, + -> Condition("field", operator) + Operator.EXISTS -> Condition.exists("field") + Operator.TODAY, + Operator.TOMORROW, + Operator.THIS_WEEK, + Operator.NEXT_WEEK, + Operator.LAST_WEEK, + Operator.THIS_MONTH, + Operator.LAST_MONTH, + -> Condition("field", operator) + Operator.BEFORE_TODAY -> Condition.beforeToday("field", "12:30:00") + Operator.RECENT_DAYS, + Operator.EARLIER_DAYS, + -> Condition("field", operator, 2) + Operator.RAW -> Condition.raw("{}") + } + + private class CountingClock( + private val fixedInstant: Instant, + private val fixedZone: ZoneId, + ) : Clock() { + var instantReads: Int = 0 + private set + + override fun getZone(): ZoneId = fixedZone + + override fun withZone(zone: ZoneId): Clock = CountingClock(fixedInstant, zone) + + override fun instant(): Instant = fixedInstant.also { instantReads++ } + } + + companion object { + private val COLLECTION_OPERATORS = setOf( + Operator.IN, + Operator.NOT_IN, + Operator.ALL_IN, + Operator.BETWEEN, + ) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/plan/QueryPlanFingerprintTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/plan/QueryPlanFingerprintTest.kt new file mode 100644 index 00000000000..467e7c0c4de --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/plan/QueryPlanFingerprintTest.kt @@ -0,0 +1,235 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.plan + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.JunctionOperator +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedProjection +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.Utf8Json +import me.ahoo.wow.query.internal.planning.PlanningConstraints +import me.ahoo.wow.query.internal.planning.PlanningDecision +import me.ahoo.wow.query.internal.planning.PlanningFixtures +import me.ahoo.wow.query.internal.planning.QueryPlanner +import me.ahoo.wow.query.internal.value.NonEmptyList +import org.junit.jupiter.api.Test + +class QueryPlanFingerprintTest { + + private val planner = QueryPlanner() + + @Test + fun `fingerprint should be stable across schema and projection registration order`() { + val includeA = NormalizedProjection.Include( + NonEmptyList.of( + PlanningFixtures.path("state", "amount"), + PlanningFixtures.path("state", "name"), + ), + ) + val includeB = NormalizedProjection.Include( + NonEmptyList.of( + PlanningFixtures.path("state", "name"), + PlanningFixtures.path("state", "amount"), + PlanningFixtures.path("state", "name"), + ), + ) + val reorderedSchema = QueryDocumentSchema( + PlanningFixtures.schema.target, + PlanningFixtures.schema.fields.values.reversed(), + PlanningFixtures.schema.searchScopes.values.reversed(), + ) + + val first = plan(includeA, PlanningFixtures.schema) + val second = plan(includeB, reorderedSchema) + + first.fingerprint.assert().isEqualTo(second.fingerprint) + first.fingerprint.value.assert().isEqualTo( + "49a4b6842c6441a3bd5c2a4a05a2e94c49bb6e88494454204d1a05b7fd669388", + ) + (first.projection as PlannedProjection.Include).fields.values.assert().containsExactly( + PlanningFixtures.amount, + PlanningFixtures.name, + ) + } + + @Test + fun `fingerprint should preserve provenance and condition order`() { + val name = predicate(PlanningFixtures.name, "Ada") + val tenant = predicate(PlanningFixtures.tenant, "tenant-1") + val first = planCondition(name, tenant) + val swapped = planCondition(tenant, name) + val reordered = planCondition( + NormalizedCondition.Junction(JunctionOperator.OR, listOf(name, tenant)), + NormalizedCondition.All, + ) + val reverseOrder = planCondition( + NormalizedCondition.Junction(JunctionOperator.OR, listOf(tenant, name)), + NormalizedCondition.All, + ) + + first.fingerprint.assert().isNotEqualTo(swapped.fingerprint) + reordered.fingerprint.assert().isNotEqualTo(reverseOrder.fingerprint) + first.fingerprint.value.length.assert().isEqualTo(64) + QueryPlanFingerprint.VERSION.assert().isEqualTo(1) + } + + @Test + fun `fingerprint should preserve Mongo object entry order`() { + val first = NormalizedValue.ObjectValue( + linkedMapOf( + "a" to NormalizedValue.Int64(1), + "b" to NormalizedValue.Int64(2), + ), + ) + val reversed = NormalizedValue.ObjectValue( + linkedMapOf( + "b" to NormalizedValue.Int64(2), + "a" to NormalizedValue.Int64(1), + ), + ) + + fingerprintFor(first).assert().isNotEqualTo(fingerprintFor(reversed)) + } + + @Test + fun `planned collections should be defensively immutable`() { + val plan = plan(NormalizedProjection.All, PlanningFixtures.schema) + + assertThrownBy { + @Suppress("UNCHECKED_CAST") + (plan.sort as MutableList).clear() + } + assertThrownBy { + @Suppress("UNCHECKED_CAST") + (plan.requiredCapabilities.fieldRequirements as MutableMap>).clear() + } + } + + @Test + fun `fingerprint should encode record window sort origin and native binding`() { + val unbounded = planInvocation( + NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.STREAM, + QueryResultShape.DYNAMIC, + NormalizedQueryInput.Stream(PlanningFixtures.recordQuery(), 0), + ), + ) + val bounded = planInvocation( + NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.STREAM, + QueryResultShape.DYNAMIC, + NormalizedQueryInput.Stream(PlanningFixtures.recordQuery(), 1), + ), + ) + unbounded.fingerprint.assert().isNotEqualTo(bounded.fingerprint) + + val firstPage = planInvocation(PlanningFixtures.page(index = 1, size = 20, offset = 0)) + val secondPage = planInvocation(PlanningFixtures.page(index = 2, size = 20, offset = 20)) + firstPage.fingerprint.assert().isNotEqualTo(secondPage.fingerprint) + val explicitIdentity = planInvocation( + PlanningFixtures.page( + PlanningFixtures.recordQuery( + sort = listOf(PlanningFixtures.sort(PlanningFixtures.path("aggregateId"))), + ), + ), + ) + firstPage.fingerprint.assert().isNotEqualTo(explicitIdentity.fingerprint) + + val mongo = nativePlan("mongo", "{}") + mongo.fingerprint.assert().isNotEqualTo(nativePlan("mongo", "{\"x\":1}").fingerprint) + mongo.fingerprint.assert().isNotEqualTo(nativePlan("elasticsearch", "{}").fingerprint) + } + + private fun plan( + projection: NormalizedProjection, + schema: QueryDocumentSchema, + ): SingleQueryPlan = + ( + planner.plan( + PlanningFixtures.single( + PlanningFixtures.recordQuery(projection = projection), + QueryResultShape.DYNAMIC, + ), + schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + ).plan as SingleQueryPlan + + private fun planCondition( + user: NormalizedCondition, + mandatory: NormalizedCondition, + ): SingleQueryPlan = + ( + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(user)), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT, mandatory), + ) as PlanningDecision.Planned + ).plan as SingleQueryPlan + + private fun planInvocation(invocation: NormalizedQueryInvocation): QueryPlan = + ( + planner.plan( + invocation, + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + ).plan + + private fun nativePlan(backend: String, payload: String): QueryPlan { + val condition = NormalizedCondition.Native(BackendId(backend), Utf8Json(payload)) + return planInvocation(PlanningFixtures.single(PlanningFixtures.recordQuery(condition))) + } + + private fun fingerprintFor(value: NormalizedValue): PlanFingerprint { + val base = plan(NormalizedProjection.All, PlanningFixtures.schema) + val condition = PlannedCondition.Predicate(PlanningFixtures.name, PredicateOperator.EQ, value) + val plan = SingleQueryPlan.create( + base.target, + base.schemaContractId, + EnforcedFilter(condition, PlannedCondition.All), + base.resultShape, + base.projection, + base.sort, + base.requiredCapabilities, + base.semanticTier, + ) + return QueryPlanFingerprint.compute(plan) + } + + private fun predicate( + field: QueryFieldId, + value: String, + ): NormalizedCondition.Predicate { + val logical = + when (field) { + is QueryFieldId.System -> me.ahoo.wow.query.internal.normalization.LogicalField.System(field.kind) + is QueryFieldId.Path -> PlanningFixtures.path(*field.segments.toTypedArray()) + } + return NormalizedCondition.Predicate(logical, PredicateOperator.EQ, NormalizedValue.Text(value)) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/plan/QueryPlanTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/plan/QueryPlanTest.kt new file mode 100644 index 00000000000..5f001bcc986 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/plan/QueryPlanTest.kt @@ -0,0 +1,33 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.plan + +import me.ahoo.test.asserts.assertThrownBy +import org.junit.jupiter.api.Test + +class QueryPlanTest { + + @Test + fun `bounded stream and page window should reject invalid values`() { + assertThrownBy { + StreamLimit.Bounded(0) + } + assertThrownBy { + PageWindow(offset = -1, size = 10) + } + assertThrownBy { + PageWindow(offset = 0, size = 0) + } + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/PlanningFixtures.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/PlanningFixtures.kt new file mode 100644 index 00000000000..23cdd1e33ae --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/PlanningFixtures.kt @@ -0,0 +1,272 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.EmptyArraySemantics +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.QuerySearchScopeDefinition +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedDeletionScope +import me.ahoo.wow.query.internal.normalization.NormalizedProjection +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.normalization.NormalizedRecordQuery +import me.ahoo.wow.query.internal.normalization.NormalizedSort +import me.ahoo.wow.query.internal.normalization.NormalizedSortDirection +import me.ahoo.wow.query.internal.normalization.PathBasis +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.SearchScope +import me.ahoo.wow.query.internal.normalization.SearchScopeId + +internal object PlanningFixtures { + val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + val identity = QueryFieldId.System(me.ahoo.wow.query.internal.normalization.SystemFieldKind.IDENTITY) + val tenant = QueryFieldId.System(me.ahoo.wow.query.internal.normalization.SystemFieldKind.TENANT_ID) + val deleted = QueryFieldId.System(me.ahoo.wow.query.internal.normalization.SystemFieldKind.DELETED) + val state = QueryFieldId.Path(listOf("state")) + val name = QueryFieldId.Path(listOf("state", "name")) + val amount = QueryFieldId.Path(listOf("state", "amount")) + val description = QueryFieldId.Path(listOf("state", "description")) + val createdAt = QueryFieldId.Path(listOf("state", "createdAt")) + val tags = QueryFieldId.Path(listOf("state", "tags")) + val items = QueryFieldId.Path(listOf("state", "items")) + val itemName = QueryFieldId.Path(listOf("state", "items", "name")) + val itemAttributes = QueryFieldId.Path(listOf("state", "items", "attributes")) + val itemAttributeName = QueryFieldId.Path(listOf("state", "items", "attributes", "name")) + val searchScopeId = SearchScopeId("order-description") + + val schema = QueryDocumentSchema( + target = target, + fields = listOf( + field( + identity, + LogicalFieldType.Text, + setOf(PredicateOperator.EQ, PredicateOperator.IN), + setOf(FieldCapability.EXACT, FieldCapability.SORTABLE), + aliases = setOf(QueryFieldId.Path(listOf("aggregateId"))), + ), + field( + tenant, + LogicalFieldType.Text, + setOf(PredicateOperator.EQ, PredicateOperator.IN), + setOf(FieldCapability.EXACT), + ), + field( + deleted, + LogicalFieldType.Boolean, + setOf(PredicateOperator.IS_TRUE, PredicateOperator.IS_FALSE), + setOf(FieldCapability.EXACT), + ), + field(state, LogicalFieldType.Object), + field( + name, + LogicalFieldType.Text, + setOf( + PredicateOperator.EQ, + PredicateOperator.IN, + PredicateOperator.CONTAINS, + PredicateOperator.STARTS_WITH, + PredicateOperator.ENDS_WITH, + PredicateOperator.IS_NULL, + PredicateOperator.NOT_NULL, + PredicateOperator.EXISTS, + ), + setOf( + FieldCapability.EXACT, + FieldCapability.PRESENCE, + FieldCapability.LITERAL_PATTERN, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + ), + ), + field( + amount, + LogicalFieldType.Decimal, + setOf( + PredicateOperator.EQ, + PredicateOperator.IN, + PredicateOperator.GT, + PredicateOperator.GTE, + PredicateOperator.LT, + PredicateOperator.LTE, + PredicateOperator.BETWEEN, + ), + setOf( + FieldCapability.EXACT, + FieldCapability.RANGE, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + FieldCapability.AGGREGATABLE, + ), + ), + field( + description, + LogicalFieldType.Text, + emptySet(), + setOf(FieldCapability.FULL_TEXT, FieldCapability.PROJECTABLE), + ), + field( + createdAt, + LogicalFieldType.Instant, + setOf(PredicateOperator.EQ, PredicateOperator.GT, PredicateOperator.GTE), + setOf( + FieldCapability.EXACT, + FieldCapability.RANGE, + FieldCapability.SORTABLE, + FieldCapability.PROJECTABLE, + FieldCapability.AGGREGATABLE, + ), + ), + field( + tags, + LogicalFieldType.Array( + elementType = LogicalFieldType.Text, + elementNullability = Nullability.NULLABLE, + emptySemantics = EmptyArraySemantics.DISTINCT, + ), + setOf( + PredicateOperator.EQ, + PredicateOperator.IN, + PredicateOperator.NOT_IN, + PredicateOperator.ALL_IN, + ), + setOf( + FieldCapability.EXACT, + FieldCapability.PROJECTABLE, + FieldCapability.ELEMENT_NULL, + ), + ), + field( + items, + LogicalFieldType.Array( + elementType = LogicalFieldType.Object, + elementNullability = Nullability.NON_NULL, + emptySemantics = EmptyArraySemantics.DISTINCT, + ), + emptySet(), + setOf(FieldCapability.ELEMENT_MATCH, FieldCapability.PROJECTABLE), + ), + field( + itemName, + LogicalFieldType.Text, + setOf(PredicateOperator.EQ, PredicateOperator.CONTAINS), + setOf(FieldCapability.EXACT, FieldCapability.LITERAL_PATTERN, FieldCapability.PROJECTABLE), + ), + field( + itemAttributes, + LogicalFieldType.Array( + elementType = LogicalFieldType.Object, + elementNullability = Nullability.NON_NULL, + emptySemantics = EmptyArraySemantics.DISTINCT, + ), + emptySet(), + setOf(FieldCapability.ELEMENT_MATCH), + ), + field( + itemAttributeName, + LogicalFieldType.Text, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT), + ), + ), + searchScopes = listOf( + QuerySearchScopeDefinition( + id = searchScopeId, + owner = null, + fields = listOf(description), + legacyAliases = setOf(description), + ), + ), + ) + + private fun field( + id: QueryFieldId, + type: LogicalFieldType, + operators: Set = emptySet(), + capabilities: Set = emptySet(), + aliases: Set = emptySet(), + ): QueryFieldSchema = QueryFieldSchema( + id = id, + type = type, + presence = Presence.OPTIONAL, + nullability = Nullability.NULLABLE, + allowedOperators = operators, + capabilities = capabilities, + logicalAliases = aliases, + ) + + fun path(vararg segments: String, basis: PathBasis = PathBasis.ROOT): LogicalField.Path = + LogicalField.Path(segments.asList(), basis) + + fun recordQuery( + condition: NormalizedCondition = NormalizedCondition.All, + projection: NormalizedProjection = NormalizedProjection.All, + sort: List = emptyList(), + deletionScope: NormalizedDeletionScope = NormalizedDeletionScope.EXPLICIT, + ): NormalizedRecordQuery = NormalizedRecordQuery(condition, projection, sort, deletionScope) + + fun single( + query: NormalizedRecordQuery = recordQuery(), + resultShape: QueryResultShape = QueryResultShape.TYPED, + ): NormalizedQueryInvocation = NormalizedQueryInvocation( + target, + QueryOperation.SINGLE, + resultShape, + NormalizedQueryInput.Single(query), + ) + + fun page( + query: NormalizedRecordQuery = recordQuery(), + resultShape: QueryResultShape = QueryResultShape.TYPED, + index: Int = 1, + size: Int = 20, + offset: Long = 0, + ): NormalizedQueryInvocation = NormalizedQueryInvocation( + target, + QueryOperation.PAGE, + resultShape, + NormalizedQueryInput.Page(query, me.ahoo.wow.query.internal.normalization.NormalizedPage(index, size, offset)), + ) + + fun stream( + query: NormalizedRecordQuery = recordQuery(), + resultShape: QueryResultShape = QueryResultShape.TYPED, + limit: Int = 0, + ): NormalizedQueryInvocation = NormalizedQueryInvocation( + target, + QueryOperation.STREAM, + resultShape, + NormalizedQueryInput.Stream(query, limit), + ) + + fun sort(field: LogicalField, direction: NormalizedSortDirection = NormalizedSortDirection.ASC): NormalizedSort = + NormalizedSort(field, direction) + + fun legacySearch(field: LogicalField.Path): SearchScope = SearchScope.LegacyField(field) +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerAccessConstraintTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerAccessConstraintTest.kt new file mode 100644 index 00000000000..20174a1e3dd --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerAccessConstraintTest.kt @@ -0,0 +1,396 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsDimension +import me.ahoo.wow.query.internal.analytics.AnalyticsGrouping +import me.ahoo.wow.query.internal.analytics.AnalyticsMetric +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPromotion +import me.ahoo.wow.query.internal.analytics.AnalyticsOverflowPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedProjection +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.SearchScope +import me.ahoo.wow.query.internal.normalization.SystemFieldKind +import me.ahoo.wow.query.internal.normalization.Utf8Json +import me.ahoo.wow.query.internal.plan.SingleQueryPlan +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.value.NonEmptyList +import org.junit.jupiter.api.Test +import java.math.RoundingMode +import java.util.function.Consumer + +class QueryPlannerAccessConstraintTest { + private val planner = QueryPlanner() + + @Test + fun `user filter should obey allow-list while mandatory may use hidden fields`() { + val user = predicate(PlanningFixtures.path("state", "name"), NormalizedValue.Text("Ada")) + val mandatory = predicate( + LogicalField.System(SystemFieldKind.TENANT_ID), + NormalizedValue.Text("tenant-1"), + ) + val constraints = PlanningConstraints( + validationMode = QueryValidationMode.STRICT, + mandatoryCondition = mandatory, + fieldConstraint = QueryFieldConstraint(filterFields = FieldAccess.AllowList(setOf(PlanningFixtures.name))), + ) + + val decision = planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(user)), + PlanningFixtures.schema, + constraints, + ) as PlanningDecision.Planned + val plan = decision.plan as SingleQueryPlan + plan.filter.mandatory.assert().isNotEqualTo(plan.filter.user) + + assertRejected(QueryRejectionCode.FILTER_FIELD_NOT_ALLOWED, "$.input.query.condition.field") { + planner.plan( + PlanningFixtures.single( + PlanningFixtures.recordQuery( + predicate(PlanningFixtures.path("state", "amount"), NormalizedValue.Int64(1)), + ), + ), + PlanningFixtures.schema, + constraints, + ) + } + } + + @Test + fun `projection sort and search constraints should reject at original request path`() { + val projection = NormalizedProjection.Include(NonEmptyList.of(PlanningFixtures.path("state", "amount"))) + assertRejected(QueryRejectionCode.PROJECTION_FIELD_NOT_ALLOWED, "$.input.query.projection.fields[0]") { + planner.plan( + PlanningFixtures.single( + PlanningFixtures.recordQuery(projection = projection), + QueryResultShape.DYNAMIC, + ), + PlanningFixtures.schema, + PlanningConstraints( + QueryValidationMode.STRICT, + fieldConstraint = QueryFieldConstraint(projectionFields = FieldAccess.DenyAll), + ), + ) + } + assertRejected(QueryRejectionCode.PROJECTION_FIELD_NOT_ALLOWED, "$.input.query.projection") { + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(projection = projection)), + PlanningFixtures.schema, + PlanningConstraints( + QueryValidationMode.COMPATIBLE, + fieldConstraint = QueryFieldConstraint(projectionFields = FieldAccess.DenyAll), + ), + ) + } + + assertRejected(QueryRejectionCode.SORT_FIELD_NOT_ALLOWED, "$.input.query.sort[0].field") { + planner.plan( + PlanningFixtures.page( + PlanningFixtures.recordQuery( + sort = listOf(PlanningFixtures.sort(PlanningFixtures.path("state", "amount"))), + ), + ), + PlanningFixtures.schema, + PlanningConstraints( + QueryValidationMode.STRICT, + fieldConstraint = QueryFieldConstraint(sortFields = FieldAccess.DenyAll), + ), + ) + } + + val search = NormalizedCondition.Search( + SearchScope.Named(PlanningFixtures.searchScopeId), + "distributed systems", + ) + assertRejected(QueryRejectionCode.SEARCH_SCOPE_NOT_ALLOWED, "$.input.query.condition.scope") { + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(search)), + PlanningFixtures.schema, + PlanningConstraints( + QueryValidationMode.STRICT, + fieldConstraint = QueryFieldConstraint(searchScopes = SearchScopeAccess.DenyAll), + ), + ) + } + } + + @Test + fun `restricted projection should reject full typed and exclude result shapes without fallback`() { + listOf(QueryValidationMode.STRICT, QueryValidationMode.COMPATIBLE).forEach { mode -> + assertRejected(QueryRejectionCode.PROJECTION_FIELD_NOT_ALLOWED, "$.input.query.projection") { + planner.plan( + PlanningFixtures.single(), + PlanningFixtures.schema, + PlanningConstraints( + mode, + fieldConstraint = QueryFieldConstraint(projectionFields = FieldAccess.DenyAll), + ), + ) + } + assertRejected(QueryRejectionCode.PROJECTION_FIELD_NOT_ALLOWED, "$.input.query.projection") { + planner.plan( + PlanningFixtures.single( + PlanningFixtures.recordQuery( + projection = NormalizedProjection.Exclude( + NonEmptyList.of(PlanningFixtures.path("state", "name")), + ), + ), + QueryResultShape.DYNAMIC, + ), + PlanningFixtures.schema, + PlanningConstraints( + mode, + fieldConstraint = QueryFieldConstraint( + projectionFields = FieldAccess.AllowList(setOf(PlanningFixtures.name)), + ), + ), + ) + } + } + } + + @Test + fun `result constraint should reject unbounded stream and oversized page`() { + val maximum = ResultPlanningConstraint.MaximumRecords(10) + val stream = NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.STREAM, + QueryResultShape.DYNAMIC, + NormalizedQueryInput.Stream(PlanningFixtures.recordQuery(), limit = 0), + ) + assertRejected( + QueryRejectionCode.RESULT_LIMIT_EXCEEDED, + "$.input.limit", + QueryRejectionCategory.BUDGET_EXCEEDED, + ) { + planner.plan( + stream, + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT, resultConstraint = maximum), + ) + } + assertRejected( + QueryRejectionCode.RESULT_LIMIT_EXCEEDED, + "$.input.page", + QueryRejectionCategory.BUDGET_EXCEEDED, + ) { + planner.plan( + PlanningFixtures.page(size = 11), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT, resultConstraint = maximum), + ) + } + + listOf( + stream.copy(input = NormalizedQueryInput.Stream(PlanningFixtures.recordQuery(), limit = 10)), + PlanningFixtures.page(size = 10), + ).forEach { invocation -> + planner.plan( + invocation, + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT, resultConstraint = maximum), + ).assert().isInstanceOf(PlanningDecision.Planned::class.java) + } + } + + @Test + fun `analytics dimensions and metrics should use independent access dimensions`() { + val dimensionQuery = AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.By( + NonEmptyList.of( + AnalyticsDimension(AnalyticsAlias("amount"), PlanningFixtures.path("state", "amount")), + ), + ), + NonEmptyList.of(AnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + ) + assertRejected( + QueryRejectionCode.ANALYTICS_DIMENSION_FIELD_NOT_ALLOWED, + "$.input.query.grouping.dimensions[0].field", + ) { + planAnalytics( + dimensionQuery, + QueryFieldConstraint(analyticsDimensionFields = FieldAccess.DenyAll), + ) + } + + val metricQuery = AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.Global, + NonEmptyList.of( + AnalyticsMetric.Sum(AnalyticsAlias("total"), PlanningFixtures.path("state", "amount")), + ), + numericPolicy = AnalyticsNumericPolicy( + AnalyticsNumericPromotion.DECIMAL128, + precision = 34, + scale = 8, + roundingMode = RoundingMode.HALF_EVEN, + overflowPolicy = AnalyticsOverflowPolicy.REJECT, + ), + ) + assertRejected( + QueryRejectionCode.ANALYTICS_METRIC_FIELD_NOT_ALLOWED, + "$.input.query.metrics[0].field", + ) { + planAnalytics( + metricQuery, + QueryFieldConstraint(analyticsMetricFields = FieldAccess.DenyAll), + ) + } + } + + @Test + fun `unknown resources should not bypass restricted policy through compatible fallback`() { + val missing = PlanningFixtures.path("state", "secret") + val restricted = QueryFieldConstraint( + filterFields = FieldAccess.AllowList(setOf(PlanningFixtures.name)), + projectionFields = FieldAccess.AllowList(setOf(PlanningFixtures.name)), + sortFields = FieldAccess.AllowList(setOf(PlanningFixtures.name)), + searchScopes = SearchScopeAccess.AllowList(setOf(PlanningFixtures.searchScopeId)), + ) + assertRejected(QueryRejectionCode.FILTER_FIELD_NOT_ALLOWED, "$.input.query.condition.field") { + planner.plan( + PlanningFixtures.single( + PlanningFixtures.recordQuery(predicate(missing, NormalizedValue.Text("secret"))), + ), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.COMPATIBLE, fieldConstraint = restricted), + ) + } + assertRejected(QueryRejectionCode.PROJECTION_FIELD_NOT_ALLOWED, "$.input.query.projection.fields[0]") { + planner.plan( + PlanningFixtures.single( + PlanningFixtures.recordQuery(projection = NormalizedProjection.Include(NonEmptyList.of(missing))), + QueryResultShape.DYNAMIC, + ), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.COMPATIBLE, fieldConstraint = restricted), + ) + } + assertRejected(QueryRejectionCode.SORT_FIELD_NOT_ALLOWED, "$.input.query.sort[0].field") { + planner.plan( + PlanningFixtures.page( + PlanningFixtures.recordQuery( + projection = NormalizedProjection.Include( + NonEmptyList.of(PlanningFixtures.path("state", "name")), + ), + sort = listOf(PlanningFixtures.sort(missing)), + ), + QueryResultShape.DYNAMIC, + ), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.COMPATIBLE, fieldConstraint = restricted), + ) + } + listOf(missing, PlanningFixtures.path("aggregateId")).forEach { legacyField -> + assertRejected(QueryRejectionCode.SEARCH_SCOPE_NOT_ALLOWED, "$.input.query.condition.scope") { + planner.plan( + PlanningFixtures.single( + PlanningFixtures.recordQuery( + NormalizedCondition.Search(SearchScope.LegacyField(legacyField), "secret"), + ), + ), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.COMPATIBLE, fieldConstraint = restricted), + ) + } + } + } + + @Test + fun `native backend access should reject before compatible fallback and permit only allowed backend`() { + val native = NormalizedCondition.Native(BackendId("mongo"), Utf8Json("{}")) + listOf( + NativeBackendAccess.DenyAll, + NativeBackendAccess.AllowList(setOf(BackendId("elasticsearch"))), + ).forEach { access -> + assertRejected(QueryRejectionCode.NATIVE_BACKEND_NOT_ALLOWED, "$.input.query.condition.backendId") { + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(native)), + PlanningFixtures.schema, + PlanningConstraints( + QueryValidationMode.COMPATIBLE, + fieldConstraint = QueryFieldConstraint(nativeBackends = access), + ), + ) + } + } + + val decision = planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(native)), + PlanningFixtures.schema, + PlanningConstraints( + QueryValidationMode.COMPATIBLE, + fieldConstraint = QueryFieldConstraint( + nativeBackends = NativeBackendAccess.AllowList(setOf(BackendId("mongo"))), + ), + ), + ) as PlanningDecision.Planned + decision.plan.requiredCapabilities.nativeBackend.assert().isEqualTo(BackendId("mongo")) + + val source = linkedSetOf(BackendId("mongo")) + val access = NativeBackendAccess.AllowList(source) + source += BackendId("elasticsearch") + access.permits(BackendId("mongo")).assert().isTrue() + access.permits(BackendId("elasticsearch")).assert().isFalse() + access.assert().isEqualTo(NativeBackendAccess.AllowList(setOf(BackendId("mongo")))) + } + + private fun planAnalytics(query: AnalyticsQuery, fieldConstraint: QueryFieldConstraint) { + planner.plan( + NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.ANALYZE, + QueryResultShape.ANALYTICS, + NormalizedQueryInput.Analytics(query), + ), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT, fieldConstraint = fieldConstraint), + ) + } + + private fun predicate(field: LogicalField, value: NormalizedValue): NormalizedCondition.Predicate = + NormalizedCondition.Predicate(field, PredicateOperator.EQ, value) + + private fun assertRejected( + code: QueryRejectionCode, + path: String, + category: QueryRejectionCategory = QueryRejectionCategory.ACCESS_DENIED, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo(path) + }, + ) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerAnalyticsFingerprintTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerAnalyticsFingerprintTest.kt new file mode 100644 index 00000000000..e2e258d8079 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerAnalyticsFingerprintTest.kt @@ -0,0 +1,134 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsBucketOrder +import me.ahoo.wow.query.internal.analytics.AnalyticsBucketWindow +import me.ahoo.wow.query.internal.analytics.AnalyticsDimension +import me.ahoo.wow.query.internal.analytics.AnalyticsGrouping +import me.ahoo.wow.query.internal.analytics.AnalyticsMetric +import me.ahoo.wow.query.internal.analytics.AnalyticsMissingPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPromotion +import me.ahoo.wow.query.internal.analytics.AnalyticsOverflowPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.plan.AnalyticsQueryPlan +import me.ahoo.wow.query.internal.value.NonEmptyList +import org.junit.jupiter.api.Test +import java.math.RoundingMode + +class QueryPlannerAnalyticsFingerprintTest { + private val planner = QueryPlanner() + + @Test + fun `analytics fingerprint should encode semantics and canonicalize equivalent input`() { + val query = grouped() + val base = plan(query) + plan(query.copy(bucketOrder = AnalyticsBucketOrder.DimensionKeyAscending)).fingerprint.assert() + .isEqualTo(base.fingerprint) + + val excludeMissing = query.copy( + grouping = AnalyticsGrouping.By( + NonEmptyList.of( + AnalyticsDimension( + AnalyticsAlias("amount"), + PlanningFixtures.path("state", "amount"), + AnalyticsMissingPolicy.EXCLUDE, + ), + ), + ), + ) + val average = query.copy( + metrics = NonEmptyList.of( + AnalyticsMetric.DocumentCount(AnalyticsAlias("count")), + AnalyticsMetric.Average(AnalyticsAlias("total"), PlanningFixtures.path("state", "amount")), + ), + ) + listOf( + excludeMissing, + grouped(metricAlias = AnalyticsAlias("renamed")), + average, + query.copy(bucketWindow = AnalyticsBucketWindow.First(24)), + query.copy(numericPolicy = numericPolicy().copy(precision = 33)), + query.copy(numericPolicy = numericPolicy().copy(scale = 7)), + query.copy(numericPolicy = numericPolicy().copy(roundingMode = RoundingMode.DOWN)), + ).forEach { variant -> + plan(variant).fingerprint.assert().isNotEqualTo(base.fingerprint) + } + + base.fingerprint.value.assert().isEqualTo( + "82300c5c723bdd295ce3f86800c2d89e93c7101e1152072070975f3bf88f4889", + ) + plan(global().copy(bucketWindow = AnalyticsBucketWindow.First(1))).fingerprint.assert().isEqualTo( + plan(global().copy(bucketWindow = AnalyticsBucketWindow.First(10_000))).fingerprint, + ) + } + + private fun plan(query: AnalyticsQuery): AnalyticsQueryPlan { + val invocation = NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.ANALYZE, + QueryResultShape.ANALYTICS, + NormalizedQueryInput.Analytics(query), + ) + return ( + planner.plan( + invocation, + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + ).plan as AnalyticsQueryPlan + } + + private fun global(): AnalyticsQuery = AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.Global, + NonEmptyList.of(AnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + ) + + private fun grouped(metricAlias: AnalyticsAlias = AnalyticsAlias("total")): AnalyticsQuery = AnalyticsQuery( + userCondition = NormalizedCondition.All, + grouping = AnalyticsGrouping.By( + NonEmptyList.of( + AnalyticsDimension( + AnalyticsAlias("amount"), + PlanningFixtures.path("state", "amount"), + AnalyticsMissingPolicy.AS_NULL_BUCKET, + ), + ), + ), + metrics = NonEmptyList.of( + AnalyticsMetric.DocumentCount(AnalyticsAlias("count")), + AnalyticsMetric.Sum(metricAlias, PlanningFixtures.path("state", "amount")), + ), + bucketWindow = AnalyticsBucketWindow.First(25), + numericPolicy = numericPolicy(), + ) + + private fun numericPolicy(): AnalyticsNumericPolicy = AnalyticsNumericPolicy( + AnalyticsNumericPromotion.DECIMAL128, + precision = 34, + scale = 8, + roundingMode = RoundingMode.HALF_EVEN, + overflowPolicy = AnalyticsOverflowPolicy.REJECT, + ) +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerAnalyticsMatrixTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerAnalyticsMatrixTest.kt new file mode 100644 index 00000000000..c48fdbbc34a --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerAnalyticsMatrixTest.kt @@ -0,0 +1,311 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsBucketWindow +import me.ahoo.wow.query.internal.analytics.AnalyticsDimension +import me.ahoo.wow.query.internal.analytics.AnalyticsGrouping +import me.ahoo.wow.query.internal.analytics.AnalyticsMetric +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPromotion +import me.ahoo.wow.query.internal.analytics.AnalyticsOverflowPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.analytics.DecodedAnalyticsCursor +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.plan.AnalyticsQueryPlan +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsMetric +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.value.NonEmptyList +import org.junit.jupiter.api.Test +import java.math.BigDecimal +import java.math.RoundingMode +import java.time.Instant +import java.util.function.Consumer + +class QueryPlannerAnalyticsMatrixTest { + + private val planner = QueryPlanner() + private val cursorText = QueryFieldId.Path(listOf("state", "cursorText")) + private val cursorBoolean = QueryFieldId.Path(listOf("state", "cursorBoolean")) + private val cursorInt64 = QueryFieldId.Path(listOf("state", "cursorInt64")) + private val itemAmount = QueryFieldId.Path(listOf("state", "items", "amount")) + private val schema = QueryDocumentSchema( + PlanningFixtures.target, + PlanningFixtures.schema.fields.values + listOf( + scalarField(cursorText, LogicalFieldType.Text), + scalarField(cursorBoolean, LogicalFieldType.Boolean), + scalarField(cursorInt64, LogicalFieldType.Int64), + scalarField(itemAmount, LogicalFieldType.Decimal), + ), + PlanningFixtures.schema.searchScopes.values, + ) + + @Test + fun `metric matrix should plan every portable numeric and instant aggregation`() { + val cases = listOf( + MetricCase( + AnalyticsMetric.Min(alias("numericMin"), amount()), + PlannedAnalyticsMetric.Min::class.java, + true + ), + MetricCase( + AnalyticsMetric.Max(alias("numericMax"), amount()), + PlannedAnalyticsMetric.Max::class.java, + true + ), + MetricCase( + AnalyticsMetric.Sum(alias("numericSum"), amount()), + PlannedAnalyticsMetric.Sum::class.java, + true + ), + MetricCase( + AnalyticsMetric.Average(alias("numericAverage"), amount()), + PlannedAnalyticsMetric.Average::class.java, + true, + ), + MetricCase( + AnalyticsMetric.Min(alias("instantMin"), PlanningFixtures.path("state", "createdAt")), + PlannedAnalyticsMetric.Min::class.java, + false, + ), + MetricCase( + AnalyticsMetric.Max(alias("instantMax"), PlanningFixtures.path("state", "createdAt")), + PlannedAnalyticsMetric.Max::class.java, + false, + ), + ) + + cases.forEach { case -> + val plan = plan(metricQuery(case.metric, case.needsNumericPolicy)) + val metric = plan.metrics.values.single() + case.plannedType.isInstance(metric).assert().isTrue() + metricField(metric).assert().isEqualTo( + if (case.needsNumericPolicy) PlanningFixtures.amount else PlanningFixtures.createdAt, + ) + } + } + + @Test + fun `nested numeric metrics should fail before backend compilation`() { + val query = metricQuery( + AnalyticsMetric.Sum(alias("nestedTotal"), PlanningFixtures.path("state", "items", "amount")), + needsNumericPolicy = true, + ) + + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.ANALYTICS_METRIC_TYPE_UNSUPPORTED, + "$.input.query.metrics[0].field", + ) { + plan(query) + } + } + + @Test + fun `cursor matrix should preserve exact canonical scalar types`() { + cursorCases().forEach { case -> + val query = grouped(listOf(case)) + val initial = plan(query) + val cursor = cursor(initial, listOf(case), listOf(case.value)) + + checkNotNull( + plan(query.copy(bucketWindow = AnalyticsBucketWindow.After(25, cursor))) + .bucketWindow.afterKey, + ).values.assert().containsExactly(case.value) + assertInvalidCursor( + query, + cursor.copy(afterKey = NonEmptyList.of(case.wrongType)), + "$.input.query.bucketWindow.cursor.afterKey[0]", + ) + } + } + + @Test + fun `compound cursor should bind alias order arity and every key`() { + val cases = cursorCases() + val query = grouped(cases) + val initial = plan(query) + val values = cases.map(CursorCase::value) + val cursor = cursor(initial, cases, values) + + checkNotNull( + plan(query.copy(bucketWindow = AnalyticsBucketWindow.After(25, cursor))) + .bucketWindow.afterKey, + ).values.assert().isEqualTo(values) + assertInvalidCursor( + query, + cursor.copy(dimensionAliases = nonEmpty(cases.map(CursorCase::alias).reversed())), + "$.input.query.bucketWindow.cursor.dimensionAliases", + ) + assertInvalidCursor( + query, + cursor.copy(afterKey = nonEmpty(values.dropLast(1))), + "$.input.query.bucketWindow.cursor.afterKey", + ) + } + + private fun cursorCases(): List = listOf( + CursorCase(alias("text"), cursorText, NormalizedValue.Text("A"), NormalizedValue.Int64(1)), + CursorCase( + alias("boolean"), + cursorBoolean, + NormalizedValue.BooleanValue(true), + NormalizedValue.Text("true"), + ), + CursorCase(alias("int64"), cursorInt64, NormalizedValue.Int64(10), NormalizedValue.Decimal(BigDecimal.TEN)), + CursorCase( + alias("decimal"), + PlanningFixtures.amount, + NormalizedValue.Decimal(BigDecimal.TEN), + NormalizedValue.Int64(10), + ), + CursorCase( + alias("instant"), + PlanningFixtures.createdAt, + NormalizedValue.InstantValue(Instant.parse("2026-08-07T00:00:00Z")), + NormalizedValue.Text("2026-08-07T00:00:00Z"), + ), + ) + + private fun grouped(cases: List): AnalyticsQuery = AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.By( + nonEmpty( + cases.map { case -> AnalyticsDimension(case.alias, logical(case.field)) }, + ), + ), + NonEmptyList.of(AnalyticsMetric.DocumentCount(alias("count"))), + bucketWindow = AnalyticsBucketWindow.First(25), + ) + + private fun metricQuery(metric: AnalyticsMetric, needsNumericPolicy: Boolean): AnalyticsQuery = AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.Global, + NonEmptyList.of(metric), + numericPolicy = numericPolicy().takeIf { needsNumericPolicy }, + ) + + private fun plan(query: AnalyticsQuery): AnalyticsQueryPlan = + ( + planner.plan( + NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.ANALYZE, + QueryResultShape.ANALYTICS, + NormalizedQueryInput.Analytics(query), + ), + schema, + PlanningConstraints(me.ahoo.wow.query.internal.model.QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + ).plan as AnalyticsQueryPlan + + private fun cursor( + plan: AnalyticsQueryPlan, + cases: List, + values: List, + ): DecodedAnalyticsCursor = DecodedAnalyticsCursor( + PlanningFixtures.target, + plan.fingerprint, + nonEmpty(cases.map(CursorCase::alias)), + nonEmpty(values), + ) + + private fun assertInvalidCursor( + query: AnalyticsQuery, + cursor: DecodedAnalyticsCursor, + path: String, + ) { + assertRejected(QueryRejectionCategory.INVALID_CURSOR, QueryRejectionCode.INVALID_CURSOR_BINDING, path) { + plan(query.copy(bucketWindow = AnalyticsBucketWindow.After(25, cursor))) + } + } + + private fun assertRejected( + category: QueryRejectionCategory, + code: QueryRejectionCode, + path: String, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo(path) + }, + ) + } + + private fun scalarField(id: QueryFieldId.Path, type: LogicalFieldType): QueryFieldSchema = QueryFieldSchema( + id, + type, + Presence.OPTIONAL, + Nullability.NULLABLE, + emptySet(), + setOf(FieldCapability.AGGREGATABLE), + ) + + private fun logical(field: QueryFieldId.Path) = PlanningFixtures.path(*field.segments.toTypedArray()) + + private fun amount() = PlanningFixtures.path("state", "amount") + + private fun alias(value: String) = AnalyticsAlias(value) + + private fun numericPolicy() = AnalyticsNumericPolicy( + AnalyticsNumericPromotion.DECIMAL128, + precision = 34, + scale = 8, + roundingMode = RoundingMode.HALF_EVEN, + overflowPolicy = AnalyticsOverflowPolicy.REJECT, + ) + + private fun metricField(metric: PlannedAnalyticsMetric): QueryFieldId = when (metric) { + is PlannedAnalyticsMetric.Min -> metric.field + is PlannedAnalyticsMetric.Max -> metric.field + is PlannedAnalyticsMetric.Sum -> metric.field + is PlannedAnalyticsMetric.Average -> metric.field + is PlannedAnalyticsMetric.DocumentCount -> error("Document count has no field.") + } + + private fun nonEmpty(values: List): NonEmptyList = checkNotNull(NonEmptyList.from(values)) + + private data class MetricCase( + val metric: AnalyticsMetric, + val plannedType: Class, + val needsNumericPolicy: Boolean, + ) + + private data class CursorCase( + val alias: AnalyticsAlias, + val field: QueryFieldId.Path, + val value: NormalizedValue, + val wrongType: NormalizedValue, + ) +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerAnalyticsTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerAnalyticsTest.kt new file mode 100644 index 00000000000..58cb6824082 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerAnalyticsTest.kt @@ -0,0 +1,586 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsBucketOrder +import me.ahoo.wow.query.internal.analytics.AnalyticsBucketWindow +import me.ahoo.wow.query.internal.analytics.AnalyticsCompleteness +import me.ahoo.wow.query.internal.analytics.AnalyticsCondition +import me.ahoo.wow.query.internal.analytics.AnalyticsConsistency +import me.ahoo.wow.query.internal.analytics.AnalyticsDimension +import me.ahoo.wow.query.internal.analytics.AnalyticsGrouping +import me.ahoo.wow.query.internal.analytics.AnalyticsMetric +import me.ahoo.wow.query.internal.analytics.AnalyticsMissingPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNullPlacement +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsNumericPromotion +import me.ahoo.wow.query.internal.analytics.AnalyticsOverflowPolicy +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.analytics.AnalyticsTextCollation +import me.ahoo.wow.query.internal.analytics.DecodedAnalyticsCursor +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.normalization.Utf8Json +import me.ahoo.wow.query.internal.plan.AnalyticsQueryPlan +import me.ahoo.wow.query.internal.plan.PlanFingerprint +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsBucketOrder +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsCondition +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsGrouping +import me.ahoo.wow.query.internal.plan.PlannedAnalyticsMetric +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.value.NonEmptyList +import org.junit.jupiter.api.Test +import java.math.RoundingMode +import java.util.function.Consumer + +class QueryPlannerAnalyticsTest { + + private val planner = QueryPlanner() + + @Test + fun `global document count should produce a portable exact plan`() { + val plan = plan( + AnalyticsQuery( + userCondition = NormalizedCondition.All, + grouping = AnalyticsGrouping.Global, + metrics = NonEmptyList.of(AnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + ), + ) + + plan.grouping.assert().isEqualTo(PlannedAnalyticsGrouping.Global) + plan.metrics.values.single().assert().isEqualTo( + PlannedAnalyticsMetric.DocumentCount(AnalyticsAlias("count")), + ) + plan.bucketOrder.assert().isEqualTo(PlannedAnalyticsBucketOrder.Global) + plan.bucketWindow.limit.assert().isEqualTo(1) + plan.having.assert().isEqualTo(PlannedAnalyticsCondition.All) + plan.filter.user.assert().isEqualTo(me.ahoo.wow.query.internal.plan.PlannedCondition.All) + plan.requiredConsistency.assert().isEqualTo(AnalyticsConsistency.EVENTUAL) + plan.requiredCompleteness.assert().isEqualTo(AnalyticsCompleteness.EXACT) + plan.fingerprint.value.length.assert().isEqualTo(64) + + val withUnusedPolicy = plan(global().copy(numericPolicy = numericPolicy())) + withUnusedPolicy.numericPolicy.assert().isNull() + withUnusedPolicy.fingerprint.assert().isEqualTo(plan.fingerprint) + } + + @Test + fun `grouped numeric metrics should bind canonical fields and aggregation capability`() { + val amount = AnalyticsAlias("amount") + val total = AnalyticsAlias("total") + val plan = plan(grouped(amount, total)) + val grouping = plan.grouping as PlannedAnalyticsGrouping.By + + grouping.dimensions.values.single().field.assert().isEqualTo(PlanningFixtures.amount) + grouping.dimensions.values.single().missingPolicy.assert().isEqualTo(AnalyticsMissingPolicy.AS_NULL_BUCKET) + (plan.metrics.values.last() as PlannedAnalyticsMetric.Sum).field.assert().isEqualTo(PlanningFixtures.amount) + plan.requiredCapabilities.fieldRequirements.getValue(PlanningFixtures.amount).assert() + .contains(me.ahoo.wow.query.backend.FieldCapability.AGGREGATABLE) + plan.bucketOrder.assert().isEqualTo( + PlannedAnalyticsBucketOrder.DimensionKeyAscending( + AnalyticsNullPlacement.FIRST, + AnalyticsTextCollation.BINARY, + ), + ) + } + + @Test + fun `analytics should reject unsupported grain shape and precision gaps`() { + val eventTarget = QueryTarget(PlanningFixtures.target.namedAggregate, QueryDocumentKind.EVENT_STREAM) + val eventSchema = QueryDocumentSchema( + eventTarget, + PlanningFixtures.schema.fields.values, + PlanningFixtures.schema.searchScopes.values, + ) + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.ANALYTICS_DOCUMENT_KIND_UNSUPPORTED, + "$.target.documentKind", + ) { + planner.plan(invocation(global(), eventTarget), eventSchema, constraints()) + } + + val arrayDimension = AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.By( + NonEmptyList.of( + AnalyticsDimension( + AnalyticsAlias("items"), + PlanningFixtures.path("state", "items"), + AnalyticsMissingPolicy.EXCLUDE, + ), + ), + ), + NonEmptyList.of(AnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + ) + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.ANALYTICS_DIMENSION_TYPE_UNSUPPORTED, + "$.input.query.grouping.dimensions[0].field", + ) { + plan(arrayDimension) + } + + val nestedDimension = arrayDimension.copy( + grouping = AnalyticsGrouping.By( + NonEmptyList.of( + AnalyticsDimension( + AnalyticsAlias("itemName"), + PlanningFixtures.path("state", "items", "name"), + ), + ), + ), + ) + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.ANALYTICS_DIMENSION_TYPE_UNSUPPORTED, + "$.input.query.grouping.dimensions[0].field", + ) { + plan(nestedDimension) + } + } + + @Test + fun `numeric metrics should require a supported Decimal128 policy`() { + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.ANALYTICS_NUMERIC_POLICY_REQUIRED, + "$.input.query.numericPolicy", + ) { + plan(grouped(numericPolicy = null)) + } + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.ANALYTICS_NUMERIC_POLICY_UNSUPPORTED, + "$.input.query.numericPolicy", + ) { + plan(grouped(numericPolicy = numericPolicy().copy(precision = 35))) + } + } + + @Test + fun `analytics should support snapshot consistency and reject unsupported having order and completeness`() { + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.ANALYTICS_HAVING_UNSUPPORTED, + "$.input.query.having", + ) { + plan(grouped().copy(having = AnalyticsCondition.Predicate(AnalyticsAlias("count")))) + } + + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.ANALYTICS_ORDER_UNSUPPORTED, + "$.input.query.bucketOrder", + ) { + plan(grouped().copy(bucketOrder = AnalyticsBucketOrder.MetricDescending(AnalyticsAlias("total")))) + } + + plan(grouped().copy(requiredConsistency = AnalyticsConsistency.SNAPSHOT)) + .requiredConsistency.assert().isEqualTo(AnalyticsConsistency.SNAPSHOT) + + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.ANALYTICS_COMPLETENESS_UNSUPPORTED, + "$.input.query.requiredCompleteness", + ) { + plan(grouped().copy(requiredCompleteness = AnalyticsCompleteness.APPROXIMATE)) + } + } + + @Test + fun `analytics aliases should be unique before field planning`() { + val duplicate = AnalyticsAlias("amount") + val query = AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.By( + NonEmptyList.of( + AnalyticsDimension( + duplicate, + PlanningFixtures.path("state", "amount"), + AnalyticsMissingPolicy.EXCLUDE, + ), + ), + ), + NonEmptyList.of(AnalyticsMetric.Sum(duplicate, PlanningFixtures.path("state", "missing"))), + numericPolicy = numericPolicy(), + ) + + assertRejected( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.DUPLICATE_ANALYTICS_ALIAS, + "$.input.query.metrics[0].alias", + ) { + plan(query) + } + } + + @Test + fun `analytics should allow instant min max but reject instant sum and non portable filters`() { + val instantMin = AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.Global, + NonEmptyList.of( + AnalyticsMetric.Min( + AnalyticsAlias("firstCreatedAt"), + PlanningFixtures.path("state", "createdAt"), + ), + ), + ) + (plan(instantMin).metrics.values.single() as PlannedAnalyticsMetric.Min).field.assert() + .isEqualTo(PlanningFixtures.createdAt) + + val instantSum = instantMin.copy( + metrics = NonEmptyList.of( + AnalyticsMetric.Sum( + AnalyticsAlias("sumCreatedAt"), + PlanningFixtures.path("state", "createdAt"), + ), + ), + numericPolicy = numericPolicy(), + ) + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.ANALYTICS_METRIC_TYPE_UNSUPPORTED, + "$.input.query.metrics[0].field", + ) { + plan(instantSum) + } + + val native = instantMin.copy( + userCondition = NormalizedCondition.Native(BackendId("mongo"), Utf8Json("{}")), + ) + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.CAPABILITY_UNAVAILABLE, + "$.input.query.userCondition", + ) { + val compatible = PlanningConstraints(QueryValidationMode.COMPATIBLE) + plan(native, compatible) + } + + val mandatorySearch = PlanningConstraints( + QueryValidationMode.STRICT, + mandatoryCondition = NormalizedCondition.Search( + PlanningFixtures.legacySearch(PlanningFixtures.path("state", "description")), + "secure", + ), + ) + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.CAPABILITY_UNAVAILABLE, + "$.constraints.mandatoryCondition", + ) { + plan(global(), mandatorySearch) + } + } + + @Test + fun `decoded cursor should bind target fingerprint order arity and value type`() { + val first = plan(grouped()) + val cursor = DecodedAnalyticsCursor( + target = PlanningFixtures.target, + planFingerprint = first.fingerprint, + dimensionAliases = NonEmptyList.of(AnalyticsAlias("amount")), + afterKey = NonEmptyList.of(NormalizedValue.Decimal(java.math.BigDecimal("10.00"))), + ) + val resumed = plan( + grouped().copy(bucketWindow = AnalyticsBucketWindow.After(limit = 25, cursor = cursor)), + ) + + resumed.fingerprint.assert().isEqualTo(first.fingerprint) + + val wrong = cursor.copy(planFingerprint = PlanFingerprint("f".repeat(64))) + assertRejected( + QueryRejectionCategory.INVALID_CURSOR, + QueryRejectionCode.INVALID_CURSOR_BINDING, + "$.input.query.bucketWindow.cursor.planFingerprint", + ) { + plan(grouped().copy(bucketWindow = AnalyticsBucketWindow.After(limit = 25, cursor = wrong))) + } + + val globalCursor = cursor.copy(planFingerprint = plan(global()).fingerprint) + assertRejected( + QueryRejectionCategory.INVALID_CURSOR, + QueryRejectionCode.INVALID_CURSOR_BINDING, + "$.input.query.bucketWindow.cursor", + ) { + plan(global().copy(bucketWindow = AnalyticsBucketWindow.After(limit = 1, cursor = globalCursor))) + } + } + + @Test + fun `decoded cursor should reject target order arity and type mismatches`() { + val query = grouped() + val first = plan(query) + val cursor = DecodedAnalyticsCursor( + PlanningFixtures.target, + first.fingerprint, + NonEmptyList.of(AnalyticsAlias("amount")), + NonEmptyList.of(NormalizedValue.Decimal(java.math.BigDecimal.TEN)), + ) + val eventTarget = QueryTarget(PlanningFixtures.target.namedAggregate, QueryDocumentKind.EVENT_STREAM) + + assertInvalidCursor(query, cursor.copy(target = eventTarget), "$.input.query.bucketWindow.cursor.target") + assertInvalidCursor( + query, + cursor.copy(dimensionAliases = NonEmptyList.of(AnalyticsAlias("other"))), + "$.input.query.bucketWindow.cursor.dimensionAliases", + ) + assertInvalidCursor( + query, + cursor.copy( + afterKey = NonEmptyList.of( + NormalizedValue.Decimal(java.math.BigDecimal.ONE), + NormalizedValue.Decimal(java.math.BigDecimal.TEN), + ), + ), + "$.input.query.bucketWindow.cursor.afterKey", + ) + assertInvalidCursor( + query, + cursor.copy(afterKey = NonEmptyList.of(NormalizedValue.Text("ten"))), + "$.input.query.bucketWindow.cursor.afterKey[0]", + ) + assertInvalidCursor( + query, + cursor.copy(afterKey = NonEmptyList.of(NormalizedValue.Int64(10))), + "$.input.query.bucketWindow.cursor.afterKey[0]", + ) + } + + @Test + fun `cursor should represent a missing non-null dimension as null only for null buckets`() { + val schema = schemaWithAmount(Presence.OPTIONAL, Nullability.NON_NULL) + val query = grouped() + val first = plan(query, schema = schema) + val cursor = DecodedAnalyticsCursor( + PlanningFixtures.target, + first.fingerprint, + NonEmptyList.of(AnalyticsAlias("amount")), + NonEmptyList.of(NormalizedValue.Null), + ) + + plan( + query.copy(bucketWindow = AnalyticsBucketWindow.After(25, cursor)), + schema = schema, + ).bucketWindow.afterKey.assert().isNotNull() + val excludedQuery = query.copy( + grouping = AnalyticsGrouping.By( + NonEmptyList.of( + AnalyticsDimension( + AnalyticsAlias("amount"), + PlanningFixtures.path("state", "amount"), + AnalyticsMissingPolicy.EXCLUDE, + ), + ), + ), + ) + val excludedCursor = cursor.copy(planFingerprint = plan(excludedQuery).fingerprint) + assertInvalidCursor(excludedQuery, excludedCursor, "$.input.query.bucketWindow.cursor.afterKey[0]") + + val requiredSchema = schemaWithAmount(Presence.REQUIRED, Nullability.NON_NULL) + val requiredCursor = cursor.copy(planFingerprint = plan(query, schema = requiredSchema).fingerprint) + assertInvalidCursor( + query, + requiredCursor, + "$.input.query.bucketWindow.cursor.afterKey[0]", + requiredSchema, + ) + } + + @Test + fun `analytics limits should be enforced without entering semantic fingerprint`() { + val query = grouped() + val unrestricted = plan(query) + val exactLimits = AnalyticsPlanningConstraint.Limits( + maxDimensions = 1, + maxMetrics = 2, + maxBucketLimit = 25, + ) + val limited = plan(query, constraints(exactLimits)) + + limited.fingerprint.assert().isEqualTo(unrestricted.fingerprint) + assertRejected( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.ANALYTICS_METRIC_LIMIT_EXCEEDED, + "$.input.query.metrics", + ) { + plan(query, constraints(exactLimits.copy(maxMetrics = 1))) + } + val twoDimensions = query.copy( + grouping = AnalyticsGrouping.By( + NonEmptyList.of( + AnalyticsDimension( + AnalyticsAlias("amount"), + PlanningFixtures.path("state", "amount"), + ), + AnalyticsDimension( + AnalyticsAlias("name"), + PlanningFixtures.path("state", "name"), + ), + ), + ), + ) + assertRejected( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.ANALYTICS_DIMENSION_LIMIT_EXCEEDED, + "$.input.query.grouping.dimensions", + ) { + plan(twoDimensions, constraints(exactLimits)) + } + assertRejected( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.ANALYTICS_BUCKET_LIMIT_EXCEEDED, + "$.input.query.bucketWindow.limit", + ) { + plan(query, constraints(exactLimits.copy(maxBucketLimit = 24))) + } + + val globalWithIrrelevantLimit = global().copy(bucketWindow = AnalyticsBucketWindow.First(10_000)) + plan( + globalWithIrrelevantLimit, + constraints(AnalyticsPlanningConstraint.Limits(1, 1, 1)), + ).bucketWindow.limit.assert().isEqualTo(1) + } + + private fun global(): AnalyticsQuery = AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.Global, + NonEmptyList.of(AnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + ) + + private fun grouped( + dimensionAlias: AnalyticsAlias = AnalyticsAlias("amount"), + metricAlias: AnalyticsAlias = AnalyticsAlias("total"), + numericPolicy: AnalyticsNumericPolicy? = numericPolicy(), + ): AnalyticsQuery = AnalyticsQuery( + userCondition = NormalizedCondition.All, + grouping = AnalyticsGrouping.By( + NonEmptyList.of( + AnalyticsDimension( + dimensionAlias, + PlanningFixtures.path("state", "amount"), + AnalyticsMissingPolicy.AS_NULL_BUCKET, + ), + ), + ), + metrics = NonEmptyList.of( + AnalyticsMetric.DocumentCount(AnalyticsAlias("count")), + AnalyticsMetric.Sum(metricAlias, PlanningFixtures.path("state", "amount")), + ), + bucketWindow = AnalyticsBucketWindow.First(limit = 25), + numericPolicy = numericPolicy, + ) + + private fun numericPolicy(): AnalyticsNumericPolicy = AnalyticsNumericPolicy( + promotion = AnalyticsNumericPromotion.DECIMAL128, + precision = 34, + scale = 8, + roundingMode = RoundingMode.HALF_EVEN, + overflowPolicy = AnalyticsOverflowPolicy.REJECT, + ) + + private fun plan( + query: AnalyticsQuery, + constraints: PlanningConstraints = constraints(), + schema: QueryDocumentSchema = PlanningFixtures.schema, + ): AnalyticsQueryPlan = + (planner.plan(invocation(query), schema, constraints) as PlanningDecision.Planned).plan + as AnalyticsQueryPlan + + private fun invocation( + query: AnalyticsQuery, + target: QueryTarget = PlanningFixtures.target, + ): NormalizedQueryInvocation = NormalizedQueryInvocation( + target, + QueryOperation.ANALYZE, + QueryResultShape.ANALYTICS, + NormalizedQueryInput.Analytics(query), + ) + + private fun constraints( + analytics: AnalyticsPlanningConstraint = AnalyticsPlanningConstraint.Unrestricted, + ): PlanningConstraints = PlanningConstraints(QueryValidationMode.STRICT, analyticsConstraint = analytics) + + private fun assertInvalidCursor( + query: AnalyticsQuery, + cursor: DecodedAnalyticsCursor, + path: String, + schema: QueryDocumentSchema = PlanningFixtures.schema, + ) { + assertRejected(QueryRejectionCategory.INVALID_CURSOR, QueryRejectionCode.INVALID_CURSOR_BINDING, path) { + plan( + query.copy(bucketWindow = AnalyticsBucketWindow.After(limit = 25, cursor = cursor)), + schema = schema, + ) + } + } + + private fun schemaWithAmount( + presence: Presence, + nullability: Nullability, + ): QueryDocumentSchema { + val amount = PlanningFixtures.schema.fields.getValue(PlanningFixtures.amount).let { field -> + QueryFieldSchema( + field.id, + field.type, + presence, + nullability, + field.allowedOperators, + field.capabilities, + field.logicalAliases, + ) + } + return QueryDocumentSchema( + PlanningFixtures.target, + PlanningFixtures.schema.fields.values.map { field -> + if (field.id == PlanningFixtures.amount) amount else field + }, + PlanningFixtures.schema.searchScopes.values, + ) + } + + private fun assertRejected( + category: QueryRejectionCategory, + code: QueryRejectionCode, + path: String, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo(path) + }, + ) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerIdentitySortTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerIdentitySortTest.kt new file mode 100644 index 00000000000..f2f774c4239 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerIdentitySortTest.kt @@ -0,0 +1,136 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.internal.model.QueryDocumentKind +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryTarget +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedPage +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.normalization.NormalizedSortDirection +import me.ahoo.wow.query.internal.plan.PageQueryPlan +import me.ahoo.wow.query.internal.plan.PlannedSortOrigin +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import java.util.function.Consumer + +class QueryPlannerIdentitySortTest { + private val planner = QueryPlanner() + + @Test + fun `strict page should append canonical identity tie breaker`() { + val query = PlanningFixtures.recordQuery( + sort = listOf(PlanningFixtures.sort(PlanningFixtures.path("state", "amount"))), + ) + val page = plan(PlanningFixtures.page(query), PlanningFixtures.schema) + + page.sort.map { it.field }.assert().containsExactly(PlanningFixtures.amount, PlanningFixtures.identity) + page.sort.map { it.origin }.assert().containsExactly( + PlannedSortOrigin.USER, + PlannedSortOrigin.STABILITY_TIE_BREAKER, + ) + page.page.offset.assert().isEqualTo(0L) + page.page.size.assert().isEqualTo(20) + } + + @Test + fun `target schema aliases should canonicalize snapshot and event identity sort`() { + val snapshotQuery = PlanningFixtures.recordQuery( + sort = listOf( + PlanningFixtures.sort(PlanningFixtures.path("aggregateId"), NormalizedSortDirection.DESC), + ), + ) + val snapshot = plan(PlanningFixtures.page(snapshotQuery), PlanningFixtures.schema) + snapshot.sort.map { it.field }.assert().containsExactly(PlanningFixtures.identity) + snapshot.sort.single().origin.assert().isEqualTo(PlannedSortOrigin.USER) + + val eventTarget = QueryTarget(PlanningFixtures.target.namedAggregate, QueryDocumentKind.EVENT_STREAM) + val eventSchema = eventSchema(eventTarget) + val eventQuery = PlanningFixtures.recordQuery(sort = listOf(PlanningFixtures.sort(PlanningFixtures.path("id")))) + val eventInvocation = NormalizedQueryInvocation( + eventTarget, + QueryOperation.PAGE, + QueryResultShape.TYPED, + NormalizedQueryInput.Page(eventQuery, NormalizedPage(1, 20, 0)), + ) + + plan(eventInvocation, eventSchema).sort.map { it.field }.assert().containsExactly(PlanningFixtures.identity) + } + + @Test + fun `system field alias should not become a legacy search scope`() { + val condition = NormalizedCondition.Search( + PlanningFixtures.legacySearch(PlanningFixtures.path("aggregateId")), + "id", + ) + + assertThrownBy { + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(condition)), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) + }.satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(QueryRejectionCategory.UNSUPPORTED_FEATURE) + error.rejection.code.assert().isEqualTo(QueryRejectionCode.SEARCH_SCOPE_NOT_FOUND) + error.rejection.path.toString().assert().isEqualTo("$.input.query.condition.scope") + }, + ) + } + + private fun eventSchema(target: QueryTarget): QueryDocumentSchema { + val identity = PlanningFixtures.schema.fields.getValue(PlanningFixtures.identity).let { field -> + QueryFieldSchema( + field.id, + field.type, + field.presence, + field.nullability, + field.allowedOperators, + field.capabilities, + setOf(QueryFieldId.Path(listOf("id"))), + ) + } + return QueryDocumentSchema( + target, + PlanningFixtures.schema.fields.values.map { field -> + if (field.id == PlanningFixtures.identity) identity else field + }, + PlanningFixtures.schema.searchScopes.values, + ) + } + + private fun plan( + invocation: NormalizedQueryInvocation, + schema: QueryDocumentSchema, + ): PageQueryPlan = + ( + planner.plan( + invocation, + schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + ).plan as PageQueryPlan +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerInvocationMatrixTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerInvocationMatrixTest.kt new file mode 100644 index 00000000000..88b50404a68 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerInvocationMatrixTest.kt @@ -0,0 +1,138 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.internal.analytics.AnalyticsAlias +import me.ahoo.wow.query.internal.analytics.AnalyticsGrouping +import me.ahoo.wow.query.internal.analytics.AnalyticsMetric +import me.ahoo.wow.query.internal.analytics.AnalyticsQuery +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedDeletionScope +import me.ahoo.wow.query.internal.normalization.NormalizedPage +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.value.NonEmptyList +import org.junit.jupiter.params.ParameterizedTest +import org.junit.jupiter.params.provider.Arguments +import org.junit.jupiter.params.provider.MethodSource +import java.util.function.Consumer +import java.util.stream.Stream + +class QueryPlannerInvocationMatrixTest { + + private val planner = QueryPlanner() + + @ParameterizedTest(name = "{0}/{1}/{2}") + @MethodSource("invocationMatrix") + fun `planner should enforce the complete normalized invocation matrix`( + operationName: String, + resultShapeName: String, + inputName: String, + expectedValid: Boolean, + ) { + val operation = QueryOperation.valueOf(operationName) + val resultShape = QueryResultShape.valueOf(resultShapeName) + val input = input(inputName) + val invocation = NormalizedQueryInvocation( + PlanningFixtures.target, + operation, + resultShape, + input, + ) + if (expectedValid) { + planner.plan( + invocation, + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ).assert().isInstanceOf(PlanningDecision.Planned::class.java) + return + } + assertThrownBy { + planner.plan( + invocation, + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) + }.satisfies( + Consumer { error -> + error.rejection.code.assert().isEqualTo(QueryRejectionCode.INVALID_INVOCATION) + error.rejection.path.toString().assert().isEqualTo("$.input") + }, + ) + } + + companion object { + @JvmStatic + fun invocationMatrix(): Stream { + val inputNames = listOf("single", "stream", "page", "count", "analytics") + return QueryOperation.entries.flatMap { operation -> + QueryResultShape.entries.flatMap { resultShape -> + inputNames.map { inputName -> + val input = input(inputName) + Arguments.of( + operation.name, + resultShape.name, + inputName, + accepts(operation, resultShape, input) + ) + } + } + }.stream() + } + + private fun input(name: String): NormalizedQueryInput { + val query = PlanningFixtures.recordQuery() + return when (name) { + "single" -> NormalizedQueryInput.Single(query) + "stream" -> NormalizedQueryInput.Stream(query, limit = 0) + "page" -> NormalizedQueryInput.Page(query, NormalizedPage(1, 20, 0)) + "count" -> NormalizedQueryInput.Count(NormalizedCondition.All, NormalizedDeletionScope.EXPLICIT) + "analytics" -> NormalizedQueryInput.Analytics( + AnalyticsQuery( + NormalizedCondition.All, + AnalyticsGrouping.Global, + NonEmptyList.of(AnalyticsMetric.DocumentCount(AnalyticsAlias("count"))), + ), + ) + + else -> error("Unknown input: $name") + } + } + + private fun accepts( + operation: QueryOperation, + resultShape: QueryResultShape, + input: NormalizedQueryInput, + ): Boolean = + when (operation) { + QueryOperation.SINGLE -> input is NormalizedQueryInput.Single && resultShape.isRecord() + QueryOperation.STREAM -> input is NormalizedQueryInput.Stream && resultShape.isRecord() + QueryOperation.PAGE -> input is NormalizedQueryInput.Page && resultShape.isRecord() + QueryOperation.COUNT -> input is NormalizedQueryInput.Count && resultShape == QueryResultShape.COUNT + QueryOperation.ANALYZE -> { + input is NormalizedQueryInput.Analytics && resultShape == QueryResultShape.ANALYTICS + } + } + + private fun QueryResultShape.isRecord(): Boolean = + this == QueryResultShape.TYPED || this == QueryResultShape.DYNAMIC + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerNativeTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerNativeTest.kt new file mode 100644 index 00000000000..b9e5f2f58b7 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerNativeTest.kt @@ -0,0 +1,86 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.JunctionOperator +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.Utf8Json +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import java.util.function.Consumer + +class QueryPlannerNativeTest { + private val planner = QueryPlanner() + + @Test + fun `native conditions should require one coherent backend in every validation mode`() { + val sameBackend = nativeJunction("mongo", "mongo") + val nativePlan = planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(sameBackend)), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) as PlanningDecision.Planned + nativePlan.plan.semanticTier.assert().isEqualTo(SemanticTier.NATIVE) + nativePlan.plan.requiredCapabilities.nativeBackend.assert().isEqualTo(BackendId("mongo")) + + QueryValidationMode.entries.forEach { mode -> + assertThrownBy { + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(nativeJunction("mongo", "elasticsearch"))), + PlanningFixtures.schema, + PlanningConstraints(mode), + ) + }.satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(QueryRejectionCategory.INVALID_QUERY) + error.rejection.code.assert().isEqualTo(QueryRejectionCode.NATIVE_BACKEND_CONFLICT) + error.rejection.path.toString().assert() + .isEqualTo("$.input.query.condition.children[1].backendId") + }, + ) + + assertThrownBy { + planner.plan( + PlanningFixtures.single(), + PlanningFixtures.schema, + PlanningConstraints( + mode, + mandatoryCondition = NormalizedCondition.Native(BackendId("mongo"), Utf8Json("{}")), + ), + ) + }.satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(QueryRejectionCategory.UNSUPPORTED_FEATURE) + error.rejection.code.assert().isEqualTo(QueryRejectionCode.MANDATORY_NATIVE_NOT_ALLOWED) + error.rejection.path.toString().assert().isEqualTo("$.constraints.mandatoryCondition") + }, + ) + } + } + + private fun nativeJunction(first: String, second: String): NormalizedCondition = NormalizedCondition.Junction( + JunctionOperator.AND, + listOf( + NormalizedCondition.Native(BackendId(first), Utf8Json("{}")), + NormalizedCondition.Native(BackendId(second), Utf8Json("{}")), + ), + ) +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerRecordTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerRecordTest.kt new file mode 100644 index 00000000000..6a2df2f9a9a --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerRecordTest.kt @@ -0,0 +1,637 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.CaseSensitivity +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.NormalizedDeletionScope +import me.ahoo.wow.query.internal.normalization.NormalizedPredicateOptions +import me.ahoo.wow.query.internal.normalization.NormalizedProjection +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.normalization.NormalizedSortDirection +import me.ahoo.wow.query.internal.normalization.PathBasis +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.SearchScopeId +import me.ahoo.wow.query.internal.plan.CountQueryPlan +import me.ahoo.wow.query.internal.plan.PageQueryPlan +import me.ahoo.wow.query.internal.plan.PlannedCondition +import me.ahoo.wow.query.internal.plan.PlannedProjection +import me.ahoo.wow.query.internal.plan.SemanticTier +import me.ahoo.wow.query.internal.plan.SingleQueryPlan +import me.ahoo.wow.query.internal.plan.StreamLimit +import me.ahoo.wow.query.internal.plan.StreamQueryPlan +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.value.NonEmptyList +import org.junit.jupiter.api.Test +import java.util.function.Consumer + +class QueryPlannerRecordTest { + + private val planner = QueryPlanner() + + @Test + fun `planner should create all record operation plans and preserve zero as unbounded`() { + planner.plan( + PlanningFixtures.single(), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ).planned().plan.assert().isInstanceOf(SingleQueryPlan::class.java) + + val stream = NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.STREAM, + QueryResultShape.DYNAMIC, + NormalizedQueryInput.Stream(PlanningFixtures.recordQuery(), limit = 0), + ) + val streamPlan = planner.plan(stream, PlanningFixtures.schema, constraints(QueryValidationMode.STRICT)) + .planned().plan as StreamQueryPlan + streamPlan.limit.assert().isEqualTo(StreamLimit.Unbounded) + + val bounded = stream.copy(input = NormalizedQueryInput.Stream(PlanningFixtures.recordQuery(), limit = 25)) + ( + planner.plan(bounded, PlanningFixtures.schema, constraints(QueryValidationMode.STRICT)).planned().plan + as StreamQueryPlan + ).limit.assert().isEqualTo(StreamLimit.Bounded(25)) + + val count = NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.COUNT, + QueryResultShape.COUNT, + NormalizedQueryInput.Count(NormalizedCondition.All, NormalizedDeletionScope.EXPLICIT), + ) + planner.plan(count, PlanningFixtures.schema, constraints(QueryValidationMode.STRICT)) + .planned().plan.operation.assert().isEqualTo(QueryOperation.COUNT) + } + + @Test + fun `planner should apply default active deletion without treating it as user field access`() { + val defaultActive = PlanningFixtures.recordQuery( + deletionScope = NormalizedDeletionScope.DEFAULT_ACTIVE, + ) + val constrained = constraints(QueryValidationMode.STRICT).copy( + fieldConstraint = QueryFieldConstraint(filterFields = FieldAccess.DenyAll), + ) + val recordPlan = planner.plan( + PlanningFixtures.single(defaultActive), + PlanningFixtures.schema, + constrained, + ).planned().plan as SingleQueryPlan + recordPlan.filter.user.assert().isEqualTo( + PlannedCondition.Predicate(PlanningFixtures.deleted, PredicateOperator.IS_FALSE), + ) + + val countPlan = planner.plan( + NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.COUNT, + QueryResultShape.COUNT, + NormalizedQueryInput.Count(NormalizedCondition.All, NormalizedDeletionScope.DEFAULT_ACTIVE), + ), + PlanningFixtures.schema, + constrained, + ).planned().plan as CountQueryPlan + countPlan.filter.user.assert().isEqualTo(recordPlan.filter.user) + + val explicitAll = PlanningFixtures.recordQuery( + deletionScope = NormalizedDeletionScope.EXPLICIT, + ) + val explicitPlan = planner.plan( + PlanningFixtures.single(explicitAll), + PlanningFixtures.schema, + constrained, + ).planned().plan as SingleQueryPlan + explicitPlan.filter.user.assert().isEqualTo(PlannedCondition.All) + } + + @Test + fun `compatible page should not silently change legacy ordering`() { + val page = planner.plan( + PlanningFixtures.page(), + PlanningFixtures.schema, + constraints(QueryValidationMode.COMPATIBLE), + ).planned().plan as PageQueryPlan + + page.sort.assert().isEmpty() + } + + @Test + fun `duplicate sort should reject in strict and fallback in compatible mode`() { + val field = PlanningFixtures.path("state", "amount") + val query = PlanningFixtures.recordQuery( + sort = listOf( + PlanningFixtures.sort(field), + PlanningFixtures.sort(field, NormalizedSortDirection.DESC), + ), + ) + + assertRejected( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.DUPLICATE_SORT, + "$.input.query.sort[1].field", + ) { + planner.plan( + PlanningFixtures.page(query), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ) + } + planner.plan( + PlanningFixtures.page(query), + PlanningFixtures.schema, + constraints(QueryValidationMode.COMPATIBLE), + ).assert().isInstanceOf(PlanningDecision.LegacyFallback::class.java) + } + + @Test + fun `planner should preserve mandatory provenance in plan and fallback`() { + val user = predicate(PlanningFixtures.path("state", "name"), PredicateOperator.EQ, NormalizedValue.Text("Ada")) + val mandatory = predicate( + LogicalField.System(me.ahoo.wow.query.internal.normalization.SystemFieldKind.TENANT_ID), + PredicateOperator.EQ, + NormalizedValue.Text("tenant-1"), + ) + val planned = planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(user)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT, mandatory), + ).planned().plan as SingleQueryPlan + + planned.filter.user.assert().isEqualTo( + PlannedCondition.Predicate(PlanningFixtures.name, PredicateOperator.EQ, NormalizedValue.Text("Ada")), + ) + planned.filter.mandatory.assert().isEqualTo( + PlannedCondition.Predicate(PlanningFixtures.tenant, PredicateOperator.EQ, NormalizedValue.Text("tenant-1")), + ) + (planned.filter.condition as PlannedCondition.Junction).children.values.assert().containsExactly( + planned.filter.user, + planned.filter.mandatory, + ) + + val compatibleGap = PlanningFixtures.recordQuery( + predicate( + PlanningFixtures.path("state", "unknown"), + PredicateOperator.EQ, + NormalizedValue.Text("Ada"), + ), + ) + val fallback = planner.plan( + PlanningFixtures.single(compatibleGap), + PlanningFixtures.schema, + constraints(QueryValidationMode.COMPATIBLE, mandatory), + ) as PlanningDecision.LegacyFallback + fallback.validatedMandatory.condition.assert().isEqualTo(planned.filter.mandatory) + fallback.validatedMandatory.target.assert().isEqualTo(PlanningFixtures.target) + fallback.validatedMandatory.schemaContractId.assert().isEqualTo(PlanningFixtures.schema.contractId) + fallback.issues.values.single().code.assert().isEqualTo(QueryRejectionCode.FIELD_NOT_FOUND) + } + + @Test + fun `mandatory failure should fail closed in compatible mode`() { + val missingMandatory = predicate( + PlanningFixtures.path("state", "missing"), + PredicateOperator.EQ, + NormalizedValue.Text("secret"), + ) + + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.FIELD_NOT_FOUND, + "$.constraints.mandatoryCondition.field", + ) { + planner.plan( + PlanningFixtures.single(), + PlanningFixtures.schema, + constraints(QueryValidationMode.COMPATIBLE, missingMandatory), + ) + } + } + + @Test + fun `typed projection should reject until legacy typed fallback is provably lossless`() { + val include = NormalizedProjection.Include(NonEmptyList.of(PlanningFixtures.path("state", "name"))) + val mixed = NormalizedProjection.Mixed( + NonEmptyList.of(PlanningFixtures.path("state", "name")), + NonEmptyList.of(PlanningFixtures.path("state", "amount")), + ) + + QueryValidationMode.entries.forEach { validationMode -> + listOf(include, mixed).forEach { projection -> + listOf( + PlanningFixtures.single(PlanningFixtures.recordQuery(projection = projection)), + PlanningFixtures.stream(PlanningFixtures.recordQuery(projection = projection)), + PlanningFixtures.page(PlanningFixtures.recordQuery(projection = projection)), + ).forEach { invocation -> + assertRejected( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.TYPED_PROJECTION_NOT_ALLOWED, + "$.input.query.projection", + ) { + planner.plan( + invocation, + PlanningFixtures.schema, + constraints(validationMode), + ) + } + } + } + } + } + + @Test + fun `dynamic projection should canonicalize valid fields and preserve rejection path`() { + val include = NormalizedProjection.Include(NonEmptyList.of(PlanningFixtures.path("state", "name"))) + val mixed = NormalizedProjection.Mixed( + NonEmptyList.of(PlanningFixtures.path("state", "name")), + NonEmptyList.of(PlanningFixtures.path("state", "amount")), + ) + + val dynamic = planner.plan( + PlanningFixtures.single( + PlanningFixtures.recordQuery(projection = include), + QueryResultShape.DYNAMIC, + ), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ).planned().plan as SingleQueryPlan + dynamic.projection.assert().isEqualTo(PlannedProjection.Include(NonEmptyList.of(PlanningFixtures.name))) + + assertRejected( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.INVALID_PROJECTION, + "$.input.query.projection", + ) { + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(projection = mixed), QueryResultShape.DYNAMIC), + PlanningFixtures.schema, + constraints(QueryValidationMode.COMPATIBLE), + ) + } + + val originalOrder = NormalizedProjection.Include( + NonEmptyList.of( + PlanningFixtures.path("state", "name"), + PlanningFixtures.path("aggregateId"), + ), + ) + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.CAPABILITY_UNAVAILABLE, + "$.input.query.projection.fields[1]", + ) { + planner.plan( + PlanningFixtures.single( + PlanningFixtures.recordQuery(projection = originalOrder), + QueryResultShape.DYNAMIC, + ), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ) + } + } + + @Test + fun `planner should bind nested element fields to canonical logical ids`() { + val condition = NormalizedCondition.ElementMatch( + PlanningFixtures.path("state", "items"), + predicate( + PlanningFixtures.path("name", basis = PathBasis.CURRENT_ELEMENT), + PredicateOperator.CONTAINS, + NormalizedValue.Text("book"), + ), + ) + val plan = planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(condition)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ).planned().plan as SingleQueryPlan + val element = plan.filter.user as PlannedCondition.ElementMatch + + element.field.assert().isEqualTo(PlanningFixtures.items) + (element.condition as PlannedCondition.Predicate).field.assert().isEqualTo(PlanningFixtures.itemName) + plan.requiredCapabilities.fieldRequirements.getValue(PlanningFixtures.items).assert() + .contains(FieldCapability.ELEMENT_MATCH) + plan.requiredCapabilities.fieldRequirements.getValue(PlanningFixtures.itemName).assert() + .contains(FieldCapability.LITERAL_PATTERN) + } + + @Test + fun `planner should recursively bind multi level element scopes`() { + val condition = NormalizedCondition.ElementMatch( + PlanningFixtures.path("state", "items"), + NormalizedCondition.ElementMatch( + PlanningFixtures.path("attributes", basis = PathBasis.CURRENT_ELEMENT), + predicate( + PlanningFixtures.path("name", basis = PathBasis.CURRENT_ELEMENT), + PredicateOperator.EQ, + NormalizedValue.Text("color"), + ), + ), + ) + val plan = planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(condition)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ).planned().plan as SingleQueryPlan + val outer = plan.filter.user as PlannedCondition.ElementMatch + val inner = outer.condition as PlannedCondition.ElementMatch + + outer.field.assert().isEqualTo(PlanningFixtures.items) + inner.field.assert().isEqualTo(PlanningFixtures.itemAttributes) + (inner.condition as PlannedCondition.Predicate).field.assert().isEqualTo(PlanningFixtures.itemAttributeName) + } + + @Test + fun `element scope should reject root paths instead of escaping nested semantics`() { + val condition = NormalizedCondition.ElementMatch( + PlanningFixtures.path("state", "items"), + predicate( + PlanningFixtures.path("state", "name"), + PredicateOperator.EQ, + NormalizedValue.Text("outside"), + ), + ) + + assertRejected( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.INVALID_FIELD, + "$.input.query.condition.condition.field", + ) { + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(condition)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ) + } + } + + @Test + fun `string exact literal and search should remain separate capabilities and tiers`() { + val search = NormalizedCondition.Search( + PlanningFixtures.legacySearch(PlanningFixtures.path("state", "description")), + "distributed systems", + ) + val plan = planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(search)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ).planned().plan as SingleQueryPlan + + plan.semanticTier.assert().isEqualTo(SemanticTier.SEARCH) + plan.requiredCapabilities.searchRequirements.assert().contains(SearchScopeId("order-description")) + plan.requiredCapabilities.fieldRequirements.values.flatten().assert().doesNotContain( + FieldCapability.EXACT, + FieldCapability.LITERAL_PATTERN, + ) + + val insensitiveLiteral = predicate( + PlanningFixtures.path("state", "name"), + PredicateOperator.CONTAINS, + NormalizedValue.Text("Ada"), + NormalizedPredicateOptions(CaseSensitivity.INSENSITIVE), + ) + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.CASE_INSENSITIVE_UNSUPPORTED, + "$.input.query.condition.options.caseSensitivity", + ) { + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(insensitiveLiteral)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ) + } + } + + @Test + fun `predicate operators should produce contextual logical capabilities`() { + val predicates = listOf( + predicate( + PlanningFixtures.path("state", "name"), + PredicateOperator.EQ, + NormalizedValue.Text("Ada"), + ) to (PlanningFixtures.name to FieldCapability.EXACT), + predicate( + PlanningFixtures.path("state", "amount"), + PredicateOperator.GT, + NormalizedValue.Int64(10), + ) to (PlanningFixtures.amount to FieldCapability.RANGE), + predicate( + PlanningFixtures.path("state", "name"), + PredicateOperator.EXISTS, + NormalizedValue.BooleanValue(true), + ) to (PlanningFixtures.name to FieldCapability.PRESENCE), + predicate( + PlanningFixtures.path("state", "name"), + PredicateOperator.CONTAINS, + NormalizedValue.Text("Ada"), + ) to (PlanningFixtures.name to FieldCapability.LITERAL_PATTERN), + ) + + predicates.forEach { (condition, expected) -> + val plan = planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(condition)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ).planned().plan + plan.requiredCapabilities.fieldRequirements.getValue(expected.first).assert().contains(expected.second) + } + } + + @Test + fun `compatible schema gap should be explicit fallback and strict gap should reject`() { + val physicalGuess = predicate( + PlanningFixtures.path("state", "name", "keyword"), + PredicateOperator.EQ, + NormalizedValue.Text("Ada"), + ) + val compatible = planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(physicalGuess)), + PlanningFixtures.schema, + constraints(QueryValidationMode.COMPATIBLE), + ) as PlanningDecision.LegacyFallback + compatible.issues.values.single().path.toString().assert().isEqualTo("$.input.query.condition.field") + compatible.issues.values.single().code.assert().isEqualTo(QueryRejectionCode.FIELD_NOT_FOUND) + + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.FIELD_NOT_FOUND, + "$.input.query.condition.field", + ) { + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(physicalGuess)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ) + } + } + + @Test + fun `planner should reject inconsistent normalized invocation matrix`() { + assertRejected( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.INVALID_INVOCATION, + "$.input", + ) { + planner.plan( + NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.COUNT, + QueryResultShape.COUNT, + NormalizedQueryInput.Single(PlanningFixtures.recordQuery()), + ), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ) + } + } + + @Test + fun `planner should validate normalized values against logical field types`() { + val invalid = predicate( + PlanningFixtures.path("state", "name"), + PredicateOperator.EQ, + NormalizedValue.Int64(1), + ) + + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.VALUE_TYPE_MISMATCH, + "$.input.query.condition.value", + ) { + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(invalid)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ) + } + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(invalid)), + PlanningFixtures.schema, + constraints(QueryValidationMode.COMPATIBLE), + ).assert().isInstanceOf(PlanningDecision.LegacyFallback::class.java) + } + + @Test + fun `planner should validate collection operands by operator and array element type`() { + val tags = PlanningFixtures.path("state", "tags") + val all = predicate( + tags, + PredicateOperator.ALL_IN, + NormalizedValue.ListValue(listOf(NormalizedValue.Text("blue"), NormalizedValue.Null)), + ) + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(all)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ).assert().isInstanceOf(PlanningDecision.Planned::class.java) + + val validBetween = predicate( + PlanningFixtures.path("state", "amount"), + PredicateOperator.BETWEEN, + NormalizedValue.ListValue(listOf(NormalizedValue.Int64(1), NormalizedValue.Int64(2))), + ) + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(validBetween)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ).assert().isInstanceOf(PlanningDecision.Planned::class.java) + + listOf(1, 3).forEach { arity -> + val between = predicate( + PlanningFixtures.path("state", "amount"), + PredicateOperator.BETWEEN, + NormalizedValue.ListValue(List(arity) { NormalizedValue.Int64(it.toLong()) }), + ) + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.VALUE_TYPE_MISMATCH, + "$.input.query.condition.value", + ) { + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(between)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ) + } + } + } + + @Test + fun `range operators should reject null operands`() { + listOf( + predicate( + PlanningFixtures.path("state", "amount"), + PredicateOperator.GT, + NormalizedValue.Null, + ), + predicate( + PlanningFixtures.path("state", "amount"), + PredicateOperator.BETWEEN, + NormalizedValue.ListValue(listOf(NormalizedValue.Int64(1), NormalizedValue.Null)), + ), + ).forEach { invalidRange -> + assertRejected( + QueryRejectionCategory.UNSUPPORTED_FEATURE, + QueryRejectionCode.VALUE_TYPE_MISMATCH, + "$.input.query.condition.value", + ) { + planner.plan( + PlanningFixtures.single(PlanningFixtures.recordQuery(invalidRange)), + PlanningFixtures.schema, + constraints(QueryValidationMode.STRICT), + ) + } + } + } + + private fun constraints( + validationMode: QueryValidationMode, + mandatory: NormalizedCondition = NormalizedCondition.All, + ): PlanningConstraints = PlanningConstraints(validationMode, mandatory) + + private fun predicate( + field: LogicalField, + operator: PredicateOperator, + value: NormalizedValue, + options: NormalizedPredicateOptions = NormalizedPredicateOptions(), + ): NormalizedCondition.Predicate = NormalizedCondition.Predicate(field, operator, value, options) + + private fun PlanningDecision.planned(): PlanningDecision.Planned = this as PlanningDecision.Planned + + private fun assertRejected( + category: QueryRejectionCategory, + code: QueryRejectionCode, + path: String, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo(path) + }, + ) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerResultConstraintTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerResultConstraintTest.kt new file mode 100644 index 00000000000..78a46fca950 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/planning/QueryPlannerResultConstraintTest.kt @@ -0,0 +1,104 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.planning + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.query.internal.model.QueryOperation +import me.ahoo.wow.query.internal.model.QueryResultShape +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInput +import me.ahoo.wow.query.internal.normalization.NormalizedQueryInvocation +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import org.junit.jupiter.api.Test +import java.util.function.Consumer + +class QueryPlannerResultConstraintTest { + private val planner = QueryPlanner() + + @Test + fun `static result constraints should reject unbounded stream and oversized page without fallback`() { + val stream = NormalizedQueryInvocation( + PlanningFixtures.target, + QueryOperation.STREAM, + QueryResultShape.DYNAMIC, + NormalizedQueryInput.Stream(PlanningFixtures.recordQuery(), limit = 0), + ) + assertRejected( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.UNBOUNDED_STREAM_DISALLOWED, + "$.input.limit", + ) { + planner.plan( + stream, + PlanningFixtures.schema, + PlanningConstraints( + QueryValidationMode.COMPATIBLE, + streamConstraint = StreamPlanningConstraint.BoundedOnly, + ), + ) + } + + planner.plan( + PlanningFixtures.page(index = 5, size = 20, offset = 80), + PlanningFixtures.schema, + PlanningConstraints( + QueryValidationMode.STRICT, + pageConstraint = PagePlanningConstraint.MaximumWindow(100), + ), + ).assert().isInstanceOf(PlanningDecision.Planned::class.java) + assertRejected( + QueryRejectionCategory.BUDGET_EXCEEDED, + QueryRejectionCode.PAGE_WINDOW_EXCEEDED, + "$.input.page", + ) { + planner.plan( + PlanningFixtures.page(index = 6, size = 20, offset = 100), + PlanningFixtures.schema, + PlanningConstraints( + QueryValidationMode.COMPATIBLE, + pageConstraint = PagePlanningConstraint.MaximumWindow(100), + ), + ) + } + assertRejected( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionCode.INVALID_PAGE, + "$.input.page", + ) { + planner.plan( + PlanningFixtures.page(index = 5, size = 20, offset = 81), + PlanningFixtures.schema, + PlanningConstraints(QueryValidationMode.STRICT), + ) + } + } + + private fun assertRejected( + category: QueryRejectionCategory, + code: QueryRejectionCode, + path: String, + action: () -> Unit, + ) { + assertThrownBy(action).satisfies( + Consumer { error -> + error.rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo(path) + }, + ) + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/policy/QueryExecutionContextFactoryTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/policy/QueryExecutionContextFactoryTest.kt new file mode 100644 index 00000000000..6c42b31e912 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/policy/QueryExecutionContextFactoryTest.kt @@ -0,0 +1,239 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.policy + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.query.internal.model.QueryExecutionMode +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.planning.PlanningFixtures +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejection +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import org.junit.jupiter.api.Test +import reactor.core.publisher.Mono +import reactor.kotlin.test.test +import java.time.Clock +import java.time.Instant +import java.time.ZoneId +import java.time.ZoneOffset +import java.util.concurrent.atomic.AtomicInteger + +class QueryExecutionContextFactoryTest { + private val now = Instant.parse("2026-08-07T00:00:00Z") + private val clock = Clock.fixed(now, ZoneOffset.UTC) + + @Test + fun `authority should resolve once per subscription without creation-time capture`() { + val subscriptions = AtomicInteger() + val provider = QueryAuthorityProvider { + Mono.defer { + Mono.just( + QueryAuthority.Subject( + subjectId = "subject-${subscriptions.incrementAndGet()}", + tenantId = "tenant-1", + ownerGrant = QueryOwnerGrant.Unrestricted, + spaceGrant = QuerySpaceGrant.Unrestricted, + ), + ) + } + } + val context = QueryExecutionContextFactory(provider, clock).resolve(request()) + + context.test() + .consumeNextWith { it.authority.principalId.assert().isEqualTo("subject-1") } + .verifyComplete() + context.test() + .consumeNextWith { it.authority.principalId.assert().isEqualTo("subject-2") } + .verifyComplete() + subscriptions.get().assert().isEqualTo(2) + } + + @Test + fun `missing and failed authority should reject without fail-open fallback`() { + assertRejected(QueryRejectionCode.AUTHORITY_REQUIRED, "$.executionContext.authority") { + QueryExecutionContextFactory(QueryAuthorityProvider { Mono.empty() }, clock) + .resolve(request()) + } + val failure = IllegalStateException("identity provider unavailable") + assertRejected(QueryRejectionCode.AUTHORITY_RESOLUTION_FAILED, "$.executionContext.authority", failure) { + QueryExecutionContextFactory(QueryAuthorityProvider { Mono.error(failure) }, clock) + .resolve(request()) + } + assertRejected(QueryRejectionCode.AUTHORITY_RESOLUTION_FAILED, "$.executionContext.authority", failure) { + QueryExecutionContextFactory(QueryAuthorityProvider { throw failure }, clock) + .resolve(request()) + } + val typedFailure = QueryRejectedException( + QueryRejection( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionPath.ROOT, + QueryRejectionCode.INVALID_FIELD, + ), + ) + assertRejected( + QueryRejectionCode.AUTHORITY_RESOLUTION_FAILED, + "$.executionContext.authority", + typedFailure, + ) { + QueryExecutionContextFactory(QueryAuthorityProvider { Mono.error(typedFailure) }, clock) + .resolve(request()) + } + } + + @Test + fun `expired deadline should reject before resolving authority`() { + val subscriptions = AtomicInteger() + val provider = QueryAuthorityProvider { + subscriptions.incrementAndGet() + Mono.just(QueryAuthority.System("scheduler", "retention-job")) + } + + assertRejected( + QueryRejectionCode.DEADLINE_EXPIRED, + "$.executionContext.deadline", + category = QueryRejectionCategory.BUDGET_EXCEEDED, + ) { + QueryExecutionContextFactory(provider, clock) + .resolve(request(deadline = now.minusSeconds(1))) + } + subscriptions.get().assert().isZero() + } + + @Test + fun `authority resolving after deadline should reject without emitting stale context`() { + val advancingClock = MutableClock(now) + val provider = QueryAuthorityProvider { + Mono.fromSupplier { + advancingClock.current = now.plusSeconds(31) + subject() + } + } + + assertRejected( + QueryRejectionCode.DEADLINE_EXPIRED, + "$.executionContext.deadline", + category = QueryRejectionCategory.BUDGET_EXCEEDED, + ) { + QueryExecutionContextFactory(provider, advancingClock).resolve(request()) + } + } + + @Test + fun `legacy authority should require an exact trusted grant`() { + val caller = LegacyQueryCallerId("compensation-retry") + val grant = LegacyQueryGrant( + caller, + PlanningFixtures.target, + QueryPurpose("interactive-query"), + QueryExecutionMode.LEGACY, + QueryResourceScope("tenant-1"), + ) + val denied = LegacyQueryAuthorityProvider() + assertRejected(QueryRejectionCode.LEGACY_CALLER_NOT_ALLOWED, "$.executionContext.legacyGrant") { + QueryExecutionContextFactory(denied, clock).resolve(request()) + } + + QueryExecutionContextFactory(LegacyQueryAuthorityProvider(grant), clock) + .resolve(request()) + .test() + .consumeNextWith { context -> context.authority.assert().isEqualTo(QueryAuthority.Legacy(grant)) } + .verifyComplete() + assertRejected(QueryRejectionCode.LEGACY_CALLER_NOT_ALLOWED, "$.executionContext.legacyGrant") { + QueryExecutionContextFactory(LegacyQueryAuthorityProvider(grant), clock) + .resolve(request().copy(purpose = QueryPurpose("another-purpose"))) + } + assertRejected(QueryRejectionCode.LEGACY_CALLER_NOT_ALLOWED, "$.executionContext.legacyGrant") { + QueryExecutionContextFactory(LegacyQueryAuthorityProvider(grant), clock) + .resolve(request().copy(executionMode = QueryExecutionMode.PLANNED)) + } + val shadowGrant = grant.copy(executionMode = QueryExecutionMode.SHADOW) + QueryExecutionContextFactory(LegacyQueryAuthorityProvider(shadowGrant), clock) + .resolve(request().copy(executionMode = QueryExecutionMode.SHADOW)) + .test() + .consumeNextWith { context -> context.authority.assert().isEqualTo(QueryAuthority.Legacy(shadowGrant)) } + .verifyComplete() + val mismatchedGrant = grant.copy(purpose = QueryPurpose("another-purpose")) + assertRejected(QueryRejectionCode.LEGACY_CALLER_NOT_ALLOWED, "$.executionContext.legacyGrant") { + QueryExecutionContextFactory( + QueryAuthorityProvider { Mono.just(QueryAuthority.Legacy(mismatchedGrant)) }, + clock, + ).resolve(request()) + } + } + + @Test + fun `authority collections should be defensive copies`() { + val spaces = linkedSetOf("space-1") + val authority = QueryAuthority.Subject( + "subject-1", + "tenant-1", + QueryOwnerGrant.Unrestricted, + QuerySpaceGrant.AllowList(spaces), + ) + spaces += "space-2" + + (authority.spaceGrant as QuerySpaceGrant.AllowList).spaceIds.assert().containsExactly("space-1") + } + + private fun request( + deadline: Instant? = now.plusSeconds(30), + ): QueryExecutionRequest = QueryExecutionRequest( + target = PlanningFixtures.target, + purpose = QueryPurpose("interactive-query"), + executionMode = QueryExecutionMode.LEGACY, + validationMode = QueryValidationMode.COMPATIBLE, + resourceScope = QueryResourceScope(tenantId = "tenant-1"), + deadline = deadline, + budget = QueryExecutionBudget(maxReturnedRecords = 100), + ) + + private fun subject(): QueryAuthority.Subject = QueryAuthority.Subject( + "subject-1", + "tenant-1", + QueryOwnerGrant.Unrestricted, + QuerySpaceGrant.Unrestricted, + ) + + private fun assertRejected( + code: QueryRejectionCode, + path: String, + cause: Throwable? = null, + category: QueryRejectionCategory = QueryRejectionCategory.ACCESS_DENIED, + publisher: () -> Mono<*>, + ) { + publisher().test() + .consumeErrorWith { error -> + (error as QueryRejectedException).rejection.category.assert().isEqualTo(category) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo(path) + if (cause != null) { + error.cause.assert().isSameAs(cause) + } + } + .verify() + } + + private class MutableClock( + var current: Instant, + private val zoneId: ZoneId = ZoneOffset.UTC, + ) : Clock() { + override fun getZone(): ZoneId = zoneId + + override fun withZone(zone: ZoneId): Clock = MutableClock(current, zone) + + override fun instant(): Instant = current + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/policy/QueryPolicyEnforcerTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/policy/QueryPolicyEnforcerTest.kt new file mode 100644 index 00000000000..6f24d8f33a5 --- /dev/null +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/internal/policy/QueryPolicyEnforcerTest.kt @@ -0,0 +1,344 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.query.internal.policy + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.query.backend.NormalizedValue +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.internal.model.QueryExecutionMode +import me.ahoo.wow.query.internal.model.QueryValidationMode +import me.ahoo.wow.query.internal.normalization.BackendId +import me.ahoo.wow.query.internal.normalization.LogicalField +import me.ahoo.wow.query.internal.normalization.NormalizedCondition +import me.ahoo.wow.query.internal.normalization.PredicateOperator +import me.ahoo.wow.query.internal.normalization.SystemFieldKind +import me.ahoo.wow.query.internal.normalization.Utf8Json +import me.ahoo.wow.query.internal.planning.FieldAccess +import me.ahoo.wow.query.internal.planning.PlanningFixtures +import me.ahoo.wow.query.internal.planning.QueryFieldConstraint +import me.ahoo.wow.query.internal.rejection.QueryRejectedException +import me.ahoo.wow.query.internal.rejection.QueryRejection +import me.ahoo.wow.query.internal.rejection.QueryRejectionCategory +import me.ahoo.wow.query.internal.rejection.QueryRejectionCode +import me.ahoo.wow.query.internal.rejection.QueryRejectionPath +import org.junit.jupiter.api.Test +import reactor.core.publisher.Mono +import reactor.kotlin.test.test +import java.time.Instant + +class QueryPolicyEnforcerTest { + private val context = QueryExecutionContext( + target = PlanningFixtures.target, + purpose = QueryPurpose("interactive-query"), + authority = QueryAuthority.Subject( + "subject-1", + "tenant-1", + QueryOwnerGrant.Unrestricted, + QuerySpaceGrant.Unrestricted, + ), + executionMode = QueryExecutionMode.PLANNED, + validationMode = QueryValidationMode.STRICT, + resourceScope = QueryResourceScope("tenant-1"), + deadline = Instant.parse("2026-08-07T00:01:00Z"), + budget = QueryExecutionBudget(maxReturnedRecords = 100), + ) + + @Test + fun `allow should produce immutable planning constraints with mandatory provenance`() { + val source = linkedSetOf(PlanningFixtures.name) + val mandatory = NormalizedCondition.Predicate( + LogicalField.System(SystemFieldKind.TENANT_ID), + PredicateOperator.EQ, + NormalizedValue.Text("tenant-1"), + ) + val allowance = QueryPolicyAllowance.builder() + .mandatoryCondition(mandatory) + .fieldConstraint( + QueryFieldConstraint( + filterFields = FieldAccess.AllowList(source), + projectionFields = FieldAccess.AllowList(source), + ), + ) + .build() + source += PlanningFixtures.amount + + QueryPolicyEnforcer(QueryPolicy { Mono.just(QueryPolicyDecision.Allow(allowance)) }) + .authorize(QueryPolicyInput(context, PlanningFixtures.single(), PlanningFixtures.schema)) + .test() + .consumeNextWith { constraints -> + constraints.validationMode.assert().isEqualTo(QueryValidationMode.STRICT) + constraints.mandatoryCondition.assert().isEqualTo(mandatory) + constraints.fieldConstraint.filterFields.permits(PlanningFixtures.name).assert().isTrue() + constraints.fieldConstraint.filterFields.permits(PlanningFixtures.amount).assert().isFalse() + } + .verifyComplete() + val equalAllowance = QueryPolicyAllowance.builder() + .mandatoryCondition(mandatory) + .fieldConstraint( + QueryFieldConstraint( + filterFields = FieldAccess.AllowList(setOf(PlanningFixtures.name)), + projectionFields = FieldAccess.AllowList(setOf(PlanningFixtures.name)), + ), + ) + .build() + allowance.assert().isEqualTo(equalAllowance) + allowance.hashCode().assert().isEqualTo(equalAllowance.hashCode()) + } + + @Test + fun `deny empty and error should all fail closed`() { + assertRejected(QueryRejectionCode.POLICY_DENIED) { + QueryPolicyEnforcer(QueryPolicy { Mono.just(QueryPolicyDecision.Deny(QueryPolicyDenial.TENANT_MISMATCH)) }) + .authorize(QueryPolicyInput(context, PlanningFixtures.single(), PlanningFixtures.schema)) + } + QueryPolicyEnforcer(QueryPolicy { Mono.just(QueryPolicyDecision.Deny(QueryPolicyDenial.TENANT_MISMATCH)) }) + .authorize(QueryPolicyInput(context, PlanningFixtures.single(), PlanningFixtures.schema)) + .test() + .consumeErrorWith { error -> + (error.cause as QueryPolicyDeniedException).reason.assert() + .isEqualTo(QueryPolicyDenial.TENANT_MISMATCH) + } + .verify() + assertRejected(QueryRejectionCode.POLICY_DECISION_MISSING) { + QueryPolicyEnforcer(QueryPolicy { Mono.empty() }) + .authorize(QueryPolicyInput(context, PlanningFixtures.single(), PlanningFixtures.schema)) + } + val failure = IllegalStateException("policy store unavailable") + assertRejected(QueryRejectionCode.POLICY_EVALUATION_FAILED, failure) { + QueryPolicyEnforcer(QueryPolicy { Mono.error(failure) }) + .authorize(QueryPolicyInput(context, PlanningFixtures.single(), PlanningFixtures.schema)) + } + } + + @Test + fun `policy builder should reject mandatory native condition`() { + assertRejected( + QueryRejectionCode.MANDATORY_NATIVE_NOT_ALLOWED, + expectedPath = "$.policy.mandatoryCondition", + ) { + QueryPolicyEnforcer( + QueryPolicy { + Mono.fromCallable { + QueryPolicyDecision.Allow( + QueryPolicyAllowance.builder() + .mandatoryCondition( + NormalizedCondition.Native( + BackendId("mongo"), + Utf8Json("{\"tenantId\":\"tenant-1\"}"), + ), + ) + .build(), + ) + } + }, + ).authorize(QueryPolicyInput(context, PlanningFixtures.single(), PlanningFixtures.schema)) + } + } + + @Test + fun `tenant policy should turn trusted scope into mandatory predicates and deny selector mismatch`() { + val scopedContext = context.copy( + authority = QueryAuthority.Subject( + "subject-1", + "tenant-1", + ownerGrant = QueryOwnerGrant.Only("owner-1"), + spaceGrant = QuerySpaceGrant.AllowList(setOf("space-2", "space-1")), + ), + resourceScope = QueryResourceScope("tenant-1", "owner-1", "space-1"), + ) + val input = QueryPolicyInput(scopedContext, PlanningFixtures.single(), PlanningFixtures.schema) + + QueryPolicyEnforcer(TenantIsolationQueryPolicy()).authorize(input) + .test() + .consumeNextWith { constraints -> + val junction = constraints.mandatoryCondition as NormalizedCondition.Junction + junction.children.assert().hasSize(3) + } + .verifyComplete() + + assertRejected(QueryRejectionCode.POLICY_DENIED) { + QueryPolicyEnforcer(TenantIsolationQueryPolicy()).authorize( + input.copy(executionContext = scopedContext.copy(resourceScope = QueryResourceScope("tenant-2"))), + ) + } + listOf( + QueryResourceScope("tenant-1", ownerId = "owner-2"), + QueryResourceScope("tenant-1", spaceId = "space-3"), + ).forEach { mismatchedScope -> + assertRejected(QueryRejectionCode.POLICY_DENIED) { + QueryPolicyEnforcer(TenantIsolationQueryPolicy()).authorize( + input.copy(executionContext = scopedContext.copy(resourceScope = mismatchedScope)), + ) + } + } + } + + @Test + fun `subject grants should remain mandatory when selectors are absent or fail closed`() { + val scopedContext = context.copy( + authority = QueryAuthority.Subject( + "subject-1", + "tenant-1", + ownerGrant = QueryOwnerGrant.Only("owner-1"), + spaceGrant = QuerySpaceGrant.AllowList(setOf("space-2", "space-1")), + ), + resourceScope = QueryResourceScope("tenant-1"), + ) + val input = QueryPolicyInput(scopedContext, PlanningFixtures.single(), PlanningFixtures.schema) + + QueryPolicyEnforcer(TenantIsolationQueryPolicy()).authorize(input).test() + .consumeNextWith { constraints -> + val junction = constraints.mandatoryCondition as NormalizedCondition.Junction + junction.children.assert().hasSize(3) + val spacePredicate = junction.children.single { child -> + child is NormalizedCondition.Predicate && + child.field == LogicalField.System(SystemFieldKind.SPACE_ID) + } as NormalizedCondition.Predicate + spacePredicate.operator.assert().isEqualTo(PredicateOperator.IN) + spacePredicate.value.assert().isEqualTo( + NormalizedValue.ListValue( + listOf(NormalizedValue.Text("space-1"), NormalizedValue.Text("space-2")), + ), + ) + } + .verifyComplete() + + assertRejected(QueryRejectionCode.POLICY_DENIED) { + QueryPolicyEnforcer(TenantIsolationQueryPolicy()).authorize( + input.copy( + executionContext = scopedContext.copy( + authority = QueryAuthority.Subject( + "subject-1", + "tenant-1", + ownerGrant = QueryOwnerGrant.Only("owner-1"), + spaceGrant = QuerySpaceGrant.DenyAll, + ), + ), + ), + ) + } + } + + @Test + fun `tenant service should require an explicit purpose and remain tenant scoped`() { + val serviceContext = context.copy( + authority = QueryAuthority.Service( + "compensation-service", + "tenant-1", + setOf(QueryPurpose("compensation-retry")), + ), + resourceScope = QueryResourceScope("tenant-1"), + ) + val input = QueryPolicyInput(serviceContext, PlanningFixtures.single(), PlanningFixtures.schema) + + assertRejected(QueryRejectionCode.POLICY_DENIED) { + QueryPolicyEnforcer(TenantIsolationQueryPolicy()).authorize(input) + } + QueryPolicyEnforcer(TenantIsolationQueryPolicy()).authorize( + input.copy(executionContext = serviceContext.copy(purpose = QueryPurpose("compensation-retry"))), + ).test() + .expectNextCount(1) + .verifyComplete() + assertRejected(QueryRejectionCode.POLICY_DENIED) { + QueryPolicyEnforcer(TenantIsolationQueryPolicy()).authorize( + input.copy( + executionContext = serviceContext.copy( + purpose = QueryPurpose("compensation-retry"), + resourceScope = QueryResourceScope("tenant-2"), + ), + ), + ) + } + } + + @Test + fun `system and legacy authorities should preserve explicit selectors as mandatory predicates`() { + val scope = QueryResourceScope("tenant-1", "owner-1", "space-1") + val legacyGrant = LegacyQueryGrant( + LegacyQueryCallerId("migration"), + PlanningFixtures.target, + QueryPurpose("interactive-query"), + QueryExecutionMode.LEGACY, + scope, + ) + listOf( + QueryAuthority.System("migration", "query-migration"), + QueryAuthority.Legacy(legacyGrant), + ).forEach { authority -> + QueryPolicyEnforcer(TenantIsolationQueryPolicy()).authorize( + QueryPolicyInput( + context.copy(authority = authority, resourceScope = scope), + PlanningFixtures.single(), + PlanningFixtures.schema, + ), + ).test() + .consumeNextWith { constraints -> + val junction = constraints.mandatoryCondition as NormalizedCondition.Junction + junction.children.assert().hasSize(3) + } + .verifyComplete() + } + } + + @Test + fun `policy failures should normalize typed errors and reject schema-invalid allow-lists`() { + val upstream = QueryRejectedException( + QueryRejection( + QueryRejectionCategory.INVALID_QUERY, + QueryRejectionPath.ROOT, + QueryRejectionCode.INVALID_FIELD, + ), + ) + assertRejected(QueryRejectionCode.POLICY_EVALUATION_FAILED, upstream) { + QueryPolicyEnforcer(QueryPolicy { Mono.error(upstream) }) + .authorize(QueryPolicyInput(context, PlanningFixtures.single(), PlanningFixtures.schema)) + } + + val invalidAllowance = QueryPolicyAllowance.builder() + .fieldConstraint( + QueryFieldConstraint( + filterFields = FieldAccess.AllowList( + setOf(QueryFieldId.Path(listOf("state", "missing"))), + ), + ), + ) + .build() + assertRejected( + QueryRejectionCode.POLICY_CONSTRAINT_INVALID, + expectedPath = "$.policy.fieldConstraint.filterFields", + ) { + QueryPolicyEnforcer(QueryPolicy { Mono.just(QueryPolicyDecision.Allow(invalidAllowance)) }) + .authorize(QueryPolicyInput(context, PlanningFixtures.single(), PlanningFixtures.schema)) + } + } + + private fun assertRejected( + code: QueryRejectionCode, + cause: Throwable? = null, + expectedPath: String = "$.policy", + publisher: () -> Mono<*>, + ) { + publisher().test() + .consumeErrorWith { error -> + (error as QueryRejectedException).rejection.category.assert() + .isEqualTo(QueryRejectionCategory.ACCESS_DENIED) + error.rejection.code.assert().isEqualTo(code) + error.rejection.path.toString().assert().isEqualTo(expectedPath) + if (cause != null) { + error.cause.assert().isSameAs(cause) + } + } + .verify() + } +} diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/snapshot/filter/AbacQueryFilterTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/snapshot/filter/AbacQueryFilterTest.kt index ed04cc5eee3..00923502395 100644 --- a/wow-query/src/test/kotlin/me/ahoo/wow/query/snapshot/filter/AbacQueryFilterTest.kt +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/snapshot/filter/AbacQueryFilterTest.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:Suppress("DEPRECATION") + package me.ahoo.wow.query.snapshot.filter import io.mockk.every @@ -25,6 +27,7 @@ import me.ahoo.wow.api.query.Condition import me.ahoo.wow.api.query.Operator import me.ahoo.wow.filter.FilterChain import me.ahoo.wow.query.filter.DefaultQueryContext +import me.ahoo.wow.query.filter.PreAdmissionQueryFilter import me.ahoo.wow.query.filter.QueryContext import me.ahoo.wow.query.filter.QueryType import me.ahoo.wow.query.snapshot.filter.AbacQueryFilter.Companion.toCondition @@ -36,6 +39,12 @@ import reactor.kotlin.test.test import reactor.util.context.ContextView class AbacQueryFilterTest { + + @Test + fun `legacy ABAC filter should only remain as pre-admission compatibility rewrite`() { + MockAbacQueryFilter.assert().isInstanceOf(PreAdmissionQueryFilter::class.java) + } + @Test fun `toCondition for wildcard should return condition with EXISTS operator`() { val entry: Map.Entry = mapOf("dept" to listOf("*")).entries.first() diff --git a/wow-query/src/test/kotlin/me/ahoo/wow/query/snapshot/filter/DefaultSnapshotQueryHandlerTest.kt b/wow-query/src/test/kotlin/me/ahoo/wow/query/snapshot/filter/DefaultSnapshotQueryHandlerTest.kt index 344b4919b38..f7e90abefbb 100644 --- a/wow-query/src/test/kotlin/me/ahoo/wow/query/snapshot/filter/DefaultSnapshotQueryHandlerTest.kt +++ b/wow-query/src/test/kotlin/me/ahoo/wow/query/snapshot/filter/DefaultSnapshotQueryHandlerTest.kt @@ -13,17 +13,32 @@ package me.ahoo.wow.query.snapshot.filter +import io.mockk.every +import io.mockk.mockk +import io.mockk.verify import me.ahoo.test.asserts.assert +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.filter.ErrorAccessor +import me.ahoo.wow.filter.ErrorHandler +import me.ahoo.wow.filter.FilterChain import me.ahoo.wow.filter.FilterChainBuilder import me.ahoo.wow.filter.LogErrorHandler import me.ahoo.wow.query.dsl.condition import me.ahoo.wow.query.dsl.listQuery import me.ahoo.wow.query.dsl.singleQuery +import me.ahoo.wow.query.filter.AbstractQueryHandler import me.ahoo.wow.query.filter.QueryContext +import me.ahoo.wow.query.filter.QueryType import me.ahoo.wow.query.snapshot.NoOpSnapshotQueryServiceFactory +import me.ahoo.wow.query.snapshot.SnapshotQueryService +import me.ahoo.wow.query.snapshot.SnapshotQueryServiceFactory import me.ahoo.wow.tck.mock.MOCK_AGGREGATE_METADATA import org.junit.jupiter.api.Test +import reactor.core.publisher.Mono import reactor.kotlin.test.test +import reactor.test.publisher.PublisherProbe +import java.util.concurrent.ConcurrentHashMap class DefaultSnapshotQueryHandlerTest { private val tailSnapshotQueryFilter = TailSnapshotQueryFilter(NoOpSnapshotQueryServiceFactory) @@ -102,4 +117,217 @@ class DefaultSnapshotQueryHandlerTest { } .verifyComplete() } + + @Test + fun `query methods should execute through the overridable handle hook`() { + var handled = 0 + val handler = object : AbstractQueryHandler(snapshotQueryFilterChain, LogErrorHandler()) { + override fun handle(context: QueryContext<*, *>): Mono { + handled++ + return super.handle(context) + } + } + + handler.count(MOCK_AGGREGATE_METADATA, Condition.ALL) + .test() + .expectNext(0) + .verifyComplete() + handler.list(MOCK_AGGREGATE_METADATA, listQuery { }) + .test() + .verifyComplete() + + handled.assert().isEqualTo(2) + } + + @Test + fun `should handle asynchronous query service error`() { + val failure = IllegalStateException("query failed") + val condition = condition { id("1") } + val queryService = mockk> { + every { count(condition) } returns Mono.error(failure) + } + val queryServiceFactory = mockk { + every { create(any()) } returns queryService + } + val errorHandler = mockk>> { + every { handle(any(), failure) } returns Mono.error(failure) + } + val chain = FilterChainBuilder>() + .addFilters(listOf(TailSnapshotQueryFilter(queryServiceFactory))) + .filterCondition(SnapshotQueryHandler::class) + .build() + val handler = DefaultSnapshotQueryHandler(chain, errorHandler) + + handler.count(MOCK_AGGREGATE_METADATA, condition) + .test() + .expectErrorSatisfies { + it.assert().isSameAs(failure) + } + .verify() + + verify(exactly = 1) { errorHandler.handle(any(), failure) } + } + + @Test + fun `should not recover asynchronous query service error`() { + val failure = IllegalStateException("query failed") + val condition = condition { id("1") } + val queryService = mockk> { + every { count(condition) } returns Mono.error(failure) + } + val queryServiceFactory = mockk { + every { create(any()) } returns queryService + } + val resumeErrorHandler = mockk>> { + every { handle(any(), failure) } returns Mono.empty() + } + val chain = FilterChainBuilder>() + .addFilters(listOf(TailSnapshotQueryFilter(queryServiceFactory))) + .filterCondition(SnapshotQueryHandler::class) + .build() + val handler = DefaultSnapshotQueryHandler(chain, resumeErrorHandler) + + handler.count(MOCK_AGGREGATE_METADATA, condition) + .test() + .expectErrorSatisfies { + it.assert().isSameAs(failure) + } + .verify() + + verify(exactly = 1) { resumeErrorHandler.handle(any(), failure) } + } + + @Test + fun `should delegate direct handle context`() { + val context = ErrorQueryContext(MOCK_AGGREGATE_METADATA) + val handledContexts = mutableListOf>() + val chain = FilterChain> { + handledContexts.add(it) + Mono.empty() + } + val handler = DefaultSnapshotQueryHandler(chain) + + handler.handle(context) + .test() + .verifyComplete() + + handledContexts.assert().containsExactly(context) + } + + @Test + fun `should fail closed and store error for direct handle context`() { + val failure = IllegalStateException("filter failed") + val context = ErrorQueryContext(MOCK_AGGREGATE_METADATA) + val chain = FilterChain> { + throw failure + } + val resumeErrorHandler = mockk>> { + every { handle(context, failure) } returns Mono.empty() + } + val handler = DefaultSnapshotQueryHandler(chain, resumeErrorHandler) + + handler.handle(context) + .test() + .expectErrorSatisfies { + it.assert().isSameAs(failure) + } + .verify() + + context.getError().assert().isSameAs(failure) + verify(exactly = 1) { resumeErrorHandler.handle(context, failure) } + } + + @Test + fun `should not execute result after filter chain error`() { + val failure = IllegalStateException("masking failed") + val condition = condition { id("1") } + val backendPublisher = PublisherProbe.of(Mono.just(1L)) + val queryService = mockk> { + every { count(condition) } returns backendPublisher.mono() + } + val queryServiceFactory = mockk { + every { create(any()) } returns queryService + } + val failingPostFilter = object : SnapshotQueryFilter { + override fun filter( + context: QueryContext<*, *>, + next: FilterChain>, + ): Mono { + return next.filter(context).then(Mono.error(failure)) + } + } + val resumeErrorHandler = mockk>> { + every { handle(any(), failure) } returns Mono.empty() + } + val chain = FilterChainBuilder>() + .addFilters(listOf(failingPostFilter, TailSnapshotQueryFilter(queryServiceFactory))) + .filterCondition(SnapshotQueryHandler::class) + .build() + val handler = DefaultSnapshotQueryHandler(chain, resumeErrorHandler) + + handler.count(MOCK_AGGREGATE_METADATA, condition) + .test() + .expectErrorSatisfies { + it.assert().isSameAs(failure) + } + .verify() + + backendPublisher.assertWasNotSubscribed() + verify(exactly = 1) { resumeErrorHandler.handle(any(), failure) } + } + + @Test + fun `should isolate mutable query context for each subscription`() { + val original = Condition.eq("aggregateId", "1") + val mandatory = Condition.eq("tenantId", "tenant") + val capturedConditions = mutableListOf() + val queryService = mockk> { + every { count(capture(capturedConditions)) } returns Mono.just(0) + } + val queryServiceFactory = mockk { + every { create(any()) } returns queryService + } + val mandatoryFilter = object : SnapshotQueryFilter { + override fun filter( + context: QueryContext<*, *>, + next: FilterChain>, + ): Mono { + context.asRewritableQuery().rewriteQuery { + it.appendCondition(mandatory) + } + return next.filter(context) + } + } + val chain = FilterChainBuilder>() + .addFilters(listOf(mandatoryFilter, TailSnapshotQueryFilter(queryServiceFactory))) + .filterCondition(SnapshotQueryHandler::class) + .build() + val handler = DefaultSnapshotQueryHandler(chain) + val result = handler.count(MOCK_AGGREGATE_METADATA, original) + + result.test().expectNext(0).verifyComplete() + result.test().expectNext(0).verifyComplete() + + capturedConditions.assert().hasSize(2) + capturedConditions[0].assert().isEqualTo(Condition.and(original, mandatory)) + capturedConditions[1].assert().isEqualTo(capturedConditions[0]) + } +} + +private class ErrorQueryContext( + override val namedAggregate: NamedAggregate, +) : QueryContext>, ErrorAccessor { + override val queryType: QueryType = QueryType.COUNT + override val attributes: MutableMap = ConcurrentHashMap() + private var error: Throwable? = null + + override fun setError(throwable: Throwable) { + error = throwable + } + + override fun getError(): Throwable? = error + + override fun clearError() { + error = null + } } diff --git a/wow-schema/src/main/kotlin/me/ahoo/wow/schema/WowDefinitionProviderRegistry.kt b/wow-schema/src/main/kotlin/me/ahoo/wow/schema/WowDefinitionProviderRegistry.kt index cb4552bc053..09da1c59360 100644 --- a/wow-schema/src/main/kotlin/me/ahoo/wow/schema/WowDefinitionProviderRegistry.kt +++ b/wow-schema/src/main/kotlin/me/ahoo/wow/schema/WowDefinitionProviderRegistry.kt @@ -28,6 +28,7 @@ import me.ahoo.wow.schema.typed.StateEventDefinitionProvider import me.ahoo.wow.schema.typed.query.AggregatedListQueryDefinitionProvider import me.ahoo.wow.schema.typed.query.AggregatedPagedQueryDefinitionProvider import me.ahoo.wow.schema.typed.query.AggregatedSingleQueryDefinitionProvider +import me.ahoo.wow.schema.typed.query.AnalyticsCursorDefinitionProvider import me.ahoo.wow.schema.typed.query.ConditionOptionsDefinitionProvider import me.ahoo.wow.schema.web.ServerSentEventCustomDefinitionProvider @@ -47,6 +48,7 @@ internal object WowDefinitionProviderRegistry { StateEventDefinitionProvider, ServerSentEventCustomDefinitionProvider, ConditionOptionsDefinitionProvider, + AnalyticsCursorDefinitionProvider, MapDefinitionProvider, EnumTextDefinitionProvider ) diff --git a/wow-schema/src/main/kotlin/me/ahoo/wow/schema/typed/query/AnalyticsCursorDefinitionProvider.kt b/wow-schema/src/main/kotlin/me/ahoo/wow/schema/typed/query/AnalyticsCursorDefinitionProvider.kt new file mode 100644 index 00000000000..314ea85f049 --- /dev/null +++ b/wow-schema/src/main/kotlin/me/ahoo/wow/schema/typed/query/AnalyticsCursorDefinitionProvider.kt @@ -0,0 +1,37 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.schema.typed.query + +import com.fasterxml.classmate.ResolvedType +import com.github.victools.jsonschema.generator.CustomDefinition +import com.github.victools.jsonschema.generator.CustomDefinitionProviderV2 +import com.github.victools.jsonschema.generator.SchemaGenerationContext +import me.ahoo.wow.api.query.analytics.AnalyticsCursor + +/** Keeps the opaque cursor's JSON-schema shape aligned with its delegated scalar JSON representation. */ +object AnalyticsCursorDefinitionProvider : CustomDefinitionProviderV2 { + override fun provideCustomSchemaDefinition( + javaType: ResolvedType, + context: SchemaGenerationContext, + ): CustomDefinition? { + if (javaType.erasedType != AnalyticsCursor::class.java) { + return null + } + val schema = context.generatorConfig.createObjectNode() + schema.put("type", "string") + schema.put("maxLength", AnalyticsCursor.MAX_LENGTH) + schema.put("pattern", AnalyticsCursor.PATTERN) + return CustomDefinition(schema) + } +} diff --git a/wow-schema/src/test/kotlin/me/ahoo/wow/schema/WowDefinitionProviderRegistryTest.kt b/wow-schema/src/test/kotlin/me/ahoo/wow/schema/WowDefinitionProviderRegistryTest.kt index 21d343a9910..2ee060d9d48 100644 --- a/wow-schema/src/test/kotlin/me/ahoo/wow/schema/WowDefinitionProviderRegistryTest.kt +++ b/wow-schema/src/test/kotlin/me/ahoo/wow/schema/WowDefinitionProviderRegistryTest.kt @@ -37,6 +37,7 @@ class WowDefinitionProviderRegistryTest { "StateEventDefinitionProvider", "ServerSentEventCustomDefinitionProvider", "ConditionOptionsDefinitionProvider", + "AnalyticsCursorDefinitionProvider", "MapDefinitionProvider", "EnumTextDefinitionProvider" ) diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/elasticsearch/ElasticsearchEventSourcingAutoConfiguration.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/elasticsearch/ElasticsearchEventSourcingAutoConfiguration.kt index aff564e6cf3..6e5449baa79 100644 --- a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/elasticsearch/ElasticsearchEventSourcingAutoConfiguration.kt +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/elasticsearch/ElasticsearchEventSourcingAutoConfiguration.kt @@ -22,6 +22,7 @@ import me.ahoo.wow.elasticsearch.WowJsonpMapper import me.ahoo.wow.elasticsearch.eventsourcing.ElasticsearchEventStore import me.ahoo.wow.elasticsearch.eventsourcing.ElasticsearchSnapshotStore import me.ahoo.wow.elasticsearch.query.event.ElasticsearchEventStreamQueryServiceFactory +import me.ahoo.wow.elasticsearch.query.planned.ElasticsearchSnapshotQueryBinding import me.ahoo.wow.elasticsearch.query.snapshot.ElasticsearchSnapshotQueryServiceFactory import me.ahoo.wow.eventsourcing.EventStore import me.ahoo.wow.eventsourcing.snapshot.SnapshotStore @@ -35,9 +36,11 @@ import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.EventStreamQuerySer import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.SnapshotQueryServiceFactoryBinding import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.SnapshotStoreBinding import me.ahoo.wow.spring.boot.starter.eventsourcing.snapshot.ConditionalOnSnapshotEnabled +import me.ahoo.wow.spring.boot.starter.query.StorageQueryBackendSource import org.springframework.beans.factory.ObjectProvider import org.springframework.beans.factory.annotation.Qualifier import org.springframework.boot.autoconfigure.AutoConfiguration +import org.springframework.boot.autoconfigure.condition.ConditionalOnBean import org.springframework.boot.autoconfigure.condition.ConditionalOnClass import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty @@ -185,4 +188,13 @@ class ElasticsearchEventSourcingAutoConfiguration( elasticsearchSnapshotQueryServiceFactory, ) } + + @Bean + @ConditionalOnBean(ElasticsearchSnapshotQueryBinding::class) + @ConditionalOnSnapshotEnabled + @ConditionalOnSnapshotStoreStorage(StorageType.ELASTICSEARCH) + internal fun elasticsearchPlannedQueryBackendSource( + elasticsearchClient: ReactiveElasticsearchClient, + bindings: List, + ): StorageQueryBackendSource = ElasticsearchPlannedQueryBackendSource(elasticsearchClient, bindings) } diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/elasticsearch/ElasticsearchPlannedQueryBackendSource.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/elasticsearch/ElasticsearchPlannedQueryBackendSource.kt new file mode 100644 index 00000000000..b1a93b5e1c9 --- /dev/null +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/elasticsearch/ElasticsearchPlannedQueryBackendSource.kt @@ -0,0 +1,67 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.spring.boot.starter.elasticsearch + +import me.ahoo.wow.elasticsearch.query.planned.ElasticsearchQueryBackendNotReadyException +import me.ahoo.wow.elasticsearch.query.planned.ElasticsearchSnapshotQueryBinding +import me.ahoo.wow.elasticsearch.query.planned.prepareContribution +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.spring.boot.starter.eventsourcing.StorageType +import me.ahoo.wow.spring.boot.starter.query.StorageQueryBackendPreparation +import me.ahoo.wow.spring.boot.starter.query.StorageQueryBackendSource +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient +import reactor.core.publisher.Mono +import java.util.Collections +import java.util.LinkedHashMap + +internal class ElasticsearchPlannedQueryBackendSource( + private val client: ReactiveElasticsearchClient, + bindings: List, +) : StorageQueryBackendSource { + override val storage: StorageType = StorageType.ELASTICSEARCH + private val bindings: Map + override val targets: Set + + init { + val copy = LinkedHashMap(bindings.size) + bindings.forEach { binding -> + require(copy.put(binding.schema.target, binding) == null) { + "Elasticsearch planned Query bindings must be unique per target[${binding.schema.target}]." + } + } + this.bindings = Collections.unmodifiableMap(copy) + targets = Collections.unmodifiableSet(LinkedHashSet(copy.keys)) + } + + override fun prepare(target: QueryTarget): Mono { + val binding = requireNotNull(bindings[target]) { + "Elasticsearch planned Query binding is not registered for target[$target]." + } + return binding.prepareContribution(client) + .map(StorageQueryBackendPreparation::Ready) + .onErrorResume(ElasticsearchQueryBackendNotReadyException::class.java) { + Mono.just( + StorageQueryBackendPreparation.NotReady( + binding.schema, + binding.backendId, + ), + ) + } + } +} diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRouteResolver.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRouteResolver.kt index fcd43fd5f82..83faf9c5674 100644 --- a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRouteResolver.kt +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRouteResolver.kt @@ -34,38 +34,34 @@ class StorageRouteResolver( private val defaultSnapshotStorage: StorageType = StorageType.MONGO ) { private val eventStoreBindingsByName: Map = - eventStoreBindings.associateBy { it.name } + eventStoreBindings.associateUniqueBy("EventStoreBinding", "name", EventStoreBinding::name) private val eventStoreBindingsByStorage: Map = - eventStoreBindings.mapNotNull { binding -> - binding.storage?.let { storage -> - storage to binding - } - }.toMap() + eventStoreBindings.filter { it.storage != null } + .associateUniqueBy("EventStoreBinding", "storage") { binding -> binding.storage!! } private val snapshotStoreBindingsByName: Map = - snapshotStoreBindings.associateBy { it.name } + snapshotStoreBindings.associateUniqueBy("SnapshotStoreBinding", "name", SnapshotStoreBinding::name) private val snapshotStoreBindingsByStorage: Map = - snapshotStoreBindings.mapNotNull { binding -> - binding.storage?.let { storage -> - storage to binding - } - }.toMap() + snapshotStoreBindings.filter { it.storage != null } + .associateUniqueBy("SnapshotStoreBinding", "storage") { binding -> binding.storage!! } private val eventStreamQueryServiceFactoryBindingsByName: Map = - eventStreamQueryServiceFactoryBindings.associateBy { it.name } + eventStreamQueryServiceFactoryBindings.associateUniqueBy( + "EventStreamQueryServiceFactoryBinding", + "name", + EventStreamQueryServiceFactoryBinding::name, + ) private val eventStreamQueryServiceFactoryBindingsByStorage: Map = - eventStreamQueryServiceFactoryBindings.mapNotNull { binding -> - binding.storage?.let { storage -> - storage to binding - } - }.toMap() + eventStreamQueryServiceFactoryBindings.filter { it.storage != null } + .associateUniqueBy("EventStreamQueryServiceFactoryBinding", "storage") { binding -> binding.storage!! } private val snapshotQueryServiceFactoryBindingsByName: Map = - snapshotQueryServiceFactoryBindings.associateBy { it.name } + snapshotQueryServiceFactoryBindings.associateUniqueBy( + "SnapshotQueryServiceFactoryBinding", + "name", + SnapshotQueryServiceFactoryBinding::name, + ) private val snapshotQueryServiceFactoryBindingsByStorage: Map = - snapshotQueryServiceFactoryBindings.mapNotNull { binding -> - binding.storage?.let { storage -> - storage to binding - } - }.toMap() + snapshotQueryServiceFactoryBindings.filter { it.storage != null } + .associateUniqueBy("SnapshotQueryServiceFactoryBinding", "storage") { binding -> binding.storage!! } fun resolveEventRoutes(properties: StorageRoutingProperties): ResolvedEventRoutes { val routes: Map = properties.aggregates.mapNotNull { (routeKey, aggregateRoute) -> @@ -251,6 +247,21 @@ class StorageRouteResolver( } } +private fun Iterable.associateUniqueBy( + bindingType: String, + keyName: String, + keySelector: (B) -> K, +): Map { + val result = LinkedHashMap() + for (binding in this) { + val key = keySelector(binding) + require(result.putIfAbsent(key, binding) == null) { + "Duplicate $bindingType $keyName[$key]." + } + } + return result +} + data class ResolvedEventRoutes( val defaultEventStore: EventStore, val eventRoutes: Map diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRoutingAutoConfiguration.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRoutingAutoConfiguration.kt index 38c646eddd7..a443260d45d 100644 --- a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRoutingAutoConfiguration.kt +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRoutingAutoConfiguration.kt @@ -19,10 +19,6 @@ import me.ahoo.wow.eventsourcing.RoutingEventStore import me.ahoo.wow.eventsourcing.snapshot.AggregateSnapshotStoreRegistry import me.ahoo.wow.eventsourcing.snapshot.RoutingSnapshotStore import me.ahoo.wow.eventsourcing.snapshot.SnapshotStore -import me.ahoo.wow.query.event.EventStreamQueryServiceFactory -import me.ahoo.wow.query.event.RoutingEventStreamQueryServiceFactory -import me.ahoo.wow.query.snapshot.RoutingSnapshotQueryServiceFactory -import me.ahoo.wow.query.snapshot.SnapshotQueryServiceFactory import me.ahoo.wow.spring.boot.starter.ConditionalOnWowEnabled import me.ahoo.wow.spring.boot.starter.WowAutoConfiguration import me.ahoo.wow.spring.boot.starter.elasticsearch.ElasticsearchEventSourcingAutoConfiguration @@ -132,66 +128,6 @@ class StorageRoutingAutoConfiguration { ), ) } - - @Bean - @Primary - @Conditional(OnEventStorageRouteCondition::class) - fun routingEventStreamQueryServiceFactory( - @Qualifier(WowAutoConfiguration.WOW_CURRENT_BOUNDED_CONTEXT) - namedBoundedContext: NamedBoundedContext, - eventStoreProperties: EventStoreProperties, - snapshotProperties: SnapshotProperties, - storageRoutingProperties: StorageRoutingProperties, - eventStoreBindings: List, - snapshotStoreBindings: List, - eventStreamQueryServiceFactoryBindings: List, - snapshotQueryServiceFactoryBindings: List, - ): EventStreamQueryServiceFactory { - val resolvedRoutes = StorageRouteResolver( - contextName = namedBoundedContext.contextName, - snapshotEnabled = snapshotProperties.enabled, - eventStoreBindings = eventStoreBindings, - snapshotStoreBindings = snapshotStoreBindings, - eventStreamQueryServiceFactoryBindings = eventStreamQueryServiceFactoryBindings, - snapshotQueryServiceFactoryBindings = snapshotQueryServiceFactoryBindings, - defaultEventStorage = eventStoreProperties.storage, - defaultSnapshotStorage = snapshotProperties.storage, - ).resolveEventStreamQueryServiceFactoryRoutes(storageRoutingProperties) - return RoutingEventStreamQueryServiceFactory( - defaultEventStreamQueryServiceFactory = resolvedRoutes.defaultEventStreamQueryServiceFactory, - routes = resolvedRoutes.eventStreamQueryServiceFactoryRoutes, - ) - } - - @Bean - @Primary - @Conditional(OnSnapshotStorageRouteCondition::class) - fun routingSnapshotQueryServiceFactory( - @Qualifier(WowAutoConfiguration.WOW_CURRENT_BOUNDED_CONTEXT) - namedBoundedContext: NamedBoundedContext, - eventStoreProperties: EventStoreProperties, - snapshotProperties: SnapshotProperties, - storageRoutingProperties: StorageRoutingProperties, - eventStoreBindings: List, - snapshotStoreBindings: List, - eventStreamQueryServiceFactoryBindings: List, - snapshotQueryServiceFactoryBindings: List, - ): SnapshotQueryServiceFactory { - val resolvedRoutes = StorageRouteResolver( - contextName = namedBoundedContext.contextName, - snapshotEnabled = snapshotProperties.enabled, - eventStoreBindings = eventStoreBindings, - snapshotStoreBindings = snapshotStoreBindings, - eventStreamQueryServiceFactoryBindings = eventStreamQueryServiceFactoryBindings, - snapshotQueryServiceFactoryBindings = snapshotQueryServiceFactoryBindings, - defaultEventStorage = eventStoreProperties.storage, - defaultSnapshotStorage = snapshotProperties.storage, - ).resolveSnapshotQueryServiceFactoryRoutes(storageRoutingProperties) - return RoutingSnapshotQueryServiceFactory( - defaultSnapshotQueryServiceFactory = resolvedRoutes.defaultSnapshotQueryServiceFactory, - routes = resolvedRoutes.snapshotQueryServiceFactoryRoutes, - ) - } } private class OnEventStorageRouteCondition : SpringBootCondition() { diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/mongo/MongoEventSourcingAutoConfiguration.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/mongo/MongoEventSourcingAutoConfiguration.kt index 10380eaa5da..095cd967641 100644 --- a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/mongo/MongoEventSourcingAutoConfiguration.kt +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/mongo/MongoEventSourcingAutoConfiguration.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class) + package me.ahoo.wow.spring.boot.starter.mongo import com.mongodb.reactivestreams.client.MongoClient @@ -27,6 +29,8 @@ import me.ahoo.wow.mongo.MongoSnapshotStore import me.ahoo.wow.mongo.SnapshotSchemaInitializer import me.ahoo.wow.mongo.prepare.MongoPrepareKeyFactory import me.ahoo.wow.mongo.query.event.MongoEventStreamQueryServiceFactory +import me.ahoo.wow.mongo.query.planned.MongoEventStreamQueryBinding +import me.ahoo.wow.mongo.query.planned.MongoSnapshotQueryBinding import me.ahoo.wow.mongo.query.snapshot.MongoSnapshotQueryServiceFactory import me.ahoo.wow.spring.boot.starter.ConditionalOnWowEnabled import me.ahoo.wow.spring.boot.starter.WowAutoConfiguration @@ -41,6 +45,7 @@ import me.ahoo.wow.spring.boot.starter.eventsourcing.snapshot.ConditionalOnSnaps import me.ahoo.wow.spring.boot.starter.prepare.ConditionalOnPrepareEnabled import me.ahoo.wow.spring.boot.starter.prepare.PrepareProperties import me.ahoo.wow.spring.boot.starter.prepare.PrepareStorage +import me.ahoo.wow.spring.boot.starter.query.StorageQueryBackendSource import org.springframework.beans.factory.ObjectProvider import org.springframework.beans.factory.annotation.Qualifier import org.springframework.boot.autoconfigure.AutoConfiguration @@ -189,6 +194,31 @@ class MongoEventSourcingAutoConfiguration( return SnapshotQueryServiceFactoryBinding.storage(StorageType.MONGO, mongoSnapshotQueryServiceFactory) } + @Bean + @ConditionalOnBean(MongoSnapshotQueryBinding::class) + @ConditionalOnSnapshotEnabled + @ConditionalOnSnapshotStoreStorage(StorageType.MONGO) + internal fun mongoPlannedQueryBackendSource( + mongoClient: MongoClient, + dataMongoProperties: org.springframework.boot.mongodb.autoconfigure.MongoProperties?, + bindings: List, + ): StorageQueryBackendSource = MongoPlannedQueryBackendSource.snapshot( + getMongoSnapshotDatabase(dataMongoProperties, mongoClient), + bindings, + ) + + @Bean + @ConditionalOnBean(MongoEventStreamQueryBinding::class) + @ConditionalOnEventStoreStorage(StorageType.MONGO) + internal fun mongoEventStreamPlannedQueryBackendSource( + mongoClient: MongoClient, + dataMongoProperties: org.springframework.boot.mongodb.autoconfigure.MongoProperties?, + bindings: List, + ): StorageQueryBackendSource = MongoPlannedQueryBackendSource.eventStream( + getEventStreamDatabase(dataMongoProperties, mongoClient), + bindings, + ) + private fun getMongoSnapshotDatabase( dataMongoProperties: org.springframework.boot.mongodb.autoconfigure.MongoProperties?, mongoClient: MongoClient diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/mongo/MongoPlannedQueryBackendSource.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/mongo/MongoPlannedQueryBackendSource.kt new file mode 100644 index 00000000000..893d9691955 --- /dev/null +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/mongo/MongoPlannedQueryBackendSource.kt @@ -0,0 +1,126 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.spring.boot.starter.mongo + +import com.mongodb.reactivestreams.client.MongoDatabase +import me.ahoo.wow.mongo.query.planned.MongoEventStreamQueryBinding +import me.ahoo.wow.mongo.query.planned.MongoQueryBackendNotReadyException +import me.ahoo.wow.mongo.query.planned.MongoSnapshotQueryBinding +import me.ahoo.wow.mongo.query.planned.prepareContribution +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.RecordQueryBackendContribution +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.spring.boot.starter.eventsourcing.StorageType +import me.ahoo.wow.spring.boot.starter.query.StorageQueryBackendPreparation +import me.ahoo.wow.spring.boot.starter.query.StorageQueryBackendSource +import reactor.core.publisher.Mono +import java.util.Collections +import java.util.LinkedHashMap + +internal class MongoPlannedQueryBackendSource private constructor( + private val database: MongoDatabase, + bindings: Map, +) : StorageQueryBackendSource { + override val storage: StorageType = StorageType.MONGO + private val bindings: Map = Collections.unmodifiableMap(LinkedHashMap(bindings)) + override val targets: Set + + init { + require(this.bindings.keys == this.bindings.values.mapTo(linkedSetOf()) { binding -> binding.schema.target }) { + "Mongo planned Query binding keys must match their schema targets." + } + targets = Collections.unmodifiableSet(LinkedHashSet(this.bindings.keys)) + } + + override fun prepare(target: QueryTarget): Mono { + val binding = requireNotNull(bindings[target]) { + "Mongo planned Query binding is not registered for target[$target]." + } + require(binding.namespace.databaseName == database.name) { + "Mongo planned Query binding namespace[${binding.namespace}] must use database[${database.name}]." + } + return binding.prepare( + database.getCollection(binding.namespace.collectionName), + ).map { contribution -> + StorageQueryBackendPreparation.Ready(contribution) + }.onErrorResume(MongoQueryBackendNotReadyException::class.java) { + Mono.just( + StorageQueryBackendPreparation.NotReady( + binding.schema, + binding.backendId, + ), + ) + } + } + + companion object { + fun snapshot( + database: MongoDatabase, + bindings: List, + ): MongoPlannedQueryBackendSource = MongoPlannedQueryBackendSource( + database, + uniqueBindings( + bindings.map { binding -> + MongoPlannedBinding( + binding.schema, + binding.namespace, + binding.backendId, + binding::prepareContribution, + ) + }, + ), + ) + + fun eventStream( + database: MongoDatabase, + bindings: List, + ): MongoPlannedQueryBackendSource = MongoPlannedQueryBackendSource( + database, + uniqueBindings( + bindings.map { binding -> + MongoPlannedBinding( + binding.schema, + binding.namespace, + binding.backendId, + binding::prepareContribution, + ) + }, + ), + ) + + private fun uniqueBindings(bindings: List): Map { + val result = LinkedHashMap(bindings.size) + bindings.forEach { binding -> + require(result.put(binding.schema.target, binding) == null) { + "Mongo planned Query bindings must be unique per target[${binding.schema.target}]." + } + } + return result + } + } +} + +private class MongoPlannedBinding( + val schema: QueryDocumentSchema, + val namespace: com.mongodb.MongoNamespace, + val backendId: BackendId, + val prepare: (com.mongodb.reactivestreams.client.MongoCollection) -> + Mono, +) diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryAutoConfiguration.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryAutoConfiguration.kt index a740b45995b..0c4cbda6f5e 100644 --- a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryAutoConfiguration.kt +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryAutoConfiguration.kt @@ -24,7 +24,10 @@ import me.ahoo.wow.query.event.filter.EventStreamQueryFilter import me.ahoo.wow.query.event.filter.EventStreamQueryHandler import me.ahoo.wow.query.event.filter.MaskingEventStreamQueryFilter import me.ahoo.wow.query.event.filter.TailEventStreamQueryFilter +import me.ahoo.wow.query.filter.PreAdmissionQueryFilter import me.ahoo.wow.query.filter.QueryContext +import me.ahoo.wow.query.filter.QueryFilter +import me.ahoo.wow.query.filter.RESULT_KEY import me.ahoo.wow.query.mask.EventStreamDynamicDocumentMasker import me.ahoo.wow.query.mask.EventStreamMaskerRegistry import me.ahoo.wow.query.mask.StateDataMaskerRegistry @@ -37,6 +40,7 @@ import me.ahoo.wow.query.snapshot.filter.SnapshotQueryFilter import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler import me.ahoo.wow.query.snapshot.filter.TailSnapshotQueryFilter import me.ahoo.wow.spring.boot.starter.ConditionalOnWowEnabled +import me.ahoo.wow.spring.query.AnalyticsQueryServiceRegistrar import me.ahoo.wow.spring.query.EventStreamQueryServiceRegistrar import me.ahoo.wow.spring.query.SnapshotQueryServiceRegistrar import me.ahoo.wow.spring.query.getOrNoOp @@ -53,7 +57,11 @@ import org.springframework.context.annotation.Import * @author ahoo wang */ @AutoConfiguration -@Import(SnapshotQueryServiceRegistrar::class, EventStreamQueryServiceRegistrar::class) +@Import( + SnapshotQueryServiceRegistrar::class, + EventStreamQueryServiceRegistrar::class, + AnalyticsQueryServiceRegistrar::class +) @ConditionalOnWowEnabled class QueryAutoConfiguration { @@ -85,7 +93,9 @@ class QueryAutoConfiguration { } @Bean - fun maskingEventStreamQueryFilter(eventStreamMaskerRegistry: EventStreamMaskerRegistry): EventStreamQueryFilter { + fun maskingEventStreamQueryFilter( + eventStreamMaskerRegistry: EventStreamMaskerRegistry + ): EventStreamQueryFilter { return MaskingEventStreamQueryFilter(eventStreamMaskerRegistry) } @@ -107,20 +117,26 @@ class QueryAutoConfiguration { fun snapshotQueryFilterChain( filters: List>> ): FilterChain> { - return FilterChainBuilder>() - .addFilters(filters) - .filterCondition(SnapshotQueryHandler::class) - .build() + validateQueryFilterPhases(filters) + return phasedQueryFilterChain( + filters, + SnapshotQueryHandler::class, + filters.filterIsInstance>().single(), + filters.filterIsInstance().single(), + ) } @Bean fun eventStreamQueryFilterChain( filters: List>> ): FilterChain> { - return FilterChainBuilder>() - .addFilters(filters) - .filterCondition(EventStreamQueryHandler::class) - .build() + validateQueryFilterPhases(filters) + return phasedQueryFilterChain( + filters, + EventStreamQueryHandler::class, + filters.filterIsInstance().single(), + filters.filterIsInstance().single(), + ) } @Bean("snapshotQueryErrorHandler") @@ -163,3 +179,40 @@ class QueryAutoConfiguration { return NoOpEventStreamQueryServiceFactory } } + +private fun validateQueryFilterPhases(filters: List>>) { + val unsupported = filters.filterIsInstance>().filterNot { filter -> + filter is PreAdmissionQueryFilter || + filter is TailSnapshotQueryFilter<*> || + filter is TailEventStreamQueryFilter || + filter is MaskingSnapshotQueryFilter || + filter is MaskingEventStreamQueryFilter + } + require(unsupported.isEmpty()) { + "QueryFilter must declare the pre-admission phase; post-policy result replacement is not supported: " + + unsupported.joinToString { it.javaClass.name } + } +} + +private fun phasedQueryFilterChain( + filters: List>>, + filterType: kotlin.reflect.KClass<*>, + terminal: Filter>, + masker: Filter>, +): FilterChain> { + val requestChain = FilterChainBuilder>() + .addFilters(filters.filter { it is PreAdmissionQueryFilter }) + .filterCondition(filterType) + .build() + val emptyChain = FilterChain> { reactor.core.publisher.Mono.empty() } + return FilterChain { context -> + requestChain.filter(context) + .then( + reactor.core.publisher.Mono.fromRunnable { + context.attributes.remove(RESULT_KEY) + }, + ) + .then(reactor.core.publisher.Mono.defer { terminal.filter(context, emptyChain) }) + .then(reactor.core.publisher.Mono.defer { masker.filter(context, emptyChain) }) + } +} diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperAutoConfiguration.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperAutoConfiguration.kt new file mode 100644 index 00000000000..2855f1e362f --- /dev/null +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperAutoConfiguration.kt @@ -0,0 +1,52 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.spring.boot.starter.query + +import me.ahoo.wow.query.cursor.QueryCursorLeaseConfiguration +import me.ahoo.wow.query.gateway.QueryGatewayRuntime +import me.ahoo.wow.spring.boot.starter.ConditionalOnWowEnabled +import org.springframework.boot.autoconfigure.AutoConfiguration +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty +import org.springframework.boot.context.properties.EnableConfigurationProperties +import org.springframework.context.SmartLifecycle +import org.springframework.context.annotation.Bean + +/** Installs one explicitly enabled lifecycle owner for periodic, bounded cursor lease cleanup. */ +@AutoConfiguration(after = [QueryGatewayAutoConfiguration::class]) +@ConditionalOnWowEnabled +@ConditionalOnProperty( + prefix = QueryCursorReaperProperties.PREFIX, + name = ["enabled"], + havingValue = "true", +) +@EnableConfigurationProperties(QueryCursorReaperProperties::class) +class QueryCursorReaperAutoConfiguration { + @Bean(QUERY_CURSOR_REAPER_LIFECYCLE_BEAN_NAME) + @ConditionalOnMissingBean(name = [QUERY_CURSOR_REAPER_LIFECYCLE_BEAN_NAME]) + fun queryCursorReaperLifecycle( + runtime: QueryGatewayRuntime, + @Suppress("UNUSED_PARAMETER") cursorLeaseConfiguration: QueryCursorLeaseConfiguration, + properties: QueryCursorReaperProperties, + ): SmartLifecycle = QueryCursorReaperLifecycle(runtime::reapExpiredQueryCursors, properties) + + companion object { + const val QUERY_CURSOR_REAPER_LIFECYCLE_BEAN_NAME = "queryCursorReaperLifecycle" + } +} diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperLifecycle.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperLifecycle.kt new file mode 100644 index 00000000000..a09691c0459 --- /dev/null +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperLifecycle.kt @@ -0,0 +1,109 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.spring.boot.starter.query + +import io.github.oshai.kotlinlogging.KotlinLogging +import me.ahoo.wow.spring.WOW_RUNTIME_PHASE +import org.springframework.context.SmartLifecycle +import reactor.core.Disposable +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.core.scheduler.Scheduler +import reactor.core.scheduler.Schedulers + +/** Owns the single opt-in Cursor reaper schedule and never overlaps reaping runs. */ +internal class QueryCursorReaperLifecycle( + private val reap: (Int) -> Mono, + private val properties: QueryCursorReaperProperties, + private val scheduler: Scheduler = Schedulers.newSingle("wow-query-cursor-reaper", true), + private val ownsScheduler: Boolean = true, +) : SmartLifecycle { + private enum class State { + NEW, + RUNNING, + STOPPED, + } + + private val monitor = Any() + private var subscription: Disposable? = null + + @Volatile + private var state = State.NEW + + override fun start() { + synchronized(monitor) { + when (state) { + State.RUNNING -> return + State.STOPPED -> error("Query cursor reaper cannot restart after it has stopped.") + State.NEW -> state = State.RUNNING + } + subscription = Flux.interval(properties.initialDelay, properties.interval, scheduler) + .onBackpressureDrop { + log.warn { "Skipping one Query cursor reaper tick because the previous run is still active." } + }.concatMap( + { + reapRun() + .doOnNext { reaped -> + if (reaped > 0) { + log.info { "Reaped $reaped expired Query cursor lease(s)." } + } + }.onErrorResume { error -> + log.error(error) { "Query cursor lease reaping failed; the next scheduled run remains active." } + Mono.empty() + } + }, + 1, + ).subscribe( + {}, + { error -> log.error(error) { "Query cursor reaper schedule terminated unexpectedly." } }, + ) + } + } + + override fun stop() { + val current = synchronized(monitor) { + if (state == State.STOPPED) return + state = State.STOPPED + subscription.also { subscription = null } + } + current?.dispose() + if (ownsScheduler) scheduler.dispose() + } + + override fun isRunning(): Boolean = state == State.RUNNING + + override fun getPhase(): Int = WOW_RUNTIME_PHASE + 1 + + private fun reapRun(): Mono = Flux.range(0, properties.maxBatchesPerRun) + .concatMap( + { + Mono.defer { reap(properties.batchSize) } + .switchIfEmpty(Mono.error(IllegalStateException("Query cursor reaper returned no result."))) + .map(::validateBatchResult) + }, + 1, + ).takeUntil { reaped -> reaped < properties.batchSize } + .reduce(0L, Math::addExact) + + private fun validateBatchResult(reaped: Long): Long { + check(reaped in 0..properties.batchSize.toLong()) { + "Query cursor reaper returned $reaped for batch size ${properties.batchSize}." + } + return reaped + } + + private companion object { + val log = KotlinLogging.logger {} + } +} diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperProperties.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperProperties.kt new file mode 100644 index 00000000000..d3d16da593a --- /dev/null +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperProperties.kt @@ -0,0 +1,46 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.spring.boot.starter.query + +import me.ahoo.wow.api.Wow +import me.ahoo.wow.api.naming.EnabledCapable +import org.springframework.boot.context.properties.ConfigurationProperties +import org.springframework.boot.context.properties.bind.DefaultValue +import java.time.Duration + +/** Explicit opt-in scheduling policy for bounded Query cursor lease cleanup. */ +@ConfigurationProperties(prefix = QueryCursorReaperProperties.PREFIX) +class QueryCursorReaperProperties( + @DefaultValue("false") override val enabled: Boolean = false, + @DefaultValue("PT30S") val initialDelay: Duration = Duration.ofSeconds(30), + @DefaultValue("PT1M") val interval: Duration = Duration.ofMinutes(1), + @DefaultValue("100") val batchSize: Int = 100, + @DefaultValue("10") val maxBatchesPerRun: Int = 10, +) : EnabledCapable { + init { + require(!initialDelay.isNegative) { "Query cursor reaper initial delay must not be negative." } + require(!interval.isNegative && !interval.isZero) { + "Query cursor reaper interval must be positive." + } + require(batchSize > 0) { "Query cursor reaper batch size must be positive." } + require(maxBatchesPerRun in 1..MAX_BATCHES_PER_RUN) { + "Query cursor reaper max batches per run must be between 1 and $MAX_BATCHES_PER_RUN." + } + } + + companion object { + const val PREFIX = "${Wow.WOW_PREFIX}query.cursor.reaper" + private const val MAX_BATCHES_PER_RUN = 1_000 + } +} diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryGatewayAutoConfiguration.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryGatewayAutoConfiguration.kt new file mode 100644 index 00000000000..60a531d2cb0 --- /dev/null +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryGatewayAutoConfiguration.kt @@ -0,0 +1,345 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.spring.boot.starter.query + +import me.ahoo.wow.api.naming.NamedBoundedContext +import me.ahoo.wow.configuration.MetadataSearcher +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.modeling.annotation.aggregateMetadata +import me.ahoo.wow.query.analytics.AnalyticsQueryServiceFactory +import me.ahoo.wow.query.analytics.AnalyticsQueryTrustedContextResolver +import me.ahoo.wow.query.analytics.CompositeAnalyticsQueryTrustedContextResolver +import me.ahoo.wow.query.backend.QueryBackendComposition +import me.ahoo.wow.query.cursor.QueryCursorLeaseConfiguration +import me.ahoo.wow.query.event.EventStreamQueryServiceFactory +import me.ahoo.wow.query.gateway.AnalyticsQueryGateway +import me.ahoo.wow.query.gateway.CompositeQueryTrustedContextResolver +import me.ahoo.wow.query.gateway.QueryAuthorityResolver +import me.ahoo.wow.query.gateway.QueryCallResolver +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryExecutionMode +import me.ahoo.wow.query.gateway.QueryExecutionProfiles +import me.ahoo.wow.query.gateway.QueryGateway +import me.ahoo.wow.query.gateway.QueryGatewayConfiguration +import me.ahoo.wow.query.gateway.QueryGatewayRuntime +import me.ahoo.wow.query.gateway.QueryLegacyDialectResolver +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryResultMaterializer +import me.ahoo.wow.query.gateway.QueryResultMaterializers +import me.ahoo.wow.query.gateway.QueryRuntimeHealthObserver +import me.ahoo.wow.query.gateway.QueryShadowConfiguration +import me.ahoo.wow.query.gateway.QueryShadowObserver +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.gateway.QueryTrustedContextResolver +import me.ahoo.wow.query.snapshot.SnapshotQueryServiceFactory +import me.ahoo.wow.spring.boot.starter.ConditionalOnWowEnabled +import me.ahoo.wow.spring.boot.starter.WowAutoConfiguration +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.EventStreamQueryServiceFactoryBinding +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.SnapshotQueryServiceFactoryBinding +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.StorageRoutingAutoConfiguration +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.StorageRoutingProperties +import me.ahoo.wow.spring.boot.starter.eventsourcing.snapshot.SnapshotProperties +import me.ahoo.wow.spring.boot.starter.eventsourcing.store.EventStoreProperties +import me.ahoo.wow.spring.boot.starter.webflux.WebFluxAutoConfiguration +import org.springframework.beans.factory.ObjectProvider +import org.springframework.beans.factory.SmartInitializingSingleton +import org.springframework.beans.factory.annotation.Qualifier +import org.springframework.boot.autoconfigure.AutoConfiguration +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean +import org.springframework.context.annotation.Bean +import org.springframework.context.annotation.Primary +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono + +/** + * Installs the Query Gateway as the single framework-managed application query facade. + * + * Raw MongoDB/Elasticsearch/custom factories remain storage bindings and are never injected back as the facade. + */ +@AutoConfiguration( + after = [ + QueryAutoConfiguration::class, + StorageRoutingAutoConfiguration::class, + WebFluxAutoConfiguration::class, + ], +) +@ConditionalOnWowEnabled +class QueryGatewayAutoConfiguration { + @Bean + internal fun storageBindingQueryRawServiceRegistry( + @Qualifier(WowAutoConfiguration.WOW_CURRENT_BOUNDED_CONTEXT) + namedBoundedContext: NamedBoundedContext, + storageRoutingProperties: StorageRoutingProperties, + eventStoreProperties: EventStoreProperties, + snapshotProperties: SnapshotProperties, + eventStreamBindings: List, + snapshotBindings: List, + ): StorageBindingQueryRawServiceRegistry = + StorageBindingQueryRawServiceRegistry( + contextName = namedBoundedContext.contextName, + storageRoutingProperties = storageRoutingProperties, + eventStreamBindings = eventStreamBindings, + snapshotBindings = snapshotBindings, + snapshotEnabled = snapshotProperties.enabled, + defaultEventStorage = eventStoreProperties.storage, + defaultSnapshotStorage = snapshotProperties.storage, + ) + + @Bean + @ConditionalOnMissingBean(QueryGatewayConfiguration::class) + fun queryGatewayConfiguration(): QueryGatewayConfiguration = QueryGatewayConfiguration() + + @Bean + @ConditionalOnMissingBean(QueryBackendComposition::class) + internal fun storageRoutedQueryBackendComposition( + rawServiceRegistry: StorageBindingQueryRawServiceRegistry, + sources: List, + configuration: QueryGatewayConfiguration, + executionProfilesProvider: ObjectProvider, + ): QueryBackendComposition { + val profiles = executionProfilesProvider.getIfAvailable { QueryExecutionProfiles.fixed(configuration) } + return StorageRoutedQueryBackendComposition.create( + sources = sources, + shouldPrepare = { target -> + RECORD_QUERY_OPERATIONS.any { operation -> + profiles.resolve(target, operation).executionMode != QueryExecutionMode.LEGACY + } + }, + storageResolver = rawServiceRegistry::resolveStorage, + ) + } + + @Bean + @ConditionalOnMissingBean(QueryGatewayRuntime::class) + @ConditionalOnQueryGatewayWiring + internal fun queryGatewayRuntime( + rawServiceRegistry: StorageBindingQueryRawServiceRegistry, + dialectResolverProvider: ObjectProvider, + authorityResolverProvider: ObjectProvider, + callResolverProvider: ObjectProvider, + trustedContextResolvers: List, + analyticsTrustedContextResolvers: List, + configuration: QueryGatewayConfiguration, + executionProfilesProvider: ObjectProvider, + shadowConfigurationProvider: ObjectProvider, + shadowObserverProvider: ObjectProvider, + runtimeHealthObserverProvider: ObjectProvider, + backendCompositionProvider: ObjectProvider, + cursorLeaseConfigurationProvider: ObjectProvider, + customResultMaterializers: List>, + ): QueryGatewayRuntime { + val aggregateTypes = MetadataSearcher.namedAggregateType + val standardMaterializers = createStandardMaterializers(aggregateTypes) + val dialectResolver = dialectResolverProvider.getIfAvailable { + QueryLegacyDialectResolver(rawServiceRegistry::resolveDialect) + } + val composition = backendCompositionProvider.getIfAvailable { QueryBackendComposition.EMPTY } + val authorityResolver = resolveDirectAuthorityResolver(authorityResolverProvider) + val recordTrustedResolver = resolveTrustedContextResolver( + trustedContextResolvers, + callResolverProvider, + authorityResolverProvider, + ) + val analyticsTrustedResolver = when (analyticsTrustedContextResolvers.size) { + 0 -> AnalyticsQueryTrustedContextResolver { Mono.empty() } + 1 -> analyticsTrustedContextResolvers.single() + else -> CompositeAnalyticsQueryTrustedContextResolver(analyticsTrustedContextResolvers) + } + val trustedResolver = FrozenQueryTrustedContextResolvers(recordTrustedResolver, analyticsTrustedResolver) + val profiles = executionProfilesProvider.getIfAvailable { QueryExecutionProfiles.fixed(configuration) } + val shadowConfiguration = shadowConfigurationProvider.getIfAvailable { QueryShadowConfiguration() } + val shadowObserver = shadowObserverProvider.getIfAvailable { QueryShadowObserver.NONE } + val runtimeHealthObserver = runtimeHealthObserverProvider.getIfAvailable { QueryRuntimeHealthObserver.NONE } + val cursorConfiguration = cursorLeaseConfigurationProvider.getIfAvailable() + return if (cursorConfiguration == null) { + QueryGatewayRuntime.create( + aggregateTypes.keys, + composition, + rawServiceRegistry, + dialectResolver, + authorityResolver, + trustedResolver, + standardMaterializers + customResultMaterializers, + configuration, + profiles, + shadowConfiguration, + shadowObserver, + runtimeHealthObserver, + ) + } else { + QueryGatewayRuntime.create( + aggregateTypes.keys, + composition, + cursorConfiguration, + rawServiceRegistry, + dialectResolver, + authorityResolver, + trustedResolver, + standardMaterializers + customResultMaterializers, + configuration, + profiles, + shadowConfiguration, + shadowObserver, + runtimeHealthObserver, + ) + } + } + + private fun createStandardMaterializers( + aggregateTypes: Map>, + ): List> = aggregateTypes.flatMap { (namedAggregate, aggregateType) -> + val snapshotTarget = QueryTarget(namedAggregate, QueryDocumentKind.SNAPSHOT) + val eventStreamTarget = QueryTarget(namedAggregate, QueryDocumentKind.EVENT_STREAM) + listOf>( + QueryResultMaterializers.snapshot( + snapshotTarget, + aggregateType.aggregateMetadata().state.aggregateType, + ), + QueryResultMaterializers.eventStream(eventStreamTarget), + ) + } + + @Bean + @ConditionalOnMissingBean(QueryGateway::class) + @ConditionalOnQueryGatewayWiring + fun queryGateway(runtime: QueryGatewayRuntime): QueryGateway = runtime.gateway + + @Bean + @ConditionalOnMissingBean(AnalyticsQueryGateway::class) + @ConditionalOnQueryGatewayWiring + fun analyticsQueryGateway(runtime: QueryGatewayRuntime): AnalyticsQueryGateway = runtime.analyticsGateway + + @Bean + @Primary + @ConditionalOnQueryGatewayWiring + fun analyticsQueryServiceFactory( + runtime: QueryGatewayRuntime, + ): AnalyticsQueryServiceFactory = runtime.analyticsQueryServiceFactory() + + @Bean + @ConditionalOnQueryGatewayWiring + internal fun queryGatewayRuntimeOwnership( + runtime: QueryGatewayRuntime, + gateways: List, + analyticsGateways: List, + ): SmartInitializingSingleton = SmartInitializingSingleton { + require(gateways.size == 1 && gateways.single() === runtime.gateway) { + "A custom QueryGateway cannot partially override framework Gateway wiring. " + + "Provide one complete QueryGatewayRuntime override instead." + } + require(analyticsGateways.size == 1 && analyticsGateways.single() === runtime.analyticsGateway) { + "A custom AnalyticsQueryGateway cannot partially override framework Gateway wiring. " + + "Provide one complete QueryGatewayRuntime override instead." + } + } + + @Bean + @Primary + @ConditionalOnQueryGatewayWiring + fun gatewaySnapshotQueryServiceFactory(runtime: QueryGatewayRuntime): SnapshotQueryServiceFactory = + runtime.snapshotQueryServiceFactory() + + @Bean + @Primary + @ConditionalOnQueryGatewayWiring + fun gatewayEventStreamQueryServiceFactory(runtime: QueryGatewayRuntime): EventStreamQueryServiceFactory = + runtime.eventStreamQueryServiceFactory() +} + +private class FrozenQueryTrustedContextResolvers( + private val record: QueryTrustedContextResolver, + private val analytics: AnalyticsQueryTrustedContextResolver, +) : QueryTrustedContextResolver, + AnalyticsQueryTrustedContextResolver { + override fun resolve(request: me.ahoo.wow.query.gateway.QueryTrustedContextRequest) = record.resolve(request) + + override fun resolve(request: me.ahoo.wow.query.analytics.AnalyticsQueryTrustedContextRequest) = + analytics.resolve(request) +} + +private val RECORD_QUERY_OPERATIONS = listOf( + QueryOperation.SINGLE, + QueryOperation.STREAM, + QueryOperation.PAGE, + QueryOperation.COUNT, + QueryOperation.ANALYZE, +) + +private fun failClosedQueryAuthorityResolver(): QueryAuthorityResolver = + QueryAuthorityResolver { Mono.empty() } + +private fun resolveTrustedContextResolver( + trustedResolvers: List, + callResolverProvider: ObjectProvider, + authorityResolverProvider: ObjectProvider, +): QueryTrustedContextResolver { + val callResolvers = callResolverProvider.orderedStream() + .filter { resolver -> resolver !is QueryTrustedContextResolver } + .toList() + val authorityResolvers = authorityResolverProvider.orderedStream() + .filter { resolver -> resolver !is QueryTrustedContextResolver } + .toList() + require(callResolvers.size <= 1) { + "Separate QueryCallResolver compatibility beans must be unique. " + + "Use QueryTrustedContextResolver for ordered composition." + } + if (callResolvers.isNotEmpty()) { + require(authorityResolvers.size == 1) { + "A separate QueryCallResolver compatibility bean requires exactly one QueryAuthorityResolver partner." + } + } + val directPair = callResolvers.singleOrNull()?.let { callResolver -> + val authorityResolver = authorityResolvers.single() + QueryTrustedContextResolver { request -> + callResolver.resolve(request.callRequest) + .flatMap { call -> + authorityResolver.resolve( + me.ahoo.wow.query.gateway.QueryAuthorityRequest( + call, + request.executionMode, + request.validationMode, + ), + ).map { authority -> me.ahoo.wow.query.gateway.QueryTrustedContext(call, authority) } + } + } + } + val resolvers = directPair?.let { trustedResolvers + it } ?: trustedResolvers + if (resolvers.isEmpty()) { + return QueryTrustedContextResolver { Mono.empty() } + } + return CompositeQueryTrustedContextResolver(resolvers) +} + +private fun resolveDirectAuthorityResolver( + directProvider: ObjectProvider, +): QueryAuthorityResolver { + val resolvers = directProvider.orderedStream() + .filter { resolver -> resolver !is QueryTrustedContextResolver } + .toList() + return when (resolvers.size) { + 0 -> failClosedQueryAuthorityResolver() + 1 -> resolvers.single() + else -> QueryAuthorityResolver { request -> + Flux.fromIterable(resolvers) + .concatMap { resolver -> Mono.defer { resolver.resolve(request) } } + .next() + } + } +} diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryGatewayLegacyWiringRollbackAutoConfiguration.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryGatewayLegacyWiringRollbackAutoConfiguration.kt new file mode 100644 index 00000000000..f71f219f420 --- /dev/null +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryGatewayLegacyWiringRollbackAutoConfiguration.kt @@ -0,0 +1,133 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.spring.boot.starter.query + +import io.github.oshai.kotlinlogging.KotlinLogging +import io.micrometer.core.instrument.MeterRegistry +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.query.event.EventStreamQueryServiceFactory +import me.ahoo.wow.query.snapshot.SnapshotQueryService +import me.ahoo.wow.query.snapshot.SnapshotQueryServiceFactory +import me.ahoo.wow.spring.boot.starter.ConditionalOnWowEnabled +import org.springframework.beans.factory.ObjectProvider +import org.springframework.boot.autoconfigure.AutoConfiguration +import org.springframework.boot.autoconfigure.condition.ConditionOutcome +import org.springframework.boot.autoconfigure.condition.SpringBootCondition +import org.springframework.context.annotation.Bean +import org.springframework.context.annotation.Conditional +import org.springframework.context.annotation.Primary +import org.springframework.core.type.AnnotatedTypeMetadata + +internal const val QUERY_GATEWAY_LEGACY_WIRING_ROLLBACK_KEY = + "wow.query.gateway.legacy-wiring-rollback" + +/** + * One-version emergency rollback for the application query facade wiring. + * + * The switch is deliberately explicit and observable. It only bypasses the Gateway facade; authorization, policy, + * schema, lowering, or mapping failures never select this path automatically. + */ +@AutoConfiguration(after = [QueryGatewayAutoConfiguration::class]) +@ConditionalOnWowEnabled +@ConditionalOnLegacyQueryGatewayWiringRollback +internal class QueryGatewayLegacyWiringRollbackAutoConfiguration { + @Bean + internal fun queryLegacyWiringRollback( + rawServiceSource: StorageBindingQueryRawServiceRegistry, + meterRegistries: ObjectProvider, + ): QueryLegacyWiringRollback = QueryLegacyWiringRollback(rawServiceSource, meterRegistries.getIfAvailable()) + + @Bean + @Primary + fun legacyWiringSnapshotQueryServiceFactory( + rollback: QueryLegacyWiringRollback, + ): SnapshotQueryServiceFactory = rollback.snapshotFactory + + @Bean + @Primary + fun legacyWiringEventStreamQueryServiceFactory( + rollback: QueryLegacyWiringRollback, + ): EventStreamQueryServiceFactory = rollback.eventStreamFactory +} + +@Target(AnnotationTarget.CLASS, AnnotationTarget.FUNCTION) +@Retention(AnnotationRetention.RUNTIME) +@Conditional(OnQueryGatewayWiringCondition::class) +internal annotation class ConditionalOnQueryGatewayWiring + +@Target(AnnotationTarget.CLASS, AnnotationTarget.FUNCTION) +@Retention(AnnotationRetention.RUNTIME) +@Conditional(OnLegacyQueryGatewayWiringRollbackCondition::class) +internal annotation class ConditionalOnLegacyQueryGatewayWiringRollback + +internal class OnQueryGatewayWiringCondition : SpringBootCondition() { + override fun getMatchOutcome( + context: org.springframework.context.annotation.ConditionContext, + metadata: AnnotatedTypeMetadata, + ): ConditionOutcome = wiringOutcome(!context.rollbackEnabled(), "Query Gateway") +} + +internal class OnLegacyQueryGatewayWiringRollbackCondition : SpringBootCondition() { + override fun getMatchOutcome( + context: org.springframework.context.annotation.ConditionContext, + metadata: AnnotatedTypeMetadata, + ): ConditionOutcome = wiringOutcome(context.rollbackEnabled(), "legacy query wiring rollback") +} + +private fun org.springframework.context.annotation.ConditionContext.rollbackEnabled(): Boolean { + val configured = environment.getProperty(QUERY_GATEWAY_LEGACY_WIRING_ROLLBACK_KEY) ?: return false + return when (configured.lowercase()) { + "true" -> true + "false" -> false + else -> error( + "Property[$QUERY_GATEWAY_LEGACY_WIRING_ROLLBACK_KEY] must be exactly true or false, but was [$configured].", + ) + } +} + +private fun wiringOutcome(matched: Boolean, mode: String): ConditionOutcome = + if (matched) { + ConditionOutcome.match("$mode is selected.") + } else { + ConditionOutcome.noMatch("$mode is not selected.") + } + +internal class QueryLegacyWiringRollback( + rawServiceSource: StorageBindingQueryRawServiceRegistry, + meterRegistry: MeterRegistry?, +) { + val snapshotFactory: SnapshotQueryServiceFactory = object : SnapshotQueryServiceFactory { + @Suppress("UNCHECKED_CAST") + override fun create(namedAggregate: NamedAggregate): SnapshotQueryService = + rawServiceSource.snapshot(namedAggregate) as SnapshotQueryService + } + + val eventStreamFactory: EventStreamQueryServiceFactory = EventStreamQueryServiceFactory { namedAggregate -> + rawServiceSource.eventStream(namedAggregate) + } + + init { + LOG.warn { + "Query Gateway wiring rollback is enabled by [$QUERY_GATEWAY_LEGACY_WIRING_ROLLBACK_KEY]. " + + "Framework-managed query services are bypassing admission, policy, and lifecycle enforcement. " + + "This emergency switch is supported for one migration version only." + } + meterRegistry?.counter(ROLLBACK_METRIC)?.increment() + } + + private companion object { + const val ROLLBACK_METRIC = "wow.query.gateway.legacy.wiring.rollback" + val LOG = KotlinLogging.logger {} + } +} diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/StorageBindingQueryRawServiceRegistry.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/StorageBindingQueryRawServiceRegistry.kt new file mode 100644 index 00000000000..858c965a170 --- /dev/null +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/StorageBindingQueryRawServiceRegistry.kt @@ -0,0 +1,182 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.spring.boot.starter.query + +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.modeling.materialize +import me.ahoo.wow.query.event.EventStreamQueryService +import me.ahoo.wow.query.event.EventStreamQueryServiceFactory +import me.ahoo.wow.query.event.NoOpEventStreamQueryServiceFactory +import me.ahoo.wow.query.gateway.GatewayEventStreamQueryServiceFactory +import me.ahoo.wow.query.gateway.GatewaySnapshotQueryServiceFactory +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryElementPathMode +import me.ahoo.wow.query.gateway.QueryLegacyDialect +import me.ahoo.wow.query.gateway.QueryMatchScopeMode +import me.ahoo.wow.query.gateway.QueryRawServiceSource +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.snapshot.NoOpSnapshotQueryServiceFactory +import me.ahoo.wow.query.snapshot.SnapshotQueryService +import me.ahoo.wow.query.snapshot.SnapshotQueryServiceFactory +import me.ahoo.wow.spring.boot.starter.eventsourcing.StorageType +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.EventStreamQueryServiceFactoryBinding +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.SnapshotQueryServiceFactoryBinding +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.StorageRouteResolver +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.StorageRoutingProperties + +/** Raw query route owner. It is deliberately a different type from the public application facade factories. */ +internal class StorageBindingQueryRawServiceRegistry( + contextName: String, + storageRoutingProperties: StorageRoutingProperties, + eventStreamBindings: List, + snapshotBindings: List, + snapshotEnabled: Boolean, + defaultEventStorage: StorageType, + defaultSnapshotStorage: StorageType, +) : QueryRawServiceSource { + private val eventStreamBindings = eventStreamBindings.toList().also(::rejectFacadeEventBindings) + private val snapshotBindings = snapshotBindings.toList().also(::rejectFacadeSnapshotBindings) + private val eventRoutes: RawEventRoutes + private val snapshotRoutes: RawSnapshotRoutes + + init { + val resolver = StorageRouteResolver( + contextName = contextName, + snapshotEnabled = snapshotEnabled, + eventStoreBindings = emptyList(), + snapshotStoreBindings = emptyList(), + eventStreamQueryServiceFactoryBindings = this.eventStreamBindings, + snapshotQueryServiceFactoryBindings = this.snapshotBindings, + defaultEventStorage = defaultEventStorage, + defaultSnapshotStorage = defaultSnapshotStorage, + ) + resolver.resolveEventStreamQueryServiceFactoryRoutes(storageRoutingProperties).let { resolved -> + eventRoutes = RawEventRoutes( + resolved.defaultEventStreamQueryServiceFactory, + resolved.eventStreamQueryServiceFactoryRoutes.mapKeys { (aggregate, _) -> aggregate.materialize() }, + ) + } + resolver.resolveSnapshotQueryServiceFactoryRoutes(storageRoutingProperties).let { resolved -> + snapshotRoutes = RawSnapshotRoutes( + resolved.defaultSnapshotQueryServiceFactory, + resolved.snapshotQueryServiceFactoryRoutes.mapKeys { (aggregate, _) -> aggregate.materialize() }, + ) + } + } + + override fun snapshot(namedAggregate: NamedAggregate): SnapshotQueryService<*> = + snapshotFactory(namedAggregate).create(namedAggregate.materialize()) + + override fun eventStream(namedAggregate: NamedAggregate): EventStreamQueryService = + eventStreamFactory(namedAggregate).create(namedAggregate.materialize()) + + fun resolveDialect(target: QueryTarget): QueryLegacyDialect { + val storage = resolveStorage(target) + return when (storage) { + StorageType.ELASTICSEARCH -> ELASTICSEARCH_DIALECT + StorageType.MONGO -> MONGO_DIALECT + null -> NO_OP_DIALECT + else -> throw IllegalStateException( + "Raw query route for target[$target] does not declare a supported legacy dialect.", + ) + } + } + + fun resolveStorage(target: QueryTarget): StorageType? = when (target.documentKind) { + QueryDocumentKind.SNAPSHOT -> storageOf(snapshotFactory(target.namedAggregate), snapshotBindings) + QueryDocumentKind.EVENT_STREAM -> storageOf(eventStreamFactory(target.namedAggregate), eventStreamBindings) + } + + private fun snapshotFactory(namedAggregate: NamedAggregate): SnapshotQueryServiceFactory = + snapshotRoutes.routes[namedAggregate.materialize()] ?: snapshotRoutes.defaultFactory + + private fun eventStreamFactory(namedAggregate: NamedAggregate): EventStreamQueryServiceFactory = + eventRoutes.routes[namedAggregate.materialize()] ?: eventRoutes.defaultFactory + + private fun storageOf( + factory: SnapshotQueryServiceFactory, + bindings: List, + ): StorageType? { + if (factory === NoOpSnapshotQueryServiceFactory) { + return null + } + return bindings.asSequence() + .filter { binding -> binding.snapshotQueryServiceFactory === factory } + .mapNotNull(SnapshotQueryServiceFactoryBinding::storage) + .distinct() + .singleOrNull() + ?: throw IllegalStateException( + "Raw snapshot query factory[${factory.javaClass.name}] requires an explicit QueryLegacyDialectResolver.", + ) + } + + private fun storageOf( + factory: EventStreamQueryServiceFactory, + bindings: List, + ): StorageType? { + if (factory === NoOpEventStreamQueryServiceFactory) { + return null + } + return bindings.asSequence() + .filter { binding -> binding.eventStreamQueryServiceFactory === factory } + .mapNotNull(EventStreamQueryServiceFactoryBinding::storage) + .distinct() + .singleOrNull() + ?: throw IllegalStateException( + "Raw event-stream query factory[${factory.javaClass.name}] requires an explicit " + + "QueryLegacyDialectResolver.", + ) + } + + private data class RawSnapshotRoutes( + val defaultFactory: SnapshotQueryServiceFactory, + val routes: Map, + ) + + private data class RawEventRoutes( + val defaultFactory: EventStreamQueryServiceFactory, + val routes: Map, + ) + + private companion object { + val MONGO_DIALECT = QueryLegacyDialect( + QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, + QueryMatchScopeMode.DOCUMENT, + ) + val ELASTICSEARCH_DIALECT = QueryLegacyDialect( + QueryElementPathMode.ROOT_QUALIFIED, + QueryMatchScopeMode.FIELD, + ) + val NO_OP_DIALECT = MONGO_DIALECT + + fun rejectFacadeSnapshotBindings(bindings: List) { + bindings.forEach { binding -> + require(binding.snapshotQueryServiceFactory !is GatewaySnapshotQueryServiceFactory) { + "Gateway facade factory[${binding.name}] cannot be registered as a raw snapshot query binding." + } + } + } + + fun rejectFacadeEventBindings(bindings: List) { + bindings.forEach { binding -> + require(binding.eventStreamQueryServiceFactory !is GatewayEventStreamQueryServiceFactory) { + "Gateway facade factory[${binding.name}] cannot be registered as a raw event-stream query binding." + } + } + } + } +} diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/StorageQueryBackendSource.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/StorageQueryBackendSource.kt new file mode 100644 index 00000000000..79a7b047235 --- /dev/null +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/query/StorageQueryBackendSource.kt @@ -0,0 +1,98 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.spring.boot.starter.query + +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.QueryBackendComposition +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.RecordQueryBackendContribution +import me.ahoo.wow.query.backend.RecordQueryBackendNotReady +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.spring.boot.starter.eventsourcing.StorageType +import reactor.core.publisher.Mono + +/** Storage-specific planned backend source. It is selected only after the existing aggregate storage route resolves. */ +internal interface StorageQueryBackendSource { + val storage: StorageType + val targets: Set + + fun prepare(target: QueryTarget): Mono +} + +internal sealed interface StorageQueryBackendPreparation { + data class Ready(val contribution: RecordQueryBackendContribution) : StorageQueryBackendPreparation + + data class NotReady( + val schema: QueryDocumentSchema, + val backendId: BackendId, + ) : StorageQueryBackendPreparation +} + +internal object StorageRoutedQueryBackendComposition { + fun create( + sources: List, + shouldPrepare: (QueryTarget) -> Boolean = { true }, + storageResolver: (QueryTarget) -> StorageType?, + ): QueryBackendComposition { + val sourcesByRoute = linkedMapOf, StorageQueryBackendSource>() + sources.forEach { source -> + source.targets.forEach { target -> + require(sourcesByRoute.put(target to source.storage, source) == null) { + "Duplicate planned Query Backend source for $target/${source.storage}." + } + } + } + val contributions = mutableListOf() + val notReadyBackends = mutableListOf() + val routes = linkedMapOf() + sourcesByRoute.entries.sortedWith(ROUTE_COMPARATOR).forEach { (route, source) -> + val (target, storage) = route + if (shouldPrepare(target) && storageResolver(target) == storage) { + val preparation = requireNotNull(source.prepare(target).block()) { + "Planned Query Backend source returned empty for $target/$storage." + } + val schema = when (preparation) { + is StorageQueryBackendPreparation.Ready -> preparation.contribution.schema + is StorageQueryBackendPreparation.NotReady -> preparation.schema + } + require(schema.target == target) { + "Planned Query Backend contribution target does not match its storage route." + } + when (preparation) { + is StorageQueryBackendPreparation.Ready -> { + contributions += preparation.contribution + routes[target] = preparation.contribution.backendId + } + + is StorageQueryBackendPreparation.NotReady -> { + notReadyBackends += RecordQueryBackendNotReady(schema, preparation.backendId) + routes[target] = preparation.backendId + } + } + } + } + return QueryBackendComposition(contributions, notReadyBackends, routes) + } + + private val ROUTE_COMPARATOR = compareBy, StorageQueryBackendSource>> { + it.key.first.namedAggregate.contextName + }.thenBy { it.key.first.namedAggregate.aggregateName } + .thenBy { it.key.first.documentKind.name } + .thenBy { it.key.second.name } +} diff --git a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/webflux/WebFluxAutoConfiguration.kt b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/webflux/WebFluxAutoConfiguration.kt index 0cc1d9dc1ac..70c6623abd8 100644 --- a/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/webflux/WebFluxAutoConfiguration.kt +++ b/wow-spring-boot-starter/src/main/kotlin/me/ahoo/wow/spring/boot/starter/webflux/WebFluxAutoConfiguration.kt @@ -10,6 +10,8 @@ * See the License for the specific language governing permissions and * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.spring.boot.starter.webflux import me.ahoo.wow.bi.BiDeploymentInspector @@ -23,6 +25,7 @@ import me.ahoo.wow.messaging.compensation.EventCompensateSupporter import me.ahoo.wow.modeling.state.StateAggregateFactory import me.ahoo.wow.modeling.state.StateAggregateRepository import me.ahoo.wow.openapi.RouterSpecs +import me.ahoo.wow.query.analytics.AnalyticsQueryServiceFactory import me.ahoo.wow.query.event.filter.EventStreamQueryHandler import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler import me.ahoo.wow.spring.boot.starter.ConditionalOnWowEnabled @@ -60,7 +63,10 @@ import me.ahoo.wow.webflux.route.global.GenerateBIScriptHandlerFunctionFactory import me.ahoo.wow.webflux.route.policy.BatchExecutionPolicy import me.ahoo.wow.webflux.route.policy.CommandWaitPolicy import me.ahoo.wow.webflux.route.policy.TracingPolicy +import me.ahoo.wow.webflux.route.query.AnalyticsQueryHandlerFunctionFactory import me.ahoo.wow.webflux.route.query.DefaultRewriteRequestCondition +import me.ahoo.wow.webflux.route.query.QueryWebAuthorityResolver +import me.ahoo.wow.webflux.route.query.QueryWebTransportResolvers import me.ahoo.wow.webflux.route.query.RewriteRequestCondition import org.springframework.beans.factory.ObjectProvider import org.springframework.boot.autoconfigure.AutoConfiguration @@ -88,6 +94,19 @@ import org.springframework.web.server.WebExceptionHandler name = ["org.springframework.web.server.WebFilter", "me.ahoo.wow.webflux.route.command.CommandHandlerFunction"], ) class WebFluxAutoConfiguration { + @Bean + @ConditionalOnMissingBean + fun queryWebAuthorityResolver(): QueryWebAuthorityResolver = QueryWebAuthorityResolver { + reactor.core.publisher.Mono.empty() + } + + @Bean + @Order(Ordered.HIGHEST_PRECEDENCE) + @ConditionalOnMissingBean + fun queryWebTransportResolvers( + queryWebAuthorityResolver: QueryWebAuthorityResolver, + ): QueryWebTransportResolvers = QueryWebTransportResolvers(queryWebAuthorityResolver) + @Bean @ConditionalOnMissingBean fun webFluxErrorStrategy(): WebFluxErrorStrategy { @@ -233,6 +252,18 @@ class WebFluxAutoConfiguration { ) } + @Bean + @ConditionalOnMissingBean + fun analyticsQueryHandlerFunctionFactory( + analyticsQueryServiceFactories: ObjectProvider, + exceptionHandler: RequestExceptionHandler, + ): AnalyticsQueryHandlerFunctionFactory = AnalyticsQueryHandlerFunctionFactory( + analyticsQueryServiceFactory = analyticsQueryServiceFactories.getIfAvailable { + UnavailableAnalyticsQueryServiceFactory + }, + exceptionHandler = exceptionHandler, + ) + @Bean @Order(Ordered.HIGHEST_PRECEDENCE) @ConditionalOnMissingBean @@ -324,3 +355,21 @@ class WebFluxAutoConfiguration { ).build() } } + +private object UnavailableAnalyticsQueryServiceFactory : AnalyticsQueryServiceFactory { + override fun create( + namedAggregate: me.ahoo.wow.api.modeling.NamedAggregate, + ): me.ahoo.wow.query.analytics.AnalyticsQueryService = object : me.ahoo.wow.query.analytics.AnalyticsQueryService { + override val namedAggregate: me.ahoo.wow.api.modeling.NamedAggregate = namedAggregate + + override fun analyze( + query: me.ahoo.wow.api.query.analytics.AnalyticsQuery, + ): reactor.core.publisher.Mono = reactor.core.publisher.Mono.error( + me.ahoo.wow.query.gateway.QueryExecutionException( + me.ahoo.wow.query.gateway.QueryErrorCategory.UNSUPPORTED_FEATURE, + "$.target", + "SCHEMA_NOT_REGISTERED", + ), + ) + } +} diff --git a/wow-spring-boot-starter/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports b/wow-spring-boot-starter/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports index 6f9573226e8..ed43c43e4aa 100644 --- a/wow-spring-boot-starter/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports +++ b/wow-spring-boot-starter/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports @@ -28,4 +28,7 @@ me.ahoo.wow.spring.boot.starter.webflux.WebFluxAutoConfiguration me.ahoo.wow.spring.boot.starter.webflux.WowWebClientAutoConfiguration me.ahoo.wow.spring.boot.starter.compensation.CompensationAutoConfiguration me.ahoo.wow.spring.boot.starter.query.QueryAutoConfiguration +me.ahoo.wow.spring.boot.starter.query.QueryGatewayAutoConfiguration +me.ahoo.wow.spring.boot.starter.query.QueryCursorReaperAutoConfiguration +me.ahoo.wow.spring.boot.starter.query.QueryGatewayLegacyWiringRollbackAutoConfiguration me.ahoo.wow.spring.boot.starter.cosec.CoSecAutoConfiguration diff --git a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/elasticsearch/ElasticsearchPlannedQueryBackendSourceTest.kt b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/elasticsearch/ElasticsearchPlannedQueryBackendSourceTest.kt new file mode 100644 index 00000000000..1a7305f50ef --- /dev/null +++ b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/elasticsearch/ElasticsearchPlannedQueryBackendSourceTest.kt @@ -0,0 +1,99 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.spring.boot.starter.elasticsearch + +import io.mockk.confirmVerified +import io.mockk.mockk +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.elasticsearch.query.planned.ElasticsearchFieldBinding +import me.ahoo.wow.elasticsearch.query.planned.ElasticsearchSnapshotQueryBinding +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.spring.boot.starter.eventsourcing.StorageType +import me.ahoo.wow.spring.boot.starter.query.StorageRoutedQueryBackendComposition +import org.junit.jupiter.api.Test +import org.springframework.data.elasticsearch.client.elc.ReactiveElasticsearchClient + +class ElasticsearchPlannedQueryBackendSourceTest { + @Test + fun `binding routed elsewhere should not inspect Elasticsearch`() { + val client = mockk() + val source = ElasticsearchPlannedQueryBackendSource(client, listOf(binding)) + + val composition = StorageRoutedQueryBackendComposition.create(listOf(source)) { StorageType.MONGO } + + composition.contributions.assert().isEmpty() + composition.defaultRoutes.assert().isEmpty() + confirmVerified(client) + } + + @Test + fun `bindings should be unique for each target`() { + val client = mockk() + + assertThrownBy { + ElasticsearchPlannedQueryBackendSource(client, listOf(binding, binding)) + } + confirmVerified(client) + } + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + private val schema = QueryDocumentSchema( + target, + listOf( + QueryFieldSchema( + identity, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT), + ), + ), + emptyList(), + ) + private val binding = ElasticsearchSnapshotQueryBinding( + schema, + "wow.sales.order.snapshot", + "order-query-v1", + mapOf( + identity to ElasticsearchFieldBinding( + MessageRecords.AGGREGATE_ID, + setOf(FieldCapability.EXACT), + exactField = "_id", + ), + ), + ) +} diff --git a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRouteResolverTest.kt b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRouteResolverTest.kt index dfcc94be1ee..2a9b7500b6b 100644 --- a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRouteResolverTest.kt +++ b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRouteResolverTest.kt @@ -360,6 +360,50 @@ class StorageRouteResolverTest { .defaultSnapshotQueryServiceFactory.assert().isSameAs(NoOpSnapshotQueryServiceFactory) } + @Test + fun `duplicate query factory binding name fails fast`() { + val duplicate = EventStreamQueryServiceFactoryBinding( + name = "archive-event-store", + storage = StorageType.ELASTICSEARCH, + eventStreamQueryServiceFactory = archiveEventStreamQueryServiceFactory, + ) + + val exception = assertThrows { + StorageRouteResolver( + contextName = "order-service", + snapshotEnabled = true, + eventStoreBindings = emptyList(), + snapshotStoreBindings = emptyList(), + eventStreamQueryServiceFactoryBindings = eventStreamQueryServiceFactoryBindings(true) + duplicate, + ) + } + + exception.message.assert().contains("EventStreamQueryServiceFactoryBinding") + exception.message.assert().contains("archive-event-store") + } + + @Test + fun `duplicate query factory binding storage fails fast`() { + val duplicate = SnapshotQueryServiceFactoryBinding( + name = "secondary-mongo-snapshot-query", + storage = StorageType.MONGO, + snapshotQueryServiceFactory = archiveSnapshotQueryServiceFactory, + ) + + val exception = assertThrows { + StorageRouteResolver( + contextName = "order-service", + snapshotEnabled = true, + eventStoreBindings = emptyList(), + snapshotStoreBindings = emptyList(), + snapshotQueryServiceFactoryBindings = snapshotQueryServiceFactoryBindings(true) + duplicate, + ) + } + + exception.message.assert().contains("SnapshotQueryServiceFactoryBinding") + exception.message.assert().contains(StorageType.MONGO.name) + } + private fun resolver( contextName: String = "order-service", snapshotEnabled: Boolean = true, diff --git a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRoutingAutoConfigurationTest.kt b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRoutingAutoConfigurationTest.kt index 6326c3432b4..893289fbb99 100644 --- a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRoutingAutoConfigurationTest.kt +++ b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/eventsourcing/routing/StorageRoutingAutoConfigurationTest.kt @@ -10,6 +10,9 @@ * See the License for the specific language governing permissions and * limitations under the License. */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.spring.boot.starter.eventsourcing.routing import io.mockk.every @@ -27,6 +30,11 @@ import me.ahoo.wow.modeling.MaterializedNamedAggregate import me.ahoo.wow.modeling.aggregateId import me.ahoo.wow.query.event.EventStreamQueryServiceFactory import me.ahoo.wow.query.event.NoOpEventStreamQueryService +import me.ahoo.wow.query.gateway.QueryElementPathMode +import me.ahoo.wow.query.gateway.QueryLegacyDialect +import me.ahoo.wow.query.gateway.QueryLegacyDialectResolver +import me.ahoo.wow.query.gateway.QueryMatchScopeMode +import me.ahoo.wow.query.gateway.QueryRawServiceSource import me.ahoo.wow.query.snapshot.NoOpSnapshotQueryService import me.ahoo.wow.query.snapshot.SnapshotQueryServiceFactory import me.ahoo.wow.spring.boot.starter.enableWow @@ -34,6 +42,7 @@ import me.ahoo.wow.spring.boot.starter.eventsourcing.StorageType import me.ahoo.wow.spring.boot.starter.eventsourcing.snapshot.ConditionalOnSnapshotEnabled import me.ahoo.wow.spring.boot.starter.eventsourcing.snapshot.SnapshotProperties import me.ahoo.wow.spring.boot.starter.eventsourcing.store.EventStoreProperties +import me.ahoo.wow.spring.boot.starter.query.QueryGatewayAutoConfiguration import org.junit.jupiter.api.Test import org.springframework.beans.factory.BeanCreationException import org.springframework.boot.test.context.assertj.AssertableApplicationContext @@ -59,6 +68,7 @@ class StorageRoutingAutoConfigurationTest { ) .withUserConfiguration( StorageRoutingAutoConfiguration::class.java, + QueryGatewayAutoConfiguration::class.java, StorageRoutingTestConfiguration::class.java, ) @@ -238,17 +248,17 @@ class StorageRoutingAutoConfigurationTest { } @Test - fun `event route should create primary routing event stream query service factory`() { + fun `event route should configure raw event stream query service source`() { routingContextRunner .withPropertyValues("${StorageRoutingProperties.AGGREGATES}.order.event.storage=${StorageType.REDIS_NAME}") .run { context: AssertableApplicationContext -> val stores = context.getBean(RecordingStores::class.java) - val queryServiceFactory = context.getBean(EventStreamQueryServiceFactory::class.java) + val rawServiceSource = context.getBean(QueryRawServiceSource::class.java) - queryServiceFactory.create(ORDER) + rawServiceSource.eventStream(ORDER) stores.redisEventStreamQueryServiceFactory.lastNamedAggregate.assert().isEqualTo(ORDER) - queryServiceFactory.create(CART) + rawServiceSource.eventStream(CART) stores.mongoEventStreamQueryServiceFactory.lastNamedAggregate.assert().isEqualTo(CART) } } @@ -298,19 +308,19 @@ class StorageRoutingAutoConfigurationTest { } @Test - fun `snapshot route should create primary routing snapshot query service factory`() { + fun `snapshot route should configure raw snapshot query service source`() { routingContextRunner .withPropertyValues( "${StorageRoutingProperties.AGGREGATES}.cart.snapshot.storage=${StorageType.REDIS_NAME}" ) .run { context: AssertableApplicationContext -> val stores = context.getBean(RecordingStores::class.java) - val queryServiceFactory = context.getBean(SnapshotQueryServiceFactory::class.java) + val rawServiceSource = context.getBean(QueryRawServiceSource::class.java) - queryServiceFactory.create(CART) + rawServiceSource.snapshot(CART) stores.redisSnapshotQueryServiceFactory.lastNamedAggregate.assert().isEqualTo(CART) - queryServiceFactory.create(ORDER) + rawServiceSource.snapshot(ORDER) stores.mongoSnapshotQueryServiceFactory.lastNamedAggregate.assert().isEqualTo(ORDER) } } @@ -397,6 +407,14 @@ class StorageRoutingAutoConfigurationTest { @Bean fun recordingStores(): RecordingStores = RecordingStores() + @Bean + fun queryLegacyDialectResolver(): QueryLegacyDialectResolver = QueryLegacyDialectResolver { + QueryLegacyDialect( + QueryElementPathMode.CURRENT_ELEMENT_RELATIVE, + QueryMatchScopeMode.DOCUMENT, + ) + } + @Bean fun eventStore(stores: RecordingStores): EventStore = stores.mongoEventStore diff --git a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/mongo/MongoPlannedQueryBackendSourceTest.kt b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/mongo/MongoPlannedQueryBackendSourceTest.kt new file mode 100644 index 00000000000..5cddf2b356d --- /dev/null +++ b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/mongo/MongoPlannedQueryBackendSourceTest.kt @@ -0,0 +1,128 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.spring.boot.starter.mongo + +import com.mongodb.MongoNamespace +import com.mongodb.reactivestreams.client.MongoDatabase +import io.mockk.every +import io.mockk.mockk +import io.mockk.verify +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.mongo.Documents +import me.ahoo.wow.mongo.query.planned.MongoEventStreamQueryBinding +import me.ahoo.wow.mongo.query.planned.MongoFieldBinding +import me.ahoo.wow.mongo.query.planned.MongoSnapshotQueryBinding +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.spring.boot.starter.eventsourcing.StorageType +import me.ahoo.wow.spring.boot.starter.query.StorageRoutedQueryBackendComposition +import org.junit.jupiter.api.Test + +class MongoPlannedQueryBackendSourceTest { + @Test + fun `binding for a target routed elsewhere should not inspect the Mongo database`() { + val database = mockk() + val source = MongoPlannedQueryBackendSource.snapshot(database, listOf(binding)) + + val composition = StorageRoutedQueryBackendComposition.create(listOf(source)) { StorageType.ELASTICSEARCH } + + composition.contributions.assert().isEmpty() + composition.defaultRoutes.assert().isEmpty() + verify(exactly = 0) { database.name } + } + + @Test + fun `selected binding should require the exact Mongo database`() { + val database = mockk() + every { database.name } returns "other" + val source = MongoPlannedQueryBackendSource.snapshot(database, listOf(binding)) + + assertThrownBy { + StorageRoutedQueryBackendComposition.create(listOf(source)) { StorageType.MONGO } + } + } + + @Test + fun `event stream bindings should retain an independent target and database owner`() { + val database = mockk() + val source = MongoPlannedQueryBackendSource.eventStream(database, listOf(eventBinding)) + + source.targets.assert().containsExactly(eventTarget) + verify(exactly = 0) { database.name } + } + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) + private val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + private val schema = QueryDocumentSchema( + target, + listOf( + QueryFieldSchema( + identity, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT), + ), + ), + emptyList(), + ) + private val binding = MongoSnapshotQueryBinding( + schema, + MongoNamespace("sales", "order_snapshot"), + mapOf(identity to MongoFieldBinding(Documents.ID_FIELD, setOf(FieldCapability.EXACT))), + ) + private val eventTarget = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.EVENT_STREAM, + ) + private val eventSchema = QueryDocumentSchema( + eventTarget, + listOf( + QueryFieldSchema( + identity, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT), + ), + ), + emptyList(), + ) + private val eventBinding = MongoEventStreamQueryBinding( + eventSchema, + MongoNamespace("events", "order_event_stream"), + mapOf(identity to MongoFieldBinding(Documents.ID_FIELD, setOf(FieldCapability.EXACT))), + ) +} diff --git a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryAutoConfigurationTest.kt b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryAutoConfigurationTest.kt index c2565dd9ad0..b2a986c5e6d 100644 --- a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryAutoConfigurationTest.kt +++ b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryAutoConfigurationTest.kt @@ -3,10 +3,15 @@ package me.ahoo.wow.spring.boot.starter.query import io.mockk.every import io.mockk.spyk import me.ahoo.test.asserts.assert +import me.ahoo.wow.filter.FilterChain +import me.ahoo.wow.query.event.filter.EventStreamQueryFilter import me.ahoo.wow.query.event.filter.EventStreamQueryHandler import me.ahoo.wow.query.mask.EventStreamDynamicDocumentMasker +import me.ahoo.wow.query.mask.EventStreamMaskerRegistry +import me.ahoo.wow.query.mask.StateDataMaskerRegistry import me.ahoo.wow.query.mask.StateDynamicDocumentMasker import me.ahoo.wow.query.snapshot.filter.MaskingSnapshotQueryFilter +import me.ahoo.wow.query.snapshot.filter.SnapshotQueryFilter import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler import me.ahoo.wow.query.snapshot.filter.TailSnapshotQueryFilter import me.ahoo.wow.spring.boot.starter.enableWow @@ -18,6 +23,22 @@ import org.springframework.boot.test.context.runner.ApplicationContextRunner class QueryAutoConfigurationTest { private val contextRunner = ApplicationContextRunner() + @Test + fun `existing query auto configuration JVM descriptors should remain compatible`() { + QueryAutoConfiguration::class.java + .getDeclaredMethod("maskingSnapshotQueryFilter", StateDataMaskerRegistry::class.java) + .returnType.assert().isEqualTo(SnapshotQueryFilter::class.java) + QueryAutoConfiguration::class.java + .getDeclaredMethod("maskingEventStreamQueryFilter", EventStreamMaskerRegistry::class.java) + .returnType.assert().isEqualTo(EventStreamQueryFilter::class.java) + QueryAutoConfiguration::class.java + .getDeclaredMethod("snapshotQueryFilterChain", List::class.java) + .returnType.assert().isEqualTo(FilterChain::class.java) + QueryAutoConfiguration::class.java + .getDeclaredMethod("eventStreamQueryFilterChain", List::class.java) + .returnType.assert().isEqualTo(FilterChain::class.java) + } + @Test fun `should load context with query handler beans`() { contextRunner diff --git a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperAutoConfigurationTest.kt b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperAutoConfigurationTest.kt new file mode 100644 index 00000000000..0fa542f2405 --- /dev/null +++ b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperAutoConfigurationTest.kt @@ -0,0 +1,92 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.cursor.ExperimentalQueryCursorApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.spring.boot.starter.query + +import io.mockk.every +import io.mockk.mockk +import me.ahoo.test.asserts.assert +import me.ahoo.wow.query.cursor.QueryCursorLeaseConfiguration +import me.ahoo.wow.query.gateway.QueryGatewayRuntime +import me.ahoo.wow.spring.boot.starter.enableWow +import org.junit.jupiter.api.Test +import org.springframework.boot.autoconfigure.AutoConfigurations +import org.springframework.boot.test.context.runner.ApplicationContextRunner +import org.springframework.context.SmartLifecycle +import reactor.core.publisher.Mono +import java.time.Duration + +class QueryCursorReaperAutoConfigurationTest { + private val contextRunner = ApplicationContextRunner() + .enableWow() + .withConfiguration(AutoConfigurations.of(QueryCursorReaperAutoConfiguration::class.java)) + + @Test + fun `reaper should remain disabled by default`() { + contextRunner.run { context -> + context.assert().hasNotFailed() + context.containsBean(QueryCursorReaperAutoConfiguration.QUERY_CURSOR_REAPER_LIFECYCLE_BEAN_NAME) + .assert().isFalse() + } + } + + @Test + fun `enabled reaper should require a cursor lease configuration`() { + contextRunner + .withPropertyValues("${QueryCursorReaperProperties.PREFIX}.enabled=true") + .withBean(QueryGatewayRuntime::class.java, ::runtime) + .run { context -> + context.assert().hasFailed() + generateSequence(context.startupFailure, Throwable::cause) + .mapNotNull(Throwable::message) + .joinToString("\n") + .assert().contains(QueryCursorLeaseConfiguration::class.java.name) + } + } + + @Test + fun `enabled reaper should bind policy and own one running lifecycle`() { + contextRunner + .withPropertyValues( + "${QueryCursorReaperProperties.PREFIX}.enabled=true", + "${QueryCursorReaperProperties.PREFIX}.initial-delay=PT1H", + "${QueryCursorReaperProperties.PREFIX}.interval=PT2H", + "${QueryCursorReaperProperties.PREFIX}.batch-size=7", + "${QueryCursorReaperProperties.PREFIX}.max-batches-per-run=3", + ) + .withBean(QueryGatewayRuntime::class.java, ::runtime) + .withBean(QueryCursorLeaseConfiguration::class.java, { mockk() }) + .run { context -> + context.assert().hasNotFailed() + val lifecycle = context.getBean( + QueryCursorReaperAutoConfiguration.QUERY_CURSOR_REAPER_LIFECYCLE_BEAN_NAME, + SmartLifecycle::class.java, + ) + lifecycle.isRunning.assert().isTrue() + val properties = context.getBean(QueryCursorReaperProperties::class.java) + properties.initialDelay.assert().isEqualTo(Duration.ofHours(1)) + properties.interval.assert().isEqualTo(Duration.ofHours(2)) + properties.batchSize.assert().isEqualTo(7) + properties.maxBatchesPerRun.assert().isEqualTo(3) + } + } + + private fun runtime(): QueryGatewayRuntime = mockk { + every { reapExpiredQueryCursors(any()) } returns Mono.just(0) + } +} diff --git a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperLifecycleTest.kt b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperLifecycleTest.kt new file mode 100644 index 00000000000..fa8bc364d35 --- /dev/null +++ b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryCursorReaperLifecycleTest.kt @@ -0,0 +1,117 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.spring.boot.starter.query + +import me.ahoo.test.asserts.assert +import org.junit.jupiter.api.Test +import reactor.core.publisher.Mono +import reactor.test.scheduler.VirtualTimeScheduler +import java.time.Duration +import java.util.concurrent.atomic.AtomicInteger + +class QueryCursorReaperLifecycleTest { + @Test + fun `should drain only a bounded number of batches per scheduled run`() { + val scheduler = VirtualTimeScheduler.create() + val calls = AtomicInteger() + val lifecycle = QueryCursorReaperLifecycle( + reap = { + calls.incrementAndGet() + Mono.just(2) + }, + properties = QueryCursorReaperProperties( + enabled = true, + initialDelay = Duration.ofSeconds(1), + interval = Duration.ofSeconds(10), + batchSize = 2, + maxBatchesPerRun = 3, + ), + scheduler = scheduler, + ownsScheduler = false, + ) + + lifecycle.start() + scheduler.advanceTimeBy(Duration.ofSeconds(1)) + + lifecycle.isRunning.assert().isTrue() + calls.get().assert().isEqualTo(3) + + lifecycle.stop() + scheduler.advanceTimeBy(Duration.ofMinutes(1)) + lifecycle.isRunning.assert().isFalse() + calls.get().assert().isEqualTo(3) + } + + @Test + fun `should isolate one failed run and continue at the next interval`() { + val scheduler = VirtualTimeScheduler.create() + val calls = AtomicInteger() + val lifecycle = QueryCursorReaperLifecycle( + reap = { + if (calls.getAndIncrement() == 0) { + Mono.error(IllegalStateException("cursor store unavailable")) + } else { + Mono.just(0) + } + }, + properties = QueryCursorReaperProperties( + enabled = true, + initialDelay = Duration.ofSeconds(1), + interval = Duration.ofSeconds(10), + batchSize = 2, + maxBatchesPerRun = 3, + ), + scheduler = scheduler, + ownsScheduler = false, + ) + + lifecycle.start() + scheduler.advanceTimeBy(Duration.ofSeconds(1)) + calls.get().assert().isEqualTo(1) + + scheduler.advanceTimeBy(Duration.ofSeconds(10)) + calls.get().assert().isEqualTo(2) + lifecycle.isRunning.assert().isTrue() + + lifecycle.stop() + } + + @Test + fun `should stop after a short terminal batch`() { + val scheduler = VirtualTimeScheduler.create() + val calls = AtomicInteger() + val results = ArrayDeque(listOf(2L, 1L)) + val lifecycle = QueryCursorReaperLifecycle( + reap = { + calls.incrementAndGet() + Mono.just(results.removeFirst()) + }, + properties = QueryCursorReaperProperties( + enabled = true, + initialDelay = Duration.ofSeconds(1), + interval = Duration.ofSeconds(10), + batchSize = 2, + maxBatchesPerRun = 3, + ), + scheduler = scheduler, + ownsScheduler = false, + ) + + lifecycle.start() + scheduler.advanceTimeBy(Duration.ofSeconds(1)) + + calls.get().assert().isEqualTo(2) + lifecycle.stop() + } +} diff --git a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryGatewayAutoConfigurationTest.kt b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryGatewayAutoConfigurationTest.kt new file mode 100644 index 00000000000..9c988f37058 --- /dev/null +++ b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/QueryGatewayAutoConfigurationTest.kt @@ -0,0 +1,701 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.spring.boot.starter.query + +import io.micrometer.core.instrument.simple.SimpleMeterRegistry +import io.mockk.mockk +import me.ahoo.test.asserts.assert +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DynamicDocument +import me.ahoo.wow.api.query.IListQuery +import me.ahoo.wow.api.query.IPagedQuery +import me.ahoo.wow.api.query.ISingleQuery +import me.ahoo.wow.api.query.MaterializedSnapshot +import me.ahoo.wow.api.query.PagedList +import me.ahoo.wow.configuration.MetadataSearcher +import me.ahoo.wow.filter.FilterChain +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.modeling.annotation.aggregateMetadata +import me.ahoo.wow.query.analytics.AnalyticsQueryService +import me.ahoo.wow.query.analytics.AnalyticsQueryServiceFactory +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.BackendStreamSupport +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.RecordQueryBackend +import me.ahoo.wow.query.backend.RecordQueryBackendContribution +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.event.EventStreamQueryServiceFactory +import me.ahoo.wow.query.filter.PreAdmissionQueryFilter +import me.ahoo.wow.query.filter.QueryContext +import me.ahoo.wow.query.gateway.AnalyticsQueryGateway +import me.ahoo.wow.query.gateway.GatewayEventStreamQueryServiceFactory +import me.ahoo.wow.query.gateway.GatewaySnapshotQueryServiceFactory +import me.ahoo.wow.query.gateway.QueryAuthority +import me.ahoo.wow.query.gateway.QueryAuthorityResolver +import me.ahoo.wow.query.gateway.QueryCall +import me.ahoo.wow.query.gateway.QueryCallResolver +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryExecutionException +import me.ahoo.wow.query.gateway.QueryExecutionMode +import me.ahoo.wow.query.gateway.QueryExecutionProfile +import me.ahoo.wow.query.gateway.QueryExecutionProfiles +import me.ahoo.wow.query.gateway.QueryGateway +import me.ahoo.wow.query.gateway.QueryLegacyContextResolver +import me.ahoo.wow.query.gateway.QueryLegacyGrant +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryOperationProfileKey +import me.ahoo.wow.query.gateway.QueryPurpose +import me.ahoo.wow.query.gateway.QueryRawServiceSource +import me.ahoo.wow.query.gateway.QueryResourceScope +import me.ahoo.wow.query.gateway.QueryRuntimeHealthObserver +import me.ahoo.wow.query.gateway.QueryShadowObservation +import me.ahoo.wow.query.gateway.QueryShadowObserver +import me.ahoo.wow.query.gateway.QueryShadowOutcome +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.gateway.QueryTrustedContextResolver +import me.ahoo.wow.query.gateway.QueryValidationMode +import me.ahoo.wow.query.gateway.withLegacyQueryCaller +import me.ahoo.wow.query.snapshot.SnapshotQueryService +import me.ahoo.wow.query.snapshot.SnapshotQueryServiceFactory +import me.ahoo.wow.query.snapshot.filter.SnapshotQueryFilter +import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.spring.boot.starter.enableWow +import me.ahoo.wow.spring.boot.starter.eventsourcing.StorageType +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.SnapshotQueryServiceFactoryBinding +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.StorageRoutingAutoConfiguration +import me.ahoo.wow.webflux.exception.WebFluxRequestExceptionHandler +import me.ahoo.wow.webflux.route.query.CountQueryHandlerFunction +import me.ahoo.wow.webflux.route.query.DefaultRewriteRequestCondition +import me.ahoo.wow.webflux.route.query.QueryWebTransportResolvers +import org.junit.jupiter.api.Test +import org.springframework.boot.autoconfigure.AutoConfigurations +import org.springframework.boot.test.context.assertj.AssertableApplicationContext +import org.springframework.boot.test.context.runner.ApplicationContextRunner +import org.springframework.context.annotation.Bean +import org.springframework.context.annotation.Configuration +import org.springframework.core.ResolvableType +import org.springframework.core.env.Environment +import org.springframework.http.HttpStatus +import org.springframework.mock.http.server.reactive.MockServerHttpRequest +import org.springframework.mock.web.reactive.function.server.MockServerRequest +import org.springframework.mock.web.server.MockServerWebExchange +import org.springframework.web.reactive.function.server.HandlerStrategies +import org.springframework.web.reactive.function.server.ServerResponse +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.kotlin.test.test +import reactor.test.StepVerifier +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit +import java.util.concurrent.atomic.AtomicInteger + +class QueryGatewayAutoConfigurationTest { + private val contextRunner = ApplicationContextRunner() + .enableWow() + .withConfiguration( + AutoConfigurations.of( + QueryAutoConfiguration::class.java, + StorageRoutingAutoConfiguration::class.java, + QueryGatewayAutoConfiguration::class.java, + QueryGatewayLegacyWiringRollbackAutoConfiguration::class.java, + ), + ) + + @Test + fun `framework managed factories and aggregate bean should use gateway facade`() { + contextRunner + .withUserConfiguration(TrustedQueryContextConfiguration::class.java) + .run { context: AssertableApplicationContext -> + context.assert().hasNotFailed() + context.getBean(SnapshotQueryServiceFactory::class.java) + .assert().isInstanceOf(GatewaySnapshotQueryServiceFactory::class.java) + context.getBean(EventStreamQueryServiceFactory::class.java) + .assert().isInstanceOf(GatewayEventStreamQueryServiceFactory::class.java) + context.getBean(AnalyticsQueryGateway::class.java).assert().isNotNull() + context.getBean(QueryRawServiceSource::class.java) + .assert().isInstanceOf(StorageBindingQueryRawServiceRegistry::class.java) + (context.getBean(QueryRawServiceSource::class.java) is SnapshotQueryServiceFactory) + .assert().isFalse() + + val aggregateBean = context.getBean(SNAPSHOT_QUERY_SERVICE_BEAN, SnapshotQueryService::class.java) + val aggregateType = MetadataSearcher.namedAggregateType.getValue(ORDER) + .aggregateMetadata().state.aggregateType + val genericType = ResolvableType.forClassWithGenerics( + SnapshotQueryService::class.java, + aggregateType, + ) + context.getBeanProvider(genericType).getObject().assert().isSameAs(aggregateBean) + context.getBean(SnapshotQueryServiceFactory::class.java).create(ORDER) + .assert().isSameAs(aggregateBean) + val analyticsBean = context.getBean(ANALYTICS_QUERY_SERVICE_BEAN, AnalyticsQueryService::class.java) + context.getBean(AnalyticsQueryServiceFactory::class.java).create(ORDER) + .namedAggregate.assert().isEqualTo(analyticsBean.namedAggregate) + + StepVerifier.create(aggregateBean.count(Condition.all())) + .expectNext(0) + .verifyComplete() + StepVerifier.create(context.getBean(SnapshotQueryHandler::class.java).count(ORDER, Condition.all())) + .expectNext(0) + .verifyComplete() + } + } + + @Test + fun `default facade should fail closed when trusted call context is missing`() { + contextRunner.run { context: AssertableApplicationContext -> + val queryService = context.getBean(SnapshotQueryServiceFactory::class.java).create(ORDER) + + StepVerifier.create(queryService.count(Condition.all())) + .expectErrorSatisfies { error -> + error.assert().isInstanceOf(QueryExecutionException::class.java) + (error as QueryExecutionException).code.assert().isEqualTo("QUERY_CALL_REQUIRED") + error.path.assert().isEqualTo("$.executionContext.call") + } + .verify() + } + } + + @Test + fun `pre-admission filter cannot replace the gateway result`() { + contextRunner + .withUserConfiguration( + TrustedQueryContextConfiguration::class.java, + ResultReplacingPreAdmissionFilterConfiguration::class.java, + ) + .run { context: AssertableApplicationContext -> + context.assert().hasNotFailed() + StepVerifier.create(context.getBean(SnapshotQueryHandler::class.java).count(ORDER, Condition.all())) + .expectNext(0) + .verifyComplete() + } + } + + @Test + fun `undeclared legacy query filter phase should fail startup`() { + contextRunner + .withUserConfiguration(UndeclaredQueryFilterConfiguration::class.java) + .run { context: AssertableApplicationContext -> + context.assert().hasFailed() + context.startupFailure!!.message.assert().contains("QueryFilter must declare the pre-admission phase") + } + } + + @Test + fun `process internal facade should require an exact registered legacy grant`() { + contextRunner + .withUserConfiguration(LegacyQueryGrantConfiguration::class.java) + .run { context: AssertableApplicationContext -> + val service = context.getBean(SnapshotQueryServiceFactory::class.java).create(ORDER) + service.count(Condition.all()) + .withLegacyQueryCaller("compensation-retry") + .test() + .expectNext(0) + .verifyComplete() + + service.count(Condition.all()) + .withLegacyQueryCaller("another-caller") + .test() + .expectErrorSatisfies { error -> + error.assert().isInstanceOf(QueryExecutionException::class.java) + (error as QueryExecutionException).code.assert().isEqualTo("LEGACY_CALLER_NOT_ALLOWED") + } + .verify() + } + } + + @Test + fun `direct process resolvers should remain available beside transport resolvers`() { + contextRunner + .withUserConfiguration( + EmptyTrustedContextConfiguration::class.java, + TrustedQueryContextConfiguration::class.java, + ) + .run { context: AssertableApplicationContext -> + val service = context.getBean(SnapshotQueryServiceFactory::class.java).create(ORDER) + + StepVerifier.create(service.count(Condition.all())) + .expectNext(0) + .verifyComplete() + } + } + + @Test + fun `authority-only resolver should serve direct gateway without becoming a facade pair`() { + contextRunner + .withUserConfiguration(AuthorityOnlyConfiguration::class.java) + .run { context: AssertableApplicationContext -> + context.assert().hasNotFailed() + StepVerifier.create( + context.getBean(QueryGateway::class.java).count( + QueryCall(QueryTarget(ORDER, QueryDocumentKind.SNAPSHOT), PURPOSE), + Condition.all(), + ), + ) + .expectNext(0) + .verifyComplete() + + StepVerifier.create( + context.getBean(SnapshotQueryServiceFactory::class.java) + .create(ORDER) + .count(Condition.all()), + ) + .expectErrorSatisfies { error -> + error.assert().isInstanceOf(QueryExecutionException::class.java) + (error as QueryExecutionException).code.assert().isEqualTo("QUERY_CALL_REQUIRED") + } + .verify() + } + } + + @Test + fun `web route should cross the runtime-owned authority channel before raw storage`() { + CountingSnapshotQueryService.countCalls.set(0) + contextRunner + .withUserConfiguration( + CountingRawSnapshotConfiguration::class.java, + WebTrustedContextConfiguration::class.java, + ) + .run { context: AssertableApplicationContext -> + context.assert().hasNotFailed() + writeCountRoute(context).response.statusCode.assert().isEqualTo(HttpStatus.OK) + CountingSnapshotQueryService.countCalls.get().assert().isEqualTo(1) + } + } + + @Test + fun `missing web authority should stop before raw storage`() { + CountingSnapshotQueryService.countCalls.set(0) + contextRunner + .withUserConfiguration( + CountingRawSnapshotConfiguration::class.java, + MissingWebAuthorityConfiguration::class.java, + ) + .run { context: AssertableApplicationContext -> + context.assert().hasNotFailed() + writeCountRoute(context).response.statusCode.assert().isEqualTo(HttpStatus.FORBIDDEN) + CountingSnapshotQueryService.countCalls.get().assert().isZero() + } + } + + @Test + fun `example order should rehearse storage routed shadow cut over and rollback through one facade`() { + listOf( + QueryExecutionMode.LEGACY to ExpectedRolloutCalls(raw = 1, planned = 0, prepared = 0), + QueryExecutionMode.SHADOW to ExpectedRolloutCalls(raw = 1, planned = 1, prepared = 1), + QueryExecutionMode.PLANNED to ExpectedRolloutCalls(raw = 0, planned = 1, prepared = 1), + ).forEach { (mode, expected) -> + CountingSnapshotQueryService.countCalls.set(0) + contextRunner + .withPropertyValues("$ROLLOUT_MODE_PROPERTY=${mode.name}") + .withUserConfiguration( + CountingRawSnapshotConfiguration::class.java, + WebTrustedContextConfiguration::class.java, + ExampleOrderRolloutConfiguration::class.java, + ) + .run { context: AssertableApplicationContext -> + context.assert().hasNotFailed() + val probe = context.getBean(ExampleOrderRolloutProbe::class.java) + + val exchange = writeCountRoute(context) + + exchange.response.statusCode.assert().isEqualTo(HttpStatus.OK) + exchange.response.bodyAsString.block().assert().isEqualTo("7") + if (mode == QueryExecutionMode.SHADOW) { + probe.observed.await(5, TimeUnit.SECONDS).assert().isTrue() + probe.observation!!.outcome.assert().isEqualTo(QueryShadowOutcome.MATCH) + probe.observation!!.target.assert().isEqualTo(SNAPSHOT_TARGET) + probe.observation!!.operation.assert().isEqualTo(QueryOperation.COUNT) + } else { + probe.observation.assert().isNull() + } + CountingSnapshotQueryService.countCalls.get().assert().isEqualTo(expected.raw) + probe.plannedCalls.get().assert().isEqualTo(expected.planned) + probe.prepareCalls.get().assert().isEqualTo(expected.prepared) + } + } + } + + @Test + fun `partial custom gateway override should fail with a stable diagnostic`() { + contextRunner + .withUserConfiguration(CustomGatewayConfiguration::class.java) + .run { context: AssertableApplicationContext -> + context.assert().hasFailed() + generateSequence(context.startupFailure, Throwable::cause) + .mapNotNull(Throwable::message) + .joinToString("\n") + .assert().contains("Provide one complete QueryGatewayRuntime override instead") + } + } + + @Test + fun `partial custom analytics gateway override should fail with a stable diagnostic`() { + contextRunner + .withUserConfiguration(CustomAnalyticsGatewayConfiguration::class.java) + .run { context: AssertableApplicationContext -> + context.assert().hasFailed() + generateSequence(context.startupFailure, Throwable::cause) + .mapNotNull(Throwable::message) + .joinToString("\n") + .assert().contains("Provide one complete QueryGatewayRuntime override instead") + } + } + + @Test + fun `explicit legacy wiring rollback should bypass gateway and record activation metric`() { + contextRunner + .withPropertyValues("$QUERY_GATEWAY_LEGACY_WIRING_ROLLBACK_KEY=true") + .withUserConfiguration(RollbackMetricsConfiguration::class.java) + .run { context: AssertableApplicationContext -> + context.assert().hasNotFailed() + context.getBeansOfType(QueryGateway::class.java).assert().isEmpty() + context.getBean(SnapshotQueryServiceFactory::class.java) + .assert().isNotInstanceOf(GatewaySnapshotQueryServiceFactory::class.java) + context.getBean(EventStreamQueryServiceFactory::class.java) + .assert().isNotInstanceOf(GatewayEventStreamQueryServiceFactory::class.java) + + StepVerifier.create( + context.getBean(SnapshotQueryServiceFactory::class.java) + .create(ORDER) + .count(Condition.all()), + ) + .expectNext(0) + .verifyComplete() + + context.getBean(SimpleMeterRegistry::class.java) + .find("wow.query.gateway.legacy.wiring.rollback") + .counter()!! + .count() + .assert().isEqualTo(1.0) + } + } + + @Test + fun `invalid legacy wiring rollback value should fail startup`() { + contextRunner + .withPropertyValues("$QUERY_GATEWAY_LEGACY_WIRING_ROLLBACK_KEY=treu") + .run { context: AssertableApplicationContext -> + context.assert().hasFailed() + generateSequence(context.startupFailure, Throwable::cause) + .mapNotNull(Throwable::message) + .joinToString("\n") + .assert().contains("must be exactly true or false") + } + } + + @Configuration(proxyBeanMethods = false) + class TrustedQueryContextConfiguration { + @Bean + fun queryCallResolver(): QueryCallResolver = QueryCallResolver { request -> + Mono.just(QueryCall(request.target, PURPOSE)) + } + + @Bean + fun queryAuthorityResolver(): QueryAuthorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.System("query-gateway-test", "Spring facade test")) + } + } + + @Configuration(proxyBeanMethods = false) + class ResultReplacingPreAdmissionFilterConfiguration { + @Bean + fun resultReplacingPreAdmissionFilter(): SnapshotQueryFilter = + object : SnapshotQueryFilter, PreAdmissionQueryFilter { + override fun filter( + context: QueryContext<*, *>, + next: FilterChain>, + ): Mono = next.filter(context).then( + Mono.fromRunnable { + context.asCountQuery().setResult(Mono.just(999)) + }, + ) + } + } + + @Configuration(proxyBeanMethods = false) + class EmptyTrustedContextConfiguration { + @Bean + fun emptyTrustedContextResolver(): QueryTrustedContextResolver = QueryTrustedContextResolver { Mono.empty() } + } + + @Configuration(proxyBeanMethods = false) + class AuthorityOnlyConfiguration { + @Bean + fun queryAuthorityResolver(): QueryAuthorityResolver = QueryAuthorityResolver { + Mono.just(QueryAuthority.System("direct-query-gateway-test", "Direct Gateway test")) + } + } + + @Configuration(proxyBeanMethods = false) + class CustomGatewayConfiguration { + @Bean + fun customQueryGateway(): QueryGateway = mockk(relaxed = true) + } + + @Configuration(proxyBeanMethods = false) + class CustomAnalyticsGatewayConfiguration { + @Bean + fun customAnalyticsQueryGateway(): AnalyticsQueryGateway = mockk(relaxed = true) + } + + @Configuration(proxyBeanMethods = false) + class CountingRawSnapshotConfiguration { + @Bean + fun countingSnapshotQueryServiceFactoryBinding(): SnapshotQueryServiceFactoryBinding = + SnapshotQueryServiceFactoryBinding.storage(StorageType.MONGO, CountingSnapshotQueryServiceFactory) + } + + @Configuration(proxyBeanMethods = false) + class WebTrustedContextConfiguration { + @Bean + fun webQueryTrustedContextResolver(): QueryTrustedContextResolver = QueryWebTransportResolvers { + Mono.just(QueryAuthority.System("web-query-test", "Web vertical slice")) + } + } + + @Configuration(proxyBeanMethods = false) + class MissingWebAuthorityConfiguration { + @Bean + fun webQueryTrustedContextResolver(): QueryTrustedContextResolver = QueryWebTransportResolvers { Mono.empty() } + } + + @Configuration(proxyBeanMethods = false) + class ExampleOrderRolloutConfiguration { + @Bean + fun exampleOrderRolloutProbe(): ExampleOrderRolloutProbe = ExampleOrderRolloutProbe() + + @Bean + internal fun exampleOrderPlannedBackendSource( + probe: ExampleOrderRolloutProbe, + ): StorageQueryBackendSource = object : StorageQueryBackendSource { + override val storage: StorageType = StorageType.MONGO + override val targets: Set = setOf(SNAPSHOT_TARGET) + + override fun prepare(target: QueryTarget): Mono = Mono.fromSupplier { + target.assert().isEqualTo(SNAPSHOT_TARGET) + probe.prepareCalls.incrementAndGet() + StorageQueryBackendPreparation.Ready(exampleOrderContribution(probe)) + } + } + + @Bean + fun queryExecutionProfiles(environment: Environment): QueryExecutionProfiles { + val mode = environment.getRequiredProperty(ROLLOUT_MODE_PROPERTY, QueryExecutionMode::class.java) + return QueryExecutionProfiles( + operationProfiles = mapOf( + QueryOperationProfileKey(SNAPSHOT_TARGET, QueryOperation.COUNT) to + QueryExecutionProfile(mode, QueryValidationMode.STRICT), + ), + ) + } + + @Bean + fun queryShadowObserver(probe: ExampleOrderRolloutProbe): QueryShadowObserver = + QueryShadowObserver { observation -> + probe.observation = observation + probe.observed.countDown() + } + + @Bean + fun queryRuntimeHealthObserver(): QueryRuntimeHealthObserver = QueryRuntimeHealthObserver { } + + private fun exampleOrderContribution(probe: ExampleOrderRolloutProbe): RecordQueryBackendContribution = + RecordQueryBackendContribution( + schema = EXAMPLE_ORDER_SCHEMA, + backendId = BackendId("example-order-mongo"), + supportedOperations = setOf(QueryOperation.COUNT), + streamSupport = BackendStreamSupport.NONE, + semanticTiers = setOf(SemanticTier.PORTABLE), + fieldCapabilities = EXAMPLE_ORDER_SCHEMA.fields.mapValues { (_, field) -> field.capabilities }, + backend = object : RecordQueryBackend { + override fun single( + plan: me.ahoo.wow.query.backend.BackendSingleQueryPlan, + options: QueryBackendExecutionOptions, + ) = Mono.empty() + + override fun stream( + plan: me.ahoo.wow.query.backend.BackendStreamQueryPlan, + options: QueryBackendExecutionOptions, + ) = Flux.empty() + + override fun count( + plan: BackendCountQueryPlan, + options: QueryBackendExecutionOptions, + ): Mono = Mono.fromSupplier { + plan.target.assert().isEqualTo(SNAPSHOT_TARGET) + probe.plannedCalls.incrementAndGet() + 7L + } + }, + ) + } + + class ExampleOrderRolloutProbe { + val plannedCalls = AtomicInteger() + val prepareCalls = AtomicInteger() + val observed = CountDownLatch(1) + + @Volatile + var observation: QueryShadowObservation? = null + } + + @Configuration(proxyBeanMethods = false) + class UndeclaredQueryFilterConfiguration { + @Bean + fun undeclaredQueryFilter(): SnapshotQueryFilter = object : SnapshotQueryFilter { + override fun filter( + context: QueryContext<*, *>, + next: FilterChain>, + ): Mono = next.filter(context) + } + } + + @Configuration(proxyBeanMethods = false) + class LegacyQueryGrantConfiguration { + @Bean + fun legacyQueryContextResolver(): QueryLegacyContextResolver = QueryLegacyContextResolver( + listOf( + QueryLegacyGrant( + callerId = "compensation-retry", + target = QueryTarget(ORDER, QueryDocumentKind.SNAPSHOT), + purpose = QueryPurpose("compensation-retry"), + executionMode = QueryExecutionMode.LEGACY, + resourceScope = QueryResourceScope(), + ), + ), + ) + } + + @Configuration(proxyBeanMethods = false) + class RollbackMetricsConfiguration { + @Bean + fun meterRegistry(): SimpleMeterRegistry = SimpleMeterRegistry() + } + + private fun writeCountRoute(context: AssertableApplicationContext): MockServerWebExchange { + val aggregateMetadata = MetadataSearcher.namedAggregateType.getValue(ORDER) + .aggregateMetadata() + val handler = CountQueryHandlerFunction( + aggregateMetadata, + context.getBean(SnapshotQueryHandler::class.java), + QueryDocumentKind.SNAPSHOT, + DefaultRewriteRequestCondition, + WebFluxRequestExceptionHandler(), + ) + val request = MockServerRequest.builder() + .pathVariable(MessageRecords.TENANT_ID, "tenant-1") + .body(Mono.just(Condition.all())) + val exchange = MockServerWebExchange.from(MockServerHttpRequest.post("/query").build()) + handler.handle(request) + .flatMap { response -> response.writeTo(exchange, SERVER_RESPONSE_CONTEXT) } + .block() + return exchange + } + + private object CountingSnapshotQueryServiceFactory : SnapshotQueryServiceFactory { + @Suppress("UNCHECKED_CAST") + override fun create(namedAggregate: NamedAggregate): SnapshotQueryService = + CountingSnapshotQueryService(namedAggregate) as SnapshotQueryService + } + + private class CountingSnapshotQueryService( + override val namedAggregate: NamedAggregate, + ) : SnapshotQueryService { + override val name: String = "counting-snapshot-query" + + override fun single(singleQuery: ISingleQuery): Mono> = Mono.empty() + + override fun dynamicSingle(singleQuery: ISingleQuery): Mono = Mono.empty() + + override fun list(listQuery: IListQuery): Flux> = Flux.empty() + + override fun dynamicList(listQuery: IListQuery): Flux = Flux.empty() + + override fun paged(pagedQuery: IPagedQuery): Mono>> = + Mono.just(PagedList.empty()) + + override fun dynamicPaged(pagedQuery: IPagedQuery): Mono> = + Mono.just(PagedList.empty()) + + override fun count(condition: Condition): Mono = Mono.fromSupplier { + countCalls.incrementAndGet() + 7L + } + + companion object { + val countCalls = AtomicInteger() + } + } + + private companion object { + val ORDER = MaterializedNamedAggregate("example-service", "order") + val SNAPSHOT_TARGET = QueryTarget(ORDER, QueryDocumentKind.SNAPSHOT) + val EXAMPLE_ORDER_SCHEMA = QueryDocumentSchema( + SNAPSHOT_TARGET, + listOf( + QueryFieldSchema( + QueryFieldId.System(SystemFieldKind.TENANT_ID), + LogicalFieldType.Text, + Presence.OPTIONAL, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT), + ), + QueryFieldSchema( + QueryFieldId.System(SystemFieldKind.DELETED), + LogicalFieldType.Boolean, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.IS_FALSE), + setOf(FieldCapability.EXACT), + ), + ), + emptyList(), + ) + val PURPOSE = QueryPurpose("spring-facade-test") + const val ROLLOUT_MODE_PROPERTY = "test.query.rollout-mode" + const val SNAPSHOT_QUERY_SERVICE_BEAN = "example.order.SnapshotQueryService" + const val ANALYTICS_QUERY_SERVICE_BEAN = "example.order.AnalyticsQueryService" + val SERVER_RESPONSE_CONTEXT = object : ServerResponse.Context { + private val strategies = HandlerStrategies.withDefaults() + + override fun messageWriters() = strategies.messageWriters() + + override fun viewResolvers() = strategies.viewResolvers() + } + } + + private data class ExpectedRolloutCalls( + val raw: Int, + val planned: Int, + val prepared: Int, + ) +} diff --git a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/StorageBindingQueryRawServiceRegistryTest.kt b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/StorageBindingQueryRawServiceRegistryTest.kt new file mode 100644 index 00000000000..6ded21b7deb --- /dev/null +++ b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/StorageBindingQueryRawServiceRegistryTest.kt @@ -0,0 +1,143 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.spring.boot.starter.query + +import io.mockk.mockk +import me.ahoo.test.asserts.assert +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.event.EventStreamQueryService +import me.ahoo.wow.query.event.EventStreamQueryServiceFactory +import me.ahoo.wow.query.event.NoOpEventStreamQueryService +import me.ahoo.wow.query.gateway.GatewaySnapshotQueryServiceFactory +import me.ahoo.wow.query.gateway.QueryCallResolver +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryElementPathMode +import me.ahoo.wow.query.gateway.QueryGateway +import me.ahoo.wow.query.gateway.QueryMatchScopeMode +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.snapshot.NoOpSnapshotQueryService +import me.ahoo.wow.query.snapshot.SnapshotQueryService +import me.ahoo.wow.query.snapshot.SnapshotQueryServiceFactory +import me.ahoo.wow.spring.boot.starter.eventsourcing.StorageType +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.AggregateStorageRouteProperties +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.EventStreamQueryServiceFactoryBinding +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.SnapshotQueryServiceFactoryBinding +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.StorageChannelRouteProperties +import me.ahoo.wow.spring.boot.starter.eventsourcing.routing.StorageRoutingProperties +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.assertThrows +import reactor.core.publisher.Mono + +class StorageBindingQueryRawServiceRegistryTest { + @Test + fun `exact target should resolve raw storage and matching dialect`() { + val mongoSnapshot = RecordingSnapshotFactory() + val elasticsearchSnapshot = RecordingSnapshotFactory() + val mongoEvent = RecordingEventFactory() + val elasticsearchEvent = RecordingEventFactory() + val registry = registry( + properties = StorageRoutingProperties( + aggregates = mapOf( + "order" to AggregateStorageRouteProperties( + event = StorageChannelRouteProperties(storage = StorageType.ELASTICSEARCH), + snapshot = StorageChannelRouteProperties(storage = StorageType.ELASTICSEARCH), + ), + ), + ), + eventBindings = listOf( + EventStreamQueryServiceFactoryBinding.storage(StorageType.MONGO, mongoEvent), + EventStreamQueryServiceFactoryBinding.storage(StorageType.ELASTICSEARCH, elasticsearchEvent), + ), + snapshotBindings = listOf( + SnapshotQueryServiceFactoryBinding.storage(StorageType.MONGO, mongoSnapshot), + SnapshotQueryServiceFactoryBinding.storage(StorageType.ELASTICSEARCH, elasticsearchSnapshot), + ), + ) + + registry.snapshot(ORDER) + elasticsearchSnapshot.lastTarget.assert().isEqualTo(ORDER) + registry.eventStream(ORDER) + elasticsearchEvent.lastTarget.assert().isEqualTo(ORDER) + + registry.snapshot(CART) + mongoSnapshot.lastTarget.assert().isEqualTo(CART) + registry.eventStream(CART) + mongoEvent.lastTarget.assert().isEqualTo(CART) + + val snapshotDialect = registry.resolveDialect(QueryTarget(ORDER, QueryDocumentKind.SNAPSHOT)) + snapshotDialect.elementPathMode.assert().isEqualTo(QueryElementPathMode.ROOT_QUALIFIED) + snapshotDialect.matchScopeMode.assert().isEqualTo(QueryMatchScopeMode.FIELD) + val eventDialect = registry.resolveDialect(QueryTarget(CART, QueryDocumentKind.EVENT_STREAM)) + eventDialect.elementPathMode.assert().isEqualTo(QueryElementPathMode.CURRENT_ELEMENT_RELATIVE) + eventDialect.matchScopeMode.assert().isEqualTo(QueryMatchScopeMode.DOCUMENT) + } + + @Test + fun `gateway facade can not be registered as raw storage`() { + val facade = GatewaySnapshotQueryServiceFactory( + mockk(), + QueryCallResolver { Mono.empty() }, + ) + + val exception = assertThrows { + registry( + snapshotBindings = listOf( + SnapshotQueryServiceFactoryBinding.storage(StorageType.MONGO, facade), + ), + ) + } + + exception.message.assert().contains("cannot be registered as a raw snapshot query binding") + } + + private fun registry( + properties: StorageRoutingProperties = StorageRoutingProperties(), + eventBindings: List = emptyList(), + snapshotBindings: List = emptyList(), + ): StorageBindingQueryRawServiceRegistry = StorageBindingQueryRawServiceRegistry( + contextName = "order-service", + storageRoutingProperties = properties, + eventStreamBindings = eventBindings, + snapshotBindings = snapshotBindings, + snapshotEnabled = true, + defaultEventStorage = StorageType.MONGO, + defaultSnapshotStorage = StorageType.MONGO, + ) + + private class RecordingSnapshotFactory : SnapshotQueryServiceFactory { + var lastTarget: NamedAggregate? = null + + override fun create(namedAggregate: NamedAggregate): SnapshotQueryService { + lastTarget = namedAggregate + return NoOpSnapshotQueryService(namedAggregate) + } + } + + private class RecordingEventFactory : EventStreamQueryServiceFactory { + var lastTarget: NamedAggregate? = null + + override fun create(namedAggregate: NamedAggregate): EventStreamQueryService { + lastTarget = namedAggregate + return NoOpEventStreamQueryService(namedAggregate) + } + } + + private companion object { + val ORDER = MaterializedNamedAggregate("order-service", "order") + val CART = MaterializedNamedAggregate("order-service", "cart") + } +} diff --git a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/StorageRoutedQueryBackendCompositionTest.kt b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/StorageRoutedQueryBackendCompositionTest.kt new file mode 100644 index 00000000000..f7ff8ab0796 --- /dev/null +++ b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/query/StorageRoutedQueryBackendCompositionTest.kt @@ -0,0 +1,175 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn( + me.ahoo.wow.query.backend.ExperimentalQueryBackendApi::class, + me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class, +) + +package me.ahoo.wow.spring.boot.starter.query + +import me.ahoo.test.asserts.assert +import me.ahoo.test.asserts.assertThrownBy +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.query.backend.BackendCountQueryPlan +import me.ahoo.wow.query.backend.BackendId +import me.ahoo.wow.query.backend.BackendStreamSupport +import me.ahoo.wow.query.backend.FieldCapability +import me.ahoo.wow.query.backend.LogicalFieldType +import me.ahoo.wow.query.backend.Nullability +import me.ahoo.wow.query.backend.PredicateOperator +import me.ahoo.wow.query.backend.Presence +import me.ahoo.wow.query.backend.QueryBackendExecutionOptions +import me.ahoo.wow.query.backend.QueryDocumentSchema +import me.ahoo.wow.query.backend.QueryFieldId +import me.ahoo.wow.query.backend.QueryFieldSchema +import me.ahoo.wow.query.backend.RecordQueryBackend +import me.ahoo.wow.query.backend.RecordQueryBackendContribution +import me.ahoo.wow.query.backend.SemanticTier +import me.ahoo.wow.query.backend.SystemFieldKind +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryOperation +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.spring.boot.starter.eventsourcing.StorageType +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import java.util.concurrent.atomic.AtomicInteger + +class StorageRoutedQueryBackendCompositionTest { + @Test + fun `only the source selected by the aggregate storage route should be prepared`() { + val mongoCalls = AtomicInteger() + val elasticsearchCalls = AtomicInteger() + val mongo = source(StorageType.MONGO, target, mongoCalls) + val elasticsearch = source(StorageType.ELASTICSEARCH, target, elasticsearchCalls) + + val composition = StorageRoutedQueryBackendComposition.create(listOf(mongo, elasticsearch)) { + StorageType.MONGO + } + + composition.contributions.assert().hasSize(1) + composition.defaultRoutes.getValue(target).assert().isEqualTo(BackendId("mongo")) + mongoCalls.get().assert().isEqualTo(1) + elasticsearchCalls.get().assert().isZero() + } + + @Test + fun `duplicate target and storage source should fail before preparation`() { + val calls = AtomicInteger() + + assertThrownBy { + StorageRoutedQueryBackendComposition.create( + listOf(source(StorageType.MONGO, target, calls), source(StorageType.MONGO, target, calls)), + ) { StorageType.MONGO } + } + calls.get().assert().isZero() + } + + @Test + fun `legacy-only target should not inspect planned backend readiness`() { + val calls = AtomicInteger() + + val composition = StorageRoutedQueryBackendComposition.create( + listOf(source(StorageType.MONGO, target, calls)), + storageResolver = { StorageType.MONGO }, + shouldPrepare = { false }, + ) + + composition.contributions.assert().isEmpty() + composition.notReadyBackends.assert().isEmpty() + composition.defaultRoutes.assert().isEmpty() + calls.get().assert().isZero() + } + + @Test + fun `configured but not ready source should preserve its exact route without a ready contribution`() { + val source = object : StorageQueryBackendSource { + override val storage: StorageType = StorageType.MONGO + override val targets: Set = setOf(target) + + override fun prepare(target: QueryTarget): Mono = Mono.just( + StorageQueryBackendPreparation.NotReady(schema(target), BackendId("mongo")), + ) + } + + val composition = StorageRoutedQueryBackendComposition.create(listOf(source)) { StorageType.MONGO } + + composition.contributions.assert().isEmpty() + composition.notReadyBackends.assert().hasSize(1) + composition.defaultRoutes.assert().containsEntry(target, BackendId("mongo")) + } + + private fun source( + storage: StorageType, + target: QueryTarget, + calls: AtomicInteger, + ): StorageQueryBackendSource = object : StorageQueryBackendSource { + override val storage: StorageType = storage + override val targets: Set = setOf(target) + + override fun prepare(target: QueryTarget): Mono = Mono.fromSupplier { + calls.incrementAndGet() + StorageQueryBackendPreparation.Ready(contribution(target, BackendId(storage.name.lowercase()))) + } + } + + private fun contribution(target: QueryTarget, backendId: BackendId): RecordQueryBackendContribution { + val schema = schema(target) + val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + return RecordQueryBackendContribution( + schema, + backendId, + setOf(QueryOperation.COUNT), + BackendStreamSupport.NONE, + setOf(SemanticTier.PORTABLE), + mapOf(identity to setOf(FieldCapability.EXACT)), + backend = object : RecordQueryBackend { + override fun single( + plan: me.ahoo.wow.query.backend.BackendSingleQueryPlan, + options: QueryBackendExecutionOptions, + ) = Mono.empty() + + override fun stream( + plan: me.ahoo.wow.query.backend.BackendStreamQueryPlan, + options: QueryBackendExecutionOptions, + ) = Flux.empty() + + override fun count(plan: BackendCountQueryPlan, options: QueryBackendExecutionOptions) = Mono.just(0L) + }, + ) + } + + private fun schema(target: QueryTarget): QueryDocumentSchema { + val identity = QueryFieldId.System(SystemFieldKind.IDENTITY) + return QueryDocumentSchema( + target, + listOf( + QueryFieldSchema( + identity, + LogicalFieldType.Text, + Presence.REQUIRED, + Nullability.NON_NULL, + setOf(PredicateOperator.EQ), + setOf(FieldCapability.EXACT), + ), + ), + emptyList(), + ) + } + + private val target = QueryTarget( + MaterializedNamedAggregate("sales", "order"), + QueryDocumentKind.SNAPSHOT, + ) +} diff --git a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/webflux/WebFluxAutoConfigurationTest.kt b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/webflux/WebFluxAutoConfigurationTest.kt index e0ffe27eb48..71fe2612a49 100644 --- a/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/webflux/WebFluxAutoConfigurationTest.kt +++ b/wow-spring-boot-starter/src/test/kotlin/me/ahoo/wow/spring/boot/starter/webflux/WebFluxAutoConfigurationTest.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.spring.boot.starter.webflux import io.mockk.mockk @@ -49,6 +51,7 @@ import me.ahoo.wow.openapi.contract.bi.BiScriptRequest import me.ahoo.wow.openapi.contract.bi.BiScriptTopologyMode import me.ahoo.wow.openapi.contract.bi.BiScriptTopologyRequest import me.ahoo.wow.query.event.filter.EventStreamQueryHandler +import me.ahoo.wow.query.gateway.QueryTrustedContextResolver import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler import me.ahoo.wow.spring.boot.starter.ENABLED_SUFFIX_KEY import me.ahoo.wow.spring.boot.starter.bi.BiScriptProperties @@ -79,6 +82,8 @@ import me.ahoo.wow.webflux.route.global.GenerateBIScriptHandlerFunctionFactory import me.ahoo.wow.webflux.route.policy.BatchExecutionPolicy import me.ahoo.wow.webflux.route.policy.CommandWaitPolicy import me.ahoo.wow.webflux.route.policy.TracingPolicy +import me.ahoo.wow.webflux.route.query.QueryWebAuthorityResolver +import me.ahoo.wow.webflux.route.query.QueryWebTransportResolvers import org.junit.jupiter.api.Test import org.springframework.beans.factory.ObjectProvider import org.springframework.boot.test.context.FilteredClassLoader @@ -112,6 +117,7 @@ internal class WebFluxAutoConfigurationTest { ) @Test + @Suppress("LongMethod") fun `should load context with webflux command route and exception handler`() { contextRunner .enableWow() @@ -153,6 +159,9 @@ internal class WebFluxAutoConfigurationTest { .hasSingleBean(BatchExecutionPolicy::class.java) .hasSingleBean(WebFluxProperties::class.java) .hasSingleBean(BiScriptProperties::class.java) + .hasSingleBean(QueryWebAuthorityResolver::class.java) + .hasSingleBean(QueryWebTransportResolvers::class.java) + .hasSingleBean(QueryTrustedContextResolver::class.java) val batchExecutionPolicy = context.getBean(BatchExecutionPolicy::class.java) batchExecutionPolicy.concurrency.assert().isOne() batchExecutionPolicy.prefetch.assert().isOne() diff --git a/wow-spring/src/main/kotlin/me/ahoo/wow/spring/query/AnalyticsQueryServiceRegistrar.kt b/wow-spring/src/main/kotlin/me/ahoo/wow/spring/query/AnalyticsQueryServiceRegistrar.kt new file mode 100644 index 00000000000..b21c3e39645 --- /dev/null +++ b/wow-spring/src/main/kotlin/me/ahoo/wow/spring/query/AnalyticsQueryServiceRegistrar.kt @@ -0,0 +1,66 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.spring.query + +import io.github.oshai.kotlinlogging.KotlinLogging +import me.ahoo.wow.modeling.MaterializedNamedAggregate +import me.ahoo.wow.modeling.toStringWithAlias +import me.ahoo.wow.query.analytics.AnalyticsQueryService +import me.ahoo.wow.query.analytics.AnalyticsQueryServiceFactory +import me.ahoo.wow.query.gateway.QueryErrorCategory +import me.ahoo.wow.query.gateway.QueryExecutionException +import org.springframework.beans.factory.support.BeanDefinitionBuilder +import org.springframework.beans.factory.support.BeanDefinitionRegistry +import reactor.core.publisher.Mono + +class AnalyticsQueryServiceRegistrar : QueryServiceRegistrar() { + override fun registerQueryService( + entry: Map.Entry>, + registry: BeanDefinitionRegistry, + ) { + val namedAggregate = entry.key + val beanName = "${namedAggregate.toStringWithAlias()}.AnalyticsQueryService" + log.info { "Register AnalyticsQueryService [$beanName]." } + if (registry.containsBeanDefinition(beanName)) { + log.warn { "AnalyticsQueryService [$beanName] already exists - Ignore." } + return + } + val definition = BeanDefinitionBuilder.rootBeanDefinition(AnalyticsQueryService::class.java) { + appContext.getBeanProvider(AnalyticsQueryServiceFactory::class.java).getIfAvailable() + ?.create(namedAggregate) + ?: UnavailableAnalyticsQueryService(namedAggregate) + }.beanDefinition + registry.registerBeanDefinition(beanName, definition) + } + + private companion object { + val log = KotlinLogging.logger {} + } +} + +private class UnavailableAnalyticsQueryService( + override val namedAggregate: MaterializedNamedAggregate, +) : AnalyticsQueryService { + override fun analyze( + query: me.ahoo.wow.api.query.analytics.AnalyticsQuery, + ): Mono = Mono.error( + QueryExecutionException( + QueryErrorCategory.UNSUPPORTED_FEATURE, + "$.target", + "SCHEMA_NOT_REGISTERED", + ), + ) +} diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/exception/WebFluxErrorStrategy.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/exception/WebFluxErrorStrategy.kt index 56bfd02d6b4..5d7fbc464d3 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/exception/WebFluxErrorStrategy.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/exception/WebFluxErrorStrategy.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.exception import me.ahoo.wow.api.exception.ErrorInfo @@ -19,8 +21,10 @@ import me.ahoo.wow.exception.ErrorCodes import me.ahoo.wow.exception.ErrorInfoConverterRegistrar import me.ahoo.wow.exception.toErrorInfo import me.ahoo.wow.openapi.CommonComponent +import me.ahoo.wow.query.gateway.QueryErrorCategory import me.ahoo.wow.serialization.toJsonString import me.ahoo.wow.webflux.exception.ErrorHttpStatusMapping.toHttpStatus +import org.springframework.http.HttpStatus import org.springframework.http.MediaType import org.springframework.validation.BindingResult import org.springframework.web.ErrorResponse @@ -39,7 +43,7 @@ interface WebFluxErrorStrategy { object DefaultWebFluxErrorStrategy : WebFluxErrorStrategy { override fun toServerResponse(request: ServerRequest, throwable: Throwable): Mono { val errorInfo = throwable.toWebFluxErrorInfo() - return ServerResponse.status(throwable.httpStatus(errorInfo)) + return ServerResponse.status(throwable.toWebFluxHttpStatus(errorInfo)) .contentType(MediaType.APPLICATION_JSON) .header(CommonComponent.Header.ERROR_CODE, errorInfo.errorCode) .bodyValue(errorInfo.toJsonString()) @@ -52,15 +56,49 @@ object DefaultWebFluxErrorStrategy : WebFluxErrorStrategy { } val errorInfo = throwable.toWebFluxErrorInfo() - response.statusCode = throwable.httpStatus(errorInfo) + response.statusCode = throwable.toWebFluxHttpStatus(errorInfo) response.headers.contentType = MediaType.APPLICATION_JSON response.headers.set(CommonComponent.Header.ERROR_CODE, errorInfo.errorCode) return response.writeWith(Mono.just(response.bufferFactory().wrap(errorInfo.toJsonString().toByteArray()))) } } -private fun Throwable.httpStatus(errorInfo: ErrorInfo) = - (this as? ErrorResponse)?.statusCode ?: errorInfo.toHttpStatus() +internal fun Throwable.toWebFluxHttpStatus(errorInfo: ErrorInfo) = when (this) { + is ErrorResponse -> statusCode + else -> errorInfo.toWebFluxHttpStatus() +} + +internal fun ErrorInfo.toWebFluxHttpStatus(): HttpStatus = + queryErrorCategory()?.queryHttpStatus(queryErrorCode()) ?: toHttpStatus() + +private fun ErrorInfo.queryErrorCategory(): QueryErrorCategory? { + val segments = errorCode.split('.', limit = QUERY_ERROR_CODE_SEGMENTS) + if (segments.size != QUERY_ERROR_CODE_SEGMENTS || segments.first() != QUERY_ERROR_CODE_PREFIX) { + return null + } + return QueryErrorCategory.entries.firstOrNull { category -> category.name == segments[1] } +} + +private fun ErrorInfo.queryErrorCode(): String = + errorCode.split('.', limit = QUERY_ERROR_CODE_SEGMENTS).getOrElse(2) { "" } + +private fun QueryErrorCategory.queryHttpStatus(code: String): HttpStatus = when (this) { + QueryErrorCategory.ACCESS_DENIED -> HttpStatus.FORBIDDEN + QueryErrorCategory.INVALID_QUERY, + QueryErrorCategory.INVALID_CURSOR, + QueryErrorCategory.UNSUPPORTED_FEATURE, + -> HttpStatus.BAD_REQUEST + + QueryErrorCategory.BUDGET_EXCEEDED -> + if (code == DEADLINE_EXPIRED) HttpStatus.REQUEST_TIMEOUT else HttpStatus.TOO_MANY_REQUESTS + + QueryErrorCategory.INCOMPLETE_RESULT -> HttpStatus.BAD_GATEWAY + QueryErrorCategory.BACKEND_UNAVAILABLE -> HttpStatus.SERVICE_UNAVAILABLE + QueryErrorCategory.BACKEND_TIMEOUT -> HttpStatus.GATEWAY_TIMEOUT + QueryErrorCategory.MAPPING_FAILURE, + QueryErrorCategory.INTERNAL_FAILURE, + -> HttpStatus.INTERNAL_SERVER_ERROR +} private fun Throwable.toWebFluxErrorInfo(): ErrorInfo { return when (this) { @@ -83,3 +121,6 @@ private fun Throwable.toWebFluxErrorInfo(): ErrorInfo { } private const val UNEXPECTED_SERVER_ERROR_MESSAGE = "Unexpected server error" +private const val DEADLINE_EXPIRED = "DEADLINE_EXPIRED" +private const val QUERY_ERROR_CODE_PREFIX = "Query" +private const val QUERY_ERROR_CODE_SEGMENTS = 3 diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/Responses.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/Responses.kt index 93287144029..e78d2a1631f 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/Responses.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/Responses.kt @@ -17,8 +17,8 @@ import me.ahoo.wow.api.exception.ErrorInfo import me.ahoo.wow.exception.toErrorInfo import me.ahoo.wow.openapi.CommonComponent.Header.ERROR_CODE import me.ahoo.wow.serialization.toJsonString -import me.ahoo.wow.webflux.exception.ErrorHttpStatusMapping.toHttpStatus import me.ahoo.wow.webflux.exception.RequestExceptionHandler +import me.ahoo.wow.webflux.exception.toWebFluxHttpStatus import me.ahoo.wow.webflux.route.response.DefaultWebFluxResponseStrategy import org.springframework.core.ParameterizedTypeReference import org.springframework.http.MediaType @@ -34,7 +34,7 @@ object StringServerSentEventType : ParameterizedTypeReference { val errorInfo = toErrorInfo() - val status = errorInfo.toHttpStatus() + val status = toWebFluxHttpStatus(errorInfo) return ResponseEntity.status(status) .contentType(MediaType.APPLICATION_JSON) .header(ERROR_CODE, errorInfo.errorCode) @@ -42,11 +42,16 @@ fun Throwable.toResponseEntity(): ResponseEntity { } fun ErrorInfo.toServerResponse(): Mono { - val status = toHttpStatus() + val errorInfo = if (this is Throwable) toErrorInfo() else this + val status = if (this is Throwable) { + toWebFluxHttpStatus(errorInfo) + } else { + errorInfo.toWebFluxHttpStatus() + } return ServerResponse.status(status) .contentType(MediaType.APPLICATION_JSON) - .header(ERROR_CODE, errorCode) - .bodyValue(this.toJsonString()) + .header(ERROR_CODE, errorInfo.errorCode) + .bodyValue(errorInfo.toJsonString()) } fun Mono<*>.toServerResponse( diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/CountEventStreamHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/CountEventStreamHandlerFunction.kt index e02fb329ff3..4b986e3c862 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/CountEventStreamHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/CountEventStreamHandlerFunction.kt @@ -11,10 +11,13 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.event import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys import me.ahoo.wow.query.event.filter.EventStreamQueryHandler +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.query.CountQueryHandlerFunctionFactory import me.ahoo.wow.webflux.route.query.RewriteRequestCondition @@ -26,6 +29,7 @@ class CountEventStreamHandlerFunctionFactory( ) : CountQueryHandlerFunctionFactory( handlerKey = BuiltInHttpRouteHandlerKeys.Event.COUNT, queryHandler = eventStreamQueryHandler, + documentKind = QueryDocumentKind.EVENT_STREAM, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler ) diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/ListQueryEventStreamHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/ListQueryEventStreamHandlerFunction.kt index df4ba332725..27be05722f8 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/ListQueryEventStreamHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/ListQueryEventStreamHandlerFunction.kt @@ -11,10 +11,13 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.event import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys import me.ahoo.wow.query.event.filter.EventStreamQueryHandler +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.query.ListQueryHandlerFunctionFactory import me.ahoo.wow.webflux.route.query.RewriteRequestCondition @@ -26,6 +29,7 @@ class ListQueryEventStreamHandlerFunctionFactory( ) : ListQueryHandlerFunctionFactory( handlerKey = BuiltInHttpRouteHandlerKeys.Event.LIST_QUERY, queryHandler = eventStreamQueryHandler, + documentKind = QueryDocumentKind.EVENT_STREAM, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler ) diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/LoadEventStreamHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/LoadEventStreamHandlerFunction.kt index 03081810739..574e19f9595 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/LoadEventStreamHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/LoadEventStreamHandlerFunction.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.event import me.ahoo.wow.modeling.metadata.AggregateMetadata @@ -20,10 +22,14 @@ import me.ahoo.wow.openapi.contract.HttpRouteContract import me.ahoo.wow.openapi.contract.HttpRouteHandlerMetadata import me.ahoo.wow.query.dsl.listQuery import me.ahoo.wow.query.event.filter.EventStreamQueryHandler +import me.ahoo.wow.query.filter.Contexts.writeRawRequest +import me.ahoo.wow.query.filter.QueryType +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.serialization.MessageRecords import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.AggregateRouteHandlerFunctionFactorySupport import me.ahoo.wow.webflux.route.command.getTenantIdOrDefault +import me.ahoo.wow.webflux.route.query.writeQueryWebTransport import me.ahoo.wow.webflux.route.toServerResponse import org.springframework.web.reactive.function.server.HandlerFunction import org.springframework.web.reactive.function.server.ServerRequest @@ -41,7 +47,11 @@ class LoadEventStreamHandlerFunction( val id = request.pathVariable(MessageRecords.ID) val headVersion = request.pathVariable(BatchComponent.PathVariable.HEAD_VERSION).toInt() val tailVersion = request.pathVariable(BatchComponent.PathVariable.TAIL_VERSION).toInt() - val limit = tailVersion - headVersion + 1 + require(tailVersion >= headVersion) { + "Tail version must be greater than or equal to head version." + } + val versionCount = tailVersion.toLong() - headVersion.toLong() + 1 + val limit = if (versionCount > Int.MAX_VALUE) 0 else versionCount.toInt() val listQuery = listQuery { condition { tenantId(tenantId) @@ -51,6 +61,14 @@ class LoadEventStreamHandlerFunction( limit(limit) } return eventStreamQueryHandler.dynamicList(aggregateMetadata, listQuery) + .writeRawRequest(request) + .writeQueryWebTransport( + request, + aggregateMetadata, + QueryDocumentKind.EVENT_STREAM, + QueryType.DYNAMIC_LIST, + tenantId, + ) .toServerResponse(request, exceptionHandler) } } diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/PagedQueryEventStreamHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/PagedQueryEventStreamHandlerFunction.kt index 50ec07eeb30..ec8d9ba5466 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/PagedQueryEventStreamHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/event/PagedQueryEventStreamHandlerFunction.kt @@ -11,10 +11,13 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.event import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys import me.ahoo.wow.query.event.filter.EventStreamQueryHandler +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.query.PagedQueryHandlerFunctionFactory import me.ahoo.wow.webflux.route.query.RewriteRequestCondition @@ -26,6 +29,7 @@ class PagedQueryEventStreamHandlerFunctionFactory( ) : PagedQueryHandlerFunctionFactory( handlerKey = BuiltInHttpRouteHandlerKeys.Event.PAGED_QUERY, queryHandler = eventStreamQueryHandler, + documentKind = QueryDocumentKind.EVENT_STREAM, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler ) diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/AnalyticsQueryHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/AnalyticsQueryHandlerFunction.kt new file mode 100644 index 00000000000..196485625fb --- /dev/null +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/AnalyticsQueryHandlerFunction.kt @@ -0,0 +1,58 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package me.ahoo.wow.webflux.route.query + +import me.ahoo.wow.api.query.analytics.AnalyticsQuery +import me.ahoo.wow.modeling.metadata.AggregateMetadata +import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys +import me.ahoo.wow.openapi.contract.HttpRouteContract +import me.ahoo.wow.openapi.contract.HttpRouteHandlerMetadata +import me.ahoo.wow.query.analytics.AnalyticsQueryService +import me.ahoo.wow.query.analytics.AnalyticsQueryServiceFactory +import me.ahoo.wow.webflux.exception.RequestExceptionHandler +import me.ahoo.wow.webflux.route.AggregateRouteHandlerFunctionFactorySupport +import me.ahoo.wow.webflux.route.toServerResponse +import org.springframework.web.reactive.function.server.HandlerFunction +import org.springframework.web.reactive.function.server.ServerRequest +import org.springframework.web.reactive.function.server.ServerResponse +import reactor.core.publisher.Mono + +class AnalyticsQueryHandlerFunction( + private val aggregateMetadata: AggregateMetadata<*, *>, + private val analyticsQueryService: AnalyticsQueryService, + private val exceptionHandler: RequestExceptionHandler, +) : HandlerFunction { + override fun handle(request: ServerRequest): Mono = + request.bodyToMono(AnalyticsQuery::class.java) + .flatMap(analyticsQueryService::analyze) + .writeAnalyticsQueryWebTransport(request, aggregateMetadata) + .toServerResponse(request, exceptionHandler) +} + +class AnalyticsQueryHandlerFunctionFactory( + private val analyticsQueryServiceFactory: AnalyticsQueryServiceFactory, + private val exceptionHandler: RequestExceptionHandler, +) : AggregateRouteHandlerFunctionFactorySupport(BuiltInHttpRouteHandlerKeys.Snapshot.ANALYZE) { + override fun create( + contract: HttpRouteContract, + metadata: HttpRouteHandlerMetadata.Aggregate, + ): HandlerFunction { + val aggregateMetadata = aggregateMetadata(metadata) + return AnalyticsQueryHandlerFunction( + aggregateMetadata = aggregateMetadata, + analyticsQueryService = analyticsQueryServiceFactory.create(aggregateMetadata), + exceptionHandler = exceptionHandler, + ) + } +} diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/CountQueryHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/CountQueryHandlerFunction.kt index ad47fa9d884..ff1a4a42c62 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/CountQueryHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/CountQueryHandlerFunction.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.query import me.ahoo.wow.modeling.metadata.AggregateMetadata @@ -18,6 +20,8 @@ import me.ahoo.wow.openapi.contract.HttpRouteContract import me.ahoo.wow.openapi.contract.HttpRouteHandlerMetadata import me.ahoo.wow.query.filter.Contexts.writeRawRequest import me.ahoo.wow.query.filter.QueryHandler +import me.ahoo.wow.query.filter.QueryType +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.AggregateRouteHandlerFunctionFactorySupport import me.ahoo.wow.webflux.route.query.QueryBodyExtractor.Companion.CONDITION_EXTRACTOR @@ -30,16 +34,36 @@ import reactor.core.publisher.Mono class CountQueryHandlerFunction( private val aggregateMetadata: AggregateMetadata<*, *>, private val queryHandler: QueryHandler<*>, + private val documentKind: QueryDocumentKind?, private val rewriteRequestCondition: RewriteRequestCondition, private val exceptionHandler: RequestExceptionHandler, ) : HandlerFunction { + constructor( + aggregateMetadata: AggregateMetadata<*, *>, + queryHandler: QueryHandler<*>, + rewriteRequestCondition: RewriteRequestCondition, + exceptionHandler: RequestExceptionHandler, + ) : this( + aggregateMetadata, + queryHandler, + queryHandler.queryDocumentKind(), + rewriteRequestCondition, + exceptionHandler, + ) + override fun handle(request: ServerRequest): Mono { return request.body(CONDITION_EXTRACTOR) .flatMap { val query = rewriteRequestCondition.rewrite(aggregateMetadata, request, it) queryHandler.count(aggregateMetadata, query) .writeRawRequest(request) + .writeQueryWebTransport( + request, + aggregateMetadata, + documentKind, + QueryType.COUNT, + ) }.toServerResponse(request, exceptionHandler) } } @@ -47,9 +71,23 @@ class CountQueryHandlerFunction( open class CountQueryHandlerFunctionFactory( handlerKey: String, private val queryHandler: QueryHandler<*>, + private val documentKind: QueryDocumentKind?, private val rewriteRequestCondition: RewriteRequestCondition, private val exceptionHandler: RequestExceptionHandler ) : AggregateRouteHandlerFunctionFactorySupport(handlerKey) { + constructor( + handlerKey: String, + queryHandler: QueryHandler<*>, + rewriteRequestCondition: RewriteRequestCondition, + exceptionHandler: RequestExceptionHandler, + ) : this( + handlerKey, + queryHandler, + queryHandler.queryDocumentKind(), + rewriteRequestCondition, + exceptionHandler, + ) + override fun create( contract: HttpRouteContract, metadata: HttpRouteHandlerMetadata.Aggregate @@ -61,6 +99,7 @@ open class CountQueryHandlerFunctionFactory( return CountQueryHandlerFunction( aggregateMetadata = aggregateMetadata, queryHandler = queryHandler, + documentKind = documentKind, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler ) diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/ListQueryHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/ListQueryHandlerFunction.kt index 7bc1e218dcd..bafc3587e47 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/ListQueryHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/ListQueryHandlerFunction.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.query import me.ahoo.wow.api.query.DynamicDocument @@ -19,6 +21,8 @@ import me.ahoo.wow.openapi.contract.HttpRouteContract import me.ahoo.wow.openapi.contract.HttpRouteHandlerMetadata import me.ahoo.wow.query.filter.Contexts.writeRawRequest import me.ahoo.wow.query.filter.QueryHandler +import me.ahoo.wow.query.filter.QueryType +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.AggregateRouteHandlerFunctionFactorySupport import me.ahoo.wow.webflux.route.query.QueryBodyExtractor.Companion.LIST_QUERY_EXTRACTOR @@ -32,18 +36,40 @@ import reactor.core.publisher.Mono class ListQueryHandlerFunction( private val aggregateMetadata: AggregateMetadata<*, *>, private val queryHandler: QueryHandler<*>, + private val documentKind: QueryDocumentKind?, private val rewriteRequestCondition: RewriteRequestCondition, private val exceptionHandler: RequestExceptionHandler, private val rewriteResult: (Flux) -> Flux ) : HandlerFunction { + constructor( + aggregateMetadata: AggregateMetadata<*, *>, + queryHandler: QueryHandler<*>, + rewriteRequestCondition: RewriteRequestCondition, + exceptionHandler: RequestExceptionHandler, + rewriteResult: (Flux) -> Flux, + ) : this( + aggregateMetadata, + queryHandler, + queryHandler.queryDocumentKind(), + rewriteRequestCondition, + exceptionHandler, + rewriteResult, + ) + override fun handle(request: ServerRequest): Mono { return request.body(LIST_QUERY_EXTRACTOR) .flatMapMany { val query = rewriteRequestCondition.rewrite(aggregateMetadata, request, it) val result = queryHandler.dynamicList(aggregateMetadata, query) - rewriteResult(result) + result.rewriteResultOneToOne(rewriteResult) }.writeRawRequest(request) + .writeQueryWebTransport( + request, + aggregateMetadata, + documentKind, + QueryType.DYNAMIC_LIST, + ) .toServerResponse(request, exceptionHandler) } } @@ -51,10 +77,26 @@ class ListQueryHandlerFunction( open class ListQueryHandlerFunctionFactory( handlerKey: String, private val queryHandler: QueryHandler<*>, + private val documentKind: QueryDocumentKind?, private val rewriteRequestCondition: RewriteRequestCondition, private val exceptionHandler: RequestExceptionHandler, private val rewriteResult: (Flux) -> Flux = { it } ) : AggregateRouteHandlerFunctionFactorySupport(handlerKey) { + constructor( + handlerKey: String, + queryHandler: QueryHandler<*>, + rewriteRequestCondition: RewriteRequestCondition, + exceptionHandler: RequestExceptionHandler, + rewriteResult: (Flux) -> Flux = { it }, + ) : this( + handlerKey, + queryHandler, + queryHandler.queryDocumentKind(), + rewriteRequestCondition, + exceptionHandler, + rewriteResult, + ) + override fun create( contract: HttpRouteContract, metadata: HttpRouteHandlerMetadata.Aggregate @@ -66,6 +108,7 @@ open class ListQueryHandlerFunctionFactory( return ListQueryHandlerFunction( aggregateMetadata = aggregateMetadata, queryHandler = queryHandler, + documentKind = documentKind, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler, rewriteResult = rewriteResult diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/PagedQueryHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/PagedQueryHandlerFunction.kt index 483d0f654c3..c7ce0e734a5 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/PagedQueryHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/PagedQueryHandlerFunction.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.query import me.ahoo.wow.api.query.DynamicDocument @@ -20,6 +22,8 @@ import me.ahoo.wow.openapi.contract.HttpRouteContract import me.ahoo.wow.openapi.contract.HttpRouteHandlerMetadata import me.ahoo.wow.query.filter.Contexts.writeRawRequest import me.ahoo.wow.query.filter.QueryHandler +import me.ahoo.wow.query.filter.QueryType +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.AggregateRouteHandlerFunctionFactorySupport import me.ahoo.wow.webflux.route.query.QueryBodyExtractor.Companion.PAGED_QUERY_EXTRACTOR @@ -32,18 +36,42 @@ import reactor.core.publisher.Mono class PagedQueryHandlerFunction( private val aggregateMetadata: AggregateMetadata<*, *>, private val queryHandler: QueryHandler<*>, + private val documentKind: QueryDocumentKind?, private val rewriteRequestCondition: RewriteRequestCondition, private val exceptionHandler: RequestExceptionHandler, private val rewriteResult: (Mono>) -> Mono> ) : HandlerFunction { + constructor( + aggregateMetadata: AggregateMetadata<*, *>, + queryHandler: QueryHandler<*>, + rewriteRequestCondition: RewriteRequestCondition, + exceptionHandler: RequestExceptionHandler, + rewriteResult: (Mono>) -> Mono>, + ) : this( + aggregateMetadata, + queryHandler, + queryHandler.queryDocumentKind(), + rewriteRequestCondition, + exceptionHandler, + rewriteResult, + ) + override fun handle(request: ServerRequest): Mono { return request.body(PAGED_QUERY_EXTRACTOR) .flatMap { val query = rewriteRequestCondition.rewrite(aggregateMetadata, request, it) val result = queryHandler.dynamicPaged(aggregateMetadata, query) - rewriteResult(result) + result.rewriteResultOneToOne(rewriteResult) { original, rewritten -> + original.total == rewritten.total && original.list.size == rewritten.list.size + } .writeRawRequest(request) + .writeQueryWebTransport( + request, + aggregateMetadata, + documentKind, + QueryType.DYNAMIC_PAGED, + ) }.toServerResponse(request, exceptionHandler) } } @@ -51,10 +79,26 @@ class PagedQueryHandlerFunction( open class PagedQueryHandlerFunctionFactory( handlerKey: String, private val queryHandler: QueryHandler<*>, + private val documentKind: QueryDocumentKind?, private val rewriteRequestCondition: RewriteRequestCondition, private val exceptionHandler: RequestExceptionHandler, private val rewriteResult: (Mono>) -> Mono> = { it } ) : AggregateRouteHandlerFunctionFactorySupport(handlerKey) { + constructor( + handlerKey: String, + queryHandler: QueryHandler<*>, + rewriteRequestCondition: RewriteRequestCondition, + exceptionHandler: RequestExceptionHandler, + rewriteResult: (Mono>) -> Mono> = { it }, + ) : this( + handlerKey, + queryHandler, + queryHandler.queryDocumentKind(), + rewriteRequestCondition, + exceptionHandler, + rewriteResult, + ) + override fun create( contract: HttpRouteContract, metadata: HttpRouteHandlerMetadata.Aggregate @@ -66,6 +110,7 @@ open class PagedQueryHandlerFunctionFactory( return PagedQueryHandlerFunction( aggregateMetadata = aggregateMetadata, queryHandler = queryHandler, + documentKind = documentKind, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler, rewriteResult = rewriteResult diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/QueryResultRewrite.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/QueryResultRewrite.kt new file mode 100644 index 00000000000..a2dcb62d2ad --- /dev/null +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/QueryResultRewrite.kt @@ -0,0 +1,58 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.webflux.route.query + +import me.ahoo.wow.query.gateway.QueryErrorCategory +import me.ahoo.wow.query.gateway.QueryExecutionException +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono + +/** Runs a legacy result transformer only after the Gateway source emitted one value. */ +internal fun Mono.rewriteResultOneToOne( + rewrite: (Mono) -> Mono, + validate: (original: T, rewritten: T) -> Boolean = { _, _ -> true }, +): Mono = flatMap { original -> + Mono.defer { rewrite(Mono.just(original)) } + .switchIfEmpty(Mono.error(resultRewriteContractViolation())) + .flatMap { rewritten -> + if (validate(original, rewritten)) { + Mono.just(rewritten) + } else { + Mono.error(resultRewriteContractViolation()) + } + } +} + +/** Runs a legacy result transformer once per Gateway item and requires exactly one output for that item. */ +internal fun Flux.rewriteResultOneToOne(rewrite: (Flux) -> Flux): Flux = + concatMap { original -> + Flux.defer { rewrite(Flux.just(original)) } + .take(2) + .collectList() + .flatMap { rewritten -> + if (rewritten.size == 1) { + Mono.just(rewritten.single()) + } else { + Mono.error(resultRewriteContractViolation()) + } + } + } + +private fun resultRewriteContractViolation(): QueryExecutionException = QueryExecutionException( + QueryErrorCategory.INTERNAL_FAILURE, + "$.result.rewrite", + "RESULT_REWRITE_CONTRACT_VIOLATION", +) diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/QueryWebTransport.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/QueryWebTransport.kt new file mode 100644 index 00000000000..ce58773e171 --- /dev/null +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/QueryWebTransport.kt @@ -0,0 +1,207 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.webflux.route.query + +import me.ahoo.wow.modeling.metadata.AggregateMetadata +import me.ahoo.wow.query.analytics.AnalyticsQueryTrustedContextRequest +import me.ahoo.wow.query.analytics.AnalyticsQueryTrustedContextResolver +import me.ahoo.wow.query.event.filter.EventStreamQueryHandler +import me.ahoo.wow.query.filter.QueryHandler +import me.ahoo.wow.query.filter.QueryType +import me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi +import me.ahoo.wow.query.gateway.QueryAuthority +import me.ahoo.wow.query.gateway.QueryCall +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryErrorCategory +import me.ahoo.wow.query.gateway.QueryExecutionException +import me.ahoo.wow.query.gateway.QueryPurpose +import me.ahoo.wow.query.gateway.QueryResourceScope +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.gateway.QueryTrustedContext +import me.ahoo.wow.query.gateway.QueryTrustedContextRequest +import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler +import me.ahoo.wow.webflux.route.command.getOwnerId +import me.ahoo.wow.webflux.route.command.getSpaceId +import me.ahoo.wow.webflux.route.command.getTenantId +import org.springframework.web.reactive.function.server.ServerRequest +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.util.context.ContextView + +/** + * Converts an authenticated application request context into Query Gateway authority. + * + * Implementations must derive the principal and grants from an authenticated source. Path variables and request + * headers are selectors already captured in [QueryWebAuthorityRequest.call]; they are never authority evidence. + */ +@ExperimentalQueryGatewayApi +fun interface QueryWebAuthorityResolver { + /** Empty and error signals are fail-closed by the Query Gateway. */ + fun resolve(request: QueryWebAuthorityRequest): Mono +} + +@ExperimentalQueryGatewayApi +data class QueryWebAuthorityRequest( + val call: QueryCall, + val request: ServerRequest, +) + +/** Resolves the compatibility facade call and authority from the same typed WebFlux transport marker. */ +@ExperimentalQueryGatewayApi +class QueryWebTransportResolvers( + private val webAuthorityResolver: QueryWebAuthorityResolver, +) : me.ahoo.wow.query.gateway.QueryTrustedContextResolver, + AnalyticsQueryTrustedContextResolver { + override fun resolve(request: QueryTrustedContextRequest): Mono = + Mono.deferContextual { context -> + val marker = context.queryTransportMarker() ?: return@deferContextual Mono.empty() + if (marker.call.target != request.callRequest.target || marker.queryType != request.callRequest.queryType) { + return@deferContextual Mono.error(transportMismatch("QUERY_TRANSPORT_CALL_MISMATCH")) + } + Mono.defer { webAuthorityResolver.resolve(QueryWebAuthorityRequest(marker.call, marker.request)) } + .onErrorMap(::authorityResolutionFailed) + .switchIfEmpty(Mono.error(authorityRequired())) + .map { authority -> QueryTrustedContext(marker.call, authority) } + } + + override fun resolve(request: AnalyticsQueryTrustedContextRequest): Mono = + Mono.deferContextual { context -> + val marker = context.analyticsQueryTransportMarker() ?: return@deferContextual Mono.empty() + if (marker.call.target != request.target) { + return@deferContextual Mono.error(transportMismatch("QUERY_TRANSPORT_CALL_MISMATCH")) + } + Mono.defer { webAuthorityResolver.resolve(QueryWebAuthorityRequest(marker.call, marker.request)) } + .onErrorMap(::authorityResolutionFailed) + .switchIfEmpty(Mono.error(authorityRequired())) + .map { authority -> QueryTrustedContext(marker.call, authority) } + } +} + +internal fun Mono.writeAnalyticsQueryWebTransport( + request: ServerRequest, + aggregateMetadata: AggregateMetadata<*, *>, +): Mono = contextWrite { context -> + context.put( + ANALYTICS_QUERY_TRANSPORT_MARKER_KEY, + AnalyticsQueryTransportMarker( + call = request.toQueryCall(aggregateMetadata, QueryDocumentKind.SNAPSHOT), + request = request, + ), + ) +} + +internal fun Mono.writeQueryWebTransport( + request: ServerRequest, + aggregateMetadata: AggregateMetadata<*, *>, + documentKind: QueryDocumentKind?, + queryType: QueryType, + effectiveTenantId: String? = null, +): Mono = + documentKind?.let { declaredKind -> + contextWrite { context -> + context.put( + QUERY_TRANSPORT_MARKER_KEY, + request.toMarker(aggregateMetadata, declaredKind, queryType, effectiveTenantId), + ) + } + } ?: this + +internal fun Flux.writeQueryWebTransport( + request: ServerRequest, + aggregateMetadata: AggregateMetadata<*, *>, + documentKind: QueryDocumentKind?, + queryType: QueryType, + effectiveTenantId: String? = null, +): Flux = + documentKind?.let { declaredKind -> + contextWrite { context -> + context.put( + QUERY_TRANSPORT_MARKER_KEY, + request.toMarker(aggregateMetadata, declaredKind, queryType, effectiveTenantId), + ) + } + } ?: this + +internal fun QueryHandler<*>.queryDocumentKind(): QueryDocumentKind? = when (this) { + is SnapshotQueryHandler -> QueryDocumentKind.SNAPSHOT + is EventStreamQueryHandler -> QueryDocumentKind.EVENT_STREAM + else -> null +} + +private data class QueryTransportMarker( + val call: QueryCall, + val queryType: QueryType, + val request: ServerRequest, +) + +private data class AnalyticsQueryTransportMarker( + val call: QueryCall, + val request: ServerRequest, +) + +private fun ServerRequest.toMarker( + aggregateMetadata: AggregateMetadata<*, *>, + documentKind: QueryDocumentKind, + queryType: QueryType, + effectiveTenantId: String?, +): QueryTransportMarker = QueryTransportMarker( + call = toQueryCall(aggregateMetadata, documentKind, effectiveTenantId), + queryType = queryType, + request = this, +) + +private fun ServerRequest.toQueryCall( + aggregateMetadata: AggregateMetadata<*, *>, + documentKind: QueryDocumentKind, + effectiveTenantId: String? = null, +): QueryCall = QueryCall( + target = QueryTarget(aggregateMetadata, documentKind), + purpose = WEB_QUERY_PURPOSE, + resourceScope = QueryResourceScope( + tenantId = effectiveTenantId ?: getTenantId(aggregateMetadata), + ownerId = getOwnerId(), + spaceId = getSpaceId(), + ), +) + +private fun ContextView.queryTransportMarker(): QueryTransportMarker? = + getOrDefault(QUERY_TRANSPORT_MARKER_KEY, null) + +private fun ContextView.analyticsQueryTransportMarker(): AnalyticsQueryTransportMarker? = + getOrDefault(ANALYTICS_QUERY_TRANSPORT_MARKER_KEY, null) + +private fun transportMismatch(code: String): QueryExecutionException = QueryExecutionException( + category = QueryErrorCategory.ACCESS_DENIED, + path = "$.executionContext.transport", + code = code, +) + +private fun authorityRequired(): QueryExecutionException = QueryExecutionException( + category = QueryErrorCategory.ACCESS_DENIED, + path = "$.executionContext.authority", + code = "AUTHORITY_REQUIRED", +) + +private fun authorityResolutionFailed(cause: Throwable): QueryExecutionException = QueryExecutionException( + category = QueryErrorCategory.ACCESS_DENIED, + path = "$.executionContext.authority", + code = "AUTHORITY_RESOLUTION_FAILED", + cause = cause, +) + +private val WEB_QUERY_PURPOSE = QueryPurpose("interactive-query") +private const val QUERY_TRANSPORT_MARKER_KEY = "me.ahoo.wow.query.web.transport" +private const val ANALYTICS_QUERY_TRANSPORT_MARKER_KEY = "me.ahoo.wow.query.web.analytics.transport" diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/SingleQueryHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/SingleQueryHandlerFunction.kt index fb63627c5ac..d7c5611ac3e 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/SingleQueryHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/query/SingleQueryHandlerFunction.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.query import me.ahoo.wow.api.query.DynamicDocument @@ -20,6 +22,8 @@ import me.ahoo.wow.openapi.contract.HttpRouteContract import me.ahoo.wow.openapi.contract.HttpRouteHandlerMetadata import me.ahoo.wow.query.filter.Contexts.writeRawRequest import me.ahoo.wow.query.filter.QueryHandler +import me.ahoo.wow.query.filter.QueryType +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.AggregateRouteHandlerFunctionFactorySupport import me.ahoo.wow.webflux.route.query.QueryBodyExtractor.Companion.SINGLE_QUERY_EXTRACTOR @@ -32,18 +36,40 @@ import reactor.core.publisher.Mono class SingleQueryHandlerFunction( private val aggregateMetadata: AggregateMetadata<*, *>, private val queryHandler: QueryHandler<*>, + private val documentKind: QueryDocumentKind?, private val rewriteRequestCondition: RewriteRequestCondition, private val exceptionHandler: RequestExceptionHandler, private val rewriteResult: (Mono) -> Mono ) : HandlerFunction { + constructor( + aggregateMetadata: AggregateMetadata<*, *>, + queryHandler: QueryHandler<*>, + rewriteRequestCondition: RewriteRequestCondition, + exceptionHandler: RequestExceptionHandler, + rewriteResult: (Mono) -> Mono, + ) : this( + aggregateMetadata, + queryHandler, + queryHandler.queryDocumentKind(), + rewriteRequestCondition, + exceptionHandler, + rewriteResult, + ) + override fun handle(request: ServerRequest): Mono { return request.body(SINGLE_QUERY_EXTRACTOR) .flatMap { val query = rewriteRequestCondition.rewrite(aggregateMetadata, request, it) val result = queryHandler.dynamicSingle(aggregateMetadata, query) - rewriteResult(result) + result.rewriteResultOneToOne(rewriteResult) .writeRawRequest(request) + .writeQueryWebTransport( + request, + aggregateMetadata, + documentKind, + QueryType.DYNAMIC_SINGLE, + ) .throwNotFoundIfEmpty() }.toServerResponse(request, exceptionHandler) } @@ -52,10 +78,26 @@ class SingleQueryHandlerFunction( open class SingleQueryHandlerFunctionFactory( handlerKey: String, private val queryHandler: QueryHandler<*>, + private val documentKind: QueryDocumentKind?, private val rewriteRequestCondition: RewriteRequestCondition, private val exceptionHandler: RequestExceptionHandler, private val rewriteResult: (Mono) -> Mono = { it } ) : AggregateRouteHandlerFunctionFactorySupport(handlerKey) { + constructor( + handlerKey: String, + queryHandler: QueryHandler<*>, + rewriteRequestCondition: RewriteRequestCondition, + exceptionHandler: RequestExceptionHandler, + rewriteResult: (Mono) -> Mono = { it }, + ) : this( + handlerKey, + queryHandler, + queryHandler.queryDocumentKind(), + rewriteRequestCondition, + exceptionHandler, + rewriteResult, + ) + override fun create( contract: HttpRouteContract, metadata: HttpRouteHandlerMetadata.Aggregate @@ -67,6 +109,7 @@ open class SingleQueryHandlerFunctionFactory( return SingleQueryHandlerFunction( aggregateMetadata = aggregateMetadata, queryHandler = queryHandler, + documentKind = documentKind, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler, rewriteResult = rewriteResult diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/CountSnapshotHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/CountSnapshotHandlerFunction.kt index b768dcfaa98..8dd242f7acf 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/CountSnapshotHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/CountSnapshotHandlerFunction.kt @@ -11,9 +11,12 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.snapshot import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.query.CountQueryHandlerFunctionFactory @@ -26,6 +29,7 @@ class CountSnapshotHandlerFunctionFactory( ) : CountQueryHandlerFunctionFactory( handlerKey = BuiltInHttpRouteHandlerKeys.Snapshot.COUNT, queryHandler = snapshotQueryHandler, + documentKind = QueryDocumentKind.SNAPSHOT, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler ) diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/ListQuerySnapshotHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/ListQuerySnapshotHandlerFunction.kt index 62cb4116054..558b639c0d6 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/ListQuerySnapshotHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/ListQuerySnapshotHandlerFunction.kt @@ -11,9 +11,12 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.snapshot import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.query.ListQueryHandlerFunctionFactory @@ -26,6 +29,7 @@ class ListQuerySnapshotHandlerFunctionFactory( ) : ListQueryHandlerFunctionFactory( handlerKey = BuiltInHttpRouteHandlerKeys.Snapshot.LIST_QUERY, queryHandler = snapshotQueryHandler, + documentKind = QueryDocumentKind.SNAPSHOT, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler ) diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/ListQuerySnapshotStateHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/ListQuerySnapshotStateHandlerFunction.kt index cfe4252f648..d8447c43295 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/ListQuerySnapshotStateHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/ListQuerySnapshotStateHandlerFunction.kt @@ -11,9 +11,12 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.snapshot import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler import me.ahoo.wow.query.snapshot.toStateDocument import me.ahoo.wow.webflux.exception.RequestExceptionHandler @@ -27,6 +30,7 @@ class ListQuerySnapshotStateHandlerFunctionFactory( ) : ListQueryHandlerFunctionFactory( handlerKey = BuiltInHttpRouteHandlerKeys.Snapshot.LIST_QUERY_STATE, queryHandler = snapshotQueryHandler, + documentKind = QueryDocumentKind.SNAPSHOT, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler, rewriteResult = { it.toStateDocument() } diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/LoadSnapshotHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/LoadSnapshotHandlerFunction.kt index e7d9c5152a0..b99586c3c8b 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/LoadSnapshotHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/LoadSnapshotHandlerFunction.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.snapshot import me.ahoo.wow.exception.throwNotFoundIfEmpty @@ -19,12 +21,16 @@ import me.ahoo.wow.openapi.contract.HttpRouteContract import me.ahoo.wow.openapi.contract.HttpRouteHandlerMetadata import me.ahoo.wow.openapi.metadata.AggregateRouteMetadata import me.ahoo.wow.query.dsl.singleQuery +import me.ahoo.wow.query.filter.Contexts.writeRawRequest +import me.ahoo.wow.query.filter.QueryType +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.AggregateRouteHandlerFunctionFactorySupport import me.ahoo.wow.webflux.route.command.getAggregateId import me.ahoo.wow.webflux.route.command.getOwnerId import me.ahoo.wow.webflux.route.command.getTenantIdOrDefault +import me.ahoo.wow.webflux.route.query.writeQueryWebTransport import me.ahoo.wow.webflux.route.toServerResponse import org.springframework.web.reactive.function.server.HandlerFunction import org.springframework.web.reactive.function.server.ServerRequest @@ -51,6 +57,14 @@ class LoadSnapshotHandlerFunction( } } return snapshotQueryHandler.dynamicSingle(aggregateMetadata, singleQuery) + .writeRawRequest(request) + .writeQueryWebTransport( + request, + aggregateMetadata, + QueryDocumentKind.SNAPSHOT, + QueryType.DYNAMIC_SINGLE, + tenantId, + ) .throwNotFoundIfEmpty() .toServerResponse(request, exceptionHandler) } diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/PagedQuerySnapshotHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/PagedQuerySnapshotHandlerFunction.kt index 3b94faea027..4c61f5bab71 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/PagedQuerySnapshotHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/PagedQuerySnapshotHandlerFunction.kt @@ -11,9 +11,12 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.snapshot import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.query.PagedQueryHandlerFunctionFactory @@ -26,6 +29,7 @@ class PagedQuerySnapshotHandlerFunctionFactory( ) : PagedQueryHandlerFunctionFactory( handlerKey = BuiltInHttpRouteHandlerKeys.Snapshot.PAGED_QUERY, queryHandler = snapshotQueryHandler, + documentKind = QueryDocumentKind.SNAPSHOT, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler ) diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/PagedQuerySnapshotStateHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/PagedQuerySnapshotStateHandlerFunction.kt index 849d84e3ac3..8f9f7bdfcce 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/PagedQuerySnapshotStateHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/PagedQuerySnapshotStateHandlerFunction.kt @@ -11,9 +11,12 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.snapshot import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler import me.ahoo.wow.query.snapshot.toStateDocumentPagedList import me.ahoo.wow.webflux.exception.RequestExceptionHandler @@ -27,6 +30,7 @@ class PagedQuerySnapshotStateHandlerFunctionFactory( ) : PagedQueryHandlerFunctionFactory( handlerKey = BuiltInHttpRouteHandlerKeys.Snapshot.PAGED_QUERY_STATE, queryHandler = snapshotQueryHandler, + documentKind = QueryDocumentKind.SNAPSHOT, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler, rewriteResult = { it.toStateDocumentPagedList() } diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/SingleSnapshotHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/SingleSnapshotHandlerFunction.kt index 59918cf1256..60c0032852a 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/SingleSnapshotHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/SingleSnapshotHandlerFunction.kt @@ -11,9 +11,12 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.snapshot import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler import me.ahoo.wow.webflux.exception.RequestExceptionHandler import me.ahoo.wow.webflux.route.query.RewriteRequestCondition @@ -26,6 +29,7 @@ class SingleSnapshotHandlerFunctionFactory( ) : SingleQueryHandlerFunctionFactory( handlerKey = BuiltInHttpRouteHandlerKeys.Snapshot.SINGLE, queryHandler = snapshotQueryHandler, + documentKind = QueryDocumentKind.SNAPSHOT, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler ) diff --git a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/SingleSnapshotStateHandlerFunction.kt b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/SingleSnapshotStateHandlerFunction.kt index f1f41026cbf..e88bdd2d7cb 100644 --- a/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/SingleSnapshotStateHandlerFunction.kt +++ b/wow-webflux/src/main/kotlin/me/ahoo/wow/webflux/route/snapshot/SingleSnapshotStateHandlerFunction.kt @@ -11,9 +11,12 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route.snapshot import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys +import me.ahoo.wow.query.gateway.QueryDocumentKind import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler import me.ahoo.wow.query.snapshot.toStateDocument import me.ahoo.wow.webflux.exception.RequestExceptionHandler @@ -27,6 +30,7 @@ class SingleSnapshotStateHandlerFunctionFactory( ) : SingleQueryHandlerFunctionFactory( handlerKey = BuiltInHttpRouteHandlerKeys.Snapshot.SINGLE_STATE, queryHandler = snapshotQueryHandler, + documentKind = QueryDocumentKind.SNAPSHOT, rewriteRequestCondition = rewriteRequestCondition, exceptionHandler = exceptionHandler, rewriteResult = { it.toStateDocument() } diff --git a/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/exception/WebFluxErrorStrategyTest.kt b/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/exception/WebFluxErrorStrategyTest.kt index a46c2d0df5a..e62e93e5c9d 100644 --- a/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/exception/WebFluxErrorStrategyTest.kt +++ b/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/exception/WebFluxErrorStrategyTest.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.exception import io.mockk.CapturingSlot @@ -25,6 +27,8 @@ import me.ahoo.wow.exception.ErrorCodes import me.ahoo.wow.exception.ErrorInfoConverter import me.ahoo.wow.exception.ErrorInfoConverterRegistrar import me.ahoo.wow.openapi.CommonComponent.Header.ERROR_CODE +import me.ahoo.wow.query.gateway.QueryErrorCategory +import me.ahoo.wow.query.gateway.QueryExecutionException import org.junit.jupiter.api.BeforeEach import org.junit.jupiter.api.Test import org.reactivestreams.Publisher @@ -76,6 +80,48 @@ class WebFluxErrorStrategyTest { .verifyComplete() } + @Test + fun `should map stable query error categories without changing the error envelope`() { + val request = MockServerRequest.builder() + .method(HttpMethod.POST) + .uri(URI.create("/query")) + .build() + val cases = listOf( + Triple(QueryErrorCategory.INVALID_QUERY, "INVALID_FIELD", HttpStatus.BAD_REQUEST), + Triple(QueryErrorCategory.INVALID_CURSOR, "CURSOR_TYPE_MISMATCH", HttpStatus.BAD_REQUEST), + Triple(QueryErrorCategory.UNSUPPORTED_FEATURE, "CAPABILITY_UNAVAILABLE", HttpStatus.BAD_REQUEST), + Triple(QueryErrorCategory.ACCESS_DENIED, "POLICY_DENIED", HttpStatus.FORBIDDEN), + Triple(QueryErrorCategory.BUDGET_EXCEEDED, "DEADLINE_EXPIRED", HttpStatus.REQUEST_TIMEOUT), + Triple(QueryErrorCategory.BUDGET_EXCEEDED, "RESULT_LIMIT_EXCEEDED", HttpStatus.TOO_MANY_REQUESTS), + Triple(QueryErrorCategory.INCOMPLETE_RESULT, "INCOMPLETE_RESULT", HttpStatus.BAD_GATEWAY), + Triple(QueryErrorCategory.BACKEND_UNAVAILABLE, "BACKEND_NOT_REGISTERED", HttpStatus.SERVICE_UNAVAILABLE), + Triple(QueryErrorCategory.BACKEND_TIMEOUT, "BACKEND_TIMEOUT", HttpStatus.GATEWAY_TIMEOUT), + Triple(QueryErrorCategory.MAPPING_FAILURE, "RESULT_MAPPING_FAILED", HttpStatus.INTERNAL_SERVER_ERROR), + Triple(QueryErrorCategory.INTERNAL_FAILURE, "UNEXPECTED_QUERY_FAILURE", HttpStatus.INTERNAL_SERVER_ERROR), + ) + + cases.forEach { (category, code, expectedStatus) -> + val failure = QueryExecutionException(category, "$.query", code) + WebTestClient.bindToRouterFunction( + route(POST("/query")) { + DefaultWebFluxErrorStrategy.toServerResponse(request, failure) + }, + ).build() + .post() + .uri("/query") + .exchange() + .expectStatus().isEqualTo(expectedStatus) + .expectHeader().valueEquals(ERROR_CODE, "Query.${category.name}.$code") + .expectBody(String::class.java) + .consumeWith { result -> + result.responseBody!!.assert() + .contains("\"errorCode\":\"Query.${category.name}.$code\"") + .contains("\"name\":\"$.query\"") + .contains("\"msg\":\"$code\"") + } + } + } + @Test fun `should map an unclassified throwable to a safe internal server response`() { val failure = NullPointerException("sensitive implementation detail") diff --git a/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/ResponsesKtTest.kt b/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/ResponsesKtTest.kt index 1aa4d151a7a..70e8243c086 100644 --- a/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/ResponsesKtTest.kt +++ b/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/ResponsesKtTest.kt @@ -11,6 +11,8 @@ * limitations under the License. */ +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + package me.ahoo.wow.webflux.route import io.mockk.every @@ -25,6 +27,8 @@ import me.ahoo.wow.exception.ErrorCodes import me.ahoo.wow.exception.toErrorInfo import me.ahoo.wow.id.generateGlobalId import me.ahoo.wow.openapi.CommonComponent.Header.ERROR_CODE +import me.ahoo.wow.query.gateway.QueryErrorCategory +import me.ahoo.wow.query.gateway.QueryExecutionException import me.ahoo.wow.webflux.exception.WebFluxRequestExceptionHandler import org.junit.jupiter.api.Test import org.springframework.http.HttpHeaders @@ -42,6 +46,32 @@ import reactor.kotlin.test.test class ResponsesKtTest { + @Test + fun `should preserve query status mapping in response entity adapter`() { + QueryExecutionException(QueryErrorCategory.ACCESS_DENIED, "$.policy", "POLICY_DENIED") + .toResponseEntity() + .statusCode.assert().isEqualTo(HttpStatus.FORBIDDEN) + } + + @Test + fun `should preserve query status mapping after materializing error info`() { + val cases = listOf( + Triple(QueryErrorCategory.ACCESS_DENIED, "POLICY_DENIED", HttpStatus.FORBIDDEN), + Triple(QueryErrorCategory.BUDGET_EXCEEDED, "DEADLINE_EXPIRED", HttpStatus.REQUEST_TIMEOUT), + Triple(QueryErrorCategory.BACKEND_UNAVAILABLE, "BACKEND_NOT_REGISTERED", HttpStatus.SERVICE_UNAVAILABLE), + Triple(QueryErrorCategory.INTERNAL_FAILURE, "UNEXPECTED_QUERY_FAILURE", HttpStatus.INTERNAL_SERVER_ERROR), + ) + + cases.forEach { (category, code, status) -> + QueryExecutionException(category, "$.query", code) + .toErrorInfo() + .toServerResponse() + .test() + .consumeNextWith { response -> response.statusCode().assert().isEqualTo(status) } + .verifyComplete() + } + } + @Test fun `should convert exception to response entity`() { val responseEntity = IllegalArgumentException() diff --git a/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/query/QueryResultRewriteTest.kt b/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/query/QueryResultRewriteTest.kt new file mode 100644 index 00000000000..d7d1bb7a01c --- /dev/null +++ b/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/query/QueryResultRewriteTest.kt @@ -0,0 +1,63 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.webflux.route.query + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.query.gateway.QueryExecutionException +import org.junit.jupiter.api.Test +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.test.StepVerifier +import java.util.concurrent.atomic.AtomicInteger + +class QueryResultRewriteTest { + @Test + fun `mono transformer cannot bypass the gateway source`() { + val subscriptions = AtomicInteger() + val source = Mono.defer { + subscriptions.incrementAndGet() + Mono.just("gateway") + } + + StepVerifier.create(source.rewriteResultOneToOne(rewrite = { Mono.just("rewritten") })) + .expectNext("rewritten") + .verifyComplete() + + subscriptions.get().assert().isEqualTo(1) + } + + @Test + fun `flux transformer cannot change cardinality`() { + StepVerifier.create(Flux.just("gateway").rewriteResultOneToOne { Flux.just("one", "two") }) + .expectErrorSatisfies { error -> + error.assert().isInstanceOf(QueryExecutionException::class.java) + (error as QueryExecutionException).code.assert().isEqualTo("RESULT_REWRITE_CONTRACT_VIOLATION") + } + .verify() + } + + @Test + fun `mono transformer must preserve page envelope invariants`() { + StepVerifier.create( + Mono.just("original").rewriteResultOneToOne( + rewrite = { Mono.just("rewritten") }, + validate = { original, rewritten -> original == rewritten }, + ), + ) + .expectError(QueryExecutionException::class.java) + .verify() + } +} diff --git a/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/query/QueryRouteTransportTest.kt b/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/query/QueryRouteTransportTest.kt new file mode 100644 index 00000000000..d1543962f21 --- /dev/null +++ b/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/query/QueryRouteTransportTest.kt @@ -0,0 +1,318 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.webflux.route.query + +import io.mockk.every +import io.mockk.mockk +import io.mockk.slot +import me.ahoo.test.asserts.assert +import me.ahoo.wow.api.modeling.NamedAggregate +import me.ahoo.wow.api.query.Condition +import me.ahoo.wow.api.query.DynamicDocument +import me.ahoo.wow.api.query.IListQuery +import me.ahoo.wow.api.query.IPagedQuery +import me.ahoo.wow.api.query.ISingleQuery +import me.ahoo.wow.api.query.ListQuery +import me.ahoo.wow.api.query.PagedList +import me.ahoo.wow.api.query.PagedQuery +import me.ahoo.wow.api.query.SimpleDynamicDocument.Companion.toDynamicDocument +import me.ahoo.wow.api.query.SingleQuery +import me.ahoo.wow.api.query.analytics.AnalyticsBucketWindow +import me.ahoo.wow.api.query.analytics.AnalyticsCompleteness +import me.ahoo.wow.api.query.analytics.AnalyticsConsistency +import me.ahoo.wow.api.query.analytics.AnalyticsGrouping +import me.ahoo.wow.api.query.analytics.AnalyticsMetric +import me.ahoo.wow.api.query.analytics.AnalyticsMetricKind +import me.ahoo.wow.api.query.analytics.AnalyticsPage +import me.ahoo.wow.api.query.analytics.AnalyticsQuery +import me.ahoo.wow.openapi.BatchComponent +import me.ahoo.wow.openapi.CommonComponent +import me.ahoo.wow.query.analytics.AnalyticsQueryService +import me.ahoo.wow.query.analytics.AnalyticsQueryTrustedContextRequest +import me.ahoo.wow.query.event.filter.EventStreamQueryHandler +import me.ahoo.wow.query.filter.QueryContext +import me.ahoo.wow.query.filter.QueryHandler +import me.ahoo.wow.query.filter.QueryType +import me.ahoo.wow.query.gateway.QueryAuthority +import me.ahoo.wow.query.gateway.QueryCall +import me.ahoo.wow.query.gateway.QueryCallResolutionRequest +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryExecutionMode +import me.ahoo.wow.query.gateway.QueryResourceScope +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.gateway.QueryTrustedContextRequest +import me.ahoo.wow.query.gateway.QueryValidationMode +import me.ahoo.wow.query.snapshot.filter.SnapshotQueryHandler +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.tck.mock.MOCK_AGGREGATE_METADATA +import me.ahoo.wow.webflux.exception.WebFluxRequestExceptionHandler +import me.ahoo.wow.webflux.route.RouteTestFixtures +import me.ahoo.wow.webflux.route.event.LoadEventStreamHandlerFunction +import me.ahoo.wow.webflux.route.snapshot.LoadSnapshotHandlerFunction +import org.junit.jupiter.api.Test +import org.springframework.mock.http.server.reactive.MockServerHttpRequest +import org.springframework.mock.web.reactive.function.server.MockServerRequest +import org.springframework.mock.web.server.MockServerWebExchange +import org.springframework.web.reactive.function.server.HandlerFunction +import org.springframework.web.reactive.function.server.HandlerStrategies +import org.springframework.web.reactive.function.server.ServerRequest +import org.springframework.web.reactive.function.server.ServerResponse +import reactor.core.publisher.Flux +import reactor.core.publisher.Mono +import reactor.kotlin.core.publisher.toMono +import reactor.kotlin.test.test + +class QueryRouteTransportTest { + @Test + fun `all generic query route families should publish exact transport markers`() { + QueryDocumentKind.entries.forEach { documentKind -> + val probe = TransportProbeQueryHandler(documentKind) + val handlers = listOf( + SingleQueryHandlerFunction( + MOCK_AGGREGATE_METADATA, + probe, + documentKind, + DefaultRewriteRequestCondition, + WebFluxRequestExceptionHandler(), + { it }, + ) to request(SingleQuery(Condition.all())), + ListQueryHandlerFunction( + MOCK_AGGREGATE_METADATA, + probe, + documentKind, + DefaultRewriteRequestCondition, + WebFluxRequestExceptionHandler(), + { it }, + ) to request(ListQuery(condition = Condition.all())), + PagedQueryHandlerFunction( + MOCK_AGGREGATE_METADATA, + probe, + documentKind, + DefaultRewriteRequestCondition, + WebFluxRequestExceptionHandler(), + { it }, + ) to request(PagedQuery(condition = Condition.all())), + CountQueryHandlerFunction( + MOCK_AGGREGATE_METADATA, + probe, + documentKind, + DefaultRewriteRequestCondition, + WebFluxRequestExceptionHandler(), + ) to request(Condition.all()), + ) + + handlers.forEach { (handler, request) -> handler.writeAndVerify(request) } + + probe.calls.map(QueryCallResolutionRequest::queryType).assert().containsExactly( + QueryType.DYNAMIC_SINGLE, + QueryType.DYNAMIC_LIST, + QueryType.DYNAMIC_PAGED, + QueryType.COUNT, + ) + probe.calls.forEach { resolution -> + resolution.target.assert().isEqualTo(QueryTarget(MOCK_AGGREGATE_METADATA, documentKind)) + } + probe.resolvedCalls.forEach { call -> + call.resourceScope.tenantId.assert().isEqualTo("tenant-1") + call.resourceScope.ownerId.assert().isEqualTo("owner-1") + call.resourceScope.spaceId.assert().isEqualTo("space-1") + } + } + } + + @Test + fun `both get load routes should publish the same typed transport marker`() { + val resolvers = QueryWebTransportResolvers { + Mono.just(QueryAuthority.System("route-test", "transport-marker-test")) + } + val calls = mutableListOf() + val resolvedLoadCalls = mutableListOf() + val snapshotTarget = QueryTarget(MOCK_AGGREGATE_METADATA, QueryDocumentKind.SNAPSHOT) + val snapshotHandler = mockk { + every { dynamicSingle(any(), any()) } returns resolvers + .resolve(trustedRequest(snapshotTarget, QueryType.DYNAMIC_SINGLE)) + .map { context -> context.call } + .doOnNext { call -> calls += QueryCallResolutionRequest(call.target, QueryType.DYNAMIC_SINGLE) } + .doOnNext(resolvedLoadCalls::add) + .map { mutableMapOf("value" to "snapshot").toDynamicDocument() } + } + val snapshotRequest = MockServerRequest.builder() + .pathVariable(MessageRecords.OWNER_ID, "owner-1") + .pathVariable(MessageRecords.ID, "aggregate-1") + .header(CommonComponent.Header.SPACE_ID, "space-1") + .build() + LoadSnapshotHandlerFunction( + RouteTestFixtures.MOCK_AGGREGATE_ROUTE_METADATA, + snapshotHandler, + WebFluxRequestExceptionHandler(), + ).writeAndVerify(snapshotRequest) + + val eventTarget = QueryTarget(MOCK_AGGREGATE_METADATA, QueryDocumentKind.EVENT_STREAM) + val eventQuery = slot() + val eventHandler = mockk { + every { dynamicList(any(), capture(eventQuery)) } returns resolvers + .resolve(trustedRequest(eventTarget, QueryType.DYNAMIC_LIST)) + .map { context -> context.call } + .doOnNext { call -> calls += QueryCallResolutionRequest(call.target, QueryType.DYNAMIC_LIST) } + .doOnNext(resolvedLoadCalls::add) + .flatMapMany { Flux.just(mutableMapOf("value" to "event").toDynamicDocument()) } + } + val eventRequest = MockServerRequest.builder() + .pathVariable(MessageRecords.ID, "aggregate-1") + .pathVariable(BatchComponent.PathVariable.HEAD_VERSION, "0") + .pathVariable(BatchComponent.PathVariable.TAIL_VERSION, Int.MAX_VALUE.toString()) + .build() + LoadEventStreamHandlerFunction( + MOCK_AGGREGATE_METADATA, + eventHandler, + WebFluxRequestExceptionHandler(), + ).writeAndVerify(eventRequest) + + calls.assert().containsExactly( + QueryCallResolutionRequest(snapshotTarget, QueryType.DYNAMIC_SINGLE), + QueryCallResolutionRequest(eventTarget, QueryType.DYNAMIC_LIST), + ) + eventQuery.captured.limit.assert().isZero() + resolvedLoadCalls.map { call -> call.resourceScope.tenantId }.assert().containsExactly("(0)", "(0)") + } + + @Test + fun `analytics route should publish a snapshot transport marker without QueryType`() { + val calls = mutableListOf() + val resolvers = QueryWebTransportResolvers { + Mono.just(QueryAuthority.System("route-test", "analytics-transport-marker-test")) + } + val service = object : AnalyticsQueryService { + override val namedAggregate: NamedAggregate = MOCK_AGGREGATE_METADATA + + override fun analyze(query: AnalyticsQuery): Mono = resolvers.resolve( + AnalyticsQueryTrustedContextRequest( + target = QueryTarget(namedAggregate, QueryDocumentKind.SNAPSHOT), + executionMode = QueryExecutionMode.PLANNED, + validationMode = QueryValidationMode.STRICT, + ), + ).doOnNext { context -> calls += context.call } + .map { + AnalyticsPage( + buckets = emptyList(), + nextCursor = null, + consistency = AnalyticsConsistency.EVENTUAL, + completeness = AnalyticsCompleteness.EXACT, + ) + } + } + val request = request( + AnalyticsQuery( + grouping = AnalyticsGrouping.global(), + metrics = listOf(AnalyticsMetric("count", AnalyticsMetricKind.DOCUMENT_COUNT)), + window = AnalyticsBucketWindow(1), + ), + ) + + AnalyticsQueryHandlerFunction( + MOCK_AGGREGATE_METADATA, + service, + WebFluxRequestExceptionHandler(), + ).writeAndVerify(request) + + calls.single().target.assert().isEqualTo(QueryTarget(MOCK_AGGREGATE_METADATA, QueryDocumentKind.SNAPSHOT)) + calls.single().resourceScope.assert().isEqualTo(QueryResourceScope("tenant-1", "owner-1", "space-1")) + } + + private fun request(body: Any): ServerRequest = MockServerRequest.builder() + .pathVariable(MessageRecords.TENANT_ID, "tenant-1") + .pathVariable(MessageRecords.OWNER_ID, "owner-1") + .header(CommonComponent.Header.SPACE_ID, "space-1") + .body(body.toMono()) + + private fun HandlerFunction.writeAndVerify(request: ServerRequest) { + handle(request) + .flatMap { response -> + val exchange = MockServerWebExchange.from(MockServerHttpRequest.get("/query").build()) + response.writeTo(exchange, SERVER_RESPONSE_CONTEXT) + } + .test() + .verifyComplete() + } + + private class TransportProbeQueryHandler( + private val documentKind: QueryDocumentKind, + ) : QueryHandler { + val calls = mutableListOf() + val resolvedCalls = mutableListOf() + private val resolvers = QueryWebTransportResolvers { + Mono.just(QueryAuthority.System("route-test", "transport-marker-test")) + } + + override fun handle(context: QueryContext<*, *>): Mono = Mono.error(UnsupportedOperationException()) + + override fun single(namedAggregate: NamedAggregate, singleQuery: ISingleQuery): Mono = unsupported() + + override fun dynamicSingle( + namedAggregate: NamedAggregate, + singleQuery: ISingleQuery, + ): Mono = resolve(namedAggregate, QueryType.DYNAMIC_SINGLE) + .map { mutableMapOf("value" to "single").toDynamicDocument() } + + override fun list(namedAggregate: NamedAggregate, listQuery: IListQuery): Flux = unsupportedFlux() + + override fun dynamicList( + namedAggregate: NamedAggregate, + listQuery: IListQuery, + ): Flux = resolve(namedAggregate, QueryType.DYNAMIC_LIST) + .flatMapMany { Flux.just(mutableMapOf("value" to "list").toDynamicDocument()) } + + override fun paged(namedAggregate: NamedAggregate, pagedQuery: IPagedQuery): Mono> = unsupported() + + override fun dynamicPaged( + namedAggregate: NamedAggregate, + pagedQuery: IPagedQuery, + ): Mono> = resolve(namedAggregate, QueryType.DYNAMIC_PAGED) + .map { PagedList.empty() } + + override fun count(namedAggregate: NamedAggregate, condition: Condition): Mono = + resolve(namedAggregate, QueryType.COUNT).thenReturn(0) + + private fun resolve(namedAggregate: NamedAggregate, queryType: QueryType): Mono { + val resolution = QueryCallResolutionRequest(QueryTarget(namedAggregate, documentKind), queryType) + calls += resolution + return resolvers.resolve(trustedRequest(resolution.target, queryType)) + .map { context -> context.call } + .doOnNext(resolvedCalls::add) + } + + private fun unsupported(): Mono = Mono.error(UnsupportedOperationException()) + + private fun unsupportedFlux(): Flux = Flux.error(UnsupportedOperationException()) + } + + private companion object { + fun trustedRequest(target: QueryTarget, queryType: QueryType): QueryTrustedContextRequest = + QueryTrustedContextRequest( + QueryCallResolutionRequest(target, queryType), + QueryExecutionMode.LEGACY, + QueryValidationMode.COMPATIBLE, + ) + + val SERVER_RESPONSE_CONTEXT = object : ServerResponse.Context { + private val strategies = HandlerStrategies.withDefaults() + + override fun messageWriters() = strategies.messageWriters() + + override fun viewResolvers() = strategies.viewResolvers() + } + } +} diff --git a/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/query/QueryWebTransportTest.kt b/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/query/QueryWebTransportTest.kt new file mode 100644 index 00000000000..06672e8025e --- /dev/null +++ b/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/query/QueryWebTransportTest.kt @@ -0,0 +1,236 @@ +/* + * Copyright [2021-present] [ahoo wang (https://github.com/Ahoo-Wang)]. + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:OptIn(me.ahoo.wow.query.gateway.ExperimentalQueryGatewayApi::class) + +package me.ahoo.wow.webflux.route.query + +import me.ahoo.test.asserts.assert +import me.ahoo.wow.openapi.CommonComponent +import me.ahoo.wow.query.analytics.AnalyticsQueryTrustedContextRequest +import me.ahoo.wow.query.filter.QueryType +import me.ahoo.wow.query.gateway.CompositeQueryTrustedContextResolver +import me.ahoo.wow.query.gateway.QueryAuthority +import me.ahoo.wow.query.gateway.QueryCallResolutionRequest +import me.ahoo.wow.query.gateway.QueryDocumentKind +import me.ahoo.wow.query.gateway.QueryErrorCategory +import me.ahoo.wow.query.gateway.QueryExecutionException +import me.ahoo.wow.query.gateway.QueryExecutionMode +import me.ahoo.wow.query.gateway.QueryLegacyContextResolver +import me.ahoo.wow.query.gateway.QueryLegacyGrant +import me.ahoo.wow.query.gateway.QueryPurpose +import me.ahoo.wow.query.gateway.QueryResourceScope +import me.ahoo.wow.query.gateway.QueryTarget +import me.ahoo.wow.query.gateway.QueryTrustedContext +import me.ahoo.wow.query.gateway.QueryTrustedContextRequest +import me.ahoo.wow.query.gateway.QueryValidationMode +import me.ahoo.wow.query.gateway.withLegacyQueryCaller +import me.ahoo.wow.serialization.MessageRecords +import me.ahoo.wow.tck.mock.MOCK_AGGREGATE_METADATA +import org.junit.jupiter.api.Test +import org.springframework.mock.web.reactive.function.server.MockServerRequest +import reactor.core.publisher.Mono +import reactor.kotlin.test.test +import java.util.concurrent.atomic.AtomicReference + +class QueryWebTransportTest { + private val target = QueryTarget(MOCK_AGGREGATE_METADATA, QueryDocumentKind.SNAPSHOT) + + @Test + fun `should resolve exact call and authenticated authority from one frozen marker`() { + val authorityRequest = AtomicReference() + val resolvers = QueryWebTransportResolvers { request -> + authorityRequest.set(request) + Mono.just(QueryAuthority.Subject("subject-1", "tenant-1")) + } + val request = MockServerRequest.builder() + .pathVariable(MessageRecords.TENANT_ID, "tenant-1") + .pathVariable(MessageRecords.OWNER_ID, "owner-1") + .header(CommonComponent.Header.SPACE_ID, "space-1") + .build() + + Mono.defer { + resolvers.resolve(trustedRequest(QueryType.DYNAMIC_SINGLE)) + .map { context -> context.call to context.authority } + }.writeQueryWebTransport( + request, + MOCK_AGGREGATE_METADATA, + QueryDocumentKind.SNAPSHOT, + QueryType.DYNAMIC_SINGLE, + ).test() + .consumeNextWith { (call, authority) -> + call.target.assert().isEqualTo(target) + call.purpose.value.assert().isEqualTo("interactive-query") + call.resourceScope.tenantId.assert().isEqualTo("tenant-1") + call.resourceScope.ownerId.assert().isEqualTo("owner-1") + call.resourceScope.spaceId.assert().isEqualTo("space-1") + authority.assert().isEqualTo(QueryAuthority.Subject("subject-1", "tenant-1")) + authorityRequest.get().request.assert().isSameAs(request) + authorityRequest.get().call.assert().isEqualTo(call) + } + .verifyComplete() + } + + @Test + fun `should resolve analytics authority from a dedicated marker without QueryType`() { + val authorityRequest = AtomicReference() + val resolvers = QueryWebTransportResolvers { request -> + authorityRequest.set(request) + Mono.just(QueryAuthority.Subject("subject-1", "tenant-1")) + } + val request = MockServerRequest.builder() + .pathVariable(MessageRecords.TENANT_ID, "tenant-1") + .pathVariable(MessageRecords.OWNER_ID, "owner-1") + .header(CommonComponent.Header.SPACE_ID, "space-1") + .build() + + Mono.defer { + resolvers.resolve( + AnalyticsQueryTrustedContextRequest( + target, + QueryExecutionMode.PLANNED, + QueryValidationMode.STRICT, + ), + ) + }.writeAnalyticsQueryWebTransport(request, MOCK_AGGREGATE_METADATA) + .test() + .consumeNextWith { context -> + context.call.target.assert().isEqualTo(target) + context.call.resourceScope.assert().isEqualTo( + QueryResourceScope("tenant-1", "owner-1", "space-1"), + ) + context.authority.assert().isEqualTo(QueryAuthority.Subject("subject-1", "tenant-1")) + authorityRequest.get().request.assert().isSameAs(request) + authorityRequest.get().call.assert().isEqualTo(context.call) + } + .verifyComplete() + } + + @Test + fun `should reject a marker for another query operation`() { + val resolvers = QueryWebTransportResolvers { Mono.empty() } + val request = MockServerRequest.builder().build() + + resolvers.resolve(trustedRequest(QueryType.COUNT)) + .writeQueryWebTransport( + request, + MOCK_AGGREGATE_METADATA, + QueryDocumentKind.SNAPSHOT, + QueryType.DYNAMIC_SINGLE, + ) + .test() + .expectErrorSatisfies { error -> + error.assert().isInstanceOf(QueryExecutionException::class.java) + (error as QueryExecutionException).category.name.assert().isEqualTo("ACCESS_DENIED") + error.path.assert().isEqualTo("$.executionContext.transport") + error.code.assert().isEqualTo("QUERY_TRANSPORT_CALL_MISMATCH") + } + .verify() + } + + @Test + fun `should not fabricate call or authority without a transport marker`() { + val resolvers = QueryWebTransportResolvers { Mono.error(AssertionError("must not resolve")) } + + resolvers.resolve(trustedRequest(QueryType.COUNT)) + .test() + .verifyComplete() + } + + @Test + fun `web marker with missing authority must not fall through to a legacy grant`() { + val webResolvers = QueryWebTransportResolvers { Mono.empty() } + val legacyResolvers = QueryLegacyContextResolver( + listOf( + QueryLegacyGrant( + callerId = "legacy-caller", + target = target, + purpose = QueryPurpose("legacy-purpose"), + executionMode = QueryExecutionMode.LEGACY, + resourceScope = QueryResourceScope(), + ), + ), + ) + val composite = CompositeQueryTrustedContextResolver(listOf(webResolvers, legacyResolvers)) + val request = MockServerRequest.builder().build() + + Mono.defer { composite.resolve(trustedRequest(QueryType.DYNAMIC_SINGLE)) } + .withLegacyQueryCaller("legacy-caller") + .writeQueryWebTransport( + request, + MOCK_AGGREGATE_METADATA, + QueryDocumentKind.SNAPSHOT, + QueryType.DYNAMIC_SINGLE, + ) + .test() + .expectErrorSatisfies { error -> + error.assert().isInstanceOf(QueryExecutionException::class.java) + (error as QueryExecutionException).code.assert().isEqualTo("AUTHORITY_REQUIRED") + error.path.assert().isEqualTo("$.executionContext.authority") + } + .verify() + } + + @Test + fun `should normalize authority resolver failure`() { + val failure = IllegalStateException("authentication store unavailable") + val resolvers = QueryWebTransportResolvers { Mono.error(failure) } + + resolveWithMarker(resolvers) + .test() + .expectErrorSatisfies { error -> + error.assert().isInstanceOf(QueryExecutionException::class.java) + (error as QueryExecutionException).category.assert().isEqualTo(QueryErrorCategory.ACCESS_DENIED) + error.path.assert().isEqualTo("$.executionContext.authority") + error.code.assert().isEqualTo("AUTHORITY_RESOLUTION_FAILED") + error.cause.assert().isSameAs(failure) + } + .verify() + } + + @Test + fun `should not trust authority resolver query rejection`() { + val failure = QueryExecutionException( + QueryErrorCategory.INVALID_QUERY, + "$.forged", + "FORGED_QUERY_REJECTION", + ) + val resolvers = QueryWebTransportResolvers { Mono.error(failure) } + + resolveWithMarker(resolvers) + .test() + .expectErrorSatisfies { error -> + error.assert().isInstanceOf(QueryExecutionException::class.java) + (error as QueryExecutionException).category.assert().isEqualTo(QueryErrorCategory.ACCESS_DENIED) + error.path.assert().isEqualTo("$.executionContext.authority") + error.code.assert().isEqualTo("AUTHORITY_RESOLUTION_FAILED") + error.cause.assert().isSameAs(failure) + } + .verify() + } + + private fun resolveWithMarker(resolvers: QueryWebTransportResolvers): Mono = + Mono.defer { resolvers.resolve(trustedRequest(QueryType.DYNAMIC_SINGLE)) } + .writeQueryWebTransport( + MockServerRequest.builder().build(), + MOCK_AGGREGATE_METADATA, + QueryDocumentKind.SNAPSHOT, + QueryType.DYNAMIC_SINGLE, + ) + + private fun trustedRequest(queryType: QueryType): QueryTrustedContextRequest = QueryTrustedContextRequest( + QueryCallResolutionRequest(target, queryType), + QueryExecutionMode.LEGACY, + QueryValidationMode.COMPATIBLE, + ) +} diff --git a/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/snapshot/LoadSnapshotHandlerFunctionTest.kt b/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/snapshot/LoadSnapshotHandlerFunctionTest.kt index 0d9fa52234b..e10d914e5d5 100644 --- a/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/snapshot/LoadSnapshotHandlerFunctionTest.kt +++ b/wow-webflux/src/test/kotlin/me/ahoo/wow/webflux/route/snapshot/LoadSnapshotHandlerFunctionTest.kt @@ -13,8 +13,6 @@ package me.ahoo.wow.webflux.route.snapshot -import io.mockk.every -import io.mockk.mockk import me.ahoo.test.asserts.assert import me.ahoo.wow.id.generateGlobalId import me.ahoo.wow.openapi.contract.BuiltInHttpRouteHandlerKeys @@ -23,11 +21,9 @@ import me.ahoo.wow.webflux.exception.WebFluxRequestExceptionHandler import me.ahoo.wow.webflux.route.RouteTestFixtures import me.ahoo.wow.webflux.route.testAggregateRouteContract import org.junit.jupiter.api.Test -import org.springframework.http.HttpMethod import org.springframework.http.HttpStatus -import org.springframework.web.reactive.function.server.ServerRequest +import org.springframework.mock.web.reactive.function.server.MockServerRequest import reactor.kotlin.test.test -import java.net.URI class LoadSnapshotHandlerFunctionTest { @@ -42,13 +38,11 @@ class LoadSnapshotHandlerFunctionTest { aggregateRouteMetadata = RouteTestFixtures.MOCK_AGGREGATE_ROUTE_METADATA ) ) - val request = mockk { - every { method() } returns HttpMethod.GET - every { uri() } returns URI.create("http://localhost") - every { pathVariables()[MessageRecords.ID] } returns generateGlobalId() - every { pathVariables()[MessageRecords.TENANT_ID] } returns generateGlobalId() - every { pathVariables()[MessageRecords.OWNER_ID] } returns generateGlobalId() - } + val request = MockServerRequest.builder() + .pathVariable(MessageRecords.ID, generateGlobalId()) + .pathVariable(MessageRecords.TENANT_ID, generateGlobalId()) + .pathVariable(MessageRecords.OWNER_ID, generateGlobalId()) + .build() handlerFunction.handle(request) .test()