[BugFix] Shadow stale mapped leaves when an override replaces an object parent (#5718)

[Ystk-hsn]

Description

Problem

spath input=body output=log on an index where log is a mapped object field silently answers log.<key> references from the stale mapped leaves instead of the extracted JSON (issue #5718). Within a single row, each leaf independently reads whichever source happens to be mapped — and where log.level = 'ERROR' on the extracted value silently matches nothing.

Root cause

Three interacting behaviors:

1. OpenSearch exposes an object field to the row schema as a struct-parent column (log) plus flattened leaf columns (log.level, log.src) side by side.
2. projectPlusOverriding replaces only exact-name matches, so spath's rewrite (eval log = json_extract_all(body)) replaces the parent column but leaves the stale leaf columns in the schema.
3. QualifiedNameResolver prefers an exact-name column over map access, so log.level resolves to the stale leaf while log.msg (unmapped) falls through to the fresh map.

Fix

Add the mirror of the existing dropStructParentsFor step: when an override replaces a column that was an object/nested parent (MAP/ARRAY-typed), also drop its flattened leaf columns (dropStructChildrenFor). The replacement value then shadows the entire <name>.* subtree — issue #5718's preferred option 1.

The children-drop is type-gated on the replaced column being container-typed. A scalar column that merely shares a dotted prefix with user-created literal columns (eval x.y = 1 | eval x = 2) keeps its children, preserving the SPL1 literal-field semantics that PR #5351 restored. Same gating style as the existing parent-drop: it only fires when the override actually replaced a container column.

This also fixes a companion defect found while reproducing: with stale leaves present, a subsequent eval log.level = 'patched' fired the override path and dropStructParentsFor removed the freshly extracted map (Field [log] not found). With the stale leaves gone, the assignment creates a literal column and the parent survives, consistent with the #5185 semantics.

Behavior changes (only when an assignment collides with a mapped object/nested parent)

Query pattern	Before	After
spath output=<object parent> → leaf reference	stale mapped value, silent	extracted value
same → where on leaf	silently matches stale value	filters on extracted value
same → unrelated doc triggers dynamic mapping	extraction silently retires to null	unaffected
spath ... path=... / eval <parent> = <scalar> → leaf reference	stale value, silent	clear resolution error (mirrors the flat-keyword collision behavior)
spath collision → eval a dotted leaf	Field not found error	works; parent map survives
user literal dotted columns under a scalar prefix	preserved	preserved (type gate)

Default (fields *) output shape is unchanged — the stale leaves were already hidden by tryToRemoveNestedFields; they were only reachable by explicit reference.

Known accepted corner: a literal dotted column created under a mapped object parent is dropped together with the mapped leaves when the parent itself is overridden (provenance of individual columns is not tracked).

Testing

• New CalcitePPLSpathCollisionIT (10 tests) written against the expected semantics before the fix: 7 failed on the unfixed build reproducing every symptom (A/B/C and path-mode) plus the companion defect; 3 guard tests pinned the already-correct behaviors. All 10 pass with the fix.
• Reproduced on main (issue was reported against 3.8; confirms main is affected).
• Regression suites all green: Spath/Eval/Bin/Rex/Parse/Trendline/Flatten/Patterns/MapPath ITs (278+ tests), :ppl:test spath units, :core:test, :integ-test:yamlRestTest (includes the issues/5185.yml regression).
• Documented the collision behavior in docs/user/ppl/cmd/spath.md (prose only; no new doctest blocks).

Related Issues

Resolves #5718

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed. (N/A — not a new command)
• API changes companion pull request created. (N/A)
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created. (N/A — in-repo user manual updated)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ No major issues detected

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Add null-safety for field lookup Add null-safety check before calling getType() on the field. If getField() returns null for a non-existent field name, calling getType() will throw a NullPointerException. Filter out null fields before accessing their type. core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java [1432-1435] Set<String> overriddenContainerParents = overriddenNames.stream() - .filter(name -> isContainerType(originalRowType.getField(name, true, false).getType())) + .map(name -> originalRowType.getField(name, true, false)) + .filter(field -> field != null && isContainerType(field.getType())) + .map(field -> field.getName()) .collect(Collectors.toSet()); Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies a potential NullPointerException when getField() returns null. The overriddenNames are derived from filtering against originalFieldNameSet, so fields should exist, but defensive programming is valuable here. The improved code properly handles the null case.	Medium
General	Validate field resolution before casting Verify that context.relBuilder.field(f) doesn't return null before casting to RexNode. If a field name exists in the row type but cannot be resolved by the builder, this could cause issues. Add validation or filter out unresolvable fields. core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java [1491-1501] private void dropStructChildrenFor(String parentName, CalcitePlanContext context) { String prefix = parentName + "."; List<String> fieldNames = context.relBuilder.peek().getRowType().getFieldNames(); List<RexNode> childrenToDrop = fieldNames.stream() .filter(f -> f.startsWith(prefix)) - .map(f -> (RexNode) context.relBuilder.field(f)) + .map(f -> context.relBuilder.field(f)) + .filter(node -> node != null) + .map(node -> (RexNode) node) .toList(); if (!childrenToDrop.isEmpty()) { context.relBuilder.projectExcept(childrenToDrop); } } Suggestion importance[1-10]: 7 __ Why: This suggestion asks to verify that context.relBuilder.field(f) doesn't return null. While the field names come from the row type, adding null-safety is a reasonable defensive measure. However, since this is primarily a verification request rather than fixing a confirmed bug, the score is moderate.	Medium