[BUG] spath output field silently ignored when it collides with an existing object field's mapped subfield

[penghuo]

Query Information

PPL Command/Query:

source=coll_obj | spath input=body output=log | fields log.level

Expected Result:

log.level should return the value extracted from the JSON in body, i.e. ERROR.

log.level
---------
ERROR

Actual Result:

log.level returns the pre-existing mapped value instead. The spath extraction is silently discarded — no error, no warning.

log.level
------------
MAPPED-DEBUG

Dataset Information

Dataset/Schema Type

• OpenTelemetry (OTEL)
• Simple Schema for Observability (SS4O)
• Open Cybersecurity Schema Framework (OCSF)
• Custom (details below)

Index Mapping

log is an object field whose subfields log.level and log.src are mapped as keyword. body holds a JSON string that also contains a level key.

{
  "mappings": {
    "properties": {
      "log": {
        "properties": {
          "level": { "type": "keyword" },
          "src":   { "type": "keyword" }
        }
      },
      "body": { "type": "text" }
    }
  }
}

Sample Data

{
  "log": { "level": "MAPPED-DEBUG", "src": "real-object" },
  "body": "{\"level\":\"ERROR\",\"msg\":\"from json\"}"
}

Bug Description

Issue Summary:

When a spath output name collides with an existing object field that has mapped subfields, references to <output>.<key> silently resolve to the pre-existing mapped subfield rather than the extracted JSON value. The extraction is discarded with no error.

This is inconsistent with the flat-keyword collision case, which does not fail silently:

existing mapping of log	spath input=body output=log | fields log.level
keyword	returns the extracted value ERROR; and misusing log as a string raises a clear error: EQUAL function expects {[IP,IP],[COMPARABLE_TYPE,COMPARABLE_TYPE]}, but got [STRUCT,STRING]
object with log.level: keyword	returns the stale mapped value MAPPED-DEBUG, silently

Related but distinct from #5152 (MAP paths failing to resolve in downstream commands) and #5185 (eval dropping the MAP root). In both of those, resolution fails and the user sees an error. Here resolution succeeds and silently answers from the wrong source.

Steps to Reproduce:

1. Enable the Calcite engine:

curl -s -X PUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' \
  -d '{"transient":{"plugins.calcite.enabled":true}}'

2. Create the index with log as an object having mapped subfields:

curl -s -X PUT 'localhost:9200/coll_obj' -H 'Content-Type: application/json' -d '{
  "mappings": {"properties": {
    "log":  {"properties": {"level": {"type":"keyword"}, "src": {"type":"keyword"}}},
    "body": {"type": "text"}
  }}}'

3. Index the sample document:

curl -s -X POST 'localhost:9200/coll_obj/_doc?refresh=true' -H 'Content-Type: application/json' \
  -d '{"log":{"level":"MAPPED-DEBUG","src":"real-object"},
       "body":"{\"level\":\"ERROR\",\"msg\":\"from json\"}"}'

4. Run the query — the mapped value is returned instead of the extracted one:

source=coll_obj | spath input=body output=log | fields log.level

log.level = MAPPED-DEBUG        (expected: ERROR)

5. Symptom A — per-subfield split. Within a single row, log.* draws from two different sources depending on whether each individual leaf happens to be mapped:

source=coll_obj | spath input=body output=log | fields log.level, log.msg, log.src

log.level    | log.msg   | log.src
MAPPED-DEBUG | from json | real-object
   ^mapped      ^spath      ^mapped

log.level is mapped, so the mapped value wins. log.msg is not in the mapping, so the spath value is used. fields log alone does return the full extracted map, confirming the extraction did happen and is simply unreachable through log.level:

source=coll_obj | spath input=body output=log | fields log

{'msg': 'from json', 'level': 'ERROR'}

6. Symptom B — filters silently return wrong results. Filtering for the extracted value matches nothing, with no error; filtering for the stale mapped value matches:

source=coll_obj | spath input=body output=log | where log.level = 'ERROR' | fields log.level

(0 rows)

source=coll_obj | spath input=body output=log | where log.level = 'MAPPED-DEBUG' | fields log.level

log.level = MAPPED-DEBUG

7. Symptom C — dynamic mapping silently retires a working extraction. log.msg resolves to the spath output only because it is currently unmapped. Index one unrelated document containing log.msg so dynamic mapping adds the subfield:

curl -s -X POST 'localhost:9200/coll_obj/_doc?refresh=true' -H 'Content-Type: application/json' \
  -d '{"log":{"level":"X","src":"y","msg":"DYNAMICALLY-MAPPED"},
       "body":"{\"level\":\"E2\",\"msg\":\"json-2\"}"}'

Re-run the same query. The answer for the original, unmodified document changes from from json to null:

source=coll_obj | spath input=body output=log | fields log.msg

before:  from json
after:   null                    (original doc)
         DYNAMICALLY-MAPPED      (newly indexed doc)

8. Related oddity — path mode into a colliding name. log and log.level end up as unrelated columns, where log.level is not a subfield of log:

source=coll_obj | spath input=body output=log path=level | fields log, log.level

log   | log.level
ERROR | MAPPED-DEBUG

Impact:

Queries return stale values from the index instead of freshly extracted JSON, silently. Symptom B is the most damaging: a where clause on the extracted field returns zero rows with no error, so a dashboard or alert filtering on log.level can silently stop matching anything.

Symptom C means this is not stable over time — an extraction that works today can start returning null after an unrelated document triggers dynamic mapping, with no change to the query or to the data being queried.

Because the keyword-collision case raises a clear type error, users can reasonably assume collisions are always reported. In the object case they are not.

Expected behavior

Either behavior would be acceptable, provided it is consistent with the keyword case and never silent:

1. Preferred — spath output=log shadows the entire log.* subtree, so log.level reads the extracted value; or
2. reject the query with a clear error when output collides with an existing object field that has mapped subfields, mirroring the existing [STRUCT,STRING] error in the keyword case.

At minimum, a silent per-leaf mix of mapped and extracted values under one log.* prefix should not be possible.

Workaround

Use a non-colliding output name. Both values are then reachable and mean what was written:

source=coll_obj | spath input=body output=parsed | fields parsed.level, log.level

parsed.level | log.level
ERROR        | MAPPED-DEBUG

Environment Information

OpenSearch Version:

Branch 3.8, built from source — plugin 3.8.0.0-SNAPSHOT on OpenSearch 3.8.0-SNAPSHOT. Calcite engine enabled (plugins.calcite.enabled=true).

Additional Details:

main is expected to be affected as well — the spath and field-resolution code paths involved are unchanged between 3.8 and main. This was not verified against a main build.

docs/user/ppl/cmd/spath.md does not currently document collision behavior for the output parameter, which makes the difference between the keyword case (clear error) and the object case (silent) hard to discover.

Screenshots

N/A — all output above is from the _plugins/_ppl REST endpoint.

[Ystk-hsn]

Hi, I'd like to work on this — I've reproduced all four symptoms (A/B/C and the path-mode oddity) on current main, confirming the issue is not specific to the 3.8 branch. Root cause matches the description: the exact-name override in projectPlusOverriding replaces only the struct-parent column, leaving the stale flattened leaves in the row schema where QualifiedNameResolver's exact-name match wins. I have a fix implementing the preferred option (shadowing the entire output.* subtree, type-gated to preserve the SPL1 literal dotted-column semantics from #5351) with IT coverage — PR coming shortly.

[Ystk-hsn]

Opened #5726.