DM-55571: Czar-level EXPLAIN query support by malensek · Pull Request #1086 · lsst/qserv

malensek · 2026-08-25T02:28:33Z

This adds support for EXPLAIN <query> that runs at the czar to analyze queries before they execute. This could be particularly useful in situations where a user wants to understand how their query is being scheduled, how complex it is, or how much time it may take to execute.

Currently, the following information is reported:

Parsed SQL Representation
Qserv IR
Booleans (Y/N): needs_merge, chunked, scan_interactive
Chunks matched (0 if pruned + isDummy)
Restrictors applied
Merge SQL (if any)
Scan rating
Scan table info
Worker SQL

JSON support is also available via EXPLAIN FORMAT=JSON <query>

Known limitations:

Ideally, it should be possible to issue EXPLAIN queries to workers and collect all of the results into a single comprehensive report. If this incurs too much additional latency, we could support both "shallow" and "deep" variants, or something like EXPLAIN FULL <query>. There is no standardized spec for EXPLAIN behavior so we have some flexibility here.
The current implementation is not aware of queries that can execute entirely on the czar, such as SELECT COUNT(*). One possible workaround is directly executing SELECT queries to their fullest extent and capturing the information there instead of registering a new query type.
~~This PR will currently not merge cleanly on the xrd branch, undergoing investigation...~~ . Partially resolved by adding a new query-only ScanTableInfo to separate concerns between the query/frontend and backend layers. The rest of the minor tweaks went to a separate PR.

Testing:

EXPLAIN FORMAT=JSON
  SELECT filterId,
         COUNT(*) AS detections,
         AVG(psfFlux) AS meanPsfFlux
  FROM Source
  GROUP BY filterId
  ORDER BY detections DESC
  LIMIT 6 \G

Gives us:

*************************** 1. row ***************************
EXPLAIN: {
  "area_restrictors": "none",
  "chunks_matched": 14,
  "is_chunked": true,
  "merge_sql": "SELECT `filterId` AS `filterId`,SUM(`QS1_COUNT`) AS `detections`,(SUM(`QS3_SUM`)/SUM(`QS2_COUNT`)) AS `meanPsfFlux` FROM `qcase01`.`Source` AS `qcase01.Source` GROUP BY `filterId` ORDER BY `detections` DESC LIMIT 6",
  "needs_merge": true,
  "parsed_ir": "SelectStmt(SelectList(ValueExpr(\"filterId\", FactorOp(ValueFactor(ColumnRef(\"TableRef(\"qcase01\", \"Source\", \"qcase01.Source\")\", \"filterId\")), query::ValueExpr::NONE)), ValueExpr(\"detections\", FactorOp(ValueFactor(query::ValueFactor::AGGFUNC, FuncExpr(\"COUNT\", ValueExpr(\"\", FactorOp(ValueFactor(STAR), query::ValueExpr::NONE)))), query::ValueExpr::NONE)), ValueExpr(\"meanPsfFlux\", FactorOp(ValueFactor(query::ValueFactor::AGGFUNC, FuncExpr(\"AVG\", ValueExpr(\"\", FactorOp(ValueFactor(ColumnRef(\"TableRef(\"qcase01\", \"Source\", \"qcase01.Source\")\", \"psfFlux\")), query::ValueExpr::NONE)))), query::ValueExpr::NONE))), FromList(TableRef(\"qcase01\", \"Source\", \"qcase01.Source\")), nullptr, OrderByClause(OrderByTerm(ValueExpr(\"detections\", FactorOp(ValueFactor(query::ValueFactor::AGGFUNC, FuncExpr(\"COUNT\", ValueExpr(\"\", FactorOp(ValueFactor(STAR), query::ValueExpr::NONE)))), query::ValueExpr::NONE)), query::OrderByTerm::DESC, \"\")), GroupByClause(GroupByTerm(ValueExpr(\"filterId\", FactorOp(ValueFactor(ColumnRef(\"TableRef(\"qcase01\", \"Source\", \"qcase01.Source\")\", \"filterId\")), query::ValueExpr::NONE)), \"\")), nullptr, 0, 6)",
  "parsed_sql": "SELECT `qcase01.Source`.`filterId` AS `filterId`,COUNT(*) AS `detections`,AVG(`qcase01.Source`.`psfFlux`) AS `meanPsfFlux` FROM `qcase01`.`Source` AS `qcase01.Source` GROUP BY `filterId` ORDER BY `detections` DESC LIMIT 6",
  "scan_interactive": false,
  "scan_rating": 1,
  "scan_tables": [
    {
      "db": "qcase01",
      "lock_in_memory": true,
      "rating": 1,
      "table": "Source"
    }
  ],
  "secidx_restrictors": "none",
  "worker_sql": "SELECT `qcase01.Source`.`filterId` AS `filterId`,COUNT(*) AS `QS1_COUNT`,COUNT(`qcase01.Source`.`psfFlux`) AS `QS2_COUNT`,SUM(`qcase01.Source`.`psfFlux`) AS `QS3_SUM` FROM `qcase01`.`Source_%C\u0007C%` AS `qcase01.Source` GROUP BY `filterId` ORDER BY COUNT(*) DESC"
}
1 row in set (0.02 sec)

Copilot

Pull request overview

Adds czar-side EXPLAIN support without dispatching worker jobs.

Changes:

Parses traditional and JSON EXPLAIN syntax.
Computes query plans, restrictors, chunk coverage, and scan metadata.
Returns analysis through temporary result tables.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`src/qproc/QuerySession.h`	Exposes chunk setup.
`src/qproc/QuerySession.cc`	Centralizes chunk computation.
`src/proxy/test.sh`	Adds EXPLAIN smoke tests.
`src/proxy/mysqlProxy.lua`	Allows EXPLAIN routing.
`src/ccontrol/UserQueryType.h`	Declares EXPLAIN detection.
`src/ccontrol/UserQueryType.cc`	Parses EXPLAIN syntax.
`src/ccontrol/UserQuerySelect.cc`	Uses shared chunk setup.
`src/ccontrol/UserQueryFactory.cc`	Creates EXPLAIN queries.
`src/ccontrol/UserQueryExplain.h`	Defines the EXPLAIN query type.
`src/ccontrol/UserQueryExplain.cc`	Produces tabular or JSON reports.
`src/ccontrol/testUserQueryType.cc`	Tests EXPLAIN recognition.
`src/ccontrol/CMakeLists.txt`	Builds new implementation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

malensek · 2026-08-25T02:54:04Z

+    try {
+        // Compute chunk coverage
+        qs->setupChunking(sharedResources->secondaryIndex);


Nice catch, Copilot! That will call applyFinal on all the plugins... and it's a no-op on all but one, ScanTablePlugin! 👍

Each of these isX() helper functions are called by UserQueryFactory (sometimes multiple calls) and carried a TRACE-level log that resulted in the function name and query string being printed repeatedly. Since the information is already printed at the DEBUG level (both the query string and how it ends up being handled), these can be safely removed.

ScanTableInfo was previously used on both sides of the aisle between the front-end query analysis layer and the back-end worker dispatch. This separates them into two distinct versions to avoid dependencies reaching across the aisle.

This chunk coverage computation already relied on several parts of QuerySession, and relocating it allows us to use it both during construction of SELECT queries as well as EXPLAIN queries.

This includes a new query type for czar-level EXPLAIN support and the required plumbing to execute it. Since this version of EXPLAIN only runs on the czar, it calculates a variety of query information accessible there, such as the scan rating and number of chunks that will be involved in the query being analyzed. Note that this does *NOT* dispatch EXPLAIN queries to the workers for analysis downstream from the czar.

These tests exercise the syntax surrounding EXPLAIN queries, which is handled in UserQueryFactory and detected via regex.

fritzm

Looks good to me -- thanks!

malensek requested a balanced review from Copilot August 25, 2026 02:30

Copilot started reviewing on behalf of malensek August 25, 2026 02:31 View session

Copilot AI reviewed Aug 25, 2026

View reviewed changes

malensek force-pushed the tickets/DM-55571 branch 2 times, most recently from a1783cb to b2be30c Compare August 25, 2026 04:48

malensek force-pushed the tickets/DM-55571 branch from b2be30c to f0967ea Compare August 27, 2026 18:50

malensek added 5 commits August 27, 2026 12:50

Separate query/dispatch ScanTableInfo

66b3599

ScanTableInfo was previously used on both sides of the aisle between the front-end query analysis layer and the back-end worker dispatch. This separates them into two distinct versions to avoid dependencies reaching across the aisle.

Move setupChunking() into QuerySession

afb0734

This chunk coverage computation already relied on several parts of QuerySession, and relocating it allows us to use it both during construction of SELECT queries as well as EXPLAIN queries.

Add EXPLAIN query regex match tests

a0b8a44

These tests exercise the syntax surrounding EXPLAIN queries, which is handled in UserQueryFactory and detected via regex.

Enable EXPLAIN queries on the proxy

1dff372

malensek force-pushed the tickets/DM-55571 branch from f0967ea to 1dff372 Compare August 27, 2026 19:50

malensek mentioned this pull request Aug 27, 2026

DM-55975: Czar-level EXPLAIN support for xrd branch #1089

Open

malensek marked this pull request as ready for review August 28, 2026 22:38

fritzm approved these changes Aug 30, 2026

View reviewed changes

Navigation Menu

Uh oh!

DM-55571: Czar-level EXPLAIN query support - #1086

DM-55571: Czar-level EXPLAIN query support#1086
malensek wants to merge 6 commits into
mainfrom
tickets/DM-55571

malensek commented Aug 25, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

malensek Aug 25, 2026

Uh oh!

fritzm left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

malensek commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

malensek Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

fritzm left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

malensek commented Aug 25, 2026 •

edited

Loading