dolthub/dolt#10866: fix(sql-shell): parse input per character instead of per line

[codeaucafe]

Summary

The interactive dolt sql shell accumulated input one physical line at a time and decided "is this statement done?" by checking whether the trimmed line ended with ;. That check has no awareness of SQL quotes, comments, or the active DELIMITER, causing four bugs reported in #10860–#10862 and #10865. This PR replaces the blind suffix check with dolt's existing per-character StreamScanner (already used by batch mode) and replaces the single-statement Parse call in the shell callback with a scanner-driven split-and-execute loop.

The shell-side completeness check is wired in via a new opt-in IsComplete callback added to the dolthub/ishell fork (PR: dolthub/ishell#8). go/go.mod contains a replace directive pointing at to PR's head commit: 061915e.

Issues fixed

• #10860 — select null; select null; on one line gave syntax error at position 20 near 'select'. The accumulated buffer was fed to sqlparser.Parse once; that parser accepts only a single statement.
• #10861 — select '; was treated as complete because the line ends with ;, even though that ; is inside an open string literal.
• #10862 — select /* ; and select -- ; were treated as complete for the same reason; the ; is inside an open block or line comment.
• #10865 — pressing Enter on an empty line dropped the shell into the -> continuation prompt instead of re-issuing the primary > prompt.

Changes

go/cmd/dolt/commands/sql_statement_scanner.go

• Add NewStreamScannerWithDelimiter(r io.Reader, delimiter string) constructor. The shell processes one command at a time, so the delimiter set by a prior DELIMITER command must be passed in rather than discovered in-band (as batch mode does when scanning a whole script). NewStreamScanner delegates to this and is unchanged for existing callers.
• Add endedAtEOF bool field on StreamScanner, reset at the top of each Scan() call and set when the input runs out before the active delimiter is matched.
• Add three accessors exposing parse state to callers: InsideQuote(), InsideBlockComment(), EndedAtEOF().
• Add IsShellInputComplete(text, delimiter string) bool. Runs the scanner over the accumulated buffer and returns false when the input ends inside an open quote or block comment, or with a trailing unterminated statement. Blank/whitespace-only input returns true so an empty Enter does not trigger continuation mode (fixes Dolt shell expects more input after empty line whereas MySQL shell returns nothing #10865).

go/cmd/dolt/commands/sql.go

• Set IsComplete: IsShellInputComplete in UninterpretedConfig. The ishell library calls this after each physical line with the cumulative buffer and the current delimiter; returning true stops accumulation. This is the single change that makes completeness detection SQL-aware (fixes Dolt shell does not ignore delimiters inside quotes #10861, Dolt shell does not ignore delimiters inside comments #10862, Dolt shell expects more input after empty line whereas MySQL shell returns nothing #10865).
• Replace the single sqlparser.Parse(query) call in the Uninterpreted callback with a NewStreamScannerWithDelimiter split-and-execute loop. Each statement in the buffer is parsed and executed individually so multiple ;-separated statements on one line each run and print their own result set (fixes Dolt shell has issues parsing multiple statements in the same line #10860). Per-statement errors are printed and execution continues with the next statement, matching MySQL CLI semantics.

Testing

Unit tests (go/cmd/dolt/commands/sql_statement_scanner_test.go)

• TestIsShellInputComplete — 22 cases covering each issue branch (multi-statement, open quote, open block/line comment, empty input) plus DELIMITER cases.
• TestEndedAtEOFFlag — 3 sub-tests verifying the endedAtEOF flag is set on EOF-terminated input and cleared on delimiter-terminated input.

Integration tests (integration-tests/bats/sql-shell.bats)

Four new bats tests, one per issue, each driving a real interactive dolt sql session:

• sql-shell: multiple statements on one line (#10860)
• sql-shell: unclosed quote continues input (#10861)
• sql-shell: unclosed comment continues input (#10862)
• sql-shell: empty line stays at primary prompt (#10865)

Merge Strategy

• merge Add ishell replace directive for dolthub/dolt#11202 doltgresql#2846
  • this will allow DoltgreSQL Integration tests in Dolt PR to pass
• After Dolt tests pass, merge ishell PR: Add opt-in IsComplete callback to UninterpretedConfig ishell#8
• Merge this Dolt PR
• clean up replace statements in Doltgres and Dolt go.mod's

Fixes: #10860, #10861, #10862, #10865, #10866

[angelamayxie]

this should be dolthub/ishell

[angelamayxie]

please take a look at the failing tests

[angelamayxie]

remove this line. run go get github/com/dolthub/ishell@[commit] to update go.mod and go mod tidy to clean up go.sum

[codeaucafe]

I don't believe this works for forks/outside contributors like myself. I had to do this same thing when making changes in Doltgres that a Dolt Pr I made needed for it to work/be fixed. That's why I did it this way. I'll find those PRs later tonight to show u what I mean.

[codeaucafe]

forgot to add: but I'll also try that just to be sure again

[angelamayxie]

Okay, I think you're right this doesn't work because it's a forked repo, instead of a regular branch in the original repo.

[codeaucafe]

@angelamayxie thank you for reviewing. the following CI always fail for my outside contributor PRs:

• Benchmark Mini Sysbench...
• Benchmark SQL Correctness...
• Check for correctness_approved label...
• Check for performance_approved label...
• Check formatting, Commuters....

Tests Bats Unix Remote/Bats tests is failing at the Install Maven step. For some reason this CI gate has been failing at this step for me for my last 3-4 contributions this year. I'm not entirely sure and never got a reason when I asked and looked into it.

Test Integration with DoltgreSQL/ test-integration... is failing due to it requiring my change to ishell to pass. I had this same issue with before with a change I made to dolt that required me to make use the replace directive pointing in DoltgreSQL in order for this DoltgreSQL integration test suite to pass. This is why I state previously that this replace directive is required for forked contributions like the ones I have to make as an outside contributor. @nicktobey helped me on that change. These are the two PRs for that for reference/example:

• dolthub/dolt#10227: add dolt_status_ignored dolt system table doltgresql#2196
• dolthub/dolt#5862: Add ignore system table #10227

I made a PR the same way in Doltgres so we can get this integration test passing: dolthub/doltgresql#2846

[codeaucafe]

FYI still working on PR refactor. Made the changes, but reviewing and fixing some issues I found.

[codeaucafe]

@angelamayxie I believe this is ready for another review. Most of the new changes were done in ishell like we discussed:

ishell (the meat) — PR #8

• Relocated SQL StreamScanner into ishell so input can be scanned where the shell reads it — avoids a circular import.
• Rewrote uninterpreted read path for single streaming pass: added uninterpretedReader (adapts line-at-a-time readline into an io.Reader) and readUninterpretedCommand, replacing the old per-line readMultiLinesFunc predicate. Resolved the O(n²) re-scan and the double-scan issues. I wrote a couple unit tests and benchmark tests to verify this, but I wasn't sure if it was overkill to include it or not. If you think its worth I'll make that commit and push it up
• Added Context.Statements to give pre-scanned statements to consumers.
• Fixed Ctrl-D at continuation prompt to discard the partial statement.
• Cleanups: removed a no-op Delimiter() line, reordered a deferred-error check, trimmed stored statement text.
• (Uncommitted local: the scaling test + benchmark.)

dolt — PR #11202

• Deleted the local scanner, made shell handler consume c.Statements instead of re-scanning; repointed batch/debug/filter-branch to ishell.NewStreamScanner.
• Kept the \edit path scanning its own editor output.
• Bumped go.mod/go.sum to the pushed ishell fork pseudo-version.
• Tests: Ctrl-D discard test, large-input correctness test, tightened the quote-continuation assertion, plus the earlier per-issue tests (Dolt shell has issues parsing multiple statements in the same line #10860/Dolt shell does not ignore delimiters inside quotes #10861/Dolt shell does not ignore delimiters inside comments #10862/Dolt shell expects more input after empty line whereas MySQL shell returns nothing #10865).

doltgresql — PR #2846

• go.mod replace bump to the same ishell commit (950b93b)