Upgrade to Nim 2.2.10, bump ecosystem deps, add nimpretty formatter

[karolyi]

Nim / dependency versions:

• Nim: 2.0.8 → 2.2.10; chronos: 4.0.4 → 4.2.2; json_rpc: 0.5.0 → 0.6.0
• chronicles: 0.10.3 → 0.12.2; serialization: 0.2.6 → 0.5.2
• json_serialization: 0.2.9 → 0.4.4; stew: 0.2.0 → 0.5.0
• faststreams: 0.3.0 → 0.5.1; websock: 0.2.0 → 0.3.0; plus patch bumps for bearssl, nimcrypto, stint, zlib, httputils, testutils, unittest2, regex

Compatibility fixes for new library APIs:

• json_rpc 0.6: req.meth → req.method.get(""); decoder takes string not JsonNode; inject null id for LSP notifications, null→{} for null params
• chronicles 0.12: chronicles_disable_thread_id → chronicles_thread_ids=no; JsonNode log args wrapped in $() throughout all call sites
• Nim 2.2 gcsafe: add {.gcsafe.} to addCallback closures in ls.nim
• nimble dump: pass workingDir instead of file path as CLI argument
• lspsocketclient: manual requestId counter, rename address→connAddress, fix error-cleanup loop to use responses table, tighten proc pragmas

Formatter improvements (routes/lsp.nim, ls.nim):

• Add NlsFormatter enum and nim.formatter config option (nph | nimpretty)
• Add getNimprettyPath/getFormatterPath helpers; advertise documentFormattingProvider when either formatter binary is on PATH
• Fix pipe-full deadlock in format proc: drain stderr concurrently before waiting for process exit; add defer closeWait for process cleanup
• Fix CancelledError path: kill + reap zombie process before returning
• Fix typos in log strings (doenst, formating, modifyed)

Add test_hover.py: standalone Python LSP smoke-test script

[karolyi]

Nim 2.2.x and Dependency Upgrades

This document details all changes made to support Nim 2.2.10 and the newest
versions of the status-im / nim ecosystem libraries.

Nim Version

	Before	After
Nim	== 2.0.8	>= 2.2.10

Dependency Versions

Package	Before	After
chronos	4.0.4	4.2.2
json_rpc	0.5.0	0.6.0
chronicles	0.10.3	0.12.2
serialization	0.2.6	0.5.2
json_serialization	0.2.9	0.4.4
stew	0.2.0	0.5.0
faststreams	0.3.0	0.5.1
bearssl	0.2.5	0.2.8
nimcrypto	0.6.2	0.7.3
stint	0.8.1	0.8.2
zlib	0.1.0	0.2.0
websock	0.2.0	0.3.0
httputils	0.3.0	0.4.1
testutils	0.5.3	0.8.1
unittest2	0.2.4	0.2.5
regex	0.26.1	0.26.3
with	0.5.0	0.5.0 (explicit pin)

New transitive dependencies added to nimble.lock: zlib (now a first-class
lock entry), httputils (promoted), regex and unicodedb (from nitely).
All packages now declare nim as an explicit dependency in the lock file.

Source Code Changes

nimlangserver.nimble

Version constraints changed from strict equality to lower-bound ranges so the
project can build against any compatible future patch release.

nim.cfg / tests/all.nim.cfg / tests/config.nims

chronicles 0.12.x renamed the thread-id compile flag:

# Before
--define:"chronicles_disable_thread_id"

# After
--define:"chronicles_thread_ids=no"

The old flag was silently ignored in newer chronicles, causing unexpected
thread-id output in log lines. The new flag is the canonical switch.

ls.nim

nimble dump invocation (getNimbleDumpInfo):

The dump subcommand no longer accepts a file path as an argument in newer
Nimble. The working directory is now set explicitly instead:

# Before
arguments = @["dump", nimbleFile]

# After
workingDir = nimbleFile.parentDir(),
arguments = @["dump"],

{.gcsafe.} on addCallback closures:

Nim 2.2 enforces gcsafe more strictly in async callback context. Two
callbacks gained the pragma to satisfy the new check:

• projectNext.ns.addCallback do(fut: Future[Nimsuggest]) {.gcsafe.}:
• ls.workspaceConfiguration.addCallback do(futConfiguration: Future[JsonNode]) {.gcsafe.}:

$ conversion in chronicles log calls:

chronicles 0.12 no longer accepts arbitrary JsonNode values as log
arguments; they must be converted to string first:

# Before
res = res.read()
configuration = futConfiguration.read()

# After
res = $res.read()
configuration = $futConfiguration.read()

lstransports.nim

json_rpc 0.6 API: req.meth → req.method.get("")

The RequestRx type renamed meth to method and made it an Option[string]
in json_rpc 0.6. All call sites updated:

# Before
req.meth
# After
req.method.get("")

contentJson["method"] logging fix:

# Before
debug "[Processing Message]", request = contentJson["method"]
# After
debug "[Processing Message]", request = contentJson["method"].getStr()

LSP null-params and notification-id compatibility (json_rpc 0.6 decoder is
stricter):

The LSP spec allows "params": null and notifications without an "id" field.
json_rpc 0.6's decoder requires params to be an object/array and id to be
present on RequestRx. Two fixups are applied before decoding:

# Replace null params with empty object
if contentJson.getOrDefault("params").kind == JNull:
  contentJson["params"] = newJObject()

# Inject a null id for notifications so RequestRx.id decodes cleanly
if "id" notin contentJson:
  contentJson["id"] = newJNull()

var req = JrpcSys.decode($contentJson, RequestRx)

The re-serialisation ($contentJson) is required because JrpcSys.decode in
0.6 takes a string, not a JsonNode.

routes/lsp.nim

didChangeConfiguration log call: conf (a JsonNode) → $conf (string),
same chronicles 0.12 requirement as above.

tests/lspsocketclient.nim

Manual request-ID counter replaces the removed getNextId() helper from
json_rpc. A new requestId: int field is added to LspSocketClient and
incremented manually:

client.requestId += 1
let id = client.requestId
reqJson["id"] = %id          # was %id.num
client.responses[id] = newFut

Field rename: address → connAddress avoids shadowing a stdlib symbol
that caused a compile error under Nim 2.2:

client.connAddress = addresses[0]   # was client.address

Error-cleanup loop updated to use the correct responses table (was
incorrectly referencing the removed awaiting and batchFut fields):

for _, fut in client.responses:
  if not fut.finished:
    fut.fail(localException)
client.responses.clear()

notificationHandle pragma tightened to match the stricter async
dispatcher signature requirements in chronos 4.2:

proc notificationHandle(...): Future[void] {.gcsafe, raises: [].}

except CatchableError → except Exception in the same proc to correctly
silence all exceptions including Defects that may be raised inside Nim 2.2
string formatting.

$ on JsonNode log arguments throughout the file (same chronicles 0.12
requirement).

Formatter Support (routes/lsp.nim, ls.nim)

nimpretty as an alternative formatter

A new NlsFormatter enum and nim.formatter config option let users choose
between nph (default) and nimpretty:

type NlsFormatter* = enum
  nfNph = "nph"
  nfNimpretty = "nimpretty"

# NlsConfig gains:
formatter*: Option[NlsFormatter]

documentFormattingProvider is now advertised as true when either
formatter binary is found on $PATH, not only nph.

Pipe-full deadlock fix and CancelledError cleanup in format

The old format proc called process.waitForExit before reading stderr,
which could deadlock if the formatter wrote enough output to fill the pipe
buffer. The new implementation drains stderr concurrently:

defer: await noCancel(process.closeWait())
let stderrFuture = process.stderrStream.read()   # start draining immediately
let res =
  try:
    await process.waitForExit(InfiniteDuration)
  except CancelledError:
    # waitForExit cancels the SIGCHLD watcher without calling waitpid —
    # kill explicitly and reap the zombie before returning.
    discard process.kill()
    discard await noCancel(stderrFuture)
    try: discard await noCancel(process.waitForExit(InfiniteDuration))
    except CatchableError: discard
    return none(TextEdit)
let stderrOutput = string.fromBytes(await stderrFuture)

The defer: await noCancel(process.closeWait()) ensures the process handle is
always released, even on error paths.

Typo fixes

"doenst" → "doesn't", "formating" → "formatting",
"modifyed" → "modified" in log/message strings.

New File: test_hover.py

A standalone Python 3 manual-testing script that:

1. Spawns nimlangserver on a socket port.
2. Performs a full LSP handshake (initialize → initialized → textDocument/didOpen).
3. Waits for the window/showMessage "Nimsuggest initialized" notification.
4. Issues a textDocument/hover request and prints the result.

Useful for quickly verifying that the server starts, nimsuggest initialises, and
hover responses are returned correctly without running the full test suite.

python3 test_hover.py
python3 test_hover.py --port 19999 --line 5 --char 12 --timeout 90

[karolyi]

I've spent a whole evening with this after getting frustrated that the langserver wasn't really functioning as intended. As it turned out, this was the crucial fix:

+      # LSP allows null or absent params; json_rpc 0.6+ decoder requires array/object
+      if contentJson.getOrDefault("params").kind == JNull:
+        contentJson["params"] = newJObject()
+      # Notifications have no id; RequestRx.id is non-optional so inject null
+      # to avoid raiseIncompleteObject during decode.
+      if "id" notin contentJson:
+        contentJson["id"] = newJNull()

This made the entire langserver smooth again and my sublime text also doesn't go crazy anymore.

[jmgomez]

Thanks for your work. However before we can merge this a few changes has to be made:

• Split the PRs (fixes, updating, test coverage, formatter)
• Follow the convention for the new filenames (test_ prefix should be just t)
• Make sure CI is green

[karolyi]

Before I dissect the changes to separate commits, I need a test run which I can't start here.

[karolyi]

Need another test run but good to go afaik. The "ubuntu hash mismatch" error is nothing I can make sense of. I'm on Artix linux (on this machine), and nimble wrote these hashes.

[karolyi]

I've tried to satisfy the ubuntu test env and modified the checksum to be the one it expects, but now I have another error. People on IRC told me it's more of a ubuntu related problem now, so I'm not sure how to proceed.

Suggestions are welcome.

[karolyi]

Bump.

I still don't know what's wrong with ubuntu.

[moigagoo]

closeWait doesn't require noCancel: https://status-im.github.io/nim-chronos/concepts.html#nocancel

[moigagoo]

@karolyi Hi! Sorry for the slow response.

What is exactly failing with Ubuntu? I'm not seeing the CI run.

UPD: found it: https://github.com/nim-lang/langserver/actions/runs/26177730410/job/77729875902

UPD 2: Apparently, you're facing another error. Please send it. I'm testing locally on Ubuntu and not seeing any errors.

Split the PRs (fixes, updating, test coverage, formatter)
Follow the convention for the new filenames (test_ prefix should be just t)

These things need to be done as well. I think if you split the PR into three (Nim 2.2.10 update, deps update, and adding formatter) it will be much easier to isolate the issue with the CI.

[moigagoo]

@karolyi this is the wrong checksum though, this is why the build won't pass.

Run nimble install -ldy && nimble lock to update the lock file with the correct checksum (or simply edit it manually to 17ec440fdb89f8903db29a17898af590087d2b64).

[moigagoo]

Nimble 0.20.1 used in CI would install the version with sha1 19c7.., newer Nimble would install 17ec44...

This is why the CI failed. I've updated Nimble version, the tests pass now: #432

[moigagoo]

Duplicate lines.

[moigagoo]

Weird try-except fomatting, hard to read.

[moigagoo]

@karolyi do we need this file at all though? Hover is already coverer in tnimlangserver.nim. It's not run as part of the test suite too.

[karolyi]

I needed this to test locally with something independent from nim itself. I'd leave it there just for that, but it's your call. I'm mostly a pythonista so I needed to test the actual results in something that is proven to work.

[moigagoo]

Please remove it from the PR. I totally get the you needed it for local testing but since it's not part of the test suite and duplicates an existing test, it can be safely removed.

[moigagoo]

@jmgomez @karolyi this file is named testhelpers on master. I think it shouldn't be renamed to thelpers because it is not a test suite and the convention doesn't apply.

[moigagoo]

@karolyi please rename it back to testhelpers.nim. This will shrink the PR size.

[moigagoo]

I've cherrypicked the changes that update the deps into a separate PR: #432

[karolyi]

Feel free to cherrypick this PR to its parts, but I'd like to get some credits on it since I've already put in the work. :)

[moigagoo]

@karolyi absolutely! It is your PR, I'm just splitting it into pieces to help it get merged faster.