Resolve backend DNS on a bounded thread pool to avoid head-of-line blocking

[carlodrift]

Fixes #1833.

SeparatePoolInetNameResolver runs every backend hostname lookup on a single thread (Executors.newSingleThreadExecutor). Because the lookups are blocking, one slow resolve serializes all the others behind it: an unrelated stalled DNS query delays connecting players to otherwise-healthy servers until it returns or the JDK resolver times out. We saw fallback connections to a hostname-defined server hang for 10–18s while the same name resolved instantly on another thread.

This swaps the single-thread executor for a small bounded pool (ThreadPoolExecutor, 0–8 threads, 60s keep-alive, SynchronousQueue), so a slow lookup only ties up one thread instead of blocking all resolution. Threads are created on demand and reaped when idle, so it costs nothing at rest.

[WouterGritter]

Since we use java 21, aren't virtual threads a better fit for this?

[carlodrift]

InetAddress.getByName resolves in a native call, which pins the carrier on 21, so a stuck lookup still holds a real OS thread and virtual threads bring back the same serialization. A dedicated bounded pool stays predictable.

[WouterGritter]

Interestingly, we did use Netty's async DNS resolver here in the past: 6e7c029

But this seems to come with some downsides (as the javadoc comment on SeparatePoolInetNameResolver also suggests). Wanted to suggest this as an alternative but that seems like a no-go.

One nit:
When 8 concurrent DNS requests occur, we will now throw RejectedExecutionException on the next request (SynchronousQueue has no size).
Something like

ThreadPoolExecutor executor = new ThreadPoolExecutor(
    MAX_RESOLVE_THREADS, MAX_RESOLVE_THREADS,
    60L, TimeUnit.SECONDS,
    new LinkedBlockingQueue<>(), // unbounded -> never rejects
    new ThreadFactoryBuilder()
        .setNameFormat("Velocity DNS Resolver #%d")
        .setDaemon(true)
        .build());
executor.allowCoreThreadTimeOut(true);

Wouldn't throw and has the same queueing behavior as before, just with multiple threads (corePoolSize must also be MAX_RESOLVE_THREADS, otherwise the queue would always accept an incoming request without scaling up the amount of threads, resulting in the same single-thread behavior).

[carlodrift]

Good catch, pushed a fix.

[electronicboy]

Netty DNS just turned out to be fairly unreliable for us in the past, worked fine for 99% of people but the 1% who'd have the odd issues with it just wasn't really acceptable; Felt kinda sad because it was fairly nice when it wasn't broken, especially as it defaulted to google DNS or something rather than relying on hosting providers DNS servers

[WouterGritter]

it defaulted to google DNS or something rather than relying on hosting providers DNS servers

Yeah that sounds bad actually. If the hosting provider uses split-horizon DNS for backends or you do it manually with docker, you really need to respect the native DNS resolver