`DcgmHandle` fails to connect when hostname resolves to an IPv6 address

[bfbachmann]

I ran into this issue while trying to get NVSentinel gpu-health-monitor pods to connect to DCGM pods over IPv6 in a K8s cluster.
The DCGM client cannot connect to hosts in cases where the hostname resolves to an IPv6 address.

To Reproduce

1. Use a hostname (nvidia-dcgm.nvidia-gpu-operator.svc) that resolves to an IPv6 address. I'm running an IPv6-only cluster:

root@gpu-health-monitor-dcgm-4:/# nslookup nvidia-dcgm.nvidia-gpu-operator.svc
;; Got recursion not available from fdc3:5d58:ceb7::a
;; Got recursion not available from fdc3:5d58:ceb7::a
;; Got recursion not available from fdc3:5d58:ceb7::a
Server:         fdc3:5d58:ceb7::a
Address:        fdc3:5d58:ceb7::a#53

;; Got recursion not available from fdc3:5d58:ceb7::a
Name:   nvidia-dcgm.nvidia-gpu-operator.svc.cluster.local
Address: fdc3:5d58:ceb7::7:9d59

2. Try to connect to the IPv6 service using its hostname:

root@gpu-health-monitor-dcgm-4:/# dcgmi discovery -l --host nvidia-dcgm.nvidia-gpu-operator.svc:5555
Error: unable to establish a connection to the specified host: nvidia-dcgm.nvidia-gpu-operator.svc:5555
Error: Unable to connect to host engine. Host engine connection invalid/disconnected.

I'll get into why this is failing below.

3. Now, try connect to the IPv6 address that the hostname we used in step 2 resolves to:

root@gpu-health-monitor-dcgm-4:/# dcgmi discovery -l --host [fdc3:5d58:ceb7::7:9d59]
8 GPUs found (Active).
# ...

It works!

The Bug

I think something was missed when IPv6 support was added to DcgmIpc::ConnectTcpAsyncImpl. If you look at the function, it defaults to family AF_INET (IPv4), and only changes it to AF_INET6 (IPv6) if inet_pton succeeds in parsing the hostname as an IPv6 literal. So, if hostname is a hostname like nvidia-dcgm.nvidia-gpu-operator.svc that happens to resolve to an IPv6 address, then it'll still use AF_INET, which will force the connection to happen over IPv4, causing it to fail.

The whole failing call chain is:

DcgmHandle.__init__
  -> dcgmConnect_v3
    -> DcgmClientHandler::GetConnHandleForHostEngine
      -> TryConnectingToHostEngine
        -> DcgmIpc::ConnectTcpAsyncImpl
          -> bufferevent_socket_connect_hostname (called with wrong value for argument `family`)