The socket API connection is never re-established after the client goes away

[Ebrathul]

SocketThread connects to the socket API once, at startup
(socketthread.cpp, connect() around line 118). There is no path anywhere in
the class that dials again. Once that connection is gone, the mount stays up but
can no longer hydrate anything, for as long as it exists.

In the intended architecture the client owns the mount lifecycle, so a clean
client restart tears the mount down and brings it back. The problem is the case
where it does not: if the client crashes, or is killed, or exits without
unmounting, what is left behind is a mount that looks completely healthy —
findmnt shows it, ls and stat work, hydrated files still read — and that
fails every open of a dehydrated placeholder from then on.

Observed

Killing the client under a live mount and starting a new one on the same socket
path. The new client accepts no connection, because nothing dials it:

write: Bad file descriptor
Failed to send msg 25: V2/HYDRATE_FILE
open Failed job 25 ./test.pdf     -> EIO, immediately, and for every open after

The failure itself is well-behaved: EIO straight away, no hang, no partial
file, no corruption. It is the permanence that is the problem, and the fact that
nothing about the mount indicates it has become useless.

On main specifically

socketSendMsg() calls close(_socket) on a failed write but does not set
_socket to -1, so subsequent sends write to a closed descriptor — which the
kernel may by then have reused for something else. #54 changes that to -1 and
publishes the send failure as an explicit failed state, which is what produced
the clean EIO above. Neither version reconnects.

Why I am not sending a patch

There are at least three defensible answers and the choice is yours:

1. Redial with backoff. The mount survives a client restart. Costs a
   reconnect loop and a decision about what to do with hydrations in flight.
2. Exit and unmount when the connection drops. The mount disappears with the
   client that owns it, which matches the lifecycle model in INTEGRATION.md.
   Blunt, but it never leaves a mount that silently cannot work.
3. Keep failing, but say so. Report it through a state the client or the
   user can see, rather than only in the log of a daemon nobody is reading.

I would rather report this accurately than pick one for you.

Environment

Linux 7.1.8 (CachyOS), libfuse 3.18.2, main at 525d8c6 and the #54 branch at
0096111 — both behave the same way in this respect.

[dragotin]

I would say option 2. is the correct handling for now. We have little control if a gone client will be restarted.

[Ebrathul]

Thanks — that makes sense, and it matches what INTEGRATION.md describes.

One question about how you would want it done before anyone writes it, because
it is the difference between a small patch and a change to how the daemon is
structured.

Detecting the drop looks straightforward. The socket is non-blocking, and
readSocket() currently folds three outcomes into one:

ssize_t n = read(_socket, buf, sizeof(buf) - 1);
if (n <= 0)
    return std::string();

n == 0 means the peer closed and the client is gone, whereas n < 0 with
EAGAIN just means nothing arrived on this tick. Separating those two is a few
lines and gives a definite signal.

Acting on it is the part I would rather not decide for you. The session is
started with fuse_main(), which does not hand back a handle, so
fuse_session_exit() is not reachable from there. Two ways I can see:

1. Signal ourselves — kill(getpid(), SIGTERM) — and let the handlers
   fuse_main() installs leave the loop and unmount. Three lines, no
   restructuring.
2. Move main() to fuse_session_new / fuse_session_mount /
   fuse_session_loop, keep the handle and call fuse_session_exit(). Tidier,
   but it changes your startup and teardown, and that felt like something to ask
   about rather than arrive with.

Which would you prefer? I am happy to put it together either way.

Two smaller things while I am here:

• Should hydrations already in flight be failed explicitly before the teardown,
  so waiting open() calls get EIO promptly rather than whatever the unmount
  happens to do to them? I would lean yes.
• Fix hydration race, socket framing, and cross-desktop ignore defaults #54 rewrites a good part of socketthread.cpp, including socketSendMsg. I
  can keep this to readSocket() only, which is enough on its own since the
  timer polls constantly, so the two should not collide.

[Ebrathul]

Following up on my own question rather than leaving it hanging: I went and wrote
the version that does not touch main(), because it was quicker to measure than
to keep guessing at. It is on a branch, not a pull request — if you would rather
have the fuse_session_* restructuring I will drop this one, nothing is lost.

The detection was the easy half. readSocket() returns an empty string for three
quite different things, since the socket is non-blocking: n == 0 is the client
closing, n < 0 with EAGAIN is an idle tick (which is most ticks), and
anything else is a real error that currently passes for idle. Splitting those
apart is where the signal comes from. On EOF I fail the hydration jobs still in
flight — otherwise the open() calls waiting on them sit out the whole retry
loop for nothing — log one line, and raise SIGTERM, so the handlers
fuse_main() installs leave the loop and unmount. A flag keeps the timer from
hitting the same EOF and signalling again on every tick. 47 lines, and the only
addition to the API is SharedMap::failAllJobs().

I ran it on an isolated mount with a stub client, killing the client while a
hydration was outstanding. As things stand the mount outlives its client exactly
as described above: the waiting read was still blocked after 40 seconds and
needed SIGKILL, the mount was still there ten seconds later, every later read
failed and went on failing, and the daemon kept running. With the patch the
waiting read fails at once and the sync root is unmounted about 400 ms after the
client dies.

So this is the exit-and-unmount behaviour you asked for, done the small way. I
still have not touched main()'s use of fuse_main(), since moving it to
fuse_session_new / fuse_session_mount / fuse_session_loop changes startup
and teardown for everyone and that is your call. If you want it that way, the EOF
detection and the job-failing are unchanged and only the last line becomes
fuse_session_exit().

One thing has moved since I asked: #61 rewrites this function into
processSocketInput() and already separates n == 0 from EAGAIN. It drops the
buffer and returns, so the daemon still does not notice its client is gone, but
it does mean that if #61 goes in first this shrinks to a couple of lines inside a
branch that already exists there. I am glad to wait for that and rebase instead
of handing you two versions of the same split.

Also still open from my last comment, in case it is easier to answer now than it
was then: should hydrations already in flight be failed explicitly before the
teardown? I assumed yes and did it that way, since it is the difference between
a waiting reader getting an error immediately and one sitting out the retry loop
during an unmount.