The spool
A folder of small files between the hooks and the network. It is why a dropped connection never loses the note that says an agent is waiting for you.
nucleatedWhat it is #
Hooks write; the daemon sends. Between them is a folder:
~/.local/state/nucleated/spool/
019ab3f2-6d1c-7e4a-9b0f-3c2d1e0f4a5b.json one note, waiting to be sent
019ab3f2-7a40-7c11-8d2e-5f6a7b8c9d0e.json
rejected/ notes Watch refused for good
Each file is one event, as it will be sent, with two things beside it that stay on the box (when it was written, and the session folder's name). What is sent is the fields listed under what is kept from an event, and no others. Files are readable by the agent user only. A note is written under a temporary name and renamed into place, so the daemon never reads half a file.
The file's name is the event's id, made when the note is written. Watch remembers the ids it has seen, so sending the same note twice changes nothing. That is what makes retrying safe.
When notes are sent #
| Note | Sent |
|---|---|
| A session needs input | At once |
| Anything else | In a batch every ten seconds, or sooner if 50 are waiting, at most 100 to a request |
A note is deleted when Watch has answered that it recorded it, or that it already had it.
When sending fails #
The daemon tries again: after one second, then two, then four, doubling up to five minutes between tries, with a little randomness so a fleet of boxes coming back from the same outage does not arrive in step. It keeps going for as long as the notes are worth sending.
Nothing is lost while it waits. A box that was offline for an hour sends that hour's notes when the network returns, in the order they happened. Watch judges each note by the time it arrives, and uses the box's own clock only to put one session's notes in order, so a late batch cannot make a session look more recent than it is.
What Watch answers decides what happens next:
| Watch says | The daemon |
|---|---|
| Recorded, or already had it | Deletes the note. |
Too fast (429) |
Waits as long as Watch asks, then tries again. |
Token not accepted (401) |
Keeps every note and stops sending. nucleated status says to run nucleate login; once you have, it carries on from where it stopped. |
| This note will never be accepted | Moves it to rejected/ and does not try again. |
| Nothing, or a server error | Tries again with the delay above. |
A note is refused for good in one case that matters: it names an environment no heartbeat from this machine has named. That happens if a hook was registered with an --env that is not in the config file. Troubleshooting covers the fix.
Limits #
| Setting | Default | What happens at the limit |
|---|---|---|
max_age |
"7d" |
A note older than this is dropped unsent. By then it says nothing useful. |
max_events |
10000 |
The oldest routine notes are dropped first. A note that says a session needs input is the last thing dropped. |
Files in rejected/ follow max_age too.
Both are set in the [spool] table of the config file. At a few hundred bytes a note, a full spool is a few megabytes.
Look at it #
nucleated status shows how many notes are waiting and how old the oldest is. The files themselves are plain JSON:
ls ~/.local/state/nucleated/spool | head
A spool that is empty, or holds a handful of files a few seconds old, is healthy. A spool that keeps growing means nothing is getting through: see Troubleshooting.
With no target #
When the box is not signed in, or target is "none", there is nowhere to send a note. The daemon applies each one to its own record of the session, which is what nucleated status shows, and deletes the file.