Hear from the box when Watch cannot be reached
If Watch itself is unreachable, nothing tells you an agent is waiting. Give the daemon one address of your own and the box tells you itself, with names and times only.
nucleatedOn this page
What it covers #
Watch tells you when a box goes quiet. Nothing in Watch can tell you when Watch is the thing that cannot be reached: the box's reports go nowhere, a session stops to ask you something, and the night passes.
The notifier is the box speaking for itself. With one address of your own in the daemon's config, nucleated posts to it:
- Once when Watch has been unreachable for ten minutes, or the time you set.
- Once for each session that starts waiting on you while that lasts.
- Once when Watch can be reached again.
Every post goes to the address you set and to nowhere else. None of it goes to Nucarium. It is off unless you configure it.
Set it up #
[notify]
url = "https://ntfy.example.com/agentbox-fallback"
token_file = "/home/agent/.config/nucleated/notify-token"
after = "10m"
| Key | Default | Meaning |
|---|---|---|
url |
None | Where to post. It must be https, or http for a service on the box itself. With no url the notifier is off. |
token_file |
None | A file holding a token for that address. It is sent as a bearer token. Keep the file readable by the agent user only. |
after |
"10m" |
How long Watch must be unreachable before the first post. At least "2m". |
The daemon picks the change up by itself. The address takes the same request as a Watch alert destination, so a push topic that already receives your alerts works as it is. A topic of its own makes it plain which posts came from the box.
On a box built from the recipe, set box_notify_url in group_vars/all.yml, put a reference to the token in secret_refs.box_notify_token if the address needs one, and run:
cd ~/nucarium-box && ansible-playbook site.yml --tags day2
When it runs #
The notifier runs only on a box that reports to Watch: signed in, with the target hosted. A box with no target has no Watch to lose, and posts nothing.
Unreachable means no heartbeat has been accepted. A failed connection, a timeout and an error from Watch's side all count. Two answers do not: Watch refusing the box's token, and Watch telling the box to slow down. In both of those Watch is there and answering, and nucleated status says what it said.
Once the silence has lasted after, the box sends the first post. From then until a heartbeat is accepted, a session that moves into Needs input gets one post of its own. At the next accepted heartbeat the box sends the last post and goes quiet.
What it sends #
Each post is one request in the shape Watch's own alerts have: a title in a header, a line of plain text as the body, and X-Nucarium-Alert-Kind saying which of the three it is. These are the words, with the kind above each:
agentbox cannot reach Nucarium Watch
No heartbeat has been accepted since 02:41 UTC. This box tells you here when a session waits on you.
An agent on client-a is waiting for you
A session on client-a (agentbox) has waited since 03:05 UTC. Watch cannot be reached, so this comes from the box.
agentbox can reach Nucarium Watch again
Heartbeats are accepted again after 48 minutes.
That is the machine's name, the environment's name, and times. A post names no repo and no branch, and nothing a session said. It carries no alert id and no link to open, because Watch never saw it. Times are in UTC.
Once each, across restarts #
The daemon records what it has posted in ~/.local/state/nucleated/notify.json, so a restart in the middle of an outage repeats nothing.
A post that fails is tried again, with the same widening delays the daemon uses for events. If Watch comes back before the first post was ever delivered, neither it nor the "reachable again" post is sent: there is no point telling you about an outage that is already over.
What it cannot cover #
A box with no network. The notifier covers Watch being unreachable, not the box being cut off. A box that cannot reach anything cannot reach your address either. That case is Watch's: the box goes dark, and Watch alerts you.
A box that is off. Nothing on a dead box can speak.
Between them, the two cover each other. Watch notices a silent box; the box notices a silent Watch.
There is also a page you can open to see whether Watch is working, status.nucarium.com. It runs on the same platform as Watch, so it can go quiet when Watch does. The notifier does not: it runs on your box and posts to your address. Is Watch itself working? sets out which of the two tells you what.
What leaves the box #
The three posts above, to the one address in [notify], and only while Watch is unreachable. What leaves the box lists it beside everything else.