Pause alerts for maintenance

Before you reboot a box or take its network down, tell Watch from the box. It puts off the alerts for going dark until a time you choose, and touches nothing else.

On this page

Before planned work #

You are about to reboot the box, swap its memory or move it to another room. Watch will see the silence and buzz your phone about something you are doing on purpose. Say so first, on the box:

Boxas agent
nucleated pause 2h

The duration is a number and a unit, such as 30m or 2h. It can be at most a day: zero, or anything over 24h, is refused with exit code 2 and nothing changes.

The command prints when the pause ends, as JSON, and the running daemon sends a heartbeat at once. Watch knows about the pause before you pull the plug.

What Watch does while it lasts #

Until the time you gave, for this machine and every environment on it:

  • Watch sends no machine down and no environment down alert.
  • The machine and its environments read "Paused for maintenance until 03:10", in the inbox and on their own pages.
  • Statuses change as ever. A box that stops reporting still goes Stale and then Down, and you can see it. Only the alert waits.

Every other alert is sent as usual. A session that has waited on you past the deadline still buzzes your phone during a pause, and so does a box that goes on battery.

To hold every alert about a box, and to do it from the console, there is snooze. It is a different tool: a snoozed alert is never sent, where a paused one is sent at the end if the box is still dark. Not the maintenance pause compares the two.

When it ends #

It ends by itself, at the time you gave. To end it sooner:

Boxas agent
nucleated resume

Running nucleated pause again replaces the time.

What happens to an alert that was put off depends on the box at that moment:

When the pause ends You get
The box and its agent servers are reporting Nothing. It went dark and came back inside the pause.
The box, or an agent server, is still dark The alert that was put off, sent then.

So a pause cannot hide a box that never came back. The end is counted on Watch's clock, not the box's: if the two-hour job goes wrong and the box is still off when the two hours are up, Watch tells you.

An alert the pause releases is then treated like any other. If the machine is snoozed at that moment, or your quiet hours are running and you hold urgent alerts too, it is held and not sent.

What it does not do #

It pauses two alerts and nothing else.

  • No agent is touched. nucleated does not stop, pause or message a session, and neither does Watch. Sessions carry on until whatever you are about to do ends them. A reboot ends them: pick a quiet moment.
  • The daemon keeps working. It sends heartbeats, forwards events and runs the housekeeping as before.
  • Other alerts are not held. A session that waits, fails or gets stuck, an agent server that keeps restarting, a box on battery and a filling disk all alert as they always do.

How Watch learns of it #

nucleated pause writes the end time to ~/.local/state/nucleated/pause.json. While that time is ahead, every heartbeat carries it as paused_until: a time, and nothing else. nucleated resume removes the file, and the first heartbeat without the field ends the pause.

The box's clock is not trusted for this. Watch reads how far ahead of the heartbeat's own client_ts the time is, counts that from when the heartbeat arrived, and keeps at most a day of it. A box whose clock is an hour wrong is paused for the length you asked for.

nucleated status shows a pause that is in force and when it ends.

From your workstation #

It is an ordinary command, so it runs over SSH:

Workstation
ssh agentbox-agent nucleated pause 30m