Deadlines
How long Watch waits before it calls a box stale or down, or a waiting session overdue, and why every one of those clocks runs on Watch's time and never the box's.
WatchOn this page
The four numbers #
| Deadline | Default | What it decides |
|---|---|---|
| Heartbeat | Every 60 seconds | How often a box reports. |
| Stale | After 3 missed heartbeats | When a machine or agent server is marked Stale. |
| Down | After 10 minutes of silence | When it is marked Down, and you are alerted. |
| Waiting too long | After 15 minutes | When a Needs input session alerts you. |
So with the defaults, a box that loses power at 02:31 is stale at 02:34 and down at 02:41, and that is when your phone hears about it.
On the free plan #
A faster heartbeat is one of the things the paid plans pay for. On Seed the shortest heartbeat is five minutes, and the other deadlines stretch to fit it:
| Deadline | Seed | Grove and Orchard |
|---|---|---|
| Heartbeat | Every 5 minutes | Every 60 seconds |
| Stale | After 2 missed (10 minutes) | After 3 missed (3 minutes) |
| Down | After 15 minutes | After 10 minutes |
| Waiting too long | After 15 minutes | After 15 minutes |
A dead box takes about 15 minutes to alert on Seed and about 10 on a paid plan. A waiting session alerts after 15 minutes on every plan: that clock does not depend on the heartbeat, because the message that says a session is waiting is sent the moment it happens.
You do not set the heartbeat on the box. Watch answers every heartbeat with the interval your account expects, and nucleated follows it.
Change them #
The console's settings page shows the deadlines in force and the range your plan allows for each. Within that range they are yours to tune:
- A box on a poor connection may want a longer down deadline, so a ten-minute outage at the router does not wake you.
- If you would rather hear sooner that an agent is waiting, shorten waiting too long.
A change applies to every environment in the account at once. Clearing a field returns it to your plan's default. Two rules hold whatever you choose: no deadline can be shorter than your plan's shortest heartbeat, and down can never come before stale.
Why Watch uses its own clock #
Every last-seen time and every deadline is measured from the moment Watch received a message. The time on the box is never used to decide anything.
The reason is that a box's clock cannot be trusted, and the cost of trusting it is a false alarm or a missed one:
- A box whose clock is five minutes fast would look alive for five minutes after it died.
- A box whose clock is slow would look dead while it was still reporting.
- A box that boots without network time can believe it is years ago until it synchronises.
None of that matters to Watch. A heartbeat that arrived ten seconds ago arrived ten seconds ago, whatever the box thought the time was. That one rule is why you can believe a "down".
The box's clock is still sent with every message. Watch keeps it for two things only: to put the events of one session in the order they happened when a batch arrives late, and so you can see it when you are working out why something looks odd. A box with a badly wrong clock shows up there, and nowhere else.
What "missed" means #
Watch does not check on a box. It never connects to one. It waits, and it knows when the next heartbeat is due. If that time passes with nothing received, the heartbeat is missed.
This is why a dead box and a box with no internet look the same from Watch: in both cases nothing arrived. It is also why a stopped nucleated looks like a dark box. Watch reports what it can honestly know, which is how long it has been since it heard anything. Two last-seen times help you tell the cases apart, and so does Troubleshooting.
Late and repeated messages #
Networks drop out, and nucleated keeps what it could not send and sends it later. Watch is built for that:
- A message it has already seen is ignored. Every message carries an id made on the box, so a retry changes nothing.
- A message that arrives after a newer one for the same session is kept for the record and changes nothing. A session cannot flicker backwards.
- A message for a session that has ended is ignored. Only a new start brings a session id back to life, and then as a new session.