Troubleshooting
Find the thing you are looking at, in Watch or on the box, and read what it means and what to do. Start with the one command that answers most questions.
TroubleshootingOn this page
- Start here
- What you see in Watch
- The box is running, but Watch shows it down
- A machine stays pending
- An environment is down, but its machine is online
- Sessions never appear
- A session says needs input, but nothing is waiting
- A session shows as working, but nothing is running
- An environment is listed by name with nothing else
- An environment I removed is still there, and down
- I renamed an environment and now have two
- I got one alert and then nothing
- No alert arrived at all
- The times look wrong
- Signing a box in
- The code expired
- nucleate login cannot reach Watch
- nucleate whoami says the token is unknown
- On the box
- nucleate will not remove a session
- A session folder is still there weeks later
- A session has no add_repo tool
- A repo is missing from list_repos
- Sessions stop appearing in the vendor's app
- gh returns 401 inside sessions
- Sessions say the sandbox is unavailable
- The disk is filling up
- Agents are running out of memory
- Building and booting a box
- The playbook stops at "Read secrets"
- The playbook cannot connect
- Boot stops and asks for the disk passphrase
- The box stays off after a power cut
- ssh agentbox does not connect
- Still stuck
Start here #
Most questions about a box are answered by one command, run on the box as the agent user:
nucleated check
It reads the config, checks the token file, asks whether each agent server is alive and, on a signed-in box, sends one heartbeat. Each entry has a detail, and for whatever is wrong the detail names it. Then:
nucleated status
That is what the box believes about itself. If it matches what Watch shows, the box and Watch agree and the problem is on the box. If it does not, the problem is between them.
What you see in Watch #
The box is running, but Watch shows it down #
Down means one thing: Watch has not received a heartbeat for longer than the down deadline. The box may be fine and unable to say so. Work down this list on the box:
-
Is the daemon running?
Boxas agent systemctl --user status nucleated.service --no-pagerIf it is not, start it. If it stops again, its last words are in
journalctl --user -u nucleated.service -n 30. If it is missing after every reboot, lingering is off: runsudo loginctl enable-linger agentas a user withsudo. -
Is the box signed in, and does Watch still accept its token?
Boxas agent nucleate whoami"token": "rejected"means the token was revoked or replaced. Runnucleate login. If it says the box is not signed in, the token file is missing, or its group or others can read it: the tools will not use a token file that is not private. Fix it withchmod 600 ~/.config/nucleated/token. -
Can the box reach Watch?
Boxas agent curl -sS -o /dev/null -w '%{http_code}\n' https://api.nucarium.com/healthzAnything but
200is the network: the router, DNS, or a firewall that blocks outgoing HTTPS. -
Is the config asking it to stay quiet?
target = "none"in~/.config/nucleated/config.tomlkeeps a signed-in box from sending anything.
When the cause is fixed the box reports again within one heartbeat, Watch marks it online, and you get one "recovered" alert. Whatever sessions did meanwhile was kept in the spool and arrives in order.
A machine stays pending #
Pending means the machine was added in the console and has never reported. The box has not signed in as that machine. Run nucleate login on it and choose that machine when you approve, or see the list above if it has signed in and still says pending.
An environment is down, but its machine is online #
The box is fine. The agent server for that one login has stopped. On the box:
systemctl --user status agent-server@client-a.service --no-pager
journalctl --user -u agent-server@client-a.service -n 40 --no-pager
The usual causes are a login that has expired (run claude as the agent user and sign in again), and a server that cannot reach its vendor. A unit that is active while Watch says down means the daemon is watching a different unit: compare unit in that environment's table of the config with the unit you just looked at.
Sessions never appear #
The machine is online and its environment is listed, but starting a session changes nothing in Watch. Heartbeats are arriving and events are not, which means the hooks.
-
Are the hooks registered? In a terminal session of that login,
/hookslists them. Ifnucleated hookis not among them, register them. On a team plan, an administrator's managed settings can allow managed hooks only. -
Was the agent server restarted since? A session keeps the hooks it started with. Restart the agent server and start a new session.
-
Do the hooks know their environment? Look in the spool's
rejectedfolder:Boxas agent ls ~/.local/state/nucleated/spool/rejected | headFiles there are notes Watch refused because they name an environment no heartbeat has named. The
--envin the registered hook commands does not match anynamein the config. Print the block again with the right name and replace it. -
Can the hook find
nucleated? A hook runs with the agent server'sPATH. If~/.local/binis not on it, use the full path in the hook commands.
A session says needs input, but nothing is waiting #
A session becomes needs input in two ways that surprise people.
It finished. A turn that ended a minute ago with no reply is waiting on you, by design: an agent that stopped to ask a question looks exactly like this. Send it its next prompt, or end the session, and it clears.
You answered, and it still says waiting. Watch learns that a permission was granted when the tool it was for runs. If the row has not changed a few seconds after the agent carried on, the PostToolUse hook is missing from that login's settings. Print the block again and compare.
A session shows as working, but nothing is running #
The box stopped before the session could say it had. A power cut does this: no "stop" was ever sent. The row is honest about what Watch last heard, and the machine's own status tells you the rest. When the agent server starts again, start a new session; the old row moves to ended when its session does, and drops out of the inbox with its history.
An environment is listed by name with nothing else #
It is hidden: the account has more environments than its plan shows. It is still reporting and nothing is lost. What happens over a limit explains, and upgrading or deleting an unused environment brings it back.
An environment I removed is still there, and down #
Removing an [[environment]] table from the box's config stops the box reporting it. Watch cannot tell that from an agent server that died, so it shows it as down. Delete the environment in the console.
I renamed an environment and now have two #
The name in the box's config is how Watch knows an environment. Changing it there makes a new one. Put the old name back in the config, delete the duplicate in the console, and rename the environment in the console instead.
I got one alert and then nothing #
That is how alerts work: one when something changes, one when it clears, and silence in between. If a box went down at 02:41 and you have heard nothing since, it is still down. Why each fires once.
No alert arrived at all #
- Open the alert history in the console. If the alert is not there, nothing crossed a deadline: a session that failed does not alert, and neither does a box that only went stale.
- If it is there, look at its deliveries.
rejectedis an answer other than success from your destination,timeoutis no answer in ten seconds, andunreachableis no connection. Redirects are not followed. - Press the destination's test button and watch for it on your phone. A token that has expired at the receiving service is the usual cause.
- An alert sent while the account had no enabled destination went nowhere. It is in the history with no deliveries.
The times look wrong #
Every time in Watch is when Watch received something. If "last seen" looks right and a session's order of events looks odd, the box's clock is off. Check it:
timedatectl status | head -5
A wrong clock cannot cause a false "down" or hide a real one, because Watch does not use it for that.
Signing a box in #
The code expired #
A code lasts 15 minutes. Run nucleate login again for a new one.
nucleate login cannot reach Watch #
It exits 1 and says so. Check the box can reach https://api.nucarium.com at all, as in the list at the top of this page.
nucleate whoami says the token is unknown #
"token": "unknown" means Watch could not be reached to ask. It does not mean the token is bad.
On the box #
nucleate will not remove a session #
nucleate: refusing to remove session with unsaved work: web: 2 unpushed commit(s)
It is telling you what exists nowhere else. Push it, commit it, or decide you do not want it, then run nucleate remove-session again. The safety rule lists every reason and what clears each. Never delete a session folder with rm -rf.
A session folder is still there weeks later #
Something in it is unsaved, so neither the end of the session nor the weekly cleanup would touch it. See why:
nucleated housekeeping run cleanup --dry-run
A session has no add_repo tool #
The MCP server is not registered for that login. As the agent user, claude mcp list should show nucleate. If it does not, register it, then start a new session.
A repo is missing from list_repos #
A repo is available when its bare clone exists in the nucleus. Add it with nucleate clone, or on a box built from the recipe add it to repos in your settings and run the playbook with --tags workspace.
Sessions stop appearing in the vendor's app #
The agent server is not running. This is the same as an environment that is down: look at its unit and its journal.
gh returns 401 inside sessions #
On a box built from the recipe, agents use a short-lived GitHub token that a broker writes every 20 minutes. As the agent user:
head -c 4 /run/agent-tokens/github
It should print ghs_. If the file is missing or old, the broker is failing. As the operator:
systemctl status agent-github-token.service --no-pager
Check the app's Client ID and Installation ID in your settings, and that the app is still installed on the repo.
Sessions say the sandbox is unavailable #
bwrap --ro-bind / / true
A permission error means AppArmor is blocking it. Run the playbook with --tags users, which installs the rule that allows it.
The disk is filling up #
df -h /
Then, as the agent user, see which sessions are still on disk and why cleanup kept them:
ls ~/workspace/.worktrees/
nucleated housekeeping run cleanup --dry-run
A session with its dependencies installed takes between one and seven gigabytes. Watch shows the disk filling in the machine's health long before it is full.
Agents are running out of memory #
On a box built from the recipe, every agent session runs under one memory ceiling so that a runaway build cannot take the system down with it. Fit more memory, then raise agents_memory_high and agents_memory_max in your settings and run the playbook with --tags agent_servers.
Building and booting a box #
The playbook stops at "Read secrets" #
The 1Password CLI is not signed in, or a reference in your settings does not match an item. Test one by hand:
op read "op://Infra/agentbox/sudo-password"
The playbook cannot connect #
Check that ssh ops@192.168.1.50 hostname works without asking for a password, and that ansible_host in inventory.ini is the box's address.
Boot stops and asks for the disk passphrase #
The firmware's Secure Boot state changed, usually after a firmware update, so the TPM will not release the key. Type the passphrase at the console. Then find the binding's slot and bind it again:
sudo clevis luks list -d /dev/nvme0n1p3
sudo clevis luks regen -d /dev/nvme0n1p3 -s 1
Use the slot number the first command printed. From Watch this looks like a box that went down and stayed down, which is exactly what happened.
The box stays off after a power cut #
Check the firmware setting that powers the box on when mains returns (Assemble the box and set the firmware), and that the UPS switched its outlets off and on. Prove it works has the test.
ssh agentbox does not connect #
At the box's console, or over your home network:
sudo tailscale status
Still stuck #
The tools on the box are open source, and their issue trackers are the place for a bug: nucleate and nucleated, and the box recipe. For Watch, use the support address on your account page. Say what you saw and paste the output of nucleated check: it contains no secrets and nothing your agents said.