Skip to Content

Incidents

New. Behind the incidents feature flag (on by default). Turn it off in Settings → Feature flags or with shep settings flags disable incidents; that also turns off POST /api/alerts.

An incident is a production problem recorded in a space. Shep keeps its timeline, asks the space’s agent what is likely wrong, and can restart, roll back or scale the Kubernetes deployment behind it — after you approve, or without asking for the actions you allow. When it’s over, you resolve it with a postmortem.

Every incident is also recorded as an Incident signal (urgent for Critical and Major), so recurring outages compete for review time like any other evidence.

Before you start

  • Recording and triage need nothing extra. An incident without a workload is a record: you can triage it and keep its timeline, but runtime actions are off.
  • Runtime actions need kubectl on the machine running Shep, with a kubeconfig that can reach the cluster. Shep runs kubectl with that kubeconfig and the context you name (or your current context). It reads deployment status, events and logs during triage, and runs rollout restart, rollout undo or scale for actions.
  • Only Kubernetes deployments are supported as workloads.

Open an incident

shep incident open "Checkout 5xx" --space acme --severity critical \ --workload checkout --namespace shop --context prod
OptionMeaning
--severitycritical, major (default) or minor
-d, --detail, --urlWhat is happening; a link to the alert or dashboard
-w, --workloadThe Kubernetes deployment it concerns
-n, --namespaceIts namespace (default default)
--contextIts kubeconfig context (default: your current one)

Or use the form at the top of the Incidents page (/incidents).

Let alerts open incidents

A monitoring tool can open incidents with a space feedback key — the same key that posts customer feedback. Create one with shep feedback key create --space acme --name Grafana.

curl -X POST http://localhost:4050/api/alerts \ -H "Authorization: Bearer shep_fb_…" \ -H "Content-Type: application/json" \ -d '{ "title": "Checkout 5xx above 5%", "severity": "Critical", "detail": "Error rate on POST /checkout rose to 7% at 09:58.", "url": "https://grafana.acme.com/d/checkout", "externalId": "grafana-alert-812", "namespace": "shop", "workload": "checkout" }'
FieldRequiredMeaning
titleyesWhat is wrong
severitynoCritical, Major (default) or Minor
detailnoMore text
urlnoAn http(s) link to the alert or dashboard
externalIdnoThe alert’s id. While that incident is unresolved, a repeat adds a note instead of opening a new one
context, namespace, workloadnoWhere the workload runs

Responses: 201 opened, 200 repeat of an unresolved incident, 400 bad payload, 401 unknown or revoked key, 413 body over 16 KB. The body is { "id": "<incident id>", "duplicate": false }.

As with feedback, the endpoint is on your machine’s Shep server; a hosted monitoring tool needs a tunnel or reverse proxy you set up.

Triage

shep incident triage <incident>

Or press Triage. Shep reads the deployment’s status, recent events and logs with kubectl and records them as evidence. Then it asks the space’s agent — with no tools and nothing outside this incident — for a summary, up to three causes ranked by confidence, and at most one runtime action. The causes and the proposed action go on the timeline.

--agent <type> picks another agent; the space’s allowed agents and logins apply.

An open Minor incident on the Incidents page: deployment/search in namespace shop, a Roll back action proposed by the agent waiting for approval with Approve and Reject buttons, and a timeline with evidence, three ranked hypotheses and the proposed action

Act

An action the agent proposes waits for a person:

shep incident show <incident> # timeline and action ids shep incident approve <action> # runs at once shep incident reject <action> --reason "deploy is fine"

Or use Approve / Reject on the page. You can also act yourself; an action you ask for runs at once:

shep incident act <incident> rollback --reason "errors began with 4.12.0" shep incident act <incident> scale --replicas 6 shep incident act <incident> restart

After an action runs, Shep waits up to three minutes for the rollout to become ready and records Recovered or Not recovered. The first recovery marks an open incident Mitigated.

Let Shep act without asking

shep space config acme --auto-actions restart

Actions in the list run as soon as triage proposes them. In the browser, tick them under Runs without asking on incidents in the space’s Agent settings. shep space config acme --clear auto-actions goes back to always asking.

Rollbacks and scales change what runs in production. Allow them without approval only where that is safe.

Autopilot never runs incident actions; this setting is the only way they run without a person.

Resolve

shep incident note <incident> "Paged the DBA" shep incident resolve <incident> shep incident resolve <incident> --postmortem "$(cat postmortem.md)"

Resolving without --postmortem writes a draft from the timeline and actions: the timeline, each action with its result, and follow-ups to fill in. On the page, write the postmortem in the Postmortem box or leave it empty for the draft, then press Resolve.

Statuses: Open → Mitigated → Resolved.

CommandDoes
shep incident open <title> [options]Open an incident
shep incident ls [--open]List incidents, newest first; --open for unresolved only
shep incident show <incident>Timeline and actions
shep incident note <incident> <text>Add a note to the timeline
shep incident triage <incident> [--agent <type>]Read the workload and ask an agent for causes and one action
shep incident act <incident> <restart|rollback|scale> [-r <replicas>] [--reason <text>]Run an action now
shep incident approve <action> / reject <action> [--reason <text>]Decide on a proposed action
shep incident resolve <incident> [--postmortem <markdown>]Resolve it

Good to know

  • Workload, namespace and context names are validated before they reach kubectl and are never read as flags. Scale accepts up to 100 replicas.
  • kubectl runs without a shell, on the machine running Shep, as whoever that kubeconfig authenticates.

See also