Recurring alert runbook draft

Use this process when an alert has already caused repeat pages, handovers or incident work. You will produce a runbook that tells the responder what the alert means, what to check first, which mitigations are safe, when to escalate, and who owns each action.

This is for service owners and on-call engineers who need a usable first-response document, not a summary of past incidents. Set aside time with someone who knows the service before you publish anything.

1. Collect a bounded evidence pack

Start with one recurring alert. Do not try to document an entire service or every alert in the monitoring system.

Create a working folder or document containing:

• The alert name, exact alert text, severity, monitored service and dashboard link.
• Three to five recent alert instances, including timestamps, duration and whether a human action was needed.
• The incident notes or tickets for the most useful previous occurrences.
• Current architecture notes: dependencies, data stores, queues, deployment path and regional setup where relevant.
• Existing operating procedures, support boundaries and escalation contacts.
• The current service owner’s notes on known failure modes and actions they consider safe.

Remove credentials, access tokens, customer data and internal personal details before sharing material with the model. Replace them with labels such as [production database] or [on-call channel]. Keep the source records available, because the draft must be checked against them.

2. Separate facts from gaps

Read the evidence pack and make a two-column note before prompting. This prevents the model from filling missing operational detail with plausible wording.

Record as a fact	Record as a gap
The alert fires when a named metric crosses its stated condition	Whether the condition still matches the intended service behaviour
A previous incident was mitigated by restarting a worker	Whether a restart is currently safe, authorised and effective
A database team joined a past incident	Whether that team owns the present dependency and escalation
The alert followed a deployment twice	Whether deployment was the cause or only happened nearby in time

Ask the service owner to resolve the gaps that affect a first response. In particular, confirm actions that can change production state: restarts, rollbacks, traffic changes, queue replays, feature changes and manual data operations.

If you are using an API or workspace workflow, check the current xAI documentation overview for version-dependent implementation details. Your evidence and review process matter more than a particular interface.

3. Give the model a constrained drafting brief

Paste the evidence pack and your fact-and-gap note. Then use a prompt that requires the runbook structure and makes uncertainty visible. Replace the text in brackets.

Draft a production runbook for the recurring alert below.

Alert: [exact alert name and text]
Service: [service name]
Audience: primary on-call responder, then service owner

Use only the supplied evidence. Do not infer commands, permissions,
owners, thresholds, links or mitigations that are not supported by it.
Where evidence is missing, write: VERIFY WITH SERVICE OWNER.

Return Markdown with these sections:
1. Alert meaning
2. Immediate impact and scope
3. Initial checks, in order
4. Safe mitigation steps, with preconditions and stop conditions
5. Escalation triggers and destination team
6. Ownership table, with action, owning team, and evidence/source
7. Things not to do
8. Open questions before publication

Make each initial check observable: say what dashboard, log, deploy record
or dependency state to inspect, what result is expected, and what result
changes the next step. Keep the first-response path short.

Evidence:
[paste sanitised alert history, incident notes and operating knowledge]

The phrase “use only the supplied evidence” is important. It changes the output from a confident general-purpose procedure into a draft with visible unknowns.

4. Turn the draft into an executable runbook

Review the returned document with the person who owns the service. Edit it until every action has a clear actor and outcome.

For initial checks, use an order that reduces uncertainty without changing production. A good sequence is usually:

1. Confirm the alert is active and identify the affected service, region or tenant scope.
2. Check user impact and error or latency behaviour.
3. Check recent deployments, configuration changes and dependency health.
4. Compare current signals with the previous incidents named in the evidence.
5. Choose a mitigation only after its stated preconditions are met.

For every mitigation, add three fields if they are missing:

• Precondition: what must be true before this action is safe.
• Expected result: which signal should improve, and over what observation period your team uses.
• Stop condition: when to stop, revert or escalate rather than repeat the action.

Do not publish vague steps such as “check logs” or “restart the service”. Name the log view or dashboard, the decision it supports, and the approval needed for a state-changing action.

5. Check where the output can be wrong

The most dangerous errors are not spelling mistakes. They are incorrect certainty, missing boundaries and stale ownership.

Check each statement against the evidence pack. For each mitigation, ask: “What proves this is safe?”, “Who can authorise it?”, and “What tells the responder to stop?” If the answer is not in the evidence or confirmed by an owner, change the text to an open question or remove the step.

Use this review table before publication:

If you find this	Correct it by
An exact threshold absent from the alert definition	Remove it or mark it for owner verification
A mitigation from one old incident presented as standard practice	Add its preconditions, or move it to historical context
A named team with no current confirmation	Verify the ownership route with the service owner
A check with no decision attached	State what each result means and the next step
An action that changes production with no rollback path	Add a stop condition and escalation trigger, or remove it

6. Publish, test and keep the habit

Store the approved runbook beside the alert configuration or in the operational documentation location your team actually uses. Link the alert to it. Put the service owner, review date and source incident references at the top.

At the next real alert, ask the responder to record which step they used, which information was missing and whether the escalation route worked. Update the runbook within the incident follow-up, while the details are still known. Review it again after material changes to alert logic, dependencies, ownership or deployment practice.

When the draft does not work, do not keep asking for a better rewrite from the same thin evidence. Return to the evidence pack. Add the missing alert definition, current ownership decision or approved mitigation boundary. Then generate a new constrained draft and repeat the check with the service owner.