When a classified internal network goes down during software upgrades, operations slow, schedules scramble, and leadership fields questions. The NSA's recent NSANet disruption is a useful case study. Reports indicated the cause was likely scheduled upgrades, not a cyberattack. The agency stated that mission-critical capabilities remained operational and that its infrastructure is built with multiple layers of redundancy. Staff received a "Code Orange" message closing the campus to most employees, followed by a "Code Gray" notice directing a staggered arrival at 10 a.m. to avoid network congestion.
Most agencies and contractors don't have a written procedure that does what those two messages did. This post provides a template you can adapt today.
Purpose of the Template
This is a degraded-network operations runbook for planned maintenance and unplanned outages on internal networks. It covers decision authority, tiered workforce status codes, and staggered reporting to prevent login surges when service returns. It supports Contingency Planning (CP) controls in NIST SP 800-53 Rev 5 (CP-2, CP-4, CP-10) and Incident Response (IR-4, IR-8). If you handle Controlled Unclassified Information under NIST SP 800-171, it also supports your incident handling and system recovery practices.
Prerequisites
Before adopting the template, confirm you have:
- A current system inventory identifying mission-critical versus administrative services
- A documented Business Impact Analysis with recovery time objectives per service
- An approved emergency notification channel that doesn't depend on the affected network (SMS, out-of-band mass notification, or a phone tree)
- Named incident commander and deputy, with authority to change work schedules
- Agreement with your authorizing official or contracting officer on what outages trigger external notification. For DFARS 252.204-7012 contractors, remember that the 72-hour reporting clock applies to cyber incidents, so your runbook must include a step that rules a cyber cause in or out early.
The Template
Copy, paste, and edit the bracketed fields.
[ORGANIZATION] Degraded Network Operations Runbook
Version: [X.X] Owner: [Role] Last tested: [Date]
1. Trigger conditions
This runbook activates when [system name] is unavailable or degraded for more than [X] minutes, or when a planned maintenance window exceeds its approved duration by [X] minutes.
2. Incident commander authority
The [role] is authorized to declare a status code, adjust shift schedules, and suspend non-essential workflows. Deputy: [role].
3. Cause determination (complete within [X] hours)
- Review change records for maintenance in the last [X] hours
- Security operations confirms or rules out malicious activity
- If a cyber incident cannot be ruled out, escalate to IR-8 incident response plan and begin reporting assessment
4. Status codes
| Code | Meaning | Workforce action |
|---|---|---|
| [Color/Name 1] | Facility closed to non-essential staff | Supervisors notify shift workers individually; await updates |
| [Color/Name 2] | Facility open, staggered reporting | Report at assigned time block in Section 5 |
| [Color/Name 3] | Normal operations restored | Resume standard schedule |
5. Staggered reporting schedule
- Block A (mission-critical roles): [time]
- Block B (supporting roles): [time + X minutes]
- Block C (administrative roles): [time + X minutes]
Login surge control: [Block C may be held until services stabilize / authentication throttling enabled by [role]].
6. Communications
- Primary channel: [out-of-band system]
- Update cadence: every [X] minutes until restored
- Message owner: [role]
- External notifications: [authorizing official / contracting officer / customer] within [X] hours
7. Continuity of mission-critical functions
For each service in the BIA, list the alternate path: [redundant link / failover system / manual procedure / alternate processing site].
8. Recovery and closeout
- Mission-critical services verified by function owners
- Logs preserved for post-incident review
- After-action report due within [X] days
- Plan updated and lessons recorded
Customizing the Template
Status codes. Use names your workforce already recognizes. The point is that a one-line message tells every employee what to do. Keep the count to three or four.
Thresholds. Set trigger durations from your BIA, not from guesswork. A system with a short recovery objective needs a short trigger.
Staggered blocks. Assign blocks by role criticality, not seniority. Group users who authenticate heavily at the start of the day (analysts pulling queries, for example) into later blocks if the goal is to ease load, or earlier blocks if their work is the priority. Decide which and write it down.
Redundancy mapping. Section 7 is where the runbook earns its keep. For classified environments, redundant paths must stay within your authorization boundary and, where Commercial Solutions for Classified or other approved architectures apply, within approved configurations. Don't invent a workaround that bypasses your security controls to restore convenience.
Cause determination. The window in Section 3 should be short. A maintenance explanation is plausible, but plausible isn't confirmed. Treat the benign hypothesis as unproven until security operations signs off.
Validation Steps
- Tabletop it. Run a scenario where a patch cycle overruns and the cause is initially unclear. Confirm the incident commander can reach every role through the out-of-band channel.
- Test the notification path. Send a live test message through the channel that doesn't rely on the affected network. Measure how long delivery takes.
- Simulate the login surge. In a test environment or during a low-risk window, bring a representative user group online at once and observe authentication and application load. Compare against your staggered plan.
- Verify the redundancy claims. Every alternate path in Section 7 should be exercised, consistent with CP-4 testing requirements. A failover that has never been tested is an assumption.
- Review against your SSP. Confirm the runbook's actions match what your System Security Plan describes. If assessors compare them, inconsistencies become findings.
- Schedule retests. Test at least annually, and after any major architecture or staffing change.
Planned maintenance will keep causing outages. What separates a slowdown from a mission impact is whether your people already know what a status message means when it arrives.





