Incident Response Runbook
Security incident response procedures and playbooks
Table of Contents
Security incident response procedures for detecting, containing, and recovering from security incidents.
Incident Response Overview
Incident Response Lifecycle
- Preparation - Tools, training, procedures in place
- Detection & Analysis - Identify and assess incidents
- Containment - Stop the spread
- Eradication - Remove threat from environment
- Recovery - Restore systems to normal operation
- Lessons Learned - Post-incident review
Incident Severity Levels
Severity 1 - Critical
Active data breach
Ransomware encryption in progress
Complete service outage
Nation-state actor activity
Response: Immediate, 24/7, all hands on deck
Timeline: <15 minutes
Severity 2 - High
Suspected data breach
Malware detected on multiple systems
Major service degradation
Unauthorized access detected
Response: Urgent, senior team
Timeline: <1 hour
Severity 3 - Medium
Single compromised account
Malware on isolated system
Security control failure
Policy violation
Response: Standard business hours
Timeline: <4 hours
Severity 4 - Low
Failed login attempts
Suspicious email
Non-critical policy violation
Response: Standard process
Timeline: <24 hours
Incident Response Team
Roles and Responsibilities
Incident Commander
- Overall incident coordination
- Decision-making authority
- Stakeholder communication
- Resource allocation
Security Lead
- Technical security analysis
- Forensics coordination
- Threat intelligence
- Remediation planning
IT Lead
- Systems administration
- Network management
- Access control
- System restoration
Communications Lead
- Internal communications
- External communications (if needed)
- Legal/PR coordination
- Customer notifications
Documentation Lead
- Incident timeline tracking
- Evidence collection
- Action item tracking
- Post-incident report
Detection and Analysis
Common Detection Sources
SIEM/Log Analysis:
- Multiple failed login attempts
- Unusual outbound traffic
- Privilege escalation
- Off-hours access
Antivirus/EDR Alerts:
- Malware detection
- Suspicious process execution
- File modification
- Registry changes
User Reports:
- Suspicious emails
- Unexpected system behavior
- Password reset requests
- Account lockouts
Network Monitoring:
- Unusual traffic patterns
- Data exfiltration
- Command and control (C2) traffic
- Port scanning
Initial Assessment Questions
What happened?
- What was detected?
- When did it occur?
- How was it discovered?
What is affected?
- Which systems/users/data?
- How many assets impacted?
- What is the scope?
What is the impact?
- Data confidentiality compromised?
- System availability affected?
- Data integrity in question?
Is it contained?
- Is it still spreading?
- Can we isolate it?
- Do we need to take systems offline?
Incident Response Playbooks
Playbook 1: Ransomware Attack
Indicators:
- Files being encrypted (.encrypted, .locked extension)
- Ransom notes on desktops
- Crypto-locker processes running
- Mass file modifications
Immediate Actions (0-15 minutes):
# 1. ISOLATE AFFECTED SYSTEMS IMMEDIATELY
# Disconnect from network (DO NOT SHUT DOWN)
# Physical: Unplug network cable
# Virtual: Disable network adapter
# Windows
Disable-NetAdapter -Name "Ethernet" -Confirm:$false
# Linux
ip link set eth0 down
# 2. Identify affected systems
# Check SIEM/logs for similar activity
# Check backup systems (ensure they're not compromised)
# 3. Alert incident response team
# Send notification to IR team
# Escalate to management immediatelyContainment (15-60 minutes):
# 1. Prevent spread
# Block C2 domains/IPs at firewall
# Disable user accounts showing compromise
# Isolate network segments if needed
# Windows - Block outbound connections
New-NetFirewallRule -DisplayName "Block Ransomware C2" `
-Direction Outbound `
-RemoteAddress 198.51.100.0/24 `
-Action Block
# 2. Take snapshots/images before remediation
# VM: Create snapshot via hypervisor
# Physical: Image disk if possible
# Preserve evidence for forensics
# 3. Identify ransomware variant
# Submit sample to:
# - VirusTotal
# - Hybrid Analysis
# - ID Ransomware (id-ransomware.malwarehunterteam.com)Eradication and Recovery:
# DO NOT pay ransom (FBI recommendation)
# 1. Verify backups are clean
# Scan backup files for malware
# Verify backup integrity
# Test restoration of sample files
# 2. Wipe and rebuild affected systems
# Reinstall OS from known-good media
# Restore from clean backups
# Update and patch before reconnecting
# 3. Reset all credentials
# Force password reset for all users
# Rotate service account passwords
# Regenerate API keys
# 4. Enhance monitoring
# Increase log retention
# Monitor for reinfection
# Watch for lateral movementPrevention Checklist:
- Backups tested and offline/immutable
- Email filtering and anti-phishing in place
- EDR/antivirus up to date
- Patch management current
- Network segmentation implemented
- Principle of least privilege enforced
- MFA enabled for all admin accounts
- User security awareness training
Playbook 2: Compromised User Account
Indicators:
- Login from unusual location
- Off-hours access
- Multiple failed login attempts followed by success
- Unusual email activity (sent/forwarded emails)
- Changes to inbox rules
- Password reset requests
Immediate Actions:
# 1. Disable the compromised account
# Active Directory
Disable-ADAccount -Identity username
# Azure AD
Set-AzureADUser -ObjectId user@company.com -AccountEnabled $false
# 2. Revoke all active sessions
# Azure AD
Revoke-AzureADUserAllRefreshToken -ObjectId user@company.com
# 3. Review recent activity
# Azure AD sign-in logs
Get-AzureADAuditSignInLogs -Filter "userPrincipalName eq 'user@company.com'" -Top 50
# Check email activity (Office 365)
Search-MailboxAuditLog -Identity user@company.com -StartDate (Get-Date).AddDays(-7) -ShowDetailsInvestigation:
# 1. Check for malicious inbox rules
Get-InboxRule -Mailbox user@company.com
# Remove suspicious rules
Remove-InboxRule -Identity "Rule Name" -Mailbox user@company.com
# 2. Check for email forwarding
Get-Mailbox user@company.com | Select-Object ForwardingAddress, ForwardingSMTPAddress
# Remove forwarding
Set-Mailbox user@company.com -ForwardingAddress $null -ForwardingSMTPAddress $null
# 3. Review sent items
# Check for phishing emails sent from account
# Check for data exfiltration
# 4. Check for file access/downloads
# Review file server logs
# Check cloud storage activity (OneDrive, SharePoint)Recovery:
# 1. Reset password
Set-ADAccountPassword -Identity username -Reset -NewPassword (ConvertTo-SecureString -AsPlainText "TempP@ssw0rd!" -Force)
Set-ADUser -Identity username -ChangePasswordAtLogon $true
# 2. Re-enable account (after verification)
Enable-ADAccount -Identity username
# 3. Notify user
# Inform user of compromise
# Provide new temporary password
# Require security awareness training
# 4. Monitor account for 30 days
# Watch for unusual activity
# Review login locations
# Check for re-compromisePlaybook 3: Malware Infection
Indicators:
- Antivirus/EDR alert
- Unusual process execution
- High CPU/network usage
- Pop-ups or unexpected behavior
- Files modified/encrypted
Immediate Actions:
# 1. Isolate the system
# Disconnect from network (preserve memory)
# Do NOT shut down (loses volatile data)
# 2. Identify the malware
# Check antivirus logs
# Review process list
# Check scheduled tasks/startup items
# Windows
Get-Process | Sort-Object CPU -Descending
Get-ScheduledTask | Where-Object {$_.State -eq "Ready"}
Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Run
# Linux
ps aux --sort=-%cpu
crontab -l
cat /etc/cron*/*Containment:
# 1. Kill malicious processes
# Windows
Stop-Process -Id PID -Force
# Linux
kill -9 PID
# 2. Block C2 communication
# Add firewall rules to block malicious IPs/domains
# 3. Preserve evidence
# Take memory dump
# Windows (using DumpIt or similar)
.\DumpIt.exe
# Linux
dd if=/dev/mem of=/tmp/memory.dump bs=1M
# Capture running processes
# Windows
Get-Process | Export-Csv C:\temp\processes.csv
# Linux
ps aux > /tmp/processes.txtEradication:
# 1. Run full antivirus scan
# Update definitions first
# Run in Safe Mode if possible
# 2. Remove persistence mechanisms
# Delete scheduled tasks
# Remove startup entries
# Check browser extensions
# Review Windows services
# 3. Wipe and rebuild if critical
# Backup user data (scan first)
# Reinstall OS
# Restore from clean backupPlaybook 4: Data Breach
Indicators:
- Large data download/upload
- Unauthorized database access
- Credentials found on dark web
- Customer data exposed
- Third-party breach notification
Immediate Actions:
# 1. Contain the breach
# Identify the exposure point
# Stop ongoing exfiltration
# Revoke compromised credentials
# 2. Assess scope
# What data was exposed?
# How many records affected?
# Who is impacted (customers, employees)?
# What is the sensitivity of the data?
# 3. Preserve evidence
# Don't delete logs
# Take system snapshots
# Document timeline
# Secure forensic copiesLegal and Compliance:
# 1. Notify legal team immediately
# Breach may have legal implications
# Attorney-client privilege for communications
# 2. Determine notification requirements
# GDPR: 72 hours to notify regulator
# HIPAA: 60 days to notify affected individuals
# State breach laws: Varies by state
# PCI-DSS: Notify card brands immediately
# 3. Prepare notifications
# Regulatory notifications
# Customer notifications
# Public disclosure (if required)
# Credit monitoring offers (if applicable)
# 4. Engage third parties
# Forensics firm
# Legal counsel
# PR firm
# Insurance provider (cyber insurance)Investigation:
-- Database breach investigation
-- Check access logs
SELECT * FROM audit_log
WHERE timestamp > '2024-11-01'
AND user NOT IN (SELECT username FROM authorized_users);
-- Check for data export
SELECT * FROM query_log
WHERE query LIKE '%SELECT%'
AND rows_returned > 1000;
-- Identify affected records
SELECT customer_id, ssn, credit_card
FROM customers
WHERE customer_id IN (SELECT DISTINCT customer_id FROM breach_access_log);Recovery:
# 1. Remediate vulnerability
# Patch systems
# Fix configuration errors
# Implement additional controls
# 2. Enhanced monitoring
# Increase log retention
# Deploy additional detection
# Monitor for misuse of breached data
# 3. Strengthen security
# Implement data encryption
# Enhance access controls
# Review security policies
# Mandatory security trainingPlaybook 5: DDoS Attack
Indicators:
- Service unavailable
- Extremely high traffic volume
- Network congestion
- Slow performance
Immediate Actions:
# 1. Confirm DDoS attack
# Check traffic patterns
netstat -an | grep SYN_RECV | wc -l
# Check bandwidth usage
iftop -i eth0
# Review firewall/IPS logs
tail -f /var/log/firewall.log | grep DROP
# 2. Identify attack type
# SYN flood
# UDP flood
# HTTP flood
# DNS amplificationMitigation:
# 1. Engage DDoS mitigation service
# Cloudflare
# AWS Shield
# Akamai Prolexic
# Radware
# 2. Implement rate limiting
# Limit connections per IP
# nginx
limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
limit_req zone=one burst=20;
# iptables
iptables -A INPUT -p tcp --dport 80 -m limit --limit 25/minute --limit-burst 100 -j ACCEPT
# 3. Block attack sources
# Null-route attacking IPs
ip route add blackhole 198.51.100.0/24
# Firewall blocks
iptables -A INPUT -s 198.51.100.0/24 -j DROP
# 4. Communicate with ISP
# Request upstream filtering
# Provide attack details
# Request additional bandwidth if neededPost-Incident Activities
Incident Report Template
# Incident Report: [Incident ID]
**Incident Summary:**
- Date/Time: 2024-11-02 14:30 EST
- Severity: High (Sev 2)
- Incident Type: Ransomware Attack
- Status: Resolved
- Duration: 6 hours
**Incident Timeline:**
- 14:30 - Initial detection (user report)
- 14:35 - IR team activated
- 14:40 - Affected systems isolated
- 15:00 - Scope identified (5 workstations)
- 15:30 - Containment achieved
- 17:00 - Systems rebuilt from backup
- 18:30 - Systems back online
- 20:30 - Incident closed
**Root Cause:**
Phishing email with malicious attachment opened by user.
Lack of email filtering allowed attachment through.
User clicked on macro-enabled document.
**Impact Assessment:**
- Systems Affected: 5 workstations
- Users Impacted: 5 employees
- Data Loss: None (restored from backup)
- Downtime: 4 hours per workstation
- Estimated Cost: $10,000 (labor + recovery)
**Actions Taken:**
1. Isolated affected systems
2. Identified ransomware variant (Ryuk)
3. Blocked C2 domains at firewall
4. Rebuilt systems from clean images
5. Restored user data from backups
6. Reset user passwords
7. Enhanced email filtering
**Lessons Learned:**
- Email filtering insufficient
- User training needed
- Backup process worked well
- IR team response time good
- Need better EDR solution
**Recommendations:**
1. Implement advanced email filtering (Priority: High)
2. Deploy EDR solution (Priority: High)
3. Conduct phishing simulation training (Priority: Medium)
4. Review and update backup procedures (Priority: Medium)
5. Implement network segmentation (Priority: Low)
**Follow-up Actions:**
- [x] Systems restored and operational
- [x] Users can resume work
- [ ] Email filtering upgraded (Due: 2024-11-15)
- [ ] EDR solution deployed (Due: 2024-12-01)
- [ ] User training scheduled (Due: 2024-11-30)
- [ ] Security review completed (Due: 2024-12-15)Lessons Learned Meeting
Attendees:
- IR team members
- Management
- IT staff
- Affected department leads
Discussion Topics:
- What went well?
- What could be improved?
- Were procedures followed?
- What was unexpected?
- What changes are needed?
Action Items:
- Document with owner and due date
- Track to completion
- Update IR procedures based on findings
Incident Response Tools
Essential IR Toolkit
Investigation:
- Wireshark - Packet capture and analysis
- Sysinternals Suite - Windows system utilities
- Volatility - Memory forensics
- Autopsy - Disk forensics
- KAPE - Evidence collection
Malware Analysis:
- VirusTotal - Multi-engine malware scanner
- Hybrid Analysis - Automated malware analysis
- REMnux - Malware analysis Linux distro
- IDA Pro/Ghidra - Reverse engineering
Network Analysis:
- Zeek (Bro) - Network security monitoring
- Suricata - IDS/IPS
- NetworkMiner - Network forensics
- tcpdump - Packet capture
Log Analysis:
- Splunk / ELK Stack - SIEM
- Graylog - Log management
- Chainsaw - Windows event log analysis
Communication:
- Secure chat (Signal, Wickr)
- Conference bridge
- War room (physical or virtual)
IR Preparation Checklist
Before an Incident
- IR plan documented and distributed
- IR team identified with contact info
- IR tools and access provisioned
- Runbooks created for common scenarios
- Communication templates prepared
- Legal/PR contacts identified
- Forensics retainer established
- Cyber insurance policy in place
- Backups tested and verified
- Monitoring and detection in place
- Network diagrams updated
- Asset inventory current
- Credentials documented (secure vault)
- IR training completed
- Tabletop exercises conducted
During an Incident
- Incident declared and severity assigned
- IR team activated
- War room established
- Timeline tracking initiated
- Evidence preservation started
- Management notified
- Containment actions taken
- Regular status updates provided
- Legal/compliance considerations addressed
- External parties engaged if needed
After an Incident
- Incident report completed
- Lessons learned session held
- IR procedures updated
- Follow-up actions tracked
- Security improvements implemented
- Team debriefed
- Knowledge base updated