Security incident response procedures for detecting, containing, and recovering from security incidents.

Incident Response Overview

Incident Response Lifecycle

  1. Preparation - Tools, training, procedures in place
  2. Detection & Analysis - Identify and assess incidents
  3. Containment - Stop the spread
  4. Eradication - Remove threat from environment
  5. Recovery - Restore systems to normal operation
  6. Lessons Learned - Post-incident review

Incident Severity Levels

Severity 1 - Critical

  • Active data breach
  • Ransomware encryption in progress
  • Complete service outage
  • Nation-state actor activity

Response: Immediate, 24/7, all hands on deck Timeline: <15 minutes

Severity 2 - High

  • Suspected data breach
  • Malware detected on multiple systems
  • Major service degradation
  • Unauthorized access detected

Response: Urgent, senior team Timeline: <1 hour

Severity 3 - Medium

  • Single compromised account
  • Malware on isolated system
  • Security control failure
  • Policy violation

Response: Standard business hours Timeline: <4 hours

Severity 4 - Low

  • Failed login attempts
  • Suspicious email
  • Non-critical policy violation

Response: Standard process Timeline: <24 hours

Incident Response Team

Roles and Responsibilities

Incident Commander

  • Overall incident coordination
  • Decision-making authority
  • Stakeholder communication
  • Resource allocation

Security Lead

  • Technical security analysis
  • Forensics coordination
  • Threat intelligence
  • Remediation planning

IT Lead

  • Systems administration
  • Network management
  • Access control
  • System restoration

Communications Lead

  • Internal communications
  • External communications (if needed)
  • Legal/PR coordination
  • Customer notifications

Documentation Lead

  • Incident timeline tracking
  • Evidence collection
  • Action item tracking
  • Post-incident report

Detection and Analysis

Common Detection Sources

SIEM/Log Analysis:

  • Multiple failed login attempts
  • Unusual outbound traffic
  • Privilege escalation
  • Off-hours access

Antivirus/EDR Alerts:

  • Malware detection
  • Suspicious process execution
  • File modification
  • Registry changes

User Reports:

  • Suspicious emails
  • Unexpected system behavior
  • Password reset requests
  • Account lockouts

Network Monitoring:

  • Unusual traffic patterns
  • Data exfiltration
  • Command and control (C2) traffic
  • Port scanning

Initial Assessment Questions

  1. What happened?

    • What was detected?
    • When did it occur?
    • How was it discovered?
  2. What is affected?

    • Which systems/users/data?
    • How many assets impacted?
    • What is the scope?
  3. What is the impact?

    • Data confidentiality compromised?
    • System availability affected?
    • Data integrity in question?
  4. Is it contained?

    • Is it still spreading?
    • Can we isolate it?
    • Do we need to take systems offline?

Incident Response Playbooks

Playbook 1: Ransomware Attack

Indicators:

  • Files being encrypted (.encrypted, .locked extension)
  • Ransom notes on desktops
  • Crypto-locker processes running
  • Mass file modifications

Immediate Actions (0-15 minutes):

Lang: bash
 1# 1. ISOLATE AFFECTED SYSTEMS IMMEDIATELY
 2# Disconnect from network (DO NOT SHUT DOWN)
 3# Physical: Unplug network cable
 4# Virtual: Disable network adapter
 5
 6# Windows
 7Disable-NetAdapter -Name "Ethernet" -Confirm:$false
 8
 9# Linux
10ip link set eth0 down
11
12# 2. Identify affected systems
13# Check SIEM/logs for similar activity
14# Check backup systems (ensure they're not compromised)
15
16# 3. Alert incident response team
17# Send notification to IR team
18# Escalate to management immediately

Containment (15-60 minutes):

Lang: bash
 1# 1. Prevent spread
 2# Block C2 domains/IPs at firewall
 3# Disable user accounts showing compromise
 4# Isolate network segments if needed
 5
 6# Windows - Block outbound connections
 7New-NetFirewallRule -DisplayName "Block Ransomware C2" `
 8  -Direction Outbound `
 9  -RemoteAddress 198.51.100.0/24 `
10  -Action Block
11
12# 2. Take snapshots/images before remediation
13# VM: Create snapshot via hypervisor
14# Physical: Image disk if possible
15# Preserve evidence for forensics
16
17# 3. Identify ransomware variant
18# Submit sample to:
19# - VirusTotal
20# - Hybrid Analysis
21# - ID Ransomware (id-ransomware.malwarehunterteam.com)

Eradication and Recovery:

Lang: bash
 1# DO NOT pay ransom (FBI recommendation)
 2
 3# 1. Verify backups are clean
 4# Scan backup files for malware
 5# Verify backup integrity
 6# Test restoration of sample files
 7
 8# 2. Wipe and rebuild affected systems
 9# Reinstall OS from known-good media
10# Restore from clean backups
11# Update and patch before reconnecting
12
13# 3. Reset all credentials
14# Force password reset for all users
15# Rotate service account passwords
16# Regenerate API keys
17
18# 4. Enhance monitoring
19# Increase log retention
20# Monitor for reinfection
21# Watch for lateral movement

Prevention Checklist:

  • Backups tested and offline/immutable
  • Email filtering and anti-phishing in place
  • EDR/antivirus up to date
  • Patch management current
  • Network segmentation implemented
  • Principle of least privilege enforced
  • MFA enabled for all admin accounts
  • User security awareness training

Playbook 2: Compromised User Account

Indicators:

  • Login from unusual location
  • Off-hours access
  • Multiple failed login attempts followed by success
  • Unusual email activity (sent/forwarded emails)
  • Changes to inbox rules
  • Password reset requests

Immediate Actions:

Lang: powershell
 1# 1. Disable the compromised account
 2# Active Directory
 3Disable-ADAccount -Identity username
 4
 5# Azure AD
 6Set-AzureADUser -ObjectId user@company.com -AccountEnabled $false
 7
 8# 2. Revoke all active sessions
 9# Azure AD
10Revoke-AzureADUserAllRefreshToken -ObjectId user@company.com
11
12# 3. Review recent activity
13# Azure AD sign-in logs
14Get-AzureADAuditSignInLogs -Filter "userPrincipalName eq 'user@company.com'" -Top 50
15
16# Check email activity (Office 365)
17Search-MailboxAuditLog -Identity user@company.com -StartDate (Get-Date).AddDays(-7) -ShowDetails

Investigation:

Lang: powershell
 1# 1. Check for malicious inbox rules
 2Get-InboxRule -Mailbox user@company.com
 3
 4# Remove suspicious rules
 5Remove-InboxRule -Identity "Rule Name" -Mailbox user@company.com
 6
 7# 2. Check for email forwarding
 8Get-Mailbox user@company.com | Select-Object ForwardingAddress, ForwardingSMTPAddress
 9
10# Remove forwarding
11Set-Mailbox user@company.com -ForwardingAddress $null -ForwardingSMTPAddress $null
12
13# 3. Review sent items
14# Check for phishing emails sent from account
15# Check for data exfiltration
16
17# 4. Check for file access/downloads
18# Review file server logs
19# Check cloud storage activity (OneDrive, SharePoint)

Recovery:

Lang: powershell
 1# 1. Reset password
 2Set-ADAccountPassword -Identity username -Reset -NewPassword (ConvertTo-SecureString -AsPlainText "TempP@ssw0rd!" -Force)
 3Set-ADUser -Identity username -ChangePasswordAtLogon $true
 4
 5# 2. Re-enable account (after verification)
 6Enable-ADAccount -Identity username
 7
 8# 3. Notify user
 9# Inform user of compromise
10# Provide new temporary password
11# Require security awareness training
12
13# 4. Monitor account for 30 days
14# Watch for unusual activity
15# Review login locations
16# Check for re-compromise

Playbook 3: Malware Infection

Indicators:

  • Antivirus/EDR alert
  • Unusual process execution
  • High CPU/network usage
  • Pop-ups or unexpected behavior
  • Files modified/encrypted

Immediate Actions:

Lang: bash
 1# 1. Isolate the system
 2# Disconnect from network (preserve memory)
 3# Do NOT shut down (loses volatile data)
 4
 5# 2. Identify the malware
 6# Check antivirus logs
 7# Review process list
 8# Check scheduled tasks/startup items
 9
10# Windows
11Get-Process | Sort-Object CPU -Descending
12Get-ScheduledTask | Where-Object {$_.State -eq "Ready"}
13Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Run
14
15# Linux
16ps aux --sort=-%cpu
17crontab -l
18cat /etc/cron*/*

Containment:

Lang: bash
 1# 1. Kill malicious processes
 2# Windows
 3Stop-Process -Id PID -Force
 4
 5# Linux
 6kill -9 PID
 7
 8# 2. Block C2 communication
 9# Add firewall rules to block malicious IPs/domains
10
11# 3. Preserve evidence
12# Take memory dump
13# Windows (using DumpIt or similar)
14.\DumpIt.exe
15
16# Linux
17dd if=/dev/mem of=/tmp/memory.dump bs=1M
18
19# Capture running processes
20# Windows
21Get-Process | Export-Csv C:\temp\processes.csv
22
23# Linux
24ps aux > /tmp/processes.txt

Eradication:

Lang: bash
 1# 1. Run full antivirus scan
 2# Update definitions first
 3# Run in Safe Mode if possible
 4
 5# 2. Remove persistence mechanisms
 6# Delete scheduled tasks
 7# Remove startup entries
 8# Check browser extensions
 9# Review Windows services
10
11# 3. Wipe and rebuild if critical
12# Backup user data (scan first)
13# Reinstall OS
14# Restore from clean backup

Playbook 4: Data Breach

Indicators:

  • Large data download/upload
  • Unauthorized database access
  • Credentials found on dark web
  • Customer data exposed
  • Third-party breach notification

Immediate Actions:

Lang: bash
 1# 1. Contain the breach
 2# Identify the exposure point
 3# Stop ongoing exfiltration
 4# Revoke compromised credentials
 5
 6# 2. Assess scope
 7# What data was exposed?
 8# How many records affected?
 9# Who is impacted (customers, employees)?
10# What is the sensitivity of the data?
11
12# 3. Preserve evidence
13# Don't delete logs
14# Take system snapshots
15# Document timeline
16# Secure forensic copies

Legal and Compliance:

Lang: markdown
 1# 1. Notify legal team immediately
 2# Breach may have legal implications
 3# Attorney-client privilege for communications
 4
 5# 2. Determine notification requirements
 6# GDPR: 72 hours to notify regulator
 7# HIPAA: 60 days to notify affected individuals
 8# State breach laws: Varies by state
 9# PCI-DSS: Notify card brands immediately
10
11# 3. Prepare notifications
12# Regulatory notifications
13# Customer notifications
14# Public disclosure (if required)
15# Credit monitoring offers (if applicable)
16
17# 4. Engage third parties
18# Forensics firm
19# Legal counsel
20# PR firm
21# Insurance provider (cyber insurance)

Investigation:

Lang: sql
 1-- Database breach investigation
 2-- Check access logs
 3SELECT * FROM audit_log
 4WHERE timestamp > '2024-11-01'
 5AND user NOT IN (SELECT username FROM authorized_users);
 6
 7-- Check for data export
 8SELECT * FROM query_log
 9WHERE query LIKE '%SELECT%'
10AND rows_returned > 1000;
11
12-- Identify affected records
13SELECT customer_id, ssn, credit_card
14FROM customers
15WHERE customer_id IN (SELECT DISTINCT customer_id FROM breach_access_log);

Recovery:

Lang: markdown
 1# 1. Remediate vulnerability
 2# Patch systems
 3# Fix configuration errors
 4# Implement additional controls
 5
 6# 2. Enhanced monitoring
 7# Increase log retention
 8# Deploy additional detection
 9# Monitor for misuse of breached data
10
11# 3. Strengthen security
12# Implement data encryption
13# Enhance access controls
14# Review security policies
15# Mandatory security training

Playbook 5: DDoS Attack

Indicators:

  • Service unavailable
  • Extremely high traffic volume
  • Network congestion
  • Slow performance

Immediate Actions:

Lang: bash
 1# 1. Confirm DDoS attack
 2# Check traffic patterns
 3netstat -an | grep SYN_RECV | wc -l
 4
 5# Check bandwidth usage
 6iftop -i eth0
 7
 8# Review firewall/IPS logs
 9tail -f /var/log/firewall.log | grep DROP
10
11# 2. Identify attack type
12# SYN flood
13# UDP flood
14# HTTP flood
15# DNS amplification

Mitigation:

Lang: bash
 1# 1. Engage DDoS mitigation service
 2# Cloudflare
 3# AWS Shield
 4# Akamai Prolexic
 5# Radware
 6
 7# 2. Implement rate limiting
 8# Limit connections per IP
 9
10# nginx
11limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
12limit_req zone=one burst=20;
13
14# iptables
15iptables -A INPUT -p tcp --dport 80 -m limit --limit 25/minute --limit-burst 100 -j ACCEPT
16
17# 3. Block attack sources
18# Null-route attacking IPs
19ip route add blackhole 198.51.100.0/24
20
21# Firewall blocks
22iptables -A INPUT -s 198.51.100.0/24 -j DROP
23
24# 4. Communicate with ISP
25# Request upstream filtering
26# Provide attack details
27# Request additional bandwidth if needed

Post-Incident Activities

Incident Report Template

Lang: markdown
 1# Incident Report: [Incident ID]
 2
 3**Incident Summary:**
 4- Date/Time: 2024-11-02 14:30 EST
 5- Severity: High (Sev 2)
 6- Incident Type: Ransomware Attack
 7- Status: Resolved
 8- Duration: 6 hours
 9
10**Incident Timeline:**
11- 14:30 - Initial detection (user report)
12- 14:35 - IR team activated
13- 14:40 - Affected systems isolated
14- 15:00 - Scope identified (5 workstations)
15- 15:30 - Containment achieved
16- 17:00 - Systems rebuilt from backup
17- 18:30 - Systems back online
18- 20:30 - Incident closed
19
20**Root Cause:**
21Phishing email with malicious attachment opened by user.
22Lack of email filtering allowed attachment through.
23User clicked on macro-enabled document.
24
25**Impact Assessment:**
26- Systems Affected: 5 workstations
27- Users Impacted: 5 employees
28- Data Loss: None (restored from backup)
29- Downtime: 4 hours per workstation
30- Estimated Cost: $10,000 (labor + recovery)
31
32**Actions Taken:**
331. Isolated affected systems
342. Identified ransomware variant (Ryuk)
353. Blocked C2 domains at firewall
364. Rebuilt systems from clean images
375. Restored user data from backups
386. Reset user passwords
397. Enhanced email filtering
40
41**Lessons Learned:**
42- Email filtering insufficient
43- User training needed
44- Backup process worked well
45- IR team response time good
46- Need better EDR solution
47
48**Recommendations:**
491. Implement advanced email filtering (Priority: High)
502. Deploy EDR solution (Priority: High)
513. Conduct phishing simulation training (Priority: Medium)
524. Review and update backup procedures (Priority: Medium)
535. Implement network segmentation (Priority: Low)
54
55**Follow-up Actions:**
56- [x] Systems restored and operational
57- [x] Users can resume work
58- [ ] Email filtering upgraded (Due: 2024-11-15)
59- [ ] EDR solution deployed (Due: 2024-12-01)
60- [ ] User training scheduled (Due: 2024-11-30)
61- [ ] Security review completed (Due: 2024-12-15)

Lessons Learned Meeting

Attendees:

  • IR team members
  • Management
  • IT staff
  • Affected department leads

Discussion Topics:

  1. What went well?
  2. What could be improved?
  3. Were procedures followed?
  4. What was unexpected?
  5. What changes are needed?

Action Items:

  • Document with owner and due date
  • Track to completion
  • Update IR procedures based on findings

Incident Response Tools

Essential IR Toolkit

Investigation:

  • Wireshark - Packet capture and analysis
  • Sysinternals Suite - Windows system utilities
  • Volatility - Memory forensics
  • Autopsy - Disk forensics
  • KAPE - Evidence collection

Malware Analysis:

  • VirusTotal - Multi-engine malware scanner
  • Hybrid Analysis - Automated malware analysis
  • REMnux - Malware analysis Linux distro
  • IDA Pro/Ghidra - Reverse engineering

Network Analysis:

  • Zeek (Bro) - Network security monitoring
  • Suricata - IDS/IPS
  • NetworkMiner - Network forensics
  • tcpdump - Packet capture

Log Analysis:

  • Splunk / ELK Stack - SIEM
  • Graylog - Log management
  • Chainsaw - Windows event log analysis

Communication:

  • Secure chat (Signal, Wickr)
  • Conference bridge
  • War room (physical or virtual)

IR Preparation Checklist

Before an Incident

  • IR plan documented and distributed
  • IR team identified with contact info
  • IR tools and access provisioned
  • Runbooks created for common scenarios
  • Communication templates prepared
  • Legal/PR contacts identified
  • Forensics retainer established
  • Cyber insurance policy in place
  • Backups tested and verified
  • Monitoring and detection in place
  • Network diagrams updated
  • Asset inventory current
  • Credentials documented (secure vault)
  • IR training completed
  • Tabletop exercises conducted

During an Incident

  • Incident declared and severity assigned
  • IR team activated
  • War room established
  • Timeline tracking initiated
  • Evidence preservation started
  • Management notified
  • Containment actions taken
  • Regular status updates provided
  • Legal/compliance considerations addressed
  • External parties engaged if needed

After an Incident

  • Incident report completed
  • Lessons learned session held
  • IR procedures updated
  • Follow-up actions tracked
  • Security improvements implemented
  • Team debriefed
  • Knowledge base updated

Additional Resources