Security incident response procedures for detecting, containing, and recovering from security incidents.

Incident Response Overview

Incident Response Lifecycle

  • Preparation - Tools, training, procedures in place
  • Detection & Analysis - Identify and assess incidents
  • Containment - Stop the spread
  • Eradication - Remove threat from environment
  • Recovery - Restore systems to normal operation
  • Lessons Learned - Post-incident review

Incident Severity Levels

Severity 1 - Critical

  • Active data breach

  • Ransomware encryption in progress

  • Complete service outage

  • Nation-state actor activity

  • Response: Immediate, 24/7, all hands on deck

  • Timeline: <15 minutes

Severity 2 - High

  • Suspected data breach

  • Malware detected on multiple systems

  • Major service degradation

  • Unauthorized access detected

  • Response: Urgent, senior team

  • Timeline: <1 hour

Severity 3 - Medium

  • Single compromised account

  • Malware on isolated system

  • Security control failure

  • Policy violation

  • Response: Standard business hours

  • Timeline: <4 hours

Severity 4 - Low

  • Failed login attempts

  • Suspicious email

  • Non-critical policy violation

  • Response: Standard process

  • Timeline: <24 hours

Incident Response Team

Roles and Responsibilities

Incident Commander

  • Overall incident coordination
  • Decision-making authority
  • Stakeholder communication
  • Resource allocation

Security Lead

  • Technical security analysis
  • Forensics coordination
  • Threat intelligence
  • Remediation planning

IT Lead

  • Systems administration
  • Network management
  • Access control
  • System restoration

Communications Lead

  • Internal communications
  • External communications (if needed)
  • Legal/PR coordination
  • Customer notifications

Documentation Lead

  • Incident timeline tracking
  • Evidence collection
  • Action item tracking
  • Post-incident report

Detection and Analysis

Common Detection Sources

SIEM/Log Analysis:

  • Multiple failed login attempts
  • Unusual outbound traffic
  • Privilege escalation
  • Off-hours access

Antivirus/EDR Alerts:

  • Malware detection
  • Suspicious process execution
  • File modification
  • Registry changes

User Reports:

  • Suspicious emails
  • Unexpected system behavior
  • Password reset requests
  • Account lockouts

Network Monitoring:

  • Unusual traffic patterns
  • Data exfiltration
  • Command and control (C2) traffic
  • Port scanning

Initial Assessment Questions

What happened?

  • What was detected?
  • When did it occur?
  • How was it discovered?

What is affected?

  • Which systems/users/data?
  • How many assets impacted?
  • What is the scope?

What is the impact?

  • Data confidentiality compromised?
  • System availability affected?
  • Data integrity in question?

Is it contained?

  • Is it still spreading?
  • Can we isolate it?
  • Do we need to take systems offline?

Incident Response Playbooks

Playbook 1: Ransomware Attack

Indicators:

  • Files being encrypted (.encrypted, .locked extension)
  • Ransom notes on desktops
  • Crypto-locker processes running
  • Mass file modifications

Immediate Actions (0-15 minutes):

Lang: bash
# 1. ISOLATE AFFECTED SYSTEMS IMMEDIATELY
# Disconnect from network (DO NOT SHUT DOWN)
# Physical: Unplug network cable
# Virtual: Disable network adapter

# Windows
Disable-NetAdapter -Name "Ethernet" -Confirm:$false

# Linux
ip link set eth0 down

# 2. Identify affected systems
# Check SIEM/logs for similar activity
# Check backup systems (ensure they're not compromised)

# 3. Alert incident response team
# Send notification to IR team
# Escalate to management immediately

Containment (15-60 minutes):

Lang: bash
# 1. Prevent spread
# Block C2 domains/IPs at firewall
# Disable user accounts showing compromise
# Isolate network segments if needed

# Windows - Block outbound connections
New-NetFirewallRule -DisplayName "Block Ransomware C2" `
  -Direction Outbound `
  -RemoteAddress 198.51.100.0/24 `
  -Action Block

# 2. Take snapshots/images before remediation
# VM: Create snapshot via hypervisor
# Physical: Image disk if possible
# Preserve evidence for forensics

# 3. Identify ransomware variant
# Submit sample to:
# - VirusTotal
# - Hybrid Analysis
# - ID Ransomware (id-ransomware.malwarehunterteam.com)

Eradication and Recovery:

Lang: bash
# DO NOT pay ransom (FBI recommendation)

# 1. Verify backups are clean
# Scan backup files for malware
# Verify backup integrity
# Test restoration of sample files

# 2. Wipe and rebuild affected systems
# Reinstall OS from known-good media
# Restore from clean backups
# Update and patch before reconnecting

# 3. Reset all credentials
# Force password reset for all users
# Rotate service account passwords
# Regenerate API keys

# 4. Enhance monitoring
# Increase log retention
# Monitor for reinfection
# Watch for lateral movement

Prevention Checklist:

  • Backups tested and offline/immutable
  • Email filtering and anti-phishing in place
  • EDR/antivirus up to date
  • Patch management current
  • Network segmentation implemented
  • Principle of least privilege enforced
  • MFA enabled for all admin accounts
  • User security awareness training

Playbook 2: Compromised User Account

Indicators:

  • Login from unusual location
  • Off-hours access
  • Multiple failed login attempts followed by success
  • Unusual email activity (sent/forwarded emails)
  • Changes to inbox rules
  • Password reset requests

Immediate Actions:

Lang: powershell
# 1. Disable the compromised account
# Active Directory
Disable-ADAccount -Identity username

# Azure AD
Set-AzureADUser -ObjectId user@company.com -AccountEnabled $false

# 2. Revoke all active sessions
# Azure AD
Revoke-AzureADUserAllRefreshToken -ObjectId user@company.com

# 3. Review recent activity
# Azure AD sign-in logs
Get-AzureADAuditSignInLogs -Filter "userPrincipalName eq 'user@company.com'" -Top 50

# Check email activity (Office 365)
Search-MailboxAuditLog -Identity user@company.com -StartDate (Get-Date).AddDays(-7) -ShowDetails

Investigation:

Lang: powershell
# 1. Check for malicious inbox rules
Get-InboxRule -Mailbox user@company.com

# Remove suspicious rules
Remove-InboxRule -Identity "Rule Name" -Mailbox user@company.com

# 2. Check for email forwarding
Get-Mailbox user@company.com | Select-Object ForwardingAddress, ForwardingSMTPAddress

# Remove forwarding
Set-Mailbox user@company.com -ForwardingAddress $null -ForwardingSMTPAddress $null

# 3. Review sent items
# Check for phishing emails sent from account
# Check for data exfiltration

# 4. Check for file access/downloads
# Review file server logs
# Check cloud storage activity (OneDrive, SharePoint)

Recovery:

Lang: powershell
# 1. Reset password
Set-ADAccountPassword -Identity username -Reset -NewPassword (ConvertTo-SecureString -AsPlainText "TempP@ssw0rd!" -Force)
Set-ADUser -Identity username -ChangePasswordAtLogon $true

# 2. Re-enable account (after verification)
Enable-ADAccount -Identity username

# 3. Notify user
# Inform user of compromise
# Provide new temporary password
# Require security awareness training

# 4. Monitor account for 30 days
# Watch for unusual activity
# Review login locations
# Check for re-compromise

Playbook 3: Malware Infection

Indicators:

  • Antivirus/EDR alert
  • Unusual process execution
  • High CPU/network usage
  • Pop-ups or unexpected behavior
  • Files modified/encrypted

Immediate Actions:

Lang: bash
# 1. Isolate the system
# Disconnect from network (preserve memory)
# Do NOT shut down (loses volatile data)

# 2. Identify the malware
# Check antivirus logs
# Review process list
# Check scheduled tasks/startup items

# Windows
Get-Process | Sort-Object CPU -Descending
Get-ScheduledTask | Where-Object {$_.State -eq "Ready"}
Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Run

# Linux
ps aux --sort=-%cpu
crontab -l
cat /etc/cron*/*

Containment:

Lang: bash
# 1. Kill malicious processes
# Windows
Stop-Process -Id PID -Force

# Linux
kill -9 PID

# 2. Block C2 communication
# Add firewall rules to block malicious IPs/domains

# 3. Preserve evidence
# Take memory dump
# Windows (using DumpIt or similar)
.\DumpIt.exe

# Linux
dd if=/dev/mem of=/tmp/memory.dump bs=1M

# Capture running processes
# Windows
Get-Process | Export-Csv C:\temp\processes.csv

# Linux
ps aux > /tmp/processes.txt

Eradication:

Lang: bash
# 1. Run full antivirus scan
# Update definitions first
# Run in Safe Mode if possible

# 2. Remove persistence mechanisms
# Delete scheduled tasks
# Remove startup entries
# Check browser extensions
# Review Windows services

# 3. Wipe and rebuild if critical
# Backup user data (scan first)
# Reinstall OS
# Restore from clean backup

Playbook 4: Data Breach

Indicators:

  • Large data download/upload
  • Unauthorized database access
  • Credentials found on dark web
  • Customer data exposed
  • Third-party breach notification

Immediate Actions:

Lang: bash
# 1. Contain the breach
# Identify the exposure point
# Stop ongoing exfiltration
# Revoke compromised credentials

# 2. Assess scope
# What data was exposed?
# How many records affected?
# Who is impacted (customers, employees)?
# What is the sensitivity of the data?

# 3. Preserve evidence
# Don't delete logs
# Take system snapshots
# Document timeline
# Secure forensic copies

Legal and Compliance:

Lang: markdown
# 1. Notify legal team immediately
# Breach may have legal implications
# Attorney-client privilege for communications

# 2. Determine notification requirements
# GDPR: 72 hours to notify regulator
# HIPAA: 60 days to notify affected individuals
# State breach laws: Varies by state
# PCI-DSS: Notify card brands immediately

# 3. Prepare notifications
# Regulatory notifications
# Customer notifications
# Public disclosure (if required)
# Credit monitoring offers (if applicable)

# 4. Engage third parties
# Forensics firm
# Legal counsel
# PR firm
# Insurance provider (cyber insurance)

Investigation:

Lang: sql
-- Database breach investigation
-- Check access logs
SELECT * FROM audit_log
WHERE timestamp > '2024-11-01'
AND user NOT IN (SELECT username FROM authorized_users);

-- Check for data export
SELECT * FROM query_log
WHERE query LIKE '%SELECT%'
AND rows_returned > 1000;

-- Identify affected records
SELECT customer_id, ssn, credit_card
FROM customers
WHERE customer_id IN (SELECT DISTINCT customer_id FROM breach_access_log);

Recovery:

Lang: markdown
# 1. Remediate vulnerability
# Patch systems
# Fix configuration errors
# Implement additional controls

# 2. Enhanced monitoring
# Increase log retention
# Deploy additional detection
# Monitor for misuse of breached data

# 3. Strengthen security
# Implement data encryption
# Enhance access controls
# Review security policies
# Mandatory security training

Playbook 5: DDoS Attack

Indicators:

  • Service unavailable
  • Extremely high traffic volume
  • Network congestion
  • Slow performance

Immediate Actions:

Lang: bash
# 1. Confirm DDoS attack
# Check traffic patterns
netstat -an | grep SYN_RECV | wc -l

# Check bandwidth usage
iftop -i eth0

# Review firewall/IPS logs
tail -f /var/log/firewall.log | grep DROP

# 2. Identify attack type
# SYN flood
# UDP flood
# HTTP flood
# DNS amplification

Mitigation:

Lang: bash
# 1. Engage DDoS mitigation service
# Cloudflare
# AWS Shield
# Akamai Prolexic
# Radware

# 2. Implement rate limiting
# Limit connections per IP

# nginx
limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
limit_req zone=one burst=20;

# iptables
iptables -A INPUT -p tcp --dport 80 -m limit --limit 25/minute --limit-burst 100 -j ACCEPT

# 3. Block attack sources
# Null-route attacking IPs
ip route add blackhole 198.51.100.0/24

# Firewall blocks
iptables -A INPUT -s 198.51.100.0/24 -j DROP

# 4. Communicate with ISP
# Request upstream filtering
# Provide attack details
# Request additional bandwidth if needed

Post-Incident Activities

Incident Report Template

Lang: markdown
# Incident Report: [Incident ID]

**Incident Summary:**
- Date/Time: 2024-11-02 14:30 EST
- Severity: High (Sev 2)
- Incident Type: Ransomware Attack
- Status: Resolved
- Duration: 6 hours

**Incident Timeline:**
- 14:30 - Initial detection (user report)
- 14:35 - IR team activated
- 14:40 - Affected systems isolated
- 15:00 - Scope identified (5 workstations)
- 15:30 - Containment achieved
- 17:00 - Systems rebuilt from backup
- 18:30 - Systems back online
- 20:30 - Incident closed

**Root Cause:**
Phishing email with malicious attachment opened by user.
Lack of email filtering allowed attachment through.
User clicked on macro-enabled document.

**Impact Assessment:**
- Systems Affected: 5 workstations
- Users Impacted: 5 employees
- Data Loss: None (restored from backup)
- Downtime: 4 hours per workstation
- Estimated Cost: $10,000 (labor + recovery)

**Actions Taken:**
1. Isolated affected systems
2. Identified ransomware variant (Ryuk)
3. Blocked C2 domains at firewall
4. Rebuilt systems from clean images
5. Restored user data from backups
6. Reset user passwords
7. Enhanced email filtering

**Lessons Learned:**
- Email filtering insufficient
- User training needed
- Backup process worked well
- IR team response time good
- Need better EDR solution

**Recommendations:**
1. Implement advanced email filtering (Priority: High)
2. Deploy EDR solution (Priority: High)
3. Conduct phishing simulation training (Priority: Medium)
4. Review and update backup procedures (Priority: Medium)
5. Implement network segmentation (Priority: Low)

**Follow-up Actions:**
- [x] Systems restored and operational
- [x] Users can resume work
- [ ] Email filtering upgraded (Due: 2024-11-15)
- [ ] EDR solution deployed (Due: 2024-12-01)
- [ ] User training scheduled (Due: 2024-11-30)
- [ ] Security review completed (Due: 2024-12-15)

Lessons Learned Meeting

Attendees:

  • IR team members
  • Management
  • IT staff
  • Affected department leads

Discussion Topics:

  1. What went well?
  2. What could be improved?
  3. Were procedures followed?
  4. What was unexpected?
  5. What changes are needed?

Action Items:

  • Document with owner and due date
  • Track to completion
  • Update IR procedures based on findings

Incident Response Tools

Essential IR Toolkit

Investigation:

  • Wireshark - Packet capture and analysis
  • Sysinternals Suite - Windows system utilities
  • Volatility - Memory forensics
  • Autopsy - Disk forensics
  • KAPE - Evidence collection

Malware Analysis:

  • VirusTotal - Multi-engine malware scanner
  • Hybrid Analysis - Automated malware analysis
  • REMnux - Malware analysis Linux distro
  • IDA Pro/Ghidra - Reverse engineering

Network Analysis:

  • Zeek (Bro) - Network security monitoring
  • Suricata - IDS/IPS
  • NetworkMiner - Network forensics
  • tcpdump - Packet capture

Log Analysis:

  • Splunk / ELK Stack - SIEM
  • Graylog - Log management
  • Chainsaw - Windows event log analysis

Communication:

  • Secure chat (Signal, Wickr)
  • Conference bridge
  • War room (physical or virtual)

IR Preparation Checklist

Before an Incident

  • IR plan documented and distributed
  • IR team identified with contact info
  • IR tools and access provisioned
  • Runbooks created for common scenarios
  • Communication templates prepared
  • Legal/PR contacts identified
  • Forensics retainer established
  • Cyber insurance policy in place
  • Backups tested and verified
  • Monitoring and detection in place
  • Network diagrams updated
  • Asset inventory current
  • Credentials documented (secure vault)
  • IR training completed
  • Tabletop exercises conducted

During an Incident

  • Incident declared and severity assigned
  • IR team activated
  • War room established
  • Timeline tracking initiated
  • Evidence preservation started
  • Management notified
  • Containment actions taken
  • Regular status updates provided
  • Legal/compliance considerations addressed
  • External parties engaged if needed

After an Incident

  • Incident report completed
  • Lessons learned session held
  • IR procedures updated
  • Follow-up actions tracked
  • Security improvements implemented
  • Team debriefed
  • Knowledge base updated

Additional Resources