If you've ever dealt with a compromised Linux server, you know the instinct is to fix the problem as quickly as possible. Kill the malicious process, revoke access, patch the vulnerability, and move on.
The trouble is that jumping straight into cleanup can destroy the very evidence you need to understand what happened in the first place.
Good incident response isn't just about stopping an attack. It's about preserving enough information to reconstruct the timeline, identify the attacker's actions, determine how they got in, and prevent it from happening again.
Let's look at some practical ways to document an incident without accidentally compromising your evidence.
Treat your notes as evidence
It's easy to think of documentation as something you write up after the incident is over. In reality, your notes should start the moment you suspect something is wrong.
Every command you run changes the system in some way, so it's worth keeping a running log of what you observed, what actions you took, and when you took them.
At a minimum, try to capture:
- When the incident was discovered
- Which systems are affected
- Who reported the issue
- Commands you executed
- Files or logs you collected
- Hashes of any evidence
- Containment and recovery steps
You may also find yourself referencing browser-based dashboards during an investigation. Maybe your SIEM flagged suspicious activity, your cloud provider generated alerts, or your ticketing platform contains important notes about the incident. Those pages can change as new data comes in, so it's often useful to preserve them as they appeared.
Something like Smallpdf's online HTML to PDF converter can make it easy to save web pages as PDF files that can be archived alongside your other evidence. You want to create documentation that someone else could follow weeks or even months later.
Resist the urge to start cleaning up
This is probably the hardest part. When you spot something suspicious, your first instinct is usually to stop it immediately. Sometimes that's exactly what you should do, especially if systems or data are at immediate risk. But if the situation allows, spend a little time gathering evidence before making major changes.
For example, removing a malicious file before you've recorded where it came from, when it appeared, or what it was doing means you've lost valuable context.
Start by collecting volatile information that may disappear after a reboot or service restart.
Useful commands include:
who
w
last
ps aux
ss -tulnp
journalctl -xe
These snapshots help paint a picture of what's happening right now. Once the system changes, that information may be gone forever.
Collect evidence without modifying it
One of the biggest challenges in digital forensics is collecting evidence without accidentally changing it.
Whenever possible, avoid editing files directly on the compromised machine. If you're imaging a disk, mount it as read-only before examining its contents. If you're copying important files, verify their integrity with cryptographic hashes.
Generating a SHA-256 hash is straightforward:
sha256sum suspicious-file
Record the resulting hash in your notes. If that file is copied later, you can generate another hash to confirm nothing has changed during storage or transfer. Think of hashes as fingerprints. If the fingerprint changes, so has the evidence.
Don't overlook Linux's built-in forensic clues
Linux leaves behind a surprising amount of useful information if you know where to look.
System logs are often the first stop.
Depending on the distribution and logging configuration, you'll want to review sources such as:
- journalctl
- /var/log/auth.log
- /var/log/secure
- /var/log/syslog
- Audit logs from auditd
Authentication logs can reveal failed login attempts, privilege escalation, or unexpected SSH sessions. System logs may show service crashes, configuration changes, or software installations that line up with suspicious activity.
Don't stop there, though. Look at running processes, scheduled cron jobs, recently modified files, user accounts, SSH authorized keys, and shell history. Even small details can help explain how an attacker moved through the system.
For example, finding a recently added SSH key in a user's ~/.ssh/authorized_keys file could explain persistent unauthorized access even after passwords were changed.
Preserve your chain of custody
If multiple people are involved in the investigation, keeping track of evidence becomes even more important.
A chain of custody is simply a record showing who collected evidence, when it was collected, where it's stored, and who has accessed it since. That doesn't have to mean expensive forensic software. Even a simple spreadsheet or case management system can work, provided everyone follows the same process.
For every piece of evidence, try to record:
- Filename or evidence identifier
- Date and time collected
- Investigator
- SHA-256 hash
- Storage location
- Any transfers between team members
Automate the boring parts
Documentation tends to slip when responders are under pressure. The more routine tasks you can automate, the more consistent your investigations will become.
Many security teams use shell scripts to collect common forensic artifacts in one go. Others automatically export SIEM alerts, gather system information, or package relevant logs into timestamped archives.
Even something as simple as redirecting command output into dated text files can save time later.
For example:
mkdir incident-2026-07-28
ps aux > incident-2026-07-28/processes.txt
ss -tulnp > incident-2026-07-28/network.txt
journalctl -xe > incident-2026-07-28/journal.txt
Now you've created a structured evidence folder that's much easier to review than trying to remember which commands you ran an hour ago.
Remember that documentation is part of your security posture
Clear documentation helps you identify attack patterns, improve response playbooks, satisfy compliance requirements, and share knowledge across your security team. It also makes future investigations faster because you've already established a repeatable process.
The next time you're responding to a Linux security incident, slow down just enough to document what you're seeing before jumping into remediation. You'll thank yourself later, especially if someone asks you six months from now exactly how the incident unfolded.
