A clean automated penetration test tells you one thing: the tool found no working attack path inside the scope and time window it was given. It is silent on whether your SOC would notice an attacker who did find one. Treat those as two separate questions, and test the second one on purpose.
What automated penetration testing does well
NIST SP 800-115 describes penetration testing as assessors imitating real attacks to find ways around the security features of an application, system, or network. Automated platforms make that repeatable. They discover hosts, try known exploits and reused or default credentials, chain small weaknesses into larger access, and hand you a report with evidence.
That is useful. Automated runs cost less than a manual engagement, so you can run them monthly or after major changes. They catch regressions, such as a server rebuilt from an old image, and they confirm whether a fix actually closed the path it targeted. NIST itself recommends regular, less labor-intensive testing between periodic pentests.
The weak point is not the tool. It is what people conclude from a report with nothing red in it.
Why a clean report can mislead
It is a snapshot with a fence around it
Every test has limits: chosen subnets, supplied credentials, approved hours, and a list of techniques considered safe for production. NIST SP 800-115 notes that an assessment gives you a snapshot of security at one point in time. The week after the test, someone makes a storage bucket public, grants a contractor admin rights, or ships a new internet-facing service. The report stays the same. Your exposure does not.
Safe-mode tools also avoid destructive steps and bulk data movement by design. Sensible, but it means the most damaging stages of a real intrusion often go unexercised.
It measures reach, not response
An automated pentest answers “can an attacker get from here to there?” It rarely answers “did anyone notice?” Sometimes detection is removed from the test on purpose. The same NIST guide says administrators can set intrusion detection and other monitoring tools to ignore traffic from the testers’ IP addresses. The joint CISA living off the land guidance adds that vulnerability scanners and service accounts routinely perform privileged, suspicious-looking actions that flood analysts unless they are categorized separately. If your pentest appliance sits on an allow list, a clean run says nothing about your SOC.
That is the difference between “we couldn’t get in” and “we would have seen it.” The first is about prevention on the paths that were tested. The second is about detection and response, including on paths nobody tested.
CISA’s red team reports show how large that difference can be. After a three-month assessment of a critical infrastructure organization in 2022, CISA reported that the organization, despite a mature posture, never detected the red team, even when the team tried to provoke a response. In a 2024 advisory about a federal agency, the red team stayed undetected through the first phase; five months in, the team notified the SOC itself.
Four blind spots a clean report can hide
Identity and cloud misconfiguration chains. Many intrusions never exploit a CVE. They log in. MITRE ATT&CK’s Cloud Accounts technique (T1078.004) warns that misconfigured role assignments or role assumption policies can hand an attacker permissions well beyond an account’s intended scope. CrowdStrike’s 2026 Global Threat Report says valid account abuse made up 35% of the cloud incidents it saw in 2025. A network-focused test may never examine an over-permissioned service principal or a role that can assume a stronger one.
Lateral movement. ATT&CK’s Lateral Movement tactic (TA0008) is full of legitimate mechanisms: remote services, stolen tickets and hashes, and software deployment tools. An automated run may confirm the route is open. Whether a 2 a.m. RDP session from a finance laptop to a domain controller would page anyone is outside its remit. And attackers move fast: CrowdStrike put the average eCrime breakout time, from initial access to lateral movement, at 29 minutes in 2025.
Living off the land. Attackers increasingly use PowerShell, WMI, and other built-in tools instead of malware; CrowdStrike found that 82% of its 2025 detections were malware-free. The CISA-led guidance says CISA’s own red teams routinely use these techniques to stay hidden, and often find the target has no baseline of normal admin behavior. Exploit checks do not measure this, because nothing is being exploited.
Data exfiltration paths. Automated tools usually stop at proof of access and, rightly, do not move real data in bulk. But ATT&CK’s Exfiltration tactic (TA0010) includes sending data through legitimate web services (T1567) and copying it into another cloud account (T1537), both of which can resemble ordinary business traffic. A clean pentest leaves open whether your proxy, DLP, or cloud logs would flag it.
Pentest vs breach and attack simulation
Breach and attack simulation (BAS) and detection validation work from the other direction. Instead of searching for a way in, they replay specific attacker behaviors, usually mapped to ATT&CK techniques, and record the outcome for each: prevented, alerted, logged without an alert, or invisible.
| Automated pentest | Detection validation | |
|---|---|---|
| Core question | Can an attacker reach a target? | Would we see and stop this behavior? |
| A good result | No exploitable path found | A triaged alert, with context, fast enough to act |
| Main output | Attack paths and weak assets | Detection rate and time to detect per technique |
| Blind to | Whether the SOC noticed | Paths nobody thought to simulate |
| Cadence | Monthly or after big changes | Continuous, and after every rule or tooling change |
You need both. An exposure your SOC catches within minutes deserves a different priority than one that would run unnoticed for weeks, and you only know which is which if you test detection. More on that in why CISOs are moving from vulnerability counts to BAS.
How to measure detection, not just exposure
Detection validation turns “I think we’d catch that” into numbers. Track three per technique:
- Detection coverage: the share of simulated techniques that produced an alert someone would actually triage. A log entry with no alert does not count.
- Time to detect: from the simulated action to the alert landing in the queue. Mandiant’s M-Trends 2026 put global median dwell time at 14 days in 2025, up from 11.
- Time to respond: from alert to containment, including the handoff to whoever can disable the account or isolate the host. If triage is the slow step, that is a SOC capacity problem; see what an AI SOC changes.
Coverage is not binary. Catching one procedure for a technique does not mean you catch its variants. In the 2024 CISA engagement, defenders wrote YARA rules for one red team loader and still missed all six of the other C2 frameworks in play. Test behaviors, not just known tools.
Purple teaming is the quickest way to move these numbers. One side runs a technique, the other watches live, and both fix the gap before moving on. SCYTHE’s open Purple Team Exercise Framework is a practical template, and Red Canary’s Atomic Red Team offers small ATT&CK-mapped tests you can run yourself. CISA’s LOTL guidance likewise recommends a detection maturity model in which alerts are tested and tuned over time.
What to do now
- Map your last pentest to ATT&CK. List the techniques the tool actually executed, not just the findings. That is your tested set.
- Check what fired. Get the test’s timestamps and source IPs, then search your SIEM and EDR for alerts in that window. If the tester was allow-listed, rerun a subset without the exception.
- Choose the techniques that matter. Start with 15 to 20 drawn from reporting on groups that target your sector. Prioritize valid accounts, remote services, built-in tool abuse, and exfiltration over web services.
- Simulate and score. Run each one against a scoped, authorized target and record prevented, alerted, logged only, or unseen, plus time to detect.
- Purple team the misses. Fix rules, log sources, and alert routing, then rerun the same technique to prove the fix held.
- Put it on a schedule. Rerun after every rule change, SIEM or EDR migration, and major cloud change. Report coverage and time to detect next to pentest results.
- Watch the outside between tests. Monitor your external attack surface so a new exposure does not wait a year for the next engagement.
If you would rather not build that loop from scratch, Jutsu Red Team runs MITRE ATT&CK-mapped scenarios against scoped, authorized targets and correlates each run with SIEM events to report detection rate, time to detect, and coverage. For the outside view, Exposure produces a non-invasive external assessment from public signals. It is not a pentest and does not try to be one.