An AI SOC is a security operations center where AI agents do the first pass of triage and investigation (pulling evidence, correlating it and proposing a verdict) while people keep the decisions that carry consequences. That is the definition most practitioners would sign. The harder question is how these agents fail, and a benchmark published on October 7, 2026 gives the clearest answer we have seen so far.
Picture the kind of alert that study is full of. A web server rule fires on a burst of HTTP 400 responses. The triage agent queries the SIEM for related activity from the same source, the search comes back empty, and the agent closes the alert as benign. It was the opening of a directory enumeration. In the ALERT-BENCH study, which replayed recorded enterprise telemetry through a live SIEM, that pattern turned up again and again. We will come back to that empty search, because almost everything worth testing in an AI SOC traces to it.
What an AI SOC actually automates
The work an AI SOC takes on is the tier-1 grind, which means enrichment, alert triage, evidence gathering across SIEM, EDR, identity and cloud, timeline building and the case summary. Tines draws the line against older automation well. SOAR automated steps inside playbooks an analyst already knew how to write, while an AI SOC reasons about context the playbook did not anticipate and decides what to look at next.
The pressure behind the category is real. In a Microsoft and Omdia survey of 300 security professionals who run SOC operations, an estimated 46% of alerts proved to be false positives and 42% went uninvestigated. AI adoption inside companies is adding to the pile. A Cloud Security Alliance research note from September 14, 2026 reports that AI-related alerts grew 685% between February and June 2026 and that 94.1% of them were noise from legitimate tool use. More than 80% were suppressed automatically without an analyst looking. Something has to read those queues, and that is the job vendors are selling agents to do.
AI SOC vs traditional SOC and SOAR
In a traditional SOC an analyst pivots between consoles for every alert. SOAR moved the repeatable parts into fixed playbooks, which works until an alert does not match the playbook’s assumptions. An AI SOC replaces the fixed path with an agent that plans its own queries, so it handles the odd alert better. As the benchmark shows, it can also talk itself out of an investigation that a rigid playbook would have finished.
The other difference is accountability. A playbook fails the same way every time, which makes it auditable. An agent’s investigation differs run to run, so the record of what it searched, what came back and why it closed the alert becomes the only way to review its work. If a platform cannot show you that trace per alert, you cannot supervise it.
Is an autonomous SOC real yet?
Not in the sense the phrase suggests. Expel, which runs an AI-assisted MDR operation, says plainly that “autonomous SOC” is a marketing claim, not a production reality. The useful distinction is the one it draws between human-in-the-loop, where the agent waits for approval before acting, and human-on-the-loop, where it acts inside guardrails while a person watches and can override.
Our position is that dismissal deserves the same caution as containment. Vendors tend to gate the dramatic actions, isolating a host or disabling an account, behind approval. Closing an alert feels harmless, so it runs unattended. Yet a wrongly closed alert is the failure nobody sees, because nothing downstream ever looks at it again.
Where AI SOC agents fail: the empty search
That brings us back to the empty search. ALERT-BENCH tested five common reasoning designs for tool-using agents (single-pass tool use, the iterative ReAct loop, majority vote across several runs, self-review and chain-of-verification) on the same 1,247 alerts, with the same model and the same SIEM interface. Every one of them missed at least 40.4% of attack-related alerts, with false-negative rates ranging up to 70.2%.
| Period | Value |
|---|---|
| Direct-Prompt | 40.4% |
| Self-Consistency | 54.2% |
| AIDA | 3.1% |
The trace analysis explains why. When an agent’s searches returned no records, it dismissed attack alerts far more often. ReAct dismissed 98.6% of attacks after empty retrieval against 30.3% when at least one query returned records. Self-Refine went from 96.2% to 37.6%, and Direct-Prompt from 80.0% to 27.6%. An empty result only proves that one particular query matched nothing, yet the agents treated it as evidence that the activity was benign. Across the five designs, 211 of the 602 missed attacks ended without a single record retrieved.
Two more findings should change how you read a vendor demo. First, letting the agent review its own decision made things worse. Self-Refine fixed 16 wrong calls and broke 79 correct ones, and chain-of-verification fixed 25 and broke 132, because the review reread the same context instead of fetching new evidence. Second, more compute did not buy accuracy. The single-pass design used about 5,075 tokens per alert and scored best, while the majority-vote design used 28,002 tokens and missed 31 more attacks. It also sent only 8.5% of alerts to analysts, which looks like a great number until you see it dismissed 54.2% of the attacks.
The fix the authors built is instructive. Their AIDA design has one agent propose a decision, a second challenge it from a separate context, and a judge rule on both against a logged evidence trail, with a higher bar of proof for dismissal than for escalation. It cut the false-negative rate to 3.1% while escalating 18.4% of alerts. Note the trade, since a reliable agent sends more to your analysts, not less.
The attacker writes part of the prompt
The empty search is a failure of reasoning. The second failure is adversarial, and it follows from what an AI SOC reads all day. User agents, URLs, DNS queries and attempted usernames are written by the attacker, and the “Poisoning the Watchtower” paper showed that instructions planted in those fields reach the analyst model. Persona-hijack payloads suppressed 68% of malicious logs under a naive classifier, and summarization was the riskiest task, with context manipulation succeeding 96% of the time without defenses and 38% even with constrained output.
OWASP lists this class as ASI01, Agent Goal Hijack, in its Top 10 for Agentic Applications, next to tool misuse. For a SOC agent the consequence is concrete. An agent with permission to close alerts, suppress rules or isolate hosts can be steered by the evidence it is investigating. We covered the general defenses in our prompt injection defense guide, and they apply here with one addition, which is that raw log content should be handled as hostile input, never as context the model can trust.
What an AI SOC gets right
None of this means the category is hype, and the measured gains are real when the scope is narrow. In a randomized controlled trial of Microsoft’s Security Copilot phishing triage agent, analysts with the agent found up to 6.5 times as many true positives per minute and improved verdict accuracy by 77%, and they did not rubber-stamp its malicious verdicts. A Cloud Security Alliance benchmark run with Dropzone AI found AI-assisted analysts finished investigations 45 to 61% faster with 22 to 29% higher accuracy. Each study tests one vendor’s product, and both measure analysts working with an agent rather than agents working alone. That is the pattern we keep seeing hold up: assistance that speeds up a human verdict, not replacement of the verdict.
What to do before you buy an AI SOC platform
The research hands you a test plan, and it is cheaper to run before signing than after. Replay a few days of your own alerts with known attacks seeded in, then ask for these answers.
- Show the full investigation trace for each closed alert, including every query, its result count and the reason for closing.
- Report the false-negative rate on the seeded attacks, next to the share of alerts escalated. A low escalation rate means nothing without it.
- Explain what the agent does after an empty search. The right answer is a broader query or an escalation, never a dismissal on that basis alone.
- Demonstrate how attacker-controlled fields are isolated from instructions, and test it by planting a payload in a user agent string.
- List which actions run without approval, and whether closing an alert is one of them.
- Show whether self-review fetches new evidence or only rereads the case.
If the platform cannot pass these, run it in assist mode, where it drafts verdicts and humans close alerts. That is also how our AI-driven SOC work starts, because the trace data from that phase is what tells you which alert types can safely move to supervised automation.
Frequently asked questions
What is an AI SOC analyst?
An AI SOC analyst is a software agent that performs tier-1 work: reading an alert, querying the SIEM, EDR and identity logs for context, and proposing a verdict with its evidence. In practice it drafts the investigation and a human analyst approves or overrides it. The best published results come from that pairing, not from the agent working alone.
Will an AI SOC replace SOC analysts?
Not on current evidence. The benchmark agents missed 40% or more of attack alerts on their own, and the design that fixed it sent more alerts to people. What changes is the work. Analysts spend less time on enrichment and copy-paste and more on judging the cases agents escalate and on auditing what the agents closed.
What is an agentic SOC?
Agentic SOC is a near-synonym for AI SOC that stresses multi-step agents: systems that plan, call tools, read the results and decide the next step instead of following a fixed playbook. The term matters because agency brings the risks covered above, including attacker steering through log content and actions taken without approval.
How do you measure AI SOC accuracy?
Seed known attacks into replayed alerts and measure the false-negative rate together with the escalation rate. Either number alone misleads, since an agent that dismisses everything has a perfect escalation rate. Also sample closed alerts by hand each week, because the failures that matter never reach an analyst’s queue.
Bottom line
Before any AI SOC agent closes alerts on its own, replay your own alerts with seeded attacks and measure what it dismisses, especially after empty searches. Keep dismissal behind human review until the false-negative rate on that test is one you would accept from a new analyst.