Suppose a model reviews an API specification and notices an admin operation missing from the scope document. That's a useful question to take back to the client. It isn't permission to test the operation, and it isn't a finding.

That distinction runs through AI-assisted security work. Suggestions can help a tester notice things, organise notes, or improve a draft. Someone still has to check the source material, decide what may be tested, and establish what the result means.

Ask the provider: "Show me where AI helps, who checks its output, and where my data goes." An "AI-native" label doesn't answer any of those questions.

1. What we mean by AI-native

We plan where AI fits into the work instead of treating it as a final rewrite button. That means choosing suitable tasks, agreeing on the information it may see, and deciding how its output will be checked.

Useful tasks include comparing a specification with an asset list, organising sanitised notes, and checking a draft for missing prerequisites. These are aids to analysis. They don't establish that a security control failed.

The tester remains responsible for permission, live actions, evidence, impact, severity, and the final report. A model cannot expand the agreed scope or decide that risking customer data is acceptable.

Three rules we work by

  • Agree on the data first. A useful task isn't a reason to send unapproved client material to a model.
  • Keep testing under human control. Accounts, actions, timing, and volume must stay within the written rules.
  • Check against evidence. A finding needs a result the tester can verify, not a convincing model explanation.

2. What the research tells us

The research below is useful context, not a forecast for your pentest. These are other organisations' surveys and benchmarks. They don't measure Balhence's work or establish how much faster your test would be.

AI use among hackers has risen sharply

Bugcrowd's 2026 Inside the Mind of a Hacker announcement says its report draws from surveys and interviews with more than 2,000 participants worldwide. It reports that 82% of hackers use AI in their workflows, up from 64% in 2023. Bugcrowd identifies automation, learning, and data analysis as common uses.

That result demonstrates adoption within Bugcrowd's research community. It is not a controlled comparison of AI-assisted and non-assisted pentests, and it does not establish better coverage, accuracy, or client outcomes. It also describes hackers in a crowdsourced security context, which should not be treated as identical to a commissioned VAPT engagement with a fixed scope and rules of engagement.

Researchers report both adoption and clear limits

HackerOne's 2025 Hacker-Powered Security Report says 70% of surveyed researchers use AI tools in their workflow. Its published survey methodology states that the researcher survey included 1,825 active HackerOne researchers and was fielded between July and August 2025. HackerOne says the report was also supplemented with platform data from 1 July 2024 through 30 June 2025, covering all active customer programs.

The same report provides an important counterweight to adoption: 58% of surveyed security researchers said AI misses business logic or chained exploits, and only 12% believed it could replace them. These are self-reported views from active researchers, not a laboratory measurement, but they align with a practical weakness in model-only testing. An HTTP response can resemble a known flaw while the decisive risk depends on ownership, sequence, commercial rules, or an action in another role.

HackerOne's 2025 report announcement also says autonomous agents submitted more than 560 valid reports. That is evidence that autonomous systems can produce accepted results on its platform. It is not a count of autonomous client pentests, it does not reveal the total submission denominator in that statement, and it should not be read as a replacement rate for human researchers.

Controlled systems can find and patch real code vulnerabilities

DARPA's 2025 AI Cyber Challenge results show a different capability. Across a controlled final competition involving more than 54 million lines of code, competitor cyber reasoning systems found 54 of 63 synthetic vulnerabilities and patched 43. They also found 18 real, non-synthetic vulnerabilities in the open-source codebases and produced 11 patches for those real findings.

AIxCC was a controlled code-security challenge using systems built by multiple finalist teams. It was not a live web or API pentest, a comparison against human testers, or evidence about client authorization, production safety, business logic, or reporting. It demonstrates automated discovery and patching under competition conditions.

What the numbers support: AI adoption is high among security researchers, autonomous systems can produce valid security reports, and controlled systems can find and patch vulnerabilities at scale. The same evidence does not support claims of complete coverage, guaranteed speed, or autonomous replacement of a skilled tester.

3. The tasks where it can help

Checking scope documents for gaps

Scope information arrives in several places: a diagram, an API specification, release notes, and an email. AI can help compare approved documents and flag contradictions. The tester checks each mismatch and asks the client whether it changes the scope.

The web and API pentest scoping guide explains what needs to be agreed. A route mentioned in a document isn't automatically authorized for testing.

Asking another question

Given an approved description of a workflow, a model can suggest missing questions. What should happen to shared records when a member leaves? Does a suspended account keep an export already in progress?

The tester decides which questions apply to this product. Some suggestions will repeat existing coverage; others will misunderstand the rules. Generating a longer list doesn't make it a better test plan.

Organising observations and comparing patterns

AI can help organise sanitised observations and point out inconsistent responses. That's a lead to review, not proof. Different data, feature flags, background jobs, or caching may explain the difference.

The tester checks what changed and keeps the original evidence. A summary must never become the only record of a result.

Checking a report for missing details

A draft might forget to name the test role, explain a prerequisite, or distinguish observed impact from a possibility. AI can flag those gaps for the author. It must not fill them with invented details.

The report guide explains the detail a reader needs. Our synthetic sample report shows the format without using client evidence.

4. Who checks the work?

"Human in the loop" is vague. Ask who does each of these jobs:

  1. Before testing: confirm permission, scope, accounts, safety limits, and stop conditions.
  2. During testing: review and control live actions, including any generated material.
  3. Before calling it a finding: verify the result with controlled accounts or synthetic data.
  4. When assessing impact: separate what happened from what might happen next.
  5. When assigning severity: consider the actual access needed, affected data, and limiting controls.
  6. Before delivery: check the evidence, wording, redaction, and proposed repair.
  7. At the re-test: verify the agreed fix and record the outcome without erasing the original issue.

For our work, those responsibilities stay with the tester. If something is uncertain, the report should say so.

5. What happens to your data?

A scope document or a small response body can still contain sensitive information. Before using a model, agree on the service, purpose, permitted material, and retention terms.

We record that permission in the scope or rules of engagement. Client material doesn't go to an external model unless the use, provider, and type of data are permitted. Credentials, active session tokens, private keys, and recovery material stay out of prompts. Raw evidence stays in the agreed evidence channel unless separately approved.

Controls to settle before testing

  • Purpose: which tasks may use AI, and which may not?
  • Material: distinguish public documents, synthetic examples, sanitised traffic, code, and confidential evidence.
  • Provider: record the service, access restrictions, retention, training treatment, and relevant processing location.
  • Redaction: remove secrets, personal data, and unrelated content before an approved task.
  • Access: don't give a model unrestricted credentials or production authority.
  • Original evidence: keep captures and tester notes separately from generated summaries.
  • Cleanup: include prompts and derived files in the agreed retention and deletion rules.

If your policy prohibits external model processing, say so. The method needs to fit your data rules, not the other way around.

6. Questions worth asking before you sign

Ask for a walkthrough of the actual process, including:

  • Which parts of scoping, testing, evidence handling, and reporting use AI?
  • Can an agent send live requests, or must a tester approve and control them?
  • Which AI providers and service tiers may process engagement material?
  • Are prompts or outputs retained or used for provider training?
  • Which client data classes are prohibited from model processing?
  • How is every reported finding manually reproduced and tied to raw evidence?
  • Who decides business impact, severity, safe proof limits, and re-test status?
  • Can the provider show a useful sample report and written rules of engagement?

If a provider claims dramatic speed gains or complete coverage, ask what was measured and against what baseline. The external studies above don't establish those promises for your application.

7. Check whether AI made the work more useful

Did the document comparison reveal a real scope gap? Did the draft review catch missing evidence? How much time did a tester spend correcting plausible but wrong output? Those questions help judge whether assistance was useful.

A prompt count doesn't answer them. We don't publish speed, coverage, or finding-rate improvement percentages without comparable data from our own engagements. The report still needs to stand on the agreed scope and the evidence.

The useful result: someone can explain the finding, show how it was checked, and help your team understand the repair. How many words a model generated isn't part of that test.

Primary sources and methodology

Agree on AI use before the test

If you're planning a web application penetration test, share your AI and data-handling requirements with the scope. We'll agree on what can be used before the work begins.

Request a pentest proposal Draft your scope first