Proving LLM-Found Authorization Bugs in AI-Generated Code

1Introduction

Authorization is the hard case. Whether a route may return an object depends on who is asking and on the application's intended policy ("a user may read only their own documents"). That policy lives across RLS rules, ownership columns, and route guards — and the bug is the absence of a check relative to it. No pattern match expresses "this query should have been scoped to the caller and is not." This is precisely the class AI-generated code gets wrong most often.

The research question is narrow on purpose — not "can an LLM hack a site" (others have shown it can), but when an LLM claims a broken-access-control bug, how do we prove it is real — and decline it when it is not — without ever trusting the model's verdict? This paper makes four claims, in order of importance:

• Correctness must come from a deterministic oracle, not a model (§6, §10) — this is the contribution. A pure before/after/negative-control rule decides whether anything ships; no LLM renders the verdict, so model hallucination cannot produce a false "proven." The same rule makes the oracle a precision gate: a real-looking finding whose exploit does not fire is returned not_reproducible and refused.
• Authorization is the dominant blind spot of AI-generated code (§2).
• An LLM can autonomously discover and exploit these bugs (§7, §8) — supporting, early-stage evidence, not a benchmarked rate. The applied fix is a reference fix used as a proof vehicle, not an autonomous fixer.
• Discovery is the hard input, not a solved one (§5). A position, well supported by prior work: access-control intent is implicit in code rather than stated as a spec [Sun et al. 2011], and detecting missing checks is specification-free and false-positive-prone even for strong tools [Near & Jackson 2016].

2The authorization gap in AI-generated code

In separate testing of our production static scanner — a different system from the oracle described here, used only to characterize the problem — the pattern of misses is the telling signal: syntactic classes are caught reliably, while the residual failures cluster on authorization logic (an empty allow-list that returns "allowed," a trusted-proxy header that skips a role check). Pattern matching plateaus where authorization logic begins.

The risk is not hypothetical. CVE-2025-48757 describes RLS misconfiguration and anonymous-key exposure in apps built with the AI app-builder Lovable: an analysis found 170 of 1,645 projects (10.3%) affected, with 303 vulnerable endpoints, rated CVSS 9.3. Recent AI-assisted static tools have improved IDOR detection (Semgrep's multimodal detector reports 61% IDOR precision) — but a static claim that an IDOR "exists" is still a probability; a deploy decision needs certainty. VulX's contribution is the runtime oracle that converts a candidate into a proven, fixed result — or into nothing at all.

3Threat model and scope

Target. Web apps with a clear data-ownership model, exemplified by the Next.js / Supabase / Postgres stack: routes or edge functions that read and write rows whose ownership is declared by RLS policies and ownership columns (owner_id = auth.uid()).

Adversary. An authenticated but unprivileged user who can address objects they do not own and attempts to read or modify another user's data — Broken Object-Level Authorization (BOLA / IDOR; CWE-639, and related CWE-285/862/863), the top-ranked risk in the OWASP API Security Top 10.

What we prove vs. assume. The system assumes a candidate names the suspect route, owned resource, and object-id source. It proves, at runtime against a live instance, whether a cross-user read or write actually occurs, whether a patch eliminates it, and whether legitimate same-user access still works. The oracle's deterministic guarantees cover time-based and boolean-blind SQLi, reflected XSS, command injection, LFI/traversal, and read/write BOLA; all end-to-end evidence reported here is BOLA.

4System overview

VulX is a pipeline with a strict separation of duties: LLMs propose, a deterministic oracle disposes.

Red = LLM (proposes, may hallucinate). The oracle (the contribution) is invoked twice: once on the authored attack (must fire), once after the reference fix is applied (must not fire, control must pass). It alone issues a verdict; the LLM never does.

5Discovery: the hard problem

Discovery localizes a candidate BOLA without executing anything, recall-first ("a missed BOLA is worse than a false positive the oracle removes"), and emits candidates, never verdicts. It unions three deterministic extractors: route/sink loci via structural matching (ast-grep); data sinks and client kind (resolving whether a call runs as anon, authenticated, or — critically — service-role, which bypasses RLS); and an RLS/ownership model parsed from SQL migrations. An optional LLM inference layer only re-ranks confidence — it never drops a candidate and cannot fabricate one ("an oracle proves verdicts later, an LLM's say-so does not").

On the one real open-source target (sharespace-connect), discovery localizes the BOLA at confidence 0.99 with zero LLM calls, identifying that the route checks only that an auth header is present — never that the bearer owns the document — deterministically, with no model.

A measured recall floor (deterministic, LLM-free) MEASURED

We measured the recall of the deterministic discovery floor (the LLM-free structural pass) by running it over each app's source and asking whether it flags the oracle-proven vulnerable shape. On the six oracle-proven real-app HITs it recalls 5/6 at high confidence (≥0.7) and 6/6 at a loose threshold (the sixth, App E, only as a weak 0.45 incidental hit); on synthetic fixtures, 2/2. We report real and synthetic separately and never blend them. An independent overfit audit confirms the floor generalizes: it flagged never-seen tables and skipped a benign public-data lookup purely from structural signals (RLS state, anonymous grants, PII/ownership), with no LLM in the loop.

6The deterministic exploit oracle

The oracle is the heart of the system and the reason it is safe to run against real code. It never asks a model whether a fix worked; it measures.

Why "verified" usually lies

A common shortcut — one our own production scanner used and which we now label "heuristically verified" — is to re-test a patched endpoint and grep the response for an error string. This passes blind injections (which leak nothing), passes routes that "got fixed" by crashing (a 500 has no error string), and, in the static case, can mark a finding fixed with zero execution. A substring scan is not a proof.

Three measurements, one pure rule

• Before: fire the exploit at the unpatched app — the signal must be present (else it was a false positive).
• After: apply the patch, fire the identical probe — the signal must be gone.
• Control (negative control): a benign, legitimate request must still succeed — this catches "fixed it by breaking the app."

def decide(before, after, control_ok_after) -> Verdict:
    if before.error is not None or after.error is not None:
        return Verdict.INCONCLUSIVE      # couldn't measure -> NO PR
    if not before.present:
        return Verdict.NOT_REPRODUCIBLE  # false positive    -> NO PR
    if after.present:
        return Verdict.STILL_EXPLOITABLE # fix failed         -> NO PR
    if not control_ok_after:
        return Verdict.FIX_BROKE_APP     # broke the app      -> NO PR
    return Verdict.BLOCKED               # proven & non-breaking -> SHIP

The ship-gate. Pure, deterministic, never an LLM call. Four of five verdicts open no pull request. "No PR is safer than a false proven PR."

The precision gate: refusing what cannot be proven

The most important verdict is not blocked but not_reproducible. When discovery and the LLM produce a confident, plausible candidate but the exploit does not fire, before.present is false and the oracle refuses the finding rather than reporting it. This is where an LLM-only pipeline fails: it surfaces the plausible-but-false bug as real; the oracle cannot, because the verdict is a measurement, not an opinion (demonstrated on a real app, §8). A tool that refuses an unexploitable finding is the entire value proposition.

The BOLA signal & the negative control

For read-BOLA, the signal is whether the victim's secret marker (seeded into the victim's row, never in the attacker's request) appears in the attacker's response — regardless of HTTP status (a "fix" that returns 403 but still leaks the body must not read as fixed). The negative control is owner-reads-own: after the patch the owner must still get 2xx plus their own marker — the single test that distinguishes a real fix from a deny-all "fix," which is correctly ruled fix_broke_app. Soundness of the before/after comparison depends on a clean state reset: the backend snapshots the seeded database and resets via Postgres CREATE DATABASE … TEMPLATE between measurements.

7From candidate to certified fix: the loop

Around the oracle sits a thin loop — author attack → prove → apply fix → re-prove — whose brains and sandbox are injected behind Python protocols. The reasoning model is hosted Claude Opus 4.8; a VulX-trained model is roadmap behind the same seam.

The LLM's job: reason about the policy, author the attack. Given a candidate dossier, Opus 4.8 answers one question — does the route's enforcement match the intended access policy, and if not, how is the gap exploited? Its output is concrete and checkable — a victim object, an actor, a rationale — e.g. verbatim: "the handler fetches the note with the service-role (BYPASSRLS) client by id only and never compares note.user_id to claims.sub, so any authenticated user can read another user's note." The model emits a plan; the leak and control signals are filled from ground truth, so it can never fabricate the proof.

The fix is a proof vehicle, not an autonomous fixer. What the oracle measures is the effect of a fix. On real apps the fix is a known-good reference patch; on controlled fixtures we additionally let the LLM author the patch (passing a backdoor-safety gate) and run the adversarial battle, in which the LLM is shown its own applied patch and asked for a bypass — required to give up honestly rather than invent a payload. A productized autonomous fixer is roadmap; we never claim the fix step is autonomous on real apps.

8The oracle in action: confirm, then refuse

Every figure is from a committed run-log (simulated=false, model: claude-opus-4-8); apps were stood up in isolated, harness-provisioned test-beds (§12).

Confirm — a corpus of proven authorization bugs on real apps

Honest scope: of these six real-app HITs, only sharespace-connect was found by autonomous discovery; the other five were proven on hand-fed exploit shapes. They strengthen the proof (and the range of flaw shapes the oracle handles), not a discovery rate.

sharespace-connect (a real Lovable-built document app) was auto-found at confidence 0.97 in the autonomous run; Opus 4.8 reasoned that the edge function reads a document by id with no owner check. The oracle leaked the victim canary at HTTP 200 (before), a reference fix returned it absent at 404 (after), and the owner-reads-own control still succeeded → BLOCKED; 3 LLM calls.

App E additionally produced an identical verdict on local Docker and on E2B — a backend-independent proof point: the verdict tracks the measured exploit, not the sandbox it ran in.

Refuse — the honesty contract, three ways

Mode 1 — not_reproducible: a real flaw whose impact does not reproduce (App F). Fully deterministic (zero LLM calls). App F contains a genuine flaw: a sharing-grant table whose insert policy is WITH CHECK (auth.uid() = <granter>) and never checks the caller owns the shared record, so any user can self-grant a share. The write behaves like a real bug — the self-grant returns 201 before the fix and 403 after, while a legitimate owner-initiated share still returns 201. Yet the oracle returns NOT_REPRODUCIBLE: after the self-grant, the attacker's cross-actor read returns 200 but an empty body [] — the planted canary never appears (a latent platform footgun resolves the caller's email through an RLS-locked auth.users subquery → NULL). The authorization flaw is real; the confidentiality impact is not. A scanner flagging the missing check would report a false positive; the oracle declines to confirm.

Mode 2 — inconclusive: no baseline to form a control (App G). App G leaks real data, but it is a missing-authentication flaw (CWE-306), not BOLA (CWE-639): the app exposes everything to anonymous callers, so there is no authenticated owner-read baseline against which a negative control could be defined. Rather than fabricate one, the oracle returns INCONCLUSIVE — even though the leak is real. It cleanly separates CWE-306 from the CWE-639 the oracle is built to prove.

Mode 3 — standup-blocked: the target cannot be built from source (App H–App J). Three targets keep core tables in the Supabase/Lovable dashboard, not the repository; their RLS references tables the repo never creates. A cheap schema-completeness gate detects the gap and defers rather than guess — a provisioning limit (the off-repo-configuration ceiling, §11), which turns out to be the dominant real-world wall.

Together these are honest refusals in three modes — unproven impact (App F), no measurable control (App G), and an un-buildable target (App H–App J) — each chosen over reporting a finding the system cannot prove.

The full loop, on a controlled fixture (bola-notes)

On controlled fixtures we exercise the entire loop. On bola-notes (a service-role read by id), Opus 4.8 authored the attack (leaked a canary at 200), then proposed a minimal, framework-idiomatic patch that passed the safety gate:

   const { data: note, error } = await supabaseService
     .from("notes")
     .select("*")
     .eq("id", params.id)
 +   .eq("user_id", claims.sub)
     .single();

LLM-authored patch (verbatim, fixture). The single added line re-imposes the ownership check RLS would have enforced.

The oracle re-fired the probe (canary absent at 404), the owner-reads-own control still passed, and across 8 database resets the verdict was BLOCKED. Shown its own patch, Opus 4.8 then tried to bypass it and conceded honestly — "… the symmetric bob→alice access … is equally blocked … so the broken authorization is genuinely closed" — making the round a genuine "survived." Total LLM cost $0.084.

9The data flywheel

Each oracle-confirmed finding is distilled into a code-free signature so one proven bug can drive variant analysis across other codebases. The privacy posture is the trust gate: metadata, never code. The de-identification gate is a type-total positive allowlist — it walks the entire record and raises on anything not provably safe; keys must be known structural names, string values controlled categorical tokens; integers digit-capped and floats range/precision-capped; emails, UUIDs, JWTs, and key-shapes rejected. No LLM is in the gate — a model there would itself be an exfiltration surface. The gate was red-teamed across four passes (24 leaks → fixed → 0). It has now had its first real multi-finding fill: the batch-2 HITs were de-identified into distinct, idempotent, zero-raw records — the first time the flywheel was loaded with measured findings rather than modeled ones. The remaining step is the cross-machine live transmission to the central store (§11).

10Evaluation

This paper's contribution is the oracle, so the evaluation asks one question: does the oracle issue correct, mechanically-grounded verdicts — confirming real exploits and, just as importantly, refusing non-reproducible ones?

We deliberately do not report a detection-rate benchmark. At this stage n is far too small to support a rate, and a headline "we find X%" would invert the paper's thesis, which is about proof, not volume.

The committed oracle runs MEASURED

The wider real-application corpus — six confirmed HITs and five declined (two refused by the oracle, three deferred as un-buildable) across eleven real apps — is summarized above; the table below zooms in on the proving-ground detail: two controlled fixtures (bola-notes, bola-helpdesk) plus the autonomously-discovered sharespace-connect HIT and the App F refusal.

Baseline: do standard static tools catch these? MEASURED

We ran two industry-standard static analyzers with their standard security suites over an initial set of five real-app targets, and then over the five proven batch-2 HITs. Neither tool flagged a single authorization bug. The vulnerable files were confirmed in analysis scope, so a miss is a query/capability gap, not a coverage gap.

Four of the five proven bugs live in the Postgres/RLS layer neither tool models (CodeQL has no SQL analysis; Semgrep ships no Supabase-policy rules), and the fifth (App E) is a pure-TypeScript authorization-logic flaw outside both tools' modeled vulnerability classes. This is not a head-to-head score — it shows the class sits structurally outside pattern/dataflow SAST. Runtime proof is complementary, not redundant.

Cost MEASURED / MODELED

Measured cost is $0.065–0.11 per solve (dominated by model tokens; sandbox compute negligible). In the batch-2 corpus, cost was ~$0.09–0.10 per target — e.g. the autonomous sharespace solve at $0.0996 on a warm-E2B template. The batch runner amortizes the sandbox: a warmed template plus parallel fan-out makes N targets cost roughly one boot. For context only, MAPTA reports 83% on the XBOW broken-authorization set at ~$3.67 per repository assessment (a different unit; not a head-to-head). The deterministic oracle avoids paying a model to render a verdict — a structural cost edge, since the verdict is computation, not generation.

11Limitations and honesty

• Small n, early-stage, and hand-fed. Evidence covers eleven real apps (six confirmed, two refused, three deferred) plus two fixtures. Only sharespace-connect was auto-found; the rest were hand-fed shapes. This demonstrates the oracle's proof behavior; it establishes no discovery precision/recall.
• The fix is a reference patch on real apps. An autonomous production fixer is not claimed and is roadmap. The LLM-authored fix and adversarial battle are fixture-only.
• "Real model" ≠ trained model. Every model-driven result uses hosted Claude Opus 4.8; no VulX-trained model has been exercised.
• Model-guided, not unscaffolded. The LLM emits a plan; the leak/control signals are filled from ground truth and the test-bed is harness-provisioned — a deliberate soundness choice; we do not claim autonomous host compromise.
• Off-repo configuration ceiling (the dominant real-world wall). Many Supabase/Lovable apps keep schema/RLS in the dashboard, never in git; three of our targets (App H–App J) were confirmed-but-unprovable for this reason. The empirical case for a "connect-your-project" step over "paste-a-URL."
• Non-authz coverage is unproven end-to-end. The SQLi/XSS/cmdi/LFI probes are implemented and unit-tested offline; every committed proof is BOLA.
• Flywheel not live cross-machine; production auto-fix is heuristic; results self-reported.

12Ethics and responsible disclosure

This work studies publicly available, open-source applications. All exploitation was performed solely against local clones we stood up in isolated sandboxes — never against any third party's production system, infrastructure, or real user data. Each target ran with synthetic accounts and canary data only, and the application code was left byte-identical to its public source. One target (sharespace-connect) is our own application; the others are third-party open-source projects, identified here only in anonymized form.

13Related work

Semgrep's multimodal detector represents the AI-assisted static frontier for IDOR (61% precision); it produces strong candidates — VulX is complementary in producing proof. CodeQL pioneered multi-repository variant analysis, the model our flywheel adopts. In our own baseline, both, run with standard suites, flagged zero authorization bugs on our targets. MAPTA, an autonomous LLM pen-tester, reports 83% on broken authorization (XBOW). BACFuzz argues for instrumenting the data layer rather than trusting HTTP status — the principle behind our read-back BOLA signal. Our differentiator is the deterministic ship-gate (never an LLM verdict), the stack-grounded policy inference that scopes the attack, and the requirement that a fix survive re-attack before anything ships.

14Future work

Every oracle-confirmed run banks one labeled trajectory whose label is the oracle's verdict — the intended training set for a distilled attacker model, gated behind beating hosted Claude on held-out, oracle-confirmed exploits. The flywheel's end state is cross-codebase variant analysis: one proven-and-fixed bug, abstracted to a code-free signature, drives prove-before-surface queries across every connected codebase of the same class. The batch runner and first real flywheel fill are built and measured; the near-term priorities are autonomous discovery of the new flaw shapes, a measured discovery precision/recall on a labeled corpus, baselines on new targets, and re-attack survival on a real app.

15Conclusion

When an LLM claims a broken-access-control bug, how do we know it is real — without trusting the model? Our answer is a deterministic oracle: a pure before/after/negative-control rule that issues every verdict, so model hallucination can never produce a false "proven." The same rule is a precision gate — on a real application it refused a genuine flaw whose unauthorized write succeeded but whose data leak did not, exactly where an LLM-only pipeline would have cried wolf. Four of five verdicts open no pull request, and that is the point: a security tool earns trust by refusing to cry wolf. The evidence here is deliberately small and labeled, but it is proof, not a confidence score. Capability comes from the model; trust comes from the rule.

References

• Georgia Tech SSLab. Vibe Security Radar: Real CVEs in AI-Generated Code.
• M. Palmer et al. CVE-2025-48757: RLS misconfiguration in AI-generated (Lovable) Supabase applications. 2025.
• Semgrep. AI-Powered Detection with Semgrep (Multimodal IDOR detection). 2026.
• I. David et al. Multi-Agent Penetration Testing AI for the Web (MAPTA). arXiv:2508.20816, 2025.
• BACFuzz: Exposing the Silence on Broken Access Control Vulnerabilities in Web Applications. arXiv:2507.15984, 2025.
• F. Sun, L. Xu, Z. Su. Static Detection of Access Control Vulnerabilities in Web Applications. USENIX Security, 2011.
• J. P. Near, D. Jackson. Finding Security Bugs in Web Applications using a Catalog of Access Control Patterns. ICSE, 2016.
• OWASP. API Security Top 10 (2023): API1 Broken Object Level Authorization.

AReproducibility

The artifacts behind every claim are committed: each run is a JSON record carrying simulated:false, model id, per-step token cost, before/after signals, control result, db-reset count, and the model's verbatim rationale. The verdict is the pure function decide() above — independent of any model — so a third party can re-derive every verdict from the logged measurements. The Semgrep/CodeQL methodology and reproduce commands are in baseline_sast_eval.md. Exact run-log identifiers are available to design partners under NDA.