QEL Verification

QEL activates when Bodega's agentic loop finishes a creation task: any request that contains a creation verb (create, write, generate, build, implement, scaffold) paired with an artifact noun (files, scripts, components, endpoints, etc.). A plain question ("what does this do", "where is X handled") is not graded as a build or fix. It gets the advisory Answer check described below instead, which has no score.

The system does four things in sequence:

1. Extracts a machine-checkable contract from your request before the first LLM call - which files must be created, which framework is required, which patterns must appear in each file, and which shell commands should verify the output.
2. Injects that contract into the model's system prompt as a TASK CONTRACT block so the model knows exactly what it must produce.
3. Scores the output across five dimensions after the agent finishes writing files.
4. Runs repair nudges if the score falls below the pass threshold - injecting targeted correction instructions back into the agentic loop for up to 2–3 attempts before surfacing a failure note.

QEL was introduced in beta.19. The structured trace card, score breakdown object, and expanded framework database (Next.js, NestJS, SvelteKit, Hono, Elysia, Prisma) were added in beta.25.

Code Mode: every creation, modification or bug-fix task is verified. A modification or fix request where nothing was edited and nothing was observed to fail (for example the turn stopped at a write approval, or it only ran a passing test, build or lint command and concluded the code was already correct) is not verified: the card reads Not verified with no score, nothing is shown as a failure, and no repair attempts are spent on a turn that never edited, unless a test or build command QEL ran did fail. If a test or command you could see did fail, that still counts.

Chat Mode: a build or fix request is verified only when the turn actually wrote or edited a file, which in Chat Mode means a project folder is attached. Only writing, appending to, creating or editing a file counts. Deleting or renaming a file, or generating a document, does not. Asking Chat Mode to fix a snippet you pasted, or to write a function and show it in the reply, is answered in the chat and not graded as a build: there is nothing on disk to check. Before this rule, such a turn was nudged to edit files that did not exist and could loop until the agent gave up.

Answers: turns that are not graded as a build or fix, in either mode, go through the answer check instead. When the agent used tools to answer, Bodega checks that the files it names were actually seen in this session (read or written this turn or in an earlier turn; common config files such as package.json and tsconfig.json are always fine to mention) and adds a short note if not. A filename quoted as missing ("x.ts does not exist") is not counted as a citation. This is the Answer check: it is advisory, has no score, and is never saved as the session's verification score, a regression contract, or a pass signal for the command line tool or Loops. The optional second-model answer judge (qel.answer_judge, off by default) only runs on these answer turns, never on build or fix turns. That is deliberate: QEL grades a turn either as a build or as an answer, never both.

Read-only side chats are never verified in either mode.

Results show up in three places.

Chat sidebar - score badge Sessions that completed a QEL-verified creation task show a score badge (0–100) next to the session title in the Chat sidebar. Green = pass, red = fail. This lets you compare QEL outcomes across sessions at a glance without opening each one.

Chat mode - Verification card A collapsible card appears directly below the last assistant message after every creation task. The card header shows:

• The label Verification (hover it to see this is QEL, the Quality Enforcement Layer)
• The numeric score, color-coded: green ≥ 80, yellow ≥ 60, red < 60
• A PASS, FAIL or NOT VERIFIED badge (not verified means nothing could be checked)
• A repair #N tag if the response is the result of a repair attempt
• The deliverable filenames

Click the header to expand the card body. Inside you'll find:

• Score breakdown bars for Files / Patterns / Framework / Complete / Proof (value / max)
• An Issues section - failed files (red X), framework violations (red X), missing required patterns (orange triangle), failed proof gates (red X)
• An Optional patterns missing section for soft-pattern misses
• Proof gate output with truncated stderr from failed gates
• A collapsible Show full report section with the complete text report

Answer check card: a question turn shows a smaller card labelled Answer check with a GROUNDED or CHECK SOURCES badge. It has no /100 score, no PASS or FAIL, and no red band, because it is not a verification. Expand it to see which named files were not opened this session.

Drift strip: under the card in Code mode, the DRIFT strip re-checks results QEL verified earlier. It stays passive until you press Check now and reads Drift check - not run until a report exists.

The card always shows the newest result of the current turn, and it is cleared when you send the next message. It is not persisted to the database. Closing and reopening the session clears it.

Code mode - Verification tab in the Debug panel Open the Debug panel (Ctrl+Shift+E or click Debug in the panel sidebar), then click the Verification tab. The Verification tab shows the same data as the Chat mode card but with more vertical space and adds a repair trajectory section - a reverse-chronological list of all repair attempts for the current creation task (up to 10 iterations). A purple dot badge appears on the tab label when a trace is available and the tab is not currently active.

When no creation task has run, the Verification tab shows: No verification data. Runs automatically after creation tasks.

ACP agents (Cursor, Claude Code, Gemini CLI, Codex) do not go through QEL. Their Fleet cards show an external - not QEL-verified badge instead.

Final score = sum of all five dimensions, capped 0–100.

Structural integrity - anti-stub multiplier

Before the pattern score is finalized, StructuralVerifier scans the written code for stub functions - empty bodies, TODO comments, pass, raise NotImplementedError. The ratio of real to stub elements becomes a multiplier on the pattern score. 50% stubs halves the pattern credit. This runs on .ts, .tsx, .js, .jsx, .py, .go, and .rs files only - not on HTML, CSS, or JSON.

The contract extractor runs in under 5 ms using pure regex - no LLM call. It parses your message to produce a machine-checkable ExecutionContract that specifies:

• Deliverables: which files must be created. If you name files explicitly (create server.py and models.py), those are used directly. If you only name a framework (create a Flask API), the extractor infers the entry-point file (app.py).
• Framework: detected by keyword matching against the FRAMEWORK_DB. Recognized frameworks include: Flask, FastAPI, Django, pytest, unittest, Express, Fastify, React, Vue, Next.js, NestJS, SvelteKit, Hono, Elysia, Prisma, Gin, Echo, Fiber, Actix, Rocket, Axum, Spring, Rails.
• Hard patterns (MUST appear): framework imports, route paths, class names. Weighted 2× in scoring.
• Soft patterns (SHOULD appear): helper function names, field names. Weighted 1×.
• Proof-gate commands: inferred from file extensions.

The contract is injected into the model's system prompt as a TASK CONTRACT block before the first LLM call. It is also visible as the Deliverables section inside the expanded Verification card.

Extraction confidence can be low or medium when requests are phrased ambiguously - in those cases, fewer patterns are extracted and the score may be higher than expected for incomplete output. If you want more rigorous verification, name your files explicitly and include the framework name in the request.

As you type a creation or modification task in the chat composer, a small non-blocking Will verify (creation) / Will verify (modification) card appears above the input listing the deliverable files, framework, and route paths QEL will check against.

It is purely observational - debounced, read-only, and it never gates or alters your send. It stays hidden for plain chat and read-intent questions. Under the hood it runs the same ExecutionContract extractor (regex + static maps, under 5 ms; no LLM, DB, file-system, or network access), so the preview is the same contract you'll see post-run in the trace card's Deliverables section - just surfaced before the run.

This lets you tighten your request before generating: if the preview shows fewer files or the wrong framework than you intended, name the files explicitly or add the framework name, then re-read the updated card. What you see in the preview is exactly what the oracle will verify against.

Proof gates run real compiler and test-runner commands against the written files. When you ask for new files, commands are inferred from the file extensions in the contract:

Gates run in parallel, so a compile gate and a test gate cost one round, not two.

Edits to existing code: when a turn writes a TypeScript file, QEL also type-checks the whole project once, but only if the project root has a tsconfig.json and TypeScript is installed in the project's node_modules (QEL never downloads it). Type errors in the files the turn changed, or in files that import them, fail the check and go back to the model to repair. Errors in unrelated files are listed as pre-existing, not attributed to this turn and do not fail the turn. A project with no root tsconfig.json, or a root config that only lists project references, gets no type-check.

If the toolchain is not installed or not on PATH, the exit matches an environment-unavailable pattern (command not found, ENOENT, etc.) and the gate is scored neutral - no positive or negative points, no repair nudge. The same goes for a runner mismatch: when your project declares no test runner and the assumed Vitest finds no tests it recognises (No test suite found), that is never counted against your code. If your project does declare Vitest, an empty or uncollected suite is a real test failure. The trace card shows the stderr so you can diagnose toolchain issues.

For languages with no proof gate (Swift, Kotlin, and others), the trace card shows: No proof gate available for that extension. Code was not verified. - and a code task with no verification evidence at all parks for review rather than auto-passing.

Mid-loop micro-proof gates also run after every Nth file write during a multi-file task, using the same commands with a tighter timeout. These catch compile errors while the model is still in its tool-calling loop - before the full post-loop verification.

Execution proofs (new in beta.27): for a server task, QEL goes one step past compiling - it starts the generated app on a private local port and sends one request to the first route you asked for. A response under 500 is the strongest pass evidence; a crash on boot or a 5xx is a real failure; a missing interpreter or undetectable start command is neutral. The whole proof is capped at ~12 seconds, runs only against 127.0.0.1, and the spawned process tree is always killed when it ends. Turn it off with the qel.execution_proofs setting. The trace card shows the probe (GET /api/tasks → 200).

QEL automatically adjusts its pass threshold and timeouts by the model's size class. You don't configure this - the threshold is resolved from the model ID before the agentic loop runs.

Smaller models get longer timeouts because they generate code more slowly. A PASS needs two things: a score at or above the threshold for the model that ran, and something observed that confirms the work. A small-tier PASS at 65 / 100 is not the same bar as an xlarge PASS at 85 / 100, so factor in the tier when comparing outputs across models. A score above the bar with nothing confirming it shows NOT VERIFIED, not PASS.

When the score falls below the pass threshold, Bodega sends a structured repair nudge back to the agent as an extra message. The nudge is not free-form - it contains numbered REWRITE/PATCH instructions:

REWRITE server.py: Missing REQUIRED patterns: /api/tasks. Use Flask, not FastAPI.
PATCH models.py: Add: Task class, User class

The default repair budget is 2 attempts. A 3rd bonus attempt is granted if the score improved between iterations (a converging model). If the score stagnates or regresses, the bonus is withheld.

After the budget is exhausted, a summary note is appended to the response:

Repair budget exhausted (2 attempts). Score: 62/100. Missing: server.py. Details: ...

After 3 consecutive repair failures, the nudge is prefixed with a re-plan instruction - STOP - your repairs are not working. Step back and re-plan... - to break the model out of a stuck loop.

Test-first repair (beta.29): at the 2nd consecutive failure - earlier than the re-plan - a capable model whose project has a test runner (Vitest/Jest/pytest/go test) gets a stronger move instead of another pattern-nudge: write one failing test that pins the missing behavior, prove it fails for the right reason, then fix until it's green. A passing test is concrete proof the behavior works. It's deliberately gated to strong/medium tool-calling models - weaker local models tend to spiral writing tests they can't satisfy - and fires once, not in a loop. Toggle: agent.test_first_repair (Settings → Agent → Behavior, on by default).

Unattended runs with a capable model get at most one guess-based repair. In an Act-mode run with no one watching (the command-line tool, a loop, a benchmark) and a large or strong tool-calling model, Bodega sends one repair nudge that rests on its own pattern check, then stops. A repair backed by something observed is never limited: a failing build or test, a boot that crashed, a file your request named that is missing, or text your request quoted exactly. Small and local models, and interactive runs, keep the full repair budget.

Repair is skipped for single-file deliverables where the file was successfully written. Small local models tend to re-write the same file endlessly when given a verify nudge - skipping repair prevents that oscillation.

Truncated file detection: if a file write was cut off mid-stream (detected via [Truncated or max_tokens in the tool result), a file-continuation nudge fires first: The file was truncated. Use file_system append from exactly where it stopped.

In Chat mode you see each repair iteration as a new agent response, with a repair #N tag on the QEL card header. In Code mode the VerificationReportPanel Verification tab shows the full repair trajectory under Show repair trajectory (N iterations).

After a creation task passes QEL, Bodega can run an optional second pass that reviews the files the agent actually changed - looking for single-responsibility violations, awkward naming, and obvious bugs - and surfaces what it finds as a collapsible Code Quality section inside the verification card, grouped by file with a severity dot.

It is non-blocking: it never gates completion and never changes the QEL pass/fail verdict. It is deliberately bounded - it reviews creation tasks only, skips any file over 50 KB, caps the list at five findings, and bails to an empty result after a 5-second budget so a slow review can never hold up your run. Every finding is sanitized (paths relativized, secrets redacted) before it lands in the trace.

It is off by default. Turn it on in Settings:

• qel.code_review - enable the post-loop review.
• qel.code_review_model - the model to review with. Leave empty to reuse the model that generated the code.

Under air-gap the review only runs against a locally-served model; if none is available it simply returns nothing. The findings ride the existing QEL trace, so old sessions (and headless/CLI runs that have no card to render) are unaffected - the section just doesn't appear.

Sometimes "it compiles and matches the contract" isn't the bar you care about. The rubric grader lets you attach a free-text quality rubric to a task; after QEL passes, a one-shot grader reads the rubric and the QEL result and pins a verdict (pass / fail / inconclusive), an optional score, and a short justification to the verification card.

It is opt-in per request - the rubric travels with the message, so you decide task-by-task whether to grade. It runs on a dedicated small grader model when you configure one (qel.grader_model), falling back to the model that generated the code when you don't. Using a different model to grade is recommended: a model judging its own output shares its own blind spots.

The grader is built to never get in the way. It short-circuits to inconclusive under air-gap (grading is an LLM call), on an empty rubric, or on any error - it never breaks the stream, and the rubric is truncated to keep the prompt bounded. Like the code-review section, the result rides the existing QEL trace as an optional field, so cards without a rubric render exactly as before.

A TODO list lives for one message. A goal lives for the whole task. Type /goal API passes all auth tests with rate limiting in any AI panel and the goal becomes a real object, not a line in the chat:

• It survives every message. Its task list picks up exactly where the last run stopped - a run that hits its iteration cap reports what's left and resumes on your next message.
• An approved plan's steps become goal tasks automatically - plans stop being throwaway text.
• The plan strip above the conversation shows the goal with live progress (3/7 done), and the goal title rides into the agent's context each turn so it never loses the thread.

The agent can't just declare victory. A goal only completes when the work actually verifies (QEL pass), and right before it does, a second model is sent in to attack the result - "find what breaks against this goal." Anything it finds becomes new tasks and the agent keeps working. One attack per run, it never blocks a finish on a reviewer hiccup, and you pick the reviewer model in Settings → Agent → Behavior (goals.reviewer_model) - two different models don't share blind spots. Turn the gate off with goals.adversarial_review (on by default).

The trained companion is the /decompose skill, which turns a big objective into a goal with 3–7 verifiable tasks (each task naming its own evidence - the test that passes, the file that exists).

Every QEL result (pass/fail + score) is logged to the model_performance_log table with a rolling window of 100 results per model. To see aggregate stats:

1. Open Settings → Models & Providers → Models.
2. Click the Performance tab.

The table shows each model's QEL pass rate and average score across all sessions. Stats persist until you uninstall the app (clearing the local SQLite database resets the tracker).

If a model accumulates 3 consecutive QEL failures within a session, a boost_suggestion debug event fires suggesting you enable Cloud Boost for that task. This also appears in the trace card's debug section.

A verification result has three outcomes, not two.

• PASS - we watched the work behave correctly. A test ran and passed, an acceptance command succeeded, or the thing the task asked for was checked and found. This is the only outcome that certifies.
• FAIL - we watched something go wrong. A gate ran and failed, the target file was never touched, or the change broke a build that was fine before.
• UNVERIFIED - nothing went wrong, but nothing confirmed the work either. The edit landed, the build is green, and no test exercised the new behaviour. Common and completely normal in repositories that have no runnable test covering the path you just changed.

UNVERIFIED used to be reported as a failure. That was wrong, and it was wrong often: measured against an independent grader across 89 real tasks, Bodega solved 35 and reported failure on 29 of them. A compile proves a file still parses; it says nothing about whether the bug is fixed. Treating "I could not confirm this" the same as "I watched this break" made finished work look broken, blocked automated runs from applying correct changes, and sent the agent back to repair work that was already right.

The rule now: only something we actually observed going wrong can fail a task. An absence of evidence lowers confidence and says so plainly - it never condemns.

What this means in practice:

• Automated runs (Loops auto_if_pass) still only apply a genuine PASS. UNVERIFIED parks for you to look at, exactly as before.
• The agent no longer repairs a run it could not verify - there is nothing to repair, and redoing finished work costs you tokens.
• A modification or fix turn that wrote nothing, with no failing test or command observed, is also UNVERIFIED. It is never shown as a failed score.
• To turn an UNVERIFIED into a PASS, give the task something that runs: a test command, an acceptance command, or a named function/class to check for. That is what QEL needs to observe.

Two related honesty fixes underneath this. First, the main-loop verifier now compares the file's content before and after the edit (falling back to git diff --stat when there's no earlier read to compare against) rather than trusting a write tool's own success report - a task that edits a file and then reverts it back to its original content is UNVERIFIED, not PASS, because nothing observable actually changed. Second, a code task where literally no proof gate ran at all no longer counts as a judged pass - it used to register as a synthetic gate that quietly earned full proof credit for verifying nothing; now the absence of any gate is itself what UNVERIFIED reports.

QEL is core to the CLI's headless story: bodega run's exit code is derived directly from the verdict, --qel-threshold <N> overrides the pass bar (exit 0 iff score ≥ N), and --dry-run reports the score but forces exit 0 for report-only checks. /proof in the REPL drills into the last verification - proof gates, execution proof, answer grounding, code review, failed checks - the terminal equivalent of the trace card.