Every number says what it is: benchmark, live evaluation, internal evaluation or target. AI that works starts with claims that hold up.
held-out circuits won
Specialist-led system with frontier fallback. Equal iteration budgets. Wins rank targets met first, then external tokens. Internal evaluation.
A specialist-led system won on 9 of 10 held-out circuits against a frontier baseline, with equal iteration budgets.
Internal evaluation · specialist with frontier fallbackInside the evaluationSpecialist with frontier fallback. Targets met determine wins; external tokens break ties. Internal evaluation.
Before specialist training, the base model achieved 50% procedural accuracy on the enterprise process evaluation.
Same model · enterprise process knowledgeEnterprise billing process. Internally designed, LLM-judged evaluation. Not insurance execution accuracy. Internal evaluation.
Procedural accuracy reached 88% after training: an improvement of 38 percentage points over the 50% baseline.
52 questions · 156 blind LLM judgmentsRead the methodEnterprise billing process. Internally designed, LLM-judged evaluation. Not insurance execution accuracy. Internal evaluation.
Specialist with frontier fallback. Targets met determine wins; external tokens break ties. Internal evaluation.
What task was tested? What counted as success? What was it compared against? Who designed and judged the test? What conclusion would go beyond the evidence? The methods and scope below answer those questions.
Run with a named partner engagement. Internal and not independently audited.
An analog circuit arrives as a specification table whose targets trade off against one another. The model learns from replayable expert reasoning traces, then re-solves the circuits under a deterministic manufacturability check and real simulation.
The August 2026 internal evaluation used ten held-out circuits: six op-amps and four LDO regulators, with equal iteration budgets. The specialist-led system called a GPT-5.5 frontier fallback when it stalled; the baseline used GPT-5.5 throughout. Wins were ranked by targets met, then external tokens as a tiebreaker. Four circuits used no external calls.
Unscripted probes against a live model.
The live questions examine source precedence, missing fields and dependencies between linked claim records. The product walkthrough recreates one recorded source-conflict question and separates it from rules taken directly from the workflow documentation.
The transcript includes assumptions and imperfect answers. Individual responses should be checked against the documented process; the number of probes is not a success rate.
Explore the source-backed questionsBehavior shown in a recorded application session. This is not a production deployment or an accuracy benchmark.
The recorded run extracts an attached claim form, checks policy dates, registers a claim, handles an added instruction to create a client note, and returns to email for confirmation and marking the request as read.
The silent highlights below keep the application and actual reasoning visible in the same frame. Selected sequences are accelerated. It is a demonstration of app behavior, not a completion-rate benchmark.
Model identities were hidden from the judge in an internally designed, LLM-judged evaluation.
A compact model was fine-tuned on one enterprise billing process, then compared with a frontier chatbot and the untrained base model. The documented process was the sole authority.
Across 52 questions and 156 blind judgments, procedural accuracy was 88% for the trained specialist, 38% for the frontier chatbot and 50% for the same untrained base model. Factual accuracy was 97% / 91% / 81%; grounded relevance was 90% / 85% / 80%, in that same order. The questions comprised 16 baseline, 21 generalization and 15 trap prompts. Metric-specific denominators were not supplied, so the percentages should not be converted into counts out of 52. The same-base comparison isolates the improvement from process training more directly.
A requirement the product is being built to satisfy.
The purchase order can contain the visible fields while omitting the contextual account decision. The twin is designed to learn that decision from observed work and held-out historical orders.
It is designed to abstain with reasoning attached and remain human-approved until sustained per-customer accuracy is proven.
Three questions from the evaluation.
Follow the rule, the decision and the difference in each answer.
A biller has created the debit memo before recording the billability verdict. Must the work be reversed?
This process allows charge entry and memo creation before the verdict is recorded.
Adapted from evaluation question Q3. The source process, rather than a general best-practice assumption, determines the answer.
Permitted sequence · no reversal needed
Keep the valid work. Record the required verdict without reversing and recreating the memo.
Incorrectly treats the permitted sequence as a violation.
Also calls it a violation and adds an unsupported reversal procedure.
An invoice row has completion details but no billability note. Does that prove a required check was skipped?
The fixed/maintenance contract path skips that note by design. Other contract paths still require their checks.
Adapted from evaluation question Q7B. This is a specific contract exception, not a rule that missing notes are harmless.
Note skipped by design
Follow the required checks
Identifies the documented contract exception and checks which branch applies.
Recognizes a possible exception, but does not identify the documented contract branch.
Invents a note-overwrite explanation and an unsupported lookup.
The verbal quote is $450, while the signed purchase order says $380. What must happen before $450 can be invoiced?
Verify the disputed rate against the purchase order, quote the corrected amount, wait for customer approval, and record the awaiting-approval state.
Adapted from evaluation question Q10. The amounts are test-prompt values. No recovered revenue or production transaction is claimed.
Written approval required before invoicing $450
Requires written approval tied to the case before invoicing $450; holds the higher amount and records the waiting state.
Gives the authorization principle, but omits the concrete hold-state record.
Adds compliance and system gates that are not in the documented process.
Permitted sequence · no reversal needed
Keep the valid work. Record the required verdict without reversing and recreating the memo.
Incorrectly treats the permitted sequence as a violation.
Also calls it a violation and adds an unsupported reversal procedure.
Internal, blind, LLM-judged evaluation. Responses are summarized and anonymized; this tested process reasoning, not production execution.
See the application and the model’s reasoning together: policy checks, a form filled, an added task and the final confirmation.
The policy coverage is checked before the claim is registered.
A silent 54.5-second highlights cut using synthetic demo data. Selected sequences play at 2× speed; form recovery and the expanded reasoning have time to read. The full application and reasoning panel remain visible together. Pause or use the chapters to inspect a moment. The claim is registered and under review, not approved or paid.
Company-reported reduction in external billed tokens in the specialist engineering comparison. The benchmark retained a frontier fallback for measurable stalls; this is not a zero-external-call claim.
Neurologic’s comparison includes build costs, hardware for execution and ongoing management on our side, against frontier API token costs for the compared approach.
For every 100 external billed tokens in the compared approach, the specialist approach used 28.
Neurologic-reported, workload-specific comparison. Not independently audited. Supporting scope and assumptions available during evaluation.
The two figures measure different things. They are company-reported, workload-specific comparisons, not universal savings guarantees. Request the workload, accounting period, usage assumptions and supporting cost breakdown during your evaluation.
Request comparison details ↗action accuracy
A declared bar, not current performance.
error self-correction
A declared bar, not current performance.
autonomous task completion
A declared bar, not current performance.