What we tested, what improved and what remains to be proven. Each result includes its scope and limitations.
held-out circuits won
Specialist-led system with frontier fallback. Equal iteration budgets. Wins rank targets met first, then external tokens. Internal evaluation.
A specialist-led system won on 9 of 10 held-out circuits against a frontier baseline, with equal iteration budgets.
Internal evaluation · specialist with frontier fallbackInside the evaluationSpecialist with frontier fallback. Targets met determine wins; external tokens break ties. Internal evaluation.
Before specialist training, the base model achieved 50% procedural accuracy on the enterprise process evaluation.
Same model · enterprise process knowledgeEnterprise billing process. Internally designed, LLM-judged evaluation. Not insurance execution accuracy. Internal evaluation.
Procedural accuracy reached 88% after training: an improvement of 38 percentage points over the 50% baseline.
52 questions · 156 blind LLM judgmentsRead the methodEnterprise billing process. Internally designed, LLM-judged evaluation. Not insurance execution accuracy. Internal evaluation.
Specialist with frontier fallback. Targets met determine wins; external tokens break ties. Internal evaluation.
Each result explains the task, comparison, judging method and limits. Internal evaluations and recorded demos show different kinds of evidence; neither establishes production performance.
Run with a named partner engagement. Internal and not independently audited.
Circuit design involves competing targets. The model learns from expert attempts, then tests its proposals against design rules and a real circuit simulator.
The August 2026 internal evaluation tested ten circuits excluded from training: six op-amps and four LDO regulators. Both systems had equal iteration budgets. The specialist used GPT-5.5 when stalled; the comparison used GPT-5.5 throughout. Wins ranked targets met first, then external tokens as a tiebreaker. Four circuits needed no external calls.
Unscripted probes against a live model.
The questions test conflicting records, missing fields and linked claims. The walkthrough separates one recorded answer from rules in the workflow documentation.
The transcript includes assumptions and imperfect answers. Answers need checking against the process; the probe count is not a success rate.
Explore the source-backed questionsBehavior shown in a recorded application session. This is not a production deployment or an accuracy benchmark.
The run reads a claim form, checks coverage dates, registers the claim, adds a client note and confirms the outcome by email.
The silent film shows the app and recorded reasoning together. Selected sequences are accelerated. This demonstrates app behavior, not a completion rate.
Model identities were hidden from the judge in an internally designed, LLM-judged evaluation.
A small model learned one billing process, then answered the same questions as a frontier chatbot and its untrained counterpart. Answers were judged against that process.
Across 52 questions and 156 blind judgments, process accuracy was 88% for the trained specialist, 38% for the frontier chatbot and 50% before training. Factual accuracy was 97% / 91% / 81%; grounded relevance was 90% / 85% / 80%, in the same order. The test included 16 baseline, 21 generalization and 15 trap questions. Metric-specific denominators were not supplied: these percentages cannot be converted into counts out of 52. Comparing the same model before and after training best isolates the training effect.
A requirement the product is being built to satisfy.
An order can list every item yet leave the right account unclear. The specialist is designed to learn from expert work and be tested on unseen historical orders.
When unsure, it is designed to explain why and ask a person. Human approval remains until accuracy is proven for that customer.
Three questions from the evaluation.
Follow the rule, the decision and the difference in each answer.
A biller has created the debit memo before recording the billability verdict. Must the work be reversed?
This process allows charge entry and memo creation before the verdict is recorded.
Adapted from evaluation question Q3. The source process, rather than a general best-practice assumption, determines the answer.
Specialist: Keep the valid work. Record the required verdict without reversing and recreating the memo.
Frontier chatbot: Incorrectly treats the permitted sequence as a violation.
Untrained base model: Also calls it a violation and adds an unsupported reversal procedure.
Permitted sequence · no reversal needed
Keep the valid work. Record the required verdict without reversing and recreating the memo.
Incorrectly treats the permitted sequence as a violation.
Also calls it a violation and adds an unsupported reversal procedure.
An invoice row has completion details but no billability note. Does that prove a required check was skipped?
The fixed/maintenance contract path skips that note by design. Other contract paths still require their checks.
Adapted from evaluation question Q7B. This is a specific contract exception, not a rule that missing notes are harmless.
Specialist: Identifies the documented contract exception and checks which branch applies.
Frontier chatbot: Recognizes a possible exception, but does not identify the documented contract branch.
Untrained base model: Invents a note-overwrite explanation and an unsupported lookup.
Note skipped by design
Follow the required checks
Identifies the documented contract exception and checks which branch applies.
Recognizes a possible exception, but does not identify the documented contract branch.
Invents a note-overwrite explanation and an unsupported lookup.
The verbal quote is $450, while the signed purchase order says $380. What must happen before $450 can be invoiced?
Verify the disputed rate against the purchase order, quote the corrected amount, wait for customer approval, and record the awaiting-approval state.
Adapted from evaluation question Q10. The amounts are test-prompt values. No recovered revenue or production transaction is claimed.
Specialist: Requires written approval tied to the case before invoicing $450; holds the higher amount and records the waiting state.
Frontier chatbot: Gives the authorization principle, but omits the concrete hold-state record.
Untrained base model: Adds compliance and system gates that are not in the documented process.
Written approval required before invoicing $450
Requires written approval tied to the case before invoicing $450; holds the higher amount and records the waiting state.
Gives the authorization principle, but omits the concrete hold-state record.
Adds compliance and system gates that are not in the documented process.
Permitted sequence · no reversal needed
Keep the valid work. Record the required verdict without reversing and recreating the memo.
Incorrectly treats the permitted sequence as a violation.
Also calls it a violation and adds an unsupported reversal procedure.
Internal, blind, LLM-judged evaluation. Responses are summarized and anonymized; this tested process reasoning, not production execution.
See the application and the model’s reasoning together: policy checks, a form filled, an added task and the final confirmation.
The policy coverage is checked before the claim is registered.
A silent 54.5-second highlights cut using synthetic demo data. Selected sequences play at 2× speed; form recovery and the expanded reasoning have time to read. The full application and reasoning panel remain visible together. Pause or use the chapters to inspect a moment. The claim is registered and under review, not approved or paid.
Company-reported reduction in external billed tokens in the specialist engineering comparison. The benchmark retained a frontier fallback for measurable stalls; this is not a zero-external-call claim.
Neurologic’s comparison includes build costs, hardware for execution and ongoing management on our side, against frontier API token costs for the compared approach.
For every 100 external billed tokens in the compared approach, the specialist approach used 28.
Neurologic-reported, workload-specific comparison. Not independently audited. Supporting scope and assumptions available during evaluation.
The two figures measure different things. They are company-reported, workload-specific comparisons, not universal savings guarantees. Request the workload, accounting period, usage assumptions and supporting cost breakdown during your evaluation.
Request comparison details ↗action accuracy
A declared bar, not current performance.
error self-correction
A declared bar, not current performance.
autonomous task completion
A declared bar, not current performance.
