Architecture

Portfolio context: VITRINE is an independent job-application sample for Digitec Galaxus and an applied demonstration of the author's AgentEval work. The recruiter-facing intent and non-affiliation boundary are stated in the Digitec Galaxus value proposal; the architecture below remains company-neutral and uses only synthetic data.

For the diagram-first version of this document, open the offline VITRINE visual guide. Retrieval mechanics and the shipped UI are expanded in the retrieval deep dive and operator walkthrough.

VITRINE applies two exploration architectures to realistic, familiar e-commerce recommendation scenarios, then maps those scenarios to independent evaluation contracts. The architecture is designed to support an improvement loop: deterministic checks catch wiring and regression defects, paid LLM and stochastic plans measure behavioral quality, safety probes test the real agent under explicit opt-in, and both lanes leave durable filesystem evidence that can be compared over time.

Compile-time and cost boundaries

The repository has four compile-time layers: subject, evals, app, and tests. The subject owns catalogue facts, persona histories, the ChatClientAgent tool surface, retrieval, guardrails, and the MAF workflow. The eval project references the subject and the published AgentEval package; the app references both to coordinate runs and project their typed results. The subject references neither evaluator nor app.

Inside the eval project, execution cost is also a source boundary. Evals/Offline owns the deterministic suite, production and recommendation leaves, variant benchmarks, self-test, and diagnostic fixtures. Evals/Live owns the named paid plans, canonical live-use-case criteria, subject/judge adapters, typed trace evidence, Wilson/comparison summaries, and sanitized session store. CLI, progress, shared models/reports, and the process entry point remain at project root. This organization preserves the offline implementation as the default lane; it does not replace it with model-backed tests. OfflineSuite is the strict default plan, and OfflineDeterministic is the only remaining EvaluationExecutionProfile value; live execution is represented by the named plan/result contracts rather than a live profile.

Subject execution

The default data path is Catalogue → ConceptEmbeddingSource → HybridRetriever → guardrails → rendered recommendations. Demo02 passes one DiscoveryState message through InterestMapper → Discovery → CoverageReviewer → Ranker → Presenter. CoverageReviewer has two outgoing conditions: exit to Ranker or loop back to Discovery. MAF graph reflection, not a hand-authored diagram, supplies the topology observation used by the atomic contract.

Evaluation ownership

Pass/fail ownership is intentionally one-way. Runtime artifacts supply observations. VitrineEvalCriteria, the admitted-check registries, and the diagnostic control catalogue independently supply expected values, chance-floor derivations, applicability, and thresholds. No runtime Passed property is accepted as evidence for its own gate.

Offline chain

The no-provider lane owns deterministic observations, admitted checks, persisted benchmarks, and mutation diagnostics. It depends on the published AgentEval 0.35.0-beta package rather than a source checkout. Subject outputs are projected into typed EvalInput values; only registered evaluators and native AgentEval score objects own verdicts, census, floors, or comparisons.

The detailed gate inventory, benchmark arms, measurement states, statistics, controls, and exit rules belong to the evaluation protocol. The architecture concern is narrower: observations flow from the subject into the evaluation layer, never the reverse, and every durable result retains its producer and evaluator provenance.

Paid use-case and safety plans

The paid lane is a separate composition boundary, not a mode hidden inside the offline suite. Named plans create fresh subject and judge instances, persist one native benchmark run per arm/repetition when applicable, and write a sanitized session receipt. Eval 06 instead uses AgentEval's real RedTeamRunner and writes no synthetic benchmark facade.

Provider readiness and explicit paid confirmation are checked before execution. Runtime prompts, unrestricted responses, credentials, canaries, and raw provider exceptions do not cross the evidence boundary; bounded previews, typed measurements, allow-listed diagnostics, usage, and physical run references can. Exact plan behavior and classification rules are defined once in the evaluation protocol; the walkthrough explains how to select and read them.

Native AgentEval and measurement boundary

The current architecture rule is structural: VITRINE does not build a second general-purpose statistics, census, floor, benchmark-runner, or evaluation framework beside AgentEval. Domain-specific deterministic evaluators and explicit calibration fixtures remain where they produce the observations required by admitted AgentEval checks; former duplicate framework wrappers were removed or moved upstream.

The package boundary is deliberate: AgentEval's neutral meta lane does not merge the judge-execution, RedTeam, Memory, or benchmark evidence lanes. VITRINE adds no pack.json or parallel pack CLI; the native benchmark/arm/runner/score pipeline and standard output store are the persistence authority.

GateResult.Score and GateResult.Passed are nullable. Native census state distinguishes NotApplicable (the check could not look) from NotMeasured (the instrument did not produce a measurement); neither becomes numeric zero. Aggregate process priority is infrastructure failure 4, then missing measurement 3, then measured mandatory-gate/registered-control failure 1, then pass 0. The matched-quality diagnostic is reported but cannot change the process exit.

Runtime observation and control room

Demo01 emits request/response events at the IChatClient boundary and start/completion events around the real AIFunction await. Demo02 publishes from its actual five-executor workflow. The eval suite publishes typed gate, benchmark-persisted, and healthy/defect/recovery control progress. App adapters normalize these signals into a strict, monotonically sequenced VitrineEvent store. Audience pacing consumes a separate channel, so animation cannot delay authoritative execution.

The graph is runtime-derived. Demo01's composition root prepares one observed 13-function tool set and passes those exact objects to both the graph factory and the ChatClientAgent; tests require the graph's safe tool-set ID and every runtime endpoint/name to agree. Runtime tool objects do not cross the result or artifact boundary—only the safe identity and allow-listed names remain. Demo02 topology comes from AgentEval.MAF.MAFWorkflowAdapter over the exact prepared workflow that executes. The evaluation board maps GateResult, the standard benchmark score/census, ControlResult, and the typed LiveEvalResult without UI-side scoring. Mode-specific selection and interpretation are operator concerns documented in the walkthrough.

Mode-specific previews are available before execution. During execution, operation IDs correlate model/tool requests with their responses and drive terminal node state. Directed edges, executor counts, traversal counts, the flattened BACK route, and Demo02's executor-action shelf are all reconstructed from the same event sequence; paced presentation is a separate consumer and cannot keep the authoritative run active.

Live evaluation lifecycle correlation is equally explicit: subject, check, and trial start cards close on their typed terminal events, while SessionCompleted closes both the persistence start and the session start. A completed live session therefore leaves no stale STARTED/ACTIVE operation.

Each mode has a runtime-derived graph whose node IDs match its typed progress routes. The graph describes execution and persistence; the evaluation board projects evaluator-owned facts and never manufactures a winner or score. Operation IDs close start cards on terminal events so a completed run cannot remain visually active.

Exports and replay project the same sanitized schema-v10 screened outcome used by the UI, including answers, recommendations, verification measurements, ledger, routes, degradations, control tranches, all offline benchmark arms/runs/reference comparisons, named live-evaluation summaries, execution provenance, complete AgentEval independence provenance, and redacted Eval 06 safety census/probe facts and workload ceilings when applicable. Creation sanitizes before hashing; caller-built and deserialized artifacts are structurally rejected before integrity serialization when any string would require secret/URL redaction. HTML and the inspector verify before rendering. They never rerun the subject or evaluator.

Schema 8 added allow-listed Eval 06 probe diagnostics and typed failure stage/code/detail. Schema 9 added explicit evaluation-gate authority plus typed stochastic acceptance and trial-consistency validation. Schema 10 adds typed, sanitized live-workflow provider-stage attempt and recovery evidence. The reader still accepts integrity-valid schema-7 artifacts and reconstructs the generic safe explanation from the stored outcome/error category; it never invents or recovers a raw provider exception that the older receipt did not contain. Schema-7 through schema-9 application artifacts remain readable; new paid-session receipts use live-session schema 1.4, while the public exporter retains the strict historical schema-1.3 read path without inventing the newer evidence.

The exact per-check floor, admission, measurement-state, and ablation inventory is maintained in the repository-native MIGRATION.md ledger.