Microsoft Foundry live-run setup

Know the boundary before spending

VITRINE is offline by default. Having credentials in the environment does not select a live lane. A subject run needs an explicit LiveAzure choice; its CLI equivalent is --live --confirm-paid. A paid evaluation needs a named Eval01–Eval06 plan and the same confirmation. Real-vector queries and embedding-index rebuilds are separate paid actions behind --confirm-paid.

Current authentication contract: VITRINE constructs one Azure OpenAI resource-client shape for chat, workflow, evaluation judge, and optional embeddings. Choose api-key, default-credential for local Microsoft Entra development, or managed-identity for an Azure-hosted workload. Selection is deterministic: a selected identity path never silently falls back to an API key after an authentication failure. A Foundry project endpoint is still not accepted.

A one-case smoke can establish that this checkout reached a named deployment and produced one observed result. It does not establish stochastic reliability, production fitness, business impact, or the behavior of another model/version.

Prepare the Microsoft Foundry resource

1. Use a subscription, region, and Microsoft Foundry/Azure OpenAI resource where you are authorized to deploy models and read the resource endpoint. Availability, quota, model versions, and deployment types vary by region and subscription.
2. Deploy one chat model. The repository default is gpt-5-mini; a compatible alternative must support chat completions, tool/function calling, and the structured-output work used by the workflow and evaluator.
3. Record the deployment name you assigned. VITRINE sends this value to the SDK. A catalogue model ID or model-family label is not interchangeable with a deployment name unless you deliberately made them identical. A judge can share this deployment or use a separately named deployment through AZURE_OPENAI_JUDGE_DEPLOYMENT.
4. From the resource details, copy the Azure OpenAI inference/resource endpoint. A common endpoint form is https://<resource-name>.openai.azure.com/. Do not substitute a Foundry project endpoint containing /api/projects/...; the current application does not create an AIProjectClient.
5. Choose one authentication path. For API-key compatibility, copy a key from that same resource. For local Entra development, sign in through a supported developer tool and give that principal the Cognitive Services OpenAI User role on the resource. For an Azure-hosted app, enable a system- or user-assigned managed identity and give that identity the same role.
6. Only if you intend to use --real-vectors or rebuild the vector asset, also deploy text-embedding-3-small. Normal Demo01, Demo02, and Eval01–Eval06 runs use the authored concept-vector space by default and need no embedding deployment.

Microsoft's upstream references explain how to deploy a Foundry model and why an Azure OpenAI API call uses the deployment name rather than the underlying model name. Follow Microsoft's Foundry Entra configuration and .NET authentication guidance for role assignment and environment-specific credential choice. Those pages own portal and service details; this page owns VITRINE's configuration contract.

Choose authentication and deployments

Set the common target once in the PowerShell session that will launch VITRINE:

$vitrineAzureOpenAiResource = '<resource-name>'
$env:AZURE_OPENAI_ENDPOINT = 'https://' + $vitrineAzureOpenAiResource + '.openai.azure.com/'
$env:AZURE_OPENAI_DEPLOYMENT = '<chat-deployment-name>'

# Optional: uncomment to keep subject and judge on separate deployments.
# $env:AZURE_OPENAI_JUDGE_DEPLOYMENT = '<judge-deployment-name>'

Then choose exactly one authentication mode. For the backward-compatible API-key path:

$env:AZURE_OPENAI_AUTH_MODE = 'api-key'
$env:AZURE_OPENAI_API_KEY = Read-Host 'Azure OpenAI API key' -MaskInput

# Close this PowerShell session when finished so its secret leaves the process environment.

For local Entra development, first authenticate the developer account with Azure CLI, Azure PowerShell, Visual Studio, or VS Code, then select the local credential chain:

az login
$env:AZURE_OPENAI_AUTH_MODE = 'default-credential'
Remove-Item Env:AZURE_OPENAI_API_KEY -ErrorAction SilentlyContinue

For an Azure-hosted production process, select a deterministic managed identity. Omit the client-id variable for the system-assigned identity:

$env:AZURE_OPENAI_AUTH_MODE = 'managed-identity'

# Optional: only for a user-assigned identity; use its client GUID.
$env:AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID = '<user-assigned-client-id>'

An explicit identity mode never falls back to AZURE_OPENAI_API_KEY. Fix the role, identity, endpoint, or network problem reported by Azure; switch modes deliberately if that is the operator's intent.

Only for an embedding-asset rebuild, add:

# The committed real-vector index already names text-embedding-3-small
# for live query embedding.
$env:AZURE_OPENAI_EMBEDDING_DEPLOYMENT = 'text-embedding-3-small'

Do not put the key in source, command history, screenshots, or checked-in logs. Do not create a .env secret for this path—the current application does not load one, and a local secret file adds accidental-commit risk. Do not use setx for the secret: it makes a persistent user-level copy and does not update the current shell. Closing this PowerShell session removes the session-scoped variables.

Separate local readiness from provider connectivity

This zero-call check reports only whether the variables required by the selected mode hold non-empty values. It never prints an endpoint, key, or managed-identity client id:

$authMode = if ([string]::IsNullOrWhiteSpace($env:AZURE_OPENAI_AUTH_MODE)) {
  'api-key'
} else {
  $env:AZURE_OPENAI_AUTH_MODE
}
$required = @('AZURE_OPENAI_ENDPOINT')
if ($authMode -eq 'api-key') { $required += 'AZURE_OPENAI_API_KEY' }
$required | ForEach-Object {
  $configured = -not [string]::IsNullOrWhiteSpace(
    [Environment]::GetEnvironmentVariable($_))
  '{0}: {1}' -f $_, $(if ($configured) { 'SET' } else { 'MISSING' })
}

'Authentication mode: {0}' -f $authMode
'Subject deployment: {0}' -f $(if ($env:AZURE_OPENAI_DEPLOYMENT) {
  $env:AZURE_OPENAI_DEPLOYMENT
} else {
  'gpt-5-mini (code default; a deployment with this exact name must exist)'
})
'Judge deployment: {0}' -f $(if ($env:AZURE_OPENAI_JUDGE_DEPLOYMENT) {
  $env:AZURE_OPENAI_JUDGE_DEPLOYMENT
} else {
  'shares subject deployment'
})

SET is not CONNECTED. VITRINE's typed readiness validates an absolute HTTPS endpoint, a recognized auth mode, that mode's required local values, and resolved deployment names. It does not acquire a token, contact Foundry, validate a key or role, find a deployment, inspect its underlying model, or check quota. Only an explicitly confirmed live call establishes those facts.

Run the smallest useful live smoke

Build first without invoking a provider:

dotnet restore AgentEval.VitrineDemo.slnx --locked-mode
dotnet build AgentEval.VitrineDemo.slnx -c Release --no-restore

Then choose one paid smoke. Concept vectors remain the default in all three examples, so no live embedding call is added.

A “subject trial” is not a promise of one provider request. Agent tool loops, workflow stages, structured-output retries, and judge activity can issue multiple requests. Inspect the plan preview and reported usage; missing usage means not reported, not free.

Run the corresponding control-room smoke

dotnet run --project src/AgentEval.VitrineDemo.App -c Release --no-restore

1. For subject parity, select Agent Demo01, Nadia, and Live Azure. Review the live/paid state shown by the app, explicitly confirm paid execution when prompted, then run. Inspect the tool graph, timeline, and screened outcome.
2. For workflow parity, select Workflow Demo02, Nadia, Live Azure, and a small round cap. Confirm, run, then inspect model-stage, search, executor, and route observations. Marco is the intentional visible-loop case and can consume more model turns.
3. For evaluated parity, select Evals → Eval 01 · Agent → Nadia cross-category, leave repetitions at one, inspect the planned subject/judge workload, turn on Confirm paid execution, and run. Inspect the criterion rows as well as the terminal session state.

The App intentionally shows only Offline, Eval 01 Agent, and Eval 02 Workflow at first. Enable Advanced plans to reveal Eval 03 paired diagnostics, Eval 04/05 stochastic runs, or Eval 06 safety probes; those plans remain directly addressable from the CLI.

The CLI and UI are adapters over the same subject/evaluation cores, but two runs are still two independent provider observations. Similar wiring does not require byte-identical model output.

Add real-vector retrieval only when that is the question

--real-vectors loads the committed 99-product text-embedding-3-small index, makes a live space-identity probe, and embeds new queries live against the deployment named by the asset's model stamp. It can therefore add an embedding call even when the subject arm is otherwise offline:

dotnet run --project src/AgentEval.VitrineDemo -c Release --no-restore -- 1 --live --real-vectors --confirm-paid

The committed stamp is authoritative for query compatibility. If AZURE_OPENAI_EMBEDDING_DEPLOYMENT names something else, that configured name is not used to query this asset. To move to a different embedding deployment/space, rebuild the whole product index deliberately:

dotnet run --project src/AgentEval.VitrineDemo -c Release --no-restore -- --rebuild-embeddings --confirm-paid

The shipped catalogue rebuild makes 99 embedding calls and overwrites a committed data asset. Review the diff, model stamp, dimensions, document-template stamp, and usage report before committing it. It is maintenance—not a smoke-test prerequisite.

The retrieval deep dive explains the two spaces, identity probe, explicit fallback, and why scores cannot be compared across spaces.

Preserve evidence without publishing secrets

• Keep the deployment name, scenario, arm, timestamps, terminal state, measured/not-measured census, bounded outcome/evaluator evidence, and provider usage state.
• For Demo01 CLI, use --report <path>. In the app, save sanitized JSON and HTML after completion. Paid evals also persist below .agenteval/live and write .agenteval/live/live-sessions/<session-id>/outcome.json.
• Do not paste the endpoint, key, raw provider exceptions, unrestricted prompts/responses, Eval06 canary, or system instructions into an issue or public artifact.
• Before publishing, open the exported receipt and search for the resource host, secret fragments, personal paths, and raw prompts. Publish a sanitized copy only after that review.

A checksum detects accidental artifact changes; it is not a signature and does not prove who executed the run. See the evaluation protocol for normative paid-plan persistence and classification.

Troubleshoot by failure class

If a live run fails, retain its non-success receipt. VITRINE is designed not to replace a provider failure with an offline success. The walkthrough's status guide explains how to distinguish execution progress, evidence state, and evaluator verdict.