Promptfoo Parity Matrix

AgentV authored YAML should be Promptfoo-compatible by default for ordinary eval surfaces: prompt matrices, test rows, vars, default test data, assertions, and provider matrices all use the same broad shape. The intended authored-config differences are environment, AgentV refs, and built-in AgentV providers. Export fills those gaps for supported AgentV-native pieces so Promptfoo can validate or load the resulting config without learning AgentV semantics.

Promptfoo-shaped providers entries can be strings such as openai:gpt-4.1-mini, complete package provider strings such as package:@agentv/promptfoo-providers:CodexCliProvider or package:@agentv/promptfoo-providers/codex-cli:Provider, provider option objects with id, label, config, env, prompts, transform, delay, and inputs, or provider maps such as { "openai:gpt-4": { label, config } }. Promptfoo-native provider IDs such as openai:responses:gpt-5.4, openai:codex-sdk, and openai:codex-app-server remain directly Promptfoo-readable.

For AgentV-only provider IDs such as agentv:codex-cli, use the Promptfoo export path instead of running the AgentV YAML directly with Promptfoo. The exporter lowers supported AgentV-only IDs to Promptfoo-readable provider references such as file://.agentv/generated/promptfoo/providers/codex-cli-provider.ts and exports supported host/filesystem environment recipes to a generated Promptfoo extension such as file://.agentv/generated/promptfoo/extensions/host-environment.ts:beforeAll. The exporter passes the resolved isolated workdir through provider config, default-test vars, and metadata. Docker environment export is intentionally unsupported initially and fails with a diagnostic instead of degrading isolation, image/context, mounts, services, resources, secrets, or provenance.

Use this matrix when translating a Promptfoo-style normal eval into AgentV YAML. It documents which surfaces align directly, which AgentV surfaces are cleaner greenfield extensions, and which Promptfoo surfaces are deferred until AgentV implements equivalent semantics directly.

Decision Terms

Decision	Meaning
Align with Promptfoo	AgentV accepts the same concept, with snake_case where the field crosses the YAML boundary.
Keep AgentV divergence	AgentV intentionally uses a different shape because it is clearer for repo-native agent evals.
Keep AgentV extension	AgentV adds a capability that does not try to be Promptfoo-compatible.
Removed/rejected surface	AgentV rejects this authored surface and points to the current supported field.
Defer/future-scope	AgentV does not accept the Promptfoo surface yet. Use an AgentV primitive or wait for direct implementation.

Authored Config Matrix

Surface	Promptfoo shape	AgentV shape	Decision	Notes
TypeScript config file	promptfooconfig.ts default export typed as Promptfoo UnifiedConfig.	Explicit *.eval.ts or *.eval.mts default export typed as EvalConfig from agentv.	Keep AgentV divergence	AgentV does not accept promptfooconfig.ts by default because the config semantics intentionally diverge beyond provider declarations. Directory discovery includes explicit *.eval.ts and *.eval.mts config files, but not arbitrary helper .ts files.
Prompt matrix	Top-level prompts rendered with each test’s vars.	Top-level prompts rendered with tests[].vars and default_test.vars.	Align with Promptfoo	This is the canonical Promptfoo-compatible input shape in AgentV. Prompt entries can be inline strings, chat arrays, files, or generated prompt functions.
Test rows	tests can be inline rows or a case-file reference; rows carry vars, assert, metadata, prompt/provider filters, and expected data through vars or assertion values.	tests can be inline rows or a raw-case path; rows carry vars, assert, metadata, optional environment overrides, and run overrides.	Align with Promptfoo	AgentV uses field-local file refs such as tests: file://..., prompts: file://..., and default_test: file://...; coding-agent testbeds use environment: file://.... There is no separate imports table.
Variables	tests[].vars plus defaultTest.vars; prompt templates can reference top-level var names.	tests[].vars plus default_test.vars; templates can use {{ name }} or {{ vars.name }}.	Align with Promptfoo	Per-test vars override default vars by key.
Default test	defaultTest, inline object or file:// reference.	default_test, inline object or file:// / ref:// reference.	Align with Promptfoo	AgentV uses snake_case for YAML. Shared prompt matrix defaults belong in default_test.vars.
Reference answers	Promptfoo stores reference data in assertion values or arbitrary vars such as vars.expected_output; llm-rubric consumes it when value includes {{ expected_output }}.	Store reference answers in tests[].vars.expected_output or default_test.vars.expected_output, then consume them explicitly with assert or default_test.assert, usually type: llm-rubric and value: "Matches the reference answer: {{ expected_output }}".	Align with Promptfoo	A var named expected_output is just data. AgentV does not create hidden grading behavior for that name. Authored direct expected_output at eval, default_test, or test level is rejected in current YAML.
Output transform	defaultTest.options.transform, tests[].options.transform, and assertion-level transform.	default_test.options.transform, tests[].options.transform, and assertion-level transform.	Align with Promptfoo	Use transform to shape provider output before grading, including file-output conversions such as .xlsx to text. tests[].options.transform overrides the inherited default transform; assertion-level transform is scoped to one grader.
Deprecated postprocess	Older Promptfoo configs may still mention postprocess.	Rejected.	Removed/rejected surface	Use transform. AgentV intentionally does not adopt Promptfoo’s deprecated postprocess spelling.
Evaluate options	evaluateOptions for runtime controls.	evaluate_options for runtime controls.	Align with Promptfoo	AgentV uses evaluate_options.repeat, evaluate_options.budget_usd, and evaluate_options.max_concurrency.
Authored concurrency	Common Promptfoo usage includes runtime options such as maxConcurrency.	evaluate_options.max_concurrency.	Keep AgentV divergence	Do not author execution.max_concurrency or top-level workers in eval YAML. CLI --workers remains an operator override.
Provider selection	Promptfoo normal evals use providers; targets can alias providers in unified config.	Use top-level providers for one or more systems under test.	Align with Promptfoo	Old AgentV target/targets authoring is hard-rejected. Use CLI --provider/--providers for runtime selection.
Provider declarations	Provider entries can be strings, complete package provider strings such as package:@agentv/promptfoo-providers:CodexCliProvider or package:@agentv/promptfoo-providers/codex-cli:Provider, provider option objects with id, label, config, env, prompts, transform, delay, and inputs, or provider maps such as { "openai:gpt-4": { label, config } }.	AgentV accepts the same provider declaration layer; id is the backend/spec and label is the stable AgentV selection and result identity. AgentV also accepts providers[].environment as an inline or file:// provider-local overlay on the authored environment recipe.	Align with Promptfoo plus AgentV extension	Package provider strings must include the exported class/function segment after the final colon. Promptfoo-native provider IDs remain directly Promptfoo-readable. providers[].environment is AgentV-only environment sugar, not a new canonical testbed contract; Promptfoo export must lower supported host setup through generated extensions or omit unsupported overlays clearly.
Provider-prompt mapping	providerPromptMap maps provider identities to prompt subsets.	Rejected. Use explicit AgentV composition: separate eval suites/files for provider-specific prompt subsets, top-level prompts plus tests/default_test.vars, providers or CLI --provider, and tags/run metadata for grouping.	Removed/rejected surface	Do not author providerPromptMap or provider_prompt_map.
Direct authored input	Promptfoo prompt authoring normally goes through prompts plus vars.	Top-level input and inline tests[].input are removed from normal authored eval YAML. External raw-case imports may still carry internal input rows for compatibility.	Removed AgentV extension	Author prompt text, chat/system/user messages, and file-backed prompt content as prompts; put row data in tests[].vars and shared defaults in default_test.vars.
Authored preprocessors	Not Promptfoo’s canonical output-shaping surface.	Rejected in current authored YAML.	Removed/rejected surface	Use transform at default_test.options, tests[].options, or the assertion that needs the shaped output. Historical versioned docs may still show old preprocessor examples.
Suite assertions	assert entries can be strings or typed assertion objects.	assert entries can be strings, typed assertion objects, script graders, or AgentV extension graders.	Align with Promptfoo	Plain strings become semantic rubric checks. Use assert, not assertions, in current authored eval YAML.
Assertion grouping	type: assert-set with child assert entries, optional config, metric, weight, and threshold.	type: assert-set with child assert, optional config, metric names, weights, and parent threshold.	Align with Promptfoo	Parent config is inherited by child assertions; child config keys override shared parent keys. Without threshold, pass/fail follows nonzero-weight child assertions. With threshold, the weighted aggregate score determines pass/fail. type: composite is rejected; use assert-set.
Deterministic and rubric assertion vocabulary	Common Promptfoo types include contains, icontains, contains-any, contains-all, starts-with, regex, is-json, equals, latency, cost, javascript, python, webhook, similar, llm-rubric, and agent-rubric.	AgentV accepts the implemented overlap, including contains, icontains, contains-any, contains-all, starts-with, regex, is-json, equals, latency, cost, javascript, python, webhook, similar, llm-rubric, and agent-rubric.	Align with Promptfoo	agent-rubric runs through AgentV-native agent grader providers and writes a structured verdict file before fallback text parsing. Unsupported Promptfoo assertion names error instead of silently becoming custom assertion names.
Custom assertion terminology	Promptfoo calls normal eval custom logic assertions, with fixed code assertion types such as javascript, python, ruby, and webhook.	defineAssertion() files in .agentv/assertions/ become reusable assertion type names.	Keep AgentV extension	AgentV keeps assertion terminology and extends discovery to arbitrary assertion type names such as has-citation.
Script/custom grader terminology	Promptfoo custom code assertions are still assertion types.	defineScriptGrader() powers command-backed graders referenced with type: script and command:.	Keep AgentV divergence	Use script grader wording only for command-backed or LLM-backed scoring components that need explicit score and assertion-result control.
Skill assertions	Promptfoo includes skill-used for checking whether an agent invoked a named skill.	type: skill-used and type: not-skill-used with value: <skill> or a matcher object.	Align with Promptfoo	AgentV evaluates these against normalized tool-call and skill-use trace data.
Trajectory assertions	Promptfoo includes trajectory:tool-used, trajectory:tool-sequence, trajectory:tool-args-match, trajectory:step-count, and trajectory:goal-success.	AgentV accepts the same assertion names over AgentV-normalized traces.	Align with Promptfoo	Use these for tool presence, order, argument checks, step budgets, and LLM-judged goal success.
Other Promptfoo trace assertions	Promptfoo also includes tool-call-f1, trace-span-count, trace-span-duration, and trace-error-spans.	Not accepted yet.	Defer/future-scope	Use script assertions or execution-metrics when current AgentV primitives cover the requirement.
Coding-agent testbeds	Promptfoo normal evals do not define a typed coding-agent testbed primitive.	environment, usually inline or environment: file://..., plus distinct top-level env and lifecycle extensions.	Keep AgentV extension	environment is an AgentV extension informed by Margin local-agent ergonomics and Harbor/Terminal-Bench Docker substrate evidence. Export supports the host/filesystem subset by generating Promptfoo setup extensions and passing the resolved workdir through config, vars, and metadata. Docker export is not supported yet and fails clearly instead of degrading isolation or provenance.
Run artifacts and inspection	Promptfoo owns its own result viewer and output formats.	AgentV writes .agentv/results/<run_id>/ bundles with summary.json, .internal/index.jsonl, sidecars, and local Dashboard support.	Keep AgentV extension	AgentV-owned bundles are the source of truth for compare, Dashboard, CI, and adapters. Phoenix is link-out correlation only through safe external trace metadata.
Grading result artifacts	Promptfoo GradingResult uses aggregate pass, score, reason, recursive componentResults, optional assertion, optional namedScores, and optional metadata.	AgentV persists the same concepts as pass, score, reason, recursive component_results, optional assertion, optional named_scores, and optional metadata.	Align with Promptfoo	AgentV keeps split filesystem run bundles and snake_case persisted fields. Promptfoo import/export adapters may translate componentResults/namedScores to and from AgentV component_results/named_scores.
Compare command	Promptfoo has its own result comparison surfaces.	agentv results compare <baseline-index.jsonl> <candidate-index.jsonl>.	Keep AgentV extension	Compare consumes completed AgentV run indexes such as .agentv/results/<run_id>/.internal/index.jsonl.
CLI runtime filters	Promptfoo exposes filters such as prompt/provider/test subset flags.	AgentV supports its current CLI filters and selection fields; full Promptfoo runtime-filter parity is future work.	Defer/future-scope	Prefer current AgentV CLI flags and authored selection fields until runtime-filter parity lands.
Wire-format casing	Promptfoo config uses camelCase fields such as defaultTest and evaluateOptions.	AgentV YAML, JSONL, artifacts, and CLI JSON use snake_case; internal TypeScript uses camelCase.	Keep AgentV divergence	Translate only at process boundaries. New public wire fields should be snake_case.
Hard-rejected stale AgentV fields	Not applicable to Promptfoo.	Removed AgentV-era fields such as top-level target, targets, execution, execution.target, execution.targets, top-level budget_usd, top-level repeat/runs, and composite are rejected.	Keep AgentV divergence	Use top-level providers, evaluate_options, evaluate_options.repeat, and assert-set. Migration guidance lives in the eval migration skill reference.

Canonical Prompt-Compatible Shape

description: Release-note summarization
providers:
  - local-mini

prompts:
  - id: direct
    label: Direct
    prompt: "Summarize {{ topic }} for {{ audience }}."

default_test:
  vars:
    audience: engineers
  assert:
    - type: llm-rubric
      value: "Matches the reference answer: {{ expected_output }}"

evaluate_options:
  max_concurrency: 2

tests:
  - id: release-notes
    vars:
      topic: the July release notes
      expected_output: concise release-note summary
    assert:
      - Identifies the most important change
      - type: assert-set
        metric: release_gate
        threshold: 0.8
        assert:
          - type: contains
            value: July
          - type: llm-rubric
            value: The answer is concise and accurate.

Canonical AgentV Extension Shape

description: Repo-native direct task suite
environment: file://.agentv/environments/local-python.yaml

providers:
  - id: codex-cli
    label: codex-host
    runtime: host
  - id: claude-cli
    label: claude-host
    runtime: host

prompts:
  - - role: user
      content:
        - type: file
          value: ./instructions.md
        - type: text
          value: "{{ task }}"

tests:
  - id: refund-policy
    vars:
      task: Update the refund policy handler.
      expected_output: The handler supports the damaged-item exception.
    assert:
      - type: llm-rubric
        value: "Matches the reference answer: {{ expected_output }}"
      - type: trajectory:tool-used
        value:
          name: shell
          min: 1
      - type: script
        command: [bun, run, graders/check-refund-policy.ts]