Observability & Evaluation
AI Agent Monitoring Platform
Monitors agent health, failures and drift in production.
Observability & Evaluation
AI Evaluation System
Your AI feature's outputs are tested against a defined set of benchmarks and test cases, with a clear pass or fail report.
It is hard to say whether an AI feature is good enough when the only test is someone reading a few outputs. AI Evaluation System builds a defined set of test cases and quality benchmarks for one feature, then scores your model's outputs against them for a clear pass or fail report. You can rerun it after changes. You decide what counts as a pass.
- Clear pass or fail report for one AI feature
- Same test cases rerun after every prompt or model change
- Side-by-side comparison of models or providers on your own tests
$4,500
20–30 working days
Optional maintenance $249/mo
What it is
AI Evaluation System builds a defined set of test cases and quality benchmarks for one AI feature. It runs your model's outputs against them for a pass or fail report. It is the middle step: bigger than the Evaluation Test Suite add-on, and narrower than the ongoing AI Model Evaluation Platform.
Why businesses buy it
Without a proper test set, every change to a prompt or model is a gamble. Quality slips quietly, arguments about which model is better go nowhere, and launches rest on gut feel. After this build, one AI feature has a defined set of test cases built from your real examples, and every output is scored against them, so decisions rest on evidence. You can rerun the same tests whenever something changes. The Scope Card sets the feature, benchmark sets and systems in writing before you order, so the scope is fixed.
Problems this solves
Nobody can say whether an AI feature is good enough to ship
The Evaluation System builds a defined set of test cases and quality benchmarks for one AI feature. Your model's outputs run against them for a clear pass or fail report.
Quality is judged by a few people trying it informally
Test cases are written down and agreed, so everyone judges against the same standard. Your policies and examples, up to twenty documents, shape what counts as a pass.
A change looks fine in a demo but breaks real cases
The test set covers agreed scenarios, including the awkward ones. Running it before a change goes live shows which cases now fail, in a written findings report.
A good fit if
You have one AI feature whose output quality needs a defined set of test cases before or after a launch.
You want a benchmark report you can rerun after a prompt or model change, without an ongoing evaluation system running continuously.
You need a clear pass or fail signal against agreed quality criteria, not a live dashboard tracking drift over time.
You are choosing between models or prompt versions for one feature and need a structured comparison, not continuous monitoring.
Not the right fit if
We built your agent and you want an automated test set that runs before each change. See Evaluation Test Suite, an add-on for our own products.
You need continuous evaluation running across multiple models and versions over time, not a bounded test-case build. See AI Model Evaluation Platform, our broader ongoing evaluation infrastructure.
You need a live dashboard of latency, accuracy and error rate rather than a benchmark report. See AI Model Performance Dashboard.
You need to trace an agent's internal reasoning, not test its output quality. See AI Agent Observability.
You need to track usage volume rather than output quality. See AI Usage Monitoring Platform.
How it works
- We work with you to define the test cases and quality benchmarks that matter for your specific AI feature, drawn from real examples you provide.
- The evaluation is built vendor-neutral, so it runs your test cases against outputs from OpenAI, Google Gemini, AWS Bedrock or Azure OpenAI, whichever your feature uses.
- Each test case is scored against the agreed criteria, producing a clear pass or fail result rather than a vague quality impression.
- Results are grouped into a benchmark report, showing where outputs consistently pass, where they fail, and which test cases are borderline.
- The evaluation set is built to be rerun later, so a prompt change or a model swap can be checked against the same benchmarks again.
- We deliver the test report from the initial run, an admin guide covering how to rerun the evaluation, and a recorded handoff session.
What you provide within 72 hours
Your portal lists these the moment you order. Delivery starts the working day after everything is in and your access is checked.
- Your business information (company basics, hours and brand voice), shared during intake, or confirmation we can reuse what you gave us on an earlier order.
- The AI providers and API accounts the feature being evaluated runs on.
- Cloud or hosting access for wherever the evaluation tooling will run.
- A technical owner who can define what counts as a passing output.
- Written authorization for any testing against your live or sample data.
- Real examples of outputs, both good and bad, to build test cases from.
- Admin access and API keys for every connected system, shared by adding our team as a user or creating a key you control, never by email.
- One decision-maker who can approve the final benchmark criteria.
The 72-hour rule. Send everything on your checklist, including access, within 72 hours of ordering so your build starts on time.
Timeline
Start
business information confirmed and the feature to be evaluated agreed.
Weeks 1–2
Test cases and quality benchmarks defined and built against your example outputs.
Week 3
Evaluation connected to your chosen providers and the scoring criteria configured.
Weeks 3–4
The evaluation is run and the test report produced.
Week 4
Your review and two rounds of changes to test cases or benchmarks.
Go-live
Live, recorded handoff session, and the 30 days of support begins.
| Option | Delivery time | Price |
|---|---|---|
| Standard | 20–30 working days | $4,500 |
| Express | 15–25 working days | $5,300 |
| Rush | 10–20 working days | $6,100 |
What you receive at handover
- AI Evaluation System configured for one AI feature, with a defined set of test cases and quality benchmarks
- A pass or fail report of your model's outputs against those benchmarks
- Workflow, prompt and guardrail configuration and connections to the agreed systems
- A test report with a control test log or findings report
- An admin guide and a recorded 60-minute handoff session
- 30 days of post-launch support
Expected benefits and how to measure them
- Can give you a clear, shared view of whether a feature is ready.
- Helps catch quality problems before customers see them.
- Makes it easier to compare a change with what came before.
- Pass rate on the test set
- Record it from the first report and rerun the set before each significant change.
- Quality issues reported after release
- Count customer or staff reports about the feature, before and after evaluation was in place.
- Failed cases by type
- Group failures from each report to see which kinds of input cause trouble.
Before you start: Collect examples of good and bad outputs from the feature today. Note how quality is currently checked before release.
These are things to track, not promised results. Outcomes depend on your data, your team and your market.
Running costs
The build price is a one-time cost. Rerunning the evaluation afterwards costs roughly $20 to $300 a month in your own accounts, mostly hosting and the tooling, such as Langfuse, that stores the test cases and results.
AI model usage for each evaluation run is also billed directly to your own provider accounts, so ownership of every provider relationship stays with your team.
Ask about running costsCustom requirements
If you need test cases across more than three connected systems or AI features, or more than five benchmark sets, add a note at checkout. You get a written Change Quote, and your original order keeps its normal timeline. Larger or bespoke projects use the Custom AI request form instead.
Request a custom versionKeep it running
Monthly maintenance covers monitoring, fixes and change hours for rerunning the evaluation or adding a test case as your feature evolves. It starts after the 30 days of support and can be cancelled any month.
$249/mo (the minimum fee, includes 2 change hours)
Optional. Starts after the post-launch support period. Cancel any month.
How maintenance worksHow this compares
| Do it yourself with a SaaS tool | Hire a custom agency | Upmark Labs | |
|---|---|---|---|
| Price | An eval platform charges an ongoing subscription for continuous scoring | A custom eval build is quoted after discovery | Ours is one fixed price, shown on this page before you buy |
| Scope | An eval platform leaves you to define test cases yourself | An agency agrees scope during the project | Ours is written to your feature and test cases before you order |
| Delivery | An eval platform still needs real test-case work before it is useful | An agency build usually takes several weeks, set per project | Ours has a fixed delivery time shown before you buy, with faster options |
| Nature | An eval platform runs continuously by default, whether you need that or not | An agency deliverable depends on what was agreed | Ours is a bounded benchmark report you can rerun on demand |
Questions about this product
How much does AI Evaluation System cost?
The fixed price for each delivery speed is shown in the price and delivery panel on this page before you order. It covers everything on the Scope Card: test cases and up to five benchmark sets for one AI feature, scoring across up to three connected systems, the first pass or fail report and an admin guide with a recorded handoff session.
What does AI Evaluation System cost to run each month?
Rerunning the evaluation afterwards costs roughly $20 to $300 a month in your own accounts, mostly hosting and the tooling that stores the test cases and results. There is no separate platform license beyond that figure. AI model usage for each rerun bills separately, directly to your own provider accounts.
How is AI Evaluation System different from AI Model Evaluation Platform?
AI Evaluation System is a focused, bounded build: a defined set of test cases and benchmarks for one feature, scored once and rerunnable on demand. AI Model Evaluation Platform is broader and ongoing, running continuous evaluation across multiple models and versions with regression alerts, for teams that need evaluation running constantly rather than periodically.
Which evaluation option fits my team?
Start from what you need to test. Evaluation Test Suite is an add-on that checks an agent we built before each change. AI Evaluation System builds test cases and benchmarks for one AI feature, whoever built it. AI Model Evaluation Platform runs ongoing evaluation across several models and prompts, with regression alerts.
Can AI Evaluation System compare more than one AI provider or model?
Yes. The evaluation is built vendor-neutral, so the same test cases can be run against outputs from OpenAI, Google Gemini, AWS Bedrock or Azure OpenAI. That is useful when you are choosing between models or providers for one feature and need a structured, apples-to-apples comparison rather than a subjective read.
What do I need to provide to build the test cases?
Real examples of outputs, both good and bad, from your AI feature, plus a technical owner who can define what counts as a passing result. We also need admin access and API keys for every connected system, shared by adding our team as a user or creating a key you control, never by email, and written authorization to test against your data.
Can we rerun the evaluation or add test cases later?
Yes. Rerunning the existing evaluation, or adding a test case to an existing benchmark set, usually fits inside the monthly change hours included with maintenance. Building an entirely new benchmark set, or evaluating a fourth or fifth system beyond the three in scope, goes through a Change Quote agreed before that work starts.
Can we cancel AI Evaluation System maintenance later?
Yes. Maintenance runs month to month with no lock-in, so you can cancel any month once the 30 days of included support has ended. Unused hours do not roll over. Bigger work, such as an entirely new benchmark set, goes through a Change Quote.
Know your AI feature passes before it ships.
Order AI Evaluation System and a rerunnable benchmark report for your feature is live on a fixed delivery date.
Works well with
Observability & Evaluation
AI Model Evaluation Platform
Runs ongoing evaluations across models and prompts with regression alerts.
Observability & Evaluation
AI Model Performance Dashboard
Compares model quality, latency and cost across workflows.