AI Agents and Assistants
What is an AI agent? A plain-English guide for business owners
What an AI agent is, how it works, where it helps a small business, the risks, and a 7-point check to see if you are ready for one.
Oversight
Agent Quality Control
A quality rubric that scores completed AI agent work and flags the cases that need a person to look at.
Your AI agents finish hundreds of conversations and tasks, and nobody has time to read them all. Agent Quality Control samples that finished work, scores each item against the rubric you define, and flags anything below your bar with the exact line it failed. You get a scorecard with pass rates and trends, while your team decides what to fix.
- Weak AI responses flagged with the rubric line they failed
- Pass rate and trend scorecard for every period
- Quality checked without slowing or blocking live work
$3,400
20–30 working days
Optional maintenance $249/mo
What it is
Agent Quality Control samples finished work from any of your AI agents and scores each item against the quality rubric you define. It covers conversations, drafts and actions across several agents, and flags anything below the bar for a person. It works after the fact, on finished work, not as a live approval gate.
Why businesses buy it
When nobody reviews AI output systematically, quality problems surface as complaints, refunds or a slow loss of trust. Reading transcripts at random is tiring and inconsistent, and it rarely tells you whether things are getting better or worse. A rubric applied to a steady sample gives you a fair, repeatable measure, and shows the specific failures worth fixing first. The written scope sets the delivery channel, the data sources, the report and the alerts before you order, so the result is clear and the build has firm edges.
Problems this solves
Nobody has time to read through finished AI conversations
Agent Quality Control samples completed conversations and actions from up to three read-only sources. It scores each against your rubric, so people only read the ones flagged below the bar.
Quality reviews depend on who checks, not on a clear standard
Your rubric becomes up to twenty-five defined metrics that every sample is scored against. The same standard applies every time, whoever reviews the flagged items afterwards.
Slow quality slips go unseen until a customer complains
One dashboard or recurring report shows scores over time, delivered by email, Slack or a web dashboard. Up to five alerts warn you when results fall below the level you set.
You cannot tell whether a change to an agent made it better
Scoring is checked against three months of your historical data before launch. That gives you a reference point for comparing scores before and after a prompt or process change.
A good fit if
You already have an AI agent live and want an ongoing check on the quality of what it says and does, not a gate that slows it down.
You can define what good looks like: a rubric of the phrases, facts and steps that separate a strong response from a weak one.
You want a sample of completed work scored on a set schedule, with a trend you can show a manager or a client.
You have a reviewer who can spend a fixed block of time on the flagged cases, not open-ended monitoring of everything.
Not the right fit if
You need to stop an action before it goes out, not score it after it happened. See Agent Review Queue, which holds AI outputs for a person before they are finalized.
You need the AI's failed or unresolved runs routed to a person, not a rubric applied to completed work. See Agent Exception Handler.
You want to help a reviewer decide faster on the case already in front of them, not a scoring pass afterwards. See AI Approval Assistant.
You want this pattern for sales conversations specifically. See Human-in-the-Loop Sales Agent, or use our Custom AI enquiry form if nothing here fits.
You only need one Upmark Labs agent's conversations scored. Add the Conversation QA Scoring add-on to that agent instead.
You run a support team and want ticket conversations scored with coaching points for agents. See AI Support QA.
How it works
- Every day or week, we pull a sample of completed AI agent conversations and actions, sized to your volume, not just the ones a person happened to notice.
- Each sampled item is scored against the rubric you set: accuracy against your source material, tone, whether required steps were followed, and any rule that must never be broken.
- Anything below the score threshold you chose is flagged into a review list, with the transcript and the specific rubric line it failed, so a reviewer starts already knowing what to check.
- Patterns across flagged cases are grouped, so a repeated gap, like a policy the agent keeps missing, shows up as one finding instead of twenty separate flags.
- A scorecard goes to the person you name on the schedule you set, showing the pass rate, the trend against the last period, and the cases that need a decision.
- Findings feed back into the agent's prompt and guardrails during the build, and afterwards through monthly maintenance, so the same gap does not keep scoring low.
What you provide within 72 hours
Your portal lists these the moment you order. Delivery starts the working day after everything is in and your access is checked.
- Your business information (company basics, hours and brand voice), shared during intake.
- The workflows you want scored, and the standard each one should be judged against.
- Read-only access to each data source the rubric will sample from.
- Your definitions for the metrics that matter, plus at least three months of historical data to test against.
- The reviewer roles, their response times, and the escalation rules for anything the rubric flags.
- The report recipients and the channel it should reach, whether that is email, Slack or a dashboard.
- Admin access and API keys for every connected system, shared by adding our team as a user or creating a key you control, never by email.
- One decision-maker who can approve the rubric and sign off on the finished build.
- An API key for your own AI model account (for example OpenAI or Google Gemini), created for this build with a spending limit.
The 72-hour rule. Send everything on your checklist, including access, within 72 hours of ordering so your build starts on time.
Timeline
Start
Intake verified and the quality rubric agreed with you.
Weeks 1–2
Data sources connected and the scoring rules built against your rubric.
Week 3
Alerts and the dashboard or recurring report configured.
Weeks 3–4
Scoring tested against three months of your historical data.
Week 4
Your review and two rounds of changes to the rubric or thresholds.
Go-live
Live, recorded handoff session, support starts.
| Option | Delivery time | Price |
|---|---|---|
| Standard | 20–30 working days | $3,400 |
| Express | 15–25 working days | $4,000 |
| Rush | 10–20 working days | $4,600 |
What you receive at handover
- Quality scoring set up on up to three read-only sources of completed agent work
- Your rubric configured as up to twenty-five metrics, with prompts and guardrails
- One dashboard or recurring report, plus up to five alerts, in your chosen channel
- A test report checking scores against three months of your historical data
- An admin guide, the recorded handoff session and 30 days of post-launch support
Expected benefits and how to measure them
- Can give you a steady read on agent quality without reading every conversation.
- Helps reviewers go straight to the weakest work first.
- Makes it easier to judge whether a change to an agent helped.
- Average rubric score
- Track the weekly score in the dashboard or report and compare it with the historical check run at launch.
- Items flagged below the bar
- Count flagged items each week and watch whether the number falls as fixes go in.
- Manual quality-check hours
- Compare hours spent on manual checks in the month before launch with the month after.
Before you start: Write down your rubric and how quality is checked today, including the hours spent. Keep a sample of recent conversations you judged good and poor.
These are things to track, not promised results. Outcomes depend on your data, your team and your market.
Running costs
The build price is one-time. Running Agent Quality Control costs roughly $20 to $150 a month in your own accounts, covering AI model usage for the scoring pass plus hosting and the scheduler that pulls your samples. AI usage during the build and testing is also billed to your accounts, so every provider relationship stays yours, not ours.
Ask about running costsCustom requirements
If your rubric or data sources go past the Scope Card, add a note at checkout. You get a written Change Quote, and that work only starts once you accept it. Your original order runs on its normal timeline either way. Larger or bespoke projects go through the Custom AI request form instead.
Request a custom versionKeep it running
We recommend monthly maintenance: monitoring, fixes, and a monthly allowance of change hours, which covers new rubric lines, extra metrics and refreshed thresholds. It starts after the 30 days of support and can be cancelled any month.
$249/mo (the minimum fee, includes 2 change hours)
Optional. Starts after the post-launch support period. Cancel any month.
How maintenance worksHow this compares
| Do it yourself with a SaaS tool | Hire a custom agency | Upmark Labs | |
|---|---|---|---|
| Price | A QA analytics tool charges a recurring per-seat fee | A custom scoring build is quoted after discovery | Ours is one fixed price, shown on this page before you buy |
| Scope | A generic analytics tool leaves the rubric for you to build | An agency agrees scope during the project | Ours ships with your rubric already configured |
| Delivery | An analytics platform still needs real rubric setup before it is useful | An agency build usually takes several weeks, set per project | Ours has a fixed delivery time shown before you buy, with faster options |
| Findings | Raw dashboards leave you to spot the pattern yourself | An agency deliverable depends on what was agreed | Ours groups repeated gaps into one finding, not twenty flags |
Questions about this product
How much does Agent Quality Control cost?
The build is one fixed price, shown with each delivery speed in the panel on this page. It covers everything on the Scope Card: a scoring rubric built from your quality standard, read-only access to up to three data sources, one dashboard or recurring report, and up to five alerts. AI usage for the scoring pass and hosting are paid in your own accounts.
How is Agent Quality Control different from Agent Review Queue?
Agent Quality Control samples work that has already completed and scores it against your rubric after the fact. Agent Review Queue holds AI outputs before they are finalized, waiting on a person. Choose this build when you want an ongoing quality check on finished work, not a gate that stops anything before it goes out.
What does Agent Quality Control cost to run each month?
Running it afterwards costs roughly $20 to $150 a month in your own accounts, covering AI model usage for the scoring pass plus hosting and the scheduler that pulls your samples. There is no separate license fee on top of that figure. The exact cost depends on your sample size and how often scoring runs.
What access does Agent Quality Control need to my systems?
We need read-only access to each data source the rubric samples from, shared by adding our team as a user or creating a key you control, never by email. Your definitions for the key metrics and at least three months of historical data let us build scoring that reflects your actual standard, not a generic one.
Can we change the rubric or add metrics later?
Yes. Adjusting an existing rubric line, or adding a new metric within the twenty-five in scope, usually fits inside the monthly change hours that come with maintenance. Expanding past twenty-five metrics, or adding a second delivery channel for reports, goes through a Change Quote agreed before that work starts.
Should I buy this, AI Support QA or the Conversation QA Scoring add-on?
Choose by what you need scored. Agent Quality Control scores output from several AI agents against one standard, whoever built them. The Conversation QA Scoring add-on scores the conversations of a single Upmark Labs agent. AI Support QA is for support teams, scoring ticket conversations and highlighting coaching points.
Does Agent Quality Control ever pause or block the AI agent it is scoring?
No. It only reviews work that has already completed and reports on it; it does not intervene in a live run. If you need something to actually pause or correct an agent mid-action, that is a different build, such as the already-published AI Supervisor Agent, which watches agents in real time.
Guides on this topic
All guides Know your AI agent's quality without watching every conversation.
Order Agent Quality Control and completed AI work gets scored against your rubric, with the weak cases flagged for review, live on a fixed delivery date.
Works well with
Oversight
Agent Exception Handler
Catches technical failures in AI runs, such as errors, timeouts and unusual runs, and routes them for resolution.