Before Buying an AI Tool: Build a Benchmark Decision Sheet

A vendor score can start a conversation, but it cannot approve a purchase. Your team needs a record of which claim matters to its work, which conditions produced that claim, and which unanswered questions require a pilot. This worksheet is Romfix's proposed decision aid, not a new benchmark or a report of measured results.

Write the decision before collecting scores

Name the job you want the tool to perform. "Help with code" is too broad; "draft a patch for a repository issue that a maintainer will review" gives you an observable task. Record who reviews the result, what counts as a serious failure, and whether the tool may only suggest changes or also execute them.

Create one row for each consequential claim. Keep the original wording and a link to the underlying evaluation so that a marketing summary cannot quietly replace the test definition.

Worksheet field	What the team should record
Intended job	A concrete input, expected output and reviewer
Evidence	Evaluation version, task set and scoring rule
Executed system	Model version, instructions, tools and retrieval
Resource envelope	Attempts, tokens, runtime and allowed spending
Decision owner	Person responsible for the pilot outcome
Missing evidence	Question, responsible reviewer and next action

Apply a comparability gate

Do not put two scores in one ranking until you can explain whether their conditions match. A system allowed several attempts is being tested under a different operating budget from a single-attempt assistant. The HumanEval research illustrates why repeated sampling and functional evaluation must be described together.

Mark a comparison "not yet comparable" when the attempt budget, task definition or tool setup is missing. That status is a request for evidence, not a claim that either product is inferior. A repository repair result also leaves questions about maintainability, review burden and local setup that a function-completion score cannot answer.

Design a pilot around failure costs

Select work your team actually encounters and divide it into meaningful slices, such as routine edits, ambiguous requests and dependency failures. Keep permission checks separate from answer quality. A helpful patch is still unacceptable if obtaining it required an unauthorized action.

For each trial, record the accepted output, reviewer corrections, elapsed effort, retries and reason for rejection. Choose the acceptance criteria before examining the results. This article supplies no recommended sample size or pass percentage: those depend on the stakes and the evidence you need.

Close with one of three decisions

Use "proceed within the tested boundary", "collect missing evidence" or "stop this pilot". State the tested workload and resource ceiling beside the decision; do not generalize from that boundary to every future use. Archive the worksheet with the product configuration so a later model or tool change can trigger review.

For the permission side of the pilot, use the cyber evaluation release worksheet. For hostile retrieved material, continue with the prompt injection boundary exercise.