Choose a decision the experiment will inform
State what a result would change: adopting a transformation, moving logic to a server or declining a costly control. Choose a reconstruction task that represents a valuable product behavior and can be scored objectively. Avoid a broad task such as clone the entire company website, which makes success and failure difficult to compare.
Estimate the internal effort required to build fixtures, run trials and evaluate submissions. This is a planning estimate for the team’s work, not an attacker-cost claim. Set a resource budget that permits repeated runs and retain room for diagnosing tool or environment failures.
Build an evaluation package first
Prepare shareable authorized fixtures, normal releases and treated releases from pinned sources. Freeze a hidden behavior suite with edge cases and a build contract for reconstructed submissions. Keep the evaluator out of the agent workspace and test that the reference implementation passes before starting trials.
Include UI or API tasks only when they have a clear scoring method. A visually similar interface, correct API response and accurate architecture explanation are different outputs. If all matter, create separate dimensions rather than a single unexplained percentage.
Collect actual observations
For each run, record timestamps, model identifier, actual token or usage metadata, tool invocations, errors, intervention and frozen submission hash. Keep identical permissions and resource limits across matched conditions. Randomize order or otherwise control for operator learning.
Run enough trials to see variability and report the sample size. A timeout means the task was incomplete within a budget; a missing tool is an environment limitation. Do not relabel either as proven protection. Compare actual completed-task outcomes and distributions rather than deriving hours from file growth or a static estimator.
Define publication and deployment gates
Plan a report containing raw observations, reproducible commands, exact revisions, compatibility checks and explicit exclusions. Review artifacts for private source or credentials before public release. Link historical results to their pinned versions so later updates cannot silently rewrite what was measured.
Treat adoption as a separate decision after the experiment. A defense should satisfy runtime, signing, accessibility, diagnostic and overhead budgets before production. If a planned trial is delayed, the public page should continue to say not measured. The year in the title is a planning horizon, not a reason to imply future evidence exists.
Practical checklist
- State the engineering decision under test.
- Freeze evaluator and matched specimens.
- Capture actual usage and failures.
- Publish sample sizes and deployment overhead.
Sources and further reading
Primary references support the technical details. Procedures here are guidance; only explicitly recorded project results describe tests we ran.
Prepared with AI assistance. How this library is produced. Suggest a correction; keep private code and credentials out of public issues.