Choose a small but meaningful task
Start with an authorized synthetic application that has a real behavior contract: a pricing rule, import workflow, local data transform or multi-step form. Include edge cases that require more than matching a screenshot. Make the task small enough to repeat while retaining the boundary you want to study.
Create untreated and protected releases from the same source revision. Preserve their hashes and the transformation configuration. Check that both original applications pass the reference behavior suite before the reconstruction stage. Otherwise a broken protected program can accidentally appear difficult to clone because its own expected behavior is undefined.
Freeze the agent environment
Record operating system, runtime, analyzer revision, model identifier, prompt, tool permissions and resource limits. State whether the agent can run the specimen, use a browser, inspect network traffic or install tools. Keep those permissions equivalent across conditions and use isolated workspaces that do not expose hidden source or tests.
Randomize or counterbalance trial order where practical. A human operator who learns the answer from an untreated run should not silently guide a later protected run. Record intervention events explicitly and keep assisted trials separate from unattended ones.
Score behavior after the submission is frozen
Require the agent to produce a runnable deliverable and stop changing it before evaluation. Run the hidden tests in a fresh environment, including invalid inputs and boundary values. Report build failures, partial passes and unsupported functionality. A prose description of the architecture is a different task and needs its own rubric.
Capture actual model usage and elapsed time from the run rather than deriving them from artifact size. Store raw observations alongside summaries. Multiple trials are important because one successful or failed run can reflect model variation, transient tool trouble or a lucky inference.
Make the report independently useful
Publish fixture source when permitted, build instructions, hashes, prompts, environment details, aggregate outcomes and selected redacted logs. Document missing tools, timeouts and any differences between the reference environment and the tested deployment. Avoid exposing real customer credentials or unpublished application logic in a public benchmark.
Finally, report what the defense cost the legitimate user: extra release bytes, startup delay, runtime behavior changes, accessibility effects and signing or update failures. A useful defense must meet its release requirements while affecting the reconstruction task, not merely make the benchmark look harder.
Practical checklist
- Freeze original and protected hashes.
- Use hidden behavior tests.
- Record real usage and time.
- Repeat trials with fixed tool permissions.
- Publish deployment overhead.
Sources and further reading
Primary references support the technical details. Procedures here are guidance; only explicitly recorded project results describe tests we ran.
Prepared with AI assistance. How this library is produced. Suggest a correction; keep private code and credentials out of public issues.