Label the event precisely
An analyzer may reject an oversized header, limit nesting, truncate an inventory or stop expanding candidate values. These limits protect the analyzer from excessive work. A transformed artifact reaching one of them establishes a specific tool behavior in a particular environment. It does not establish that the underlying application logic became inaccessible.
Use a result record containing artifact hash, analyzer revision, provider, limit name, input bytes and returned status. Preserve the error text and any partial output. This makes later rechecks possible when a provider or capacity rule changes.
Keep projections distinct from actual calls
Our recorded BLA comparison contains several evidence types. Some entries invoked real REA modules. The Flutter outcome used a local projection, the EVM specimen approached an input cap, PCAP was skipped without tshark, and the WASM case reported a projected representation limit without configured Wabt. Those distinctions change the meaning of the result.
A single green top-level harness exit does not mean every provider ran or every protected specimen defeated analysis. Read individual records. Aggregate charts should show denominators and skipped cases so a partial test environment cannot accidentally look like full format coverage.
Test the alternate paths an agent can take
A reconstruction agent can sometimes ignore poisoned metadata, inspect a smaller member, use a different parser or learn behavior by running the original. A realistic trial allows a clearly defined set of ordinary tools and records which alternatives were attempted. The objective is to test the product requirement, not only the attacked analyzer branch.
Keep the fallback budget fixed between treated and untreated trials. If one side permits browser interaction or source extraction and the other does not, the result measures different task conditions. Record manual intervention separately so an unattended result is not mixed with analyst-assisted recovery.
Use two independent outcome fields
Store an analyzer outcome and a reconstruction outcome. The first might be capacity_error; the second might be passed_17_of_20_behavior_tests, failed_to_build or not_attempted. This avoids turning a tool-level refusal into an unsupported product-level verdict.
Only completed, repeated reconstruction trials can support a claim about task difficulty in the tested setting. Even then, report the task and uncertainty. Public interfaces, screenshots and accessible API responses can preserve enough information for a useful replica despite failures in one static analysis path.
Practical checklist
- Save the exact budget event.
- Separate actual, projected and skipped probes.
- Allow documented fallback tools equally.
- Score reconstructed behavior with hidden tests.
Sources and further reading
Primary references support the technical details. Procedures here are guidance; only explicitly recorded project results describe tests we ran.
Prepared with AI assistance. How this library is produced. Suggest a correction; keep private code and credentials out of public issues.