Define the legitimate-user budget
Agree on acceptable download size, disk use, startup latency and representative workload time before comparing a defense. A transformation that performs no extra hot-path computation can still increase network transfer or installer cost through added bytes. Choose budgets that reflect the actual distribution channel and user devices.
Make the measurement environment explicit: operating system, architecture, runtime, compiler, storage and relevant configuration. Keep treated and untreated builds on the same baseline. If one measurement includes a dependency download while the other uses a warm cache, record that difference rather than attributing it to the defense.
Time the pipeline in separate stages
Record build, defensive transformation, native signing, packaging and verification independently. Use a monotonic timing source in the harness and preserve repeated raw observations. Report median and spread where useful, with enough context to see outliers caused by environment noise.
Warm and cold runs answer different questions. For example, a cached compiler invocation and a first clean build should not share one unexplained average. Keep failures and retries visible because a transformation requiring frequent manual recovery can impose operational cost that a successful single-run timing misses.
Measure what reaches the customer
Record final uncompressed bytes, compressed distribution size, installed footprint and cold-start behavior. Compare representative workloads after installation rather than timing only the transformation command. Include parser, export and signature checks so a fast but invalid output does not pass the evaluation.
Our synthetic web benchmark reports local audit and staging timings for a four-file fixture. Those observations are useful for reproducing the fixture but cannot support production-throughput claims. Similarly, the recorded Swift transformation’s byte growth is a property of that specimen and configuration, not a universal size multiplier.
Keep benefit and overhead in the same report
Pair the overhead table with the actual defensive outcome: removed private artifacts, a specific analyzer budget event or a completed reconstruction trial. If the benefit is not measured, leave it unknown instead of assigning a token or dollar multiplier from file size.
A release decision should state which budget was met, which compatibility checks passed and how rollback works. This makes experimental hardening comparable with ordinary controls such as private maps or server authorization, while preventing a tool-level benchmark from becoming an unsupported economic claim about attackers.
Practical checklist
- Set customer and CI budgets in advance.
- Record matched cold and warm runs separately.
- Measure the final packaged specimen.
- Publish raw observations and missing measurements.
Sources and further reading
Primary references support the technical details. Procedures here are guidance; only explicitly recorded project results describe tests we ran.
Prepared with AI assistance. How this library is produced. Suggest a correction; keep private code and credentials out of public issues.