How to evaluate agent checkout performance
A reproducible comparison plan covering optimized Git, cache states, correctness, tail latency, and total service cost.
Measure time to a correct, task-ready input, rather than the speed of a command with a familiar name. Two “clone” operations can perform different amounts of work, provide different history, and make different durability guarantees. This guide is a proposed evaluation method; it is not a report of a production benchmark we ran.
OpenScion’s benchmark documentation is explicit about its limits: the reviewed server is in memory, Git writes compressed durable objects, and the two backends incur different process and transfer work. Those results can motivate experiments, but cannot establish production speed or savings. Pinned benchmark documentation
Freeze the workload
Choose a repository you may test and record its immutable commit ID, required paths, history-dependent commands, submodules, and external file requirements. Freeze the machine type, storage, network region, client versions, server configuration, and test script. Save a manifest of expected tracked files and executable modes.
The measured interval should start before input preparation and end when the real task can run. Include authentication, later object fetches, archive extraction, and cache misses when they occur inside that interval. Keep dependency installation and compilation separately visible so their costs do not hide the checkout effect.
Compare useful alternatives
| Variant | Why include it? |
|---|---|
| Existing pipeline | Establish the actual current experience |
| Git with only required history | Test whether full history is unnecessary |
| Filtered Git plus sparse checkout | Test reduced objects and working-tree paths |
| Existing mirror or snapshot cache | Account for optimizations already available |
| Candidate object or snapshot service | Measure the proposed service under its real guarantees |
Git’s own tools offer distinct controls for history, objects, and paths. Choose settings that preserve the task’s requirements, rather than automatically using the smallest clone. Git rev-list filtering options, Git fetch depth options
Separate cache states and load
Run cold client/cold service, cold client/warm service, and warm client cases separately. Record exactly which caches were retained; deleting a working directory does not necessarily flush an operating-system or shared service cache.
Use repeated runs and publish the individual measurements, median, tail latency, and error count. Vary concurrency to reflect the agent fleet. A fast single request does not show whether queues, locks, connection limits, or upstream misses dominate at load. Compare successful correctness checks alongside latency, including file contents and task results.
Count the service you must operate
Track client CPU, server CPU, memory, disk, transferred bytes, origin reads, and failed/retried operations. Apply actual provider prices to the measured resources. Include persistent storage, backups, idle capacity, and cache fill costs. A reduction in client time can still increase total spending.
Set an acceptance gate before examining results—for example, a chosen reduction in p95 preparation time at the target concurrency, with no correctness regressions and an acceptable total cost. That threshold is a business decision, not an industry standard.
Publish the script, pinned revisions, manifest, and raw results. If durability or authorization is missing from a prototype, label the comparison accordingly and re-run after those features exist.