Speed up file installation on Windows#3832
Conversation
smartlink.Link resolved both src and dest via ResolveUniquePath for every entry it recursed into. On Windows ResolveUniquePath calls GetLongPathName, which is a syscall, so installing a runtime with many files performed two extra syscalls per file - a major contributor to slow Windows installs. The tree root is already resolved by LinkContents (and now by the public Link), and descendant paths are built by joining an already-resolved parent with real (long) entry names, so they need no further resolution. Split the recursion into an unexported link() worker that assumes resolved inputs and does not re-resolve, keeping resolution only for the rare symlink-to-file case where the pre-existing behavior links to the dereferenced target. Also treat an "already exists" error from linkFile the same as the existing TargetExists skip, so concurrent deploys of a shared file degrade to the documented "first one wins" behavior rather than failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DeployViaLink held the depot-wide fsMutex for the entire link operation, so even though the install worker pool runs several artifacts at once, the actual file linking was serialized to one artifact at a time. On Windows, where each link is an individual (relatively slow) syscall, this made the install phase far slower than it needed to be. Linking touches no shared in-memory depot state (Exists guards its own map via mapMutex and depotPath is immutable after construction), directory creation is idempotent, and smartlink now treats an already-existing target as a skip, so it is safe to run without the lock. Drop it from DeployViaLink so linking parallelizes across the worker pool. DeployViaCopy keeps the lock: two artifacts copying the same file would race on os.Create and could corrupt the destination. That path is only used for artifacts needing file transforms, when hard links are unsupported, or in portable mode, so it remains the less common case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
FYI I ran the "Test: critical" integration tests to make sure this isn't breaking anything unexpected. The failures that did occur are sporadic and unrelated to this PR. |
| // artifacts can be linked into the runtime concurrently (the install worker pool runs several at once). | ||
| // This is safe because linking touches no shared in-memory depot state (d.Exists guards its own access | ||
| // via mapMutex, and d.depotPath is immutable after construction), directory creation is idempotent, and | ||
| // smartlink treats an already-existing target as a skip. This is a major win on Windows where each file |
There was a problem hiding this comment.
I am not terribly comfortable with this assertion. It's probably referring to this code in smartlink.go:
// Multiple artifacts can supply the same file. We do not have a better solution for this at the moment other than
// favouring the first one encountered.
if fileutils.TargetExists(dest) {
logging.Warning("Skipping linking '%s' to '%s' as it already exists", src, dest)
return nil
}If we remove the fs mutex and then come up with a better solution, we lose the mutex guarantee. If we want to remove the mutex, we should update the referenced code's comment to mention the lack of an existing mutex (the one we're removing).
I like the additional comment above DeployViaCopy that we need the mutex to avoid copying to the same destination, but I'm not convinced we can forgo the need for mutex when creating links.
There was a problem hiding this comment.
We could try making fsMutex an RWMutex and have DeployViaLink take an RLock and DeployViaCopy/Put/Undeploy take Lock. This way the safer operations which are linking can be done in parallel but copying which isn't as safe still has to be sequential.
MDrakos
left a comment
There was a problem hiding this comment.
This PR needs some tests, especially to test the concurrency element which this PR addresses.
Some ideas from my robot all run with -race:
- A deterministic unit test for the new os.IsExist branch: pre-create the target,
call Link, assert it returns nil and leaves the existing target intact. - A concurrent same-file link test in smartlink (two goroutines, same dest) to hit
the TargetExists -> linkFile -> os.IsExist path. - If we take the RWMutex: a pkg/runtime test deploying a copy artifact and a link
artifact that share a file concurrently, asserting the depot source stays
byte-for-byte identical — the regression guard for the truncation case. - A quick equivalence test for the Link refactor (regular dir / symlink->file /
symlink->dir), since only the circular case is covered today.
| // artifacts can be linked into the runtime concurrently (the install worker pool runs several at once). | ||
| // This is safe because linking touches no shared in-memory depot state (d.Exists guards its own access | ||
| // via mapMutex, and d.depotPath is immutable after construction), directory creation is idempotent, and | ||
| // smartlink treats an already-existing target as a skip. This is a major win on Windows where each file |
There was a problem hiding this comment.
We could try making fsMutex an RWMutex and have DeployViaLink take an RLock and DeployViaCopy/Put/Undeploy take Lock. This way the safer operations which are linking can be done in parallel but copying which isn't as safe still has to be sequential.
Why
Installing runtimes with the CLI is noticeably slow on Windows. Tracing the install pipeline (download → unpack into depot → deploy → transform → track) surfaced two Windows-specific costs in the deploy step, both per-file:
smartlink.Linkre-resolved both src and dest viaResolveUniquePathfor every entry it recursed into. On Windows that callsGetLongPathName— a syscall — so a runtime with many files paid ~2 extra syscalls per file.DeployViaLinkheld the depot-widefsMutexfor the entire link operation, so despite the install worker pool (maxConcurrency = 5), file linking ran one artifact at a time. On Windows each link is an individual, relatively slow syscall, so serializing them hurts.What changed
Resolve link paths once per tree instead of once per file (smartlink.go)
link()worker that assumes already-resolved inputs and does not re-resolve. The root is resolved once; descendant paths are resolved-parent + real (long) entry name, so they need no further resolution.linkFileis now treated the same as the existingTargetExistsskip, so concurrent deploys of a shared file degrade to the documented "first one wins" behavior rather than failing.Deploy linked artifacts concurrently (depot.go)
fsMutexfromDeployViaLinkso linking parallelizes across the worker pool. Safe because linking touches no shared in-memory depot state (Existsguards its own map viamapMutex,depotPathis immutable), dir creation is idempotent, and smartlink now skips already-existing targets.DeployViaCopykeeps the lock: two artifacts copying the same file would race onos.Createand could corrupt the destination. Copy is the less common path (file transforms / no hard links / portable mode).Testing
go build+go testpass forinternal/smartlinkandpkg/runtime(including the circular-symlink test).Deliberately out of scope (possible follow-ups)
ListDir+GetDirSize).🤖 Generated with Claude Code