Guard task state lifecycle and result fields with the result mutex#210
Open
damilolaedwards wants to merge 1 commit into
Open
Guard task state lifecycle and result fields with the result mutex#210damilolaedwards wants to merge 1 commit into
damilolaedwards wants to merge 1 commit into
Conversation
The task state exposes its lifecycle and result fields (started, running, skipped, timeout, start and stop times, result, error and progress) to the web api and to child task watchers via GetTaskStatus, while the scheduler and the timeout goroutine wrote those same fields without holding a lock. The concurrent read and write is a data race: a torn read of the error interface can crash a watcher goroutine, and the result can be read while it is being set, recording the wrong outcome for a task. Take the result mutex for every access to these fields. GetTaskStatus now reads the snapshot under a read lock, and the executor and timeout goroutine write the lifecycle fields under a write lock. updateTaskState is renamed to updateTaskStateLocked to document that its callers must hold the lock. GetTaskCount now guards its read of the task list, and GetTaskResultUpdateChan takes a write lock because it may create the notify channel. Add a race test covering the reader and writer paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The task state exposes its lifecycle and result fields (started, running, skipped, timeout, start and stop times, result, error and progress) to the web api and to child task watchers through
GetTaskStatus, while the scheduler and the per task timeout goroutine wrote those same fields without holding a lock.Only
setTaskResultandSetProgresstookresultMutex; the reader (GetTaskStatus) and the field writers in the executor did not. That is a data race. In practice it has two harmful outcomes:errorinterface (or a multi word time value) in a watcher goroutine can panic, and that goroutine is not covered by the executor's recover, so it takes down the process.tasks.<id>.result.Fix
Route every access to these fields through
resultMutex:GetTaskStatusbuilds its snapshot under a read lock.updateTaskStateis renamed toupdateTaskStateLockedto document that callers must hold the lock. Its callers already do (or now do).GetTaskResultUpdateChantakes a write lock because it may create the notify channel.GetTaskCountguards its read of the task list, matching the other accessors on that slice.There is no nested locking: the skip, recover and finalization paths release the mutex before calling
setTaskResult, so there is no deadlock.Tests
A new race test drives the real
setTaskResultandSetProgresswriters against the realGetTaskStatusreader. It passes under-racewith the fix and fails without the read lock onGetTaskStatus.go build ./...,go vet, and the full scheduler package under-racepass.