Skip to content

Guard task state lifecycle and result fields with the result mutex#210

Open
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix-taskstate-field-races
Open

Guard task state lifecycle and result fields with the result mutex#210
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix-taskstate-field-races

Conversation

@damilolaedwards

Copy link
Copy Markdown

Problem

The task state exposes its lifecycle and result fields (started, running, skipped, timeout, start and stop times, result, error and progress) to the web api and to child task watchers through GetTaskStatus, while the scheduler and the per task timeout goroutine wrote those same fields without holding a lock.

Only setTaskResult and SetProgress took resultMutex; the reader (GetTaskStatus) and the field writers in the executor did not. That is a data race. In practice it has two harmful outcomes:

  • A torn read of the error interface (or a multi word time value) in a watcher goroutine can panic, and that goroutine is not covered by the executor's recover, so it takes down the process.
  • The result can be read while it is being set, so a task can be reported with the wrong outcome, which then propagates to downstream tasks that read tasks.<id>.result.

Fix

Route every access to these fields through resultMutex:

  • GetTaskStatus builds its snapshot under a read lock.
  • The executor and the timeout goroutine write the lifecycle fields under a write lock (mark started, mark stopped, skip, timeout, panic recovery, and the post execution error and result reads).
  • updateTaskState is renamed to updateTaskStateLocked to document that callers must hold the lock. Its callers already do (or now do).
  • GetTaskResultUpdateChan takes a write lock because it may create the notify channel.
  • GetTaskCount guards its read of the task list, matching the other accessors on that slice.

There is no nested locking: the skip, recover and finalization paths release the mutex before calling setTaskResult, so there is no deadlock.

Tests

A new race test drives the real setTaskResult and SetProgress writers against the real GetTaskStatus reader. It passes under -race with the fix and fails without the read lock on GetTaskStatus.

go build ./..., go vet, and the full scheduler package under -race pass.

The task state exposes its lifecycle and result fields (started, running,
skipped, timeout, start and stop times, result, error and progress) to the
web api and to child task watchers via GetTaskStatus, while the scheduler
and the timeout goroutine wrote those same fields without holding a lock.
The concurrent read and write is a data race: a torn read of the error
interface can crash a watcher goroutine, and the result can be read while
it is being set, recording the wrong outcome for a task.

Take the result mutex for every access to these fields. GetTaskStatus now
reads the snapshot under a read lock, and the executor and timeout
goroutine write the lifecycle fields under a write lock. updateTaskState
is renamed to updateTaskStateLocked to document that its callers must hold
the lock. GetTaskCount now guards its read of the task list, and
GetTaskResultUpdateChan takes a write lock because it may create the
notify channel. Add a race test covering the reader and writer paths.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant