fix(serverless): force-kill worker on fitness check failure

[deanq]

Problem

A failed worker fitness check is supposed to force-kill the worker so the orchestrator restarts it. Instead, on workers with live non-daemon background threads, the worker logs Worker is unhealthy, exiting. and keeps serving jobs. Observed on a live vLLM endpoint.

Root cause

rp_fitness.run_fitness_checks calls sys.exit(1) on failure. sys.exit() only raises SystemExit, which unwinds to the top of asyncio.run(run_fitness_checks()) (worker.py:40) and begins cooperative interpreter shutdown. That shutdown blocks in threading._shutdown() joining every non-daemon thread.

worker-vllm constructs AsyncLLMEngine.from_engine_args(...) at module import — before runpod.serverless.start(...) runs, and therefore before fitness checks run. Its non-daemon background threads/loops never return, so the interpreter blocks forever on the join and the process never dies.

Reproduction (against the real run_fitness_checks)

A non-daemon thread loops forever, a failing check is registered, then asyncio.run(run_fitness_checks()) runs exactly as worker.py does:

Exit call	Result
sys.exit(1) (before)	Logs "unhealthy, exiting." then hangs (killed by timeout)
os._exit(1) (after)	Logs the same line, terminates immediately with code 1

Fix

Route the unhealthy exit through a _terminate_unhealthy helper that flushes stdio then calls os._exit(1). A fitness failure means the environment is broken and the worker must die now; os._exit bypasses thread joins, atexit handlers, and asyncgen cleanup — the required semantics.

Test plan

• New test_force_kill.py:
  • test_unhealthy_exit_terminates_with_live_non_daemon_thread — subprocess with a live non-daemon thread; fails (hangs to timeout) before the fix, exits 1 after.
  • test_terminate_unhealthy_uses_os_exit — asserts the helper calls os._exit.
• Existing SystemExit-based fitness tests kept green via an autouse fixture that redirects os._exit to raise SystemExit(1) (so it can't kill the pytest process).
• make quality-check: 525 passed, coverage 94.20% (threshold 90%).

Closes SLS-379

[capy-ai]

Capy auto-review is paused for this organization because the usage-cycle auto-review limit has been reached. Increase the limit or turn it off in billing settings to resume automatic reviews.

[Copilot]

Pull request overview

This PR changes serverless fitness-check failure handling so an unhealthy worker is force-terminated immediately (via os._exit(1)) instead of attempting cooperative shutdown via sys.exit(1), which can hang when non-daemon background threads are still running (e.g., vLLM engine threads).

Changes:

• Add _terminate_unhealthy() helper in rp_fitness that flushes stdio then calls os._exit(1) on fitness-check failure.
• Update run_fitness_checks() to use _terminate_unhealthy(1) when a check fails.
• Add regression tests that reproduce the hang scenario in a subprocess and verify os._exit is used; adjust test fixtures to prevent os._exit from killing the pytest process.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
runpod/serverless/modules/rp_fitness.py	Introduces hard-exit helper and routes fitness failures through it.
tests/test_serverless/test_modules/test_fitness/conftest.py	Autouse fixture neutralizes os._exit during in-process tests by raising SystemExit.
tests/test_serverless/test_modules/test_fitness/test_force_kill.py	Adds subprocess regression test for non-daemon thread hang and asserts helper uses os._exit.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.