Skip to content

docs: documentation drift audit and rework tracking#233

Draft
vredchenko wants to merge 1 commit into
mainfrom
docs/drift-audit
Draft

docs: documentation drift audit and rework tracking#233
vredchenko wants to merge 1 commit into
mainfrom
docs/drift-audit

Conversation

@vredchenko

Copy link
Copy Markdown
Collaborator

Summary

Draft tracking PR for the documentation rework. Rather than rewrite the docs blind, this lands a
measured audit of how far the current docs have drifted from the code, so the rework can reuse what is
still true. Full detail in doc-audit/REPORT.md; interactive view in
doc-audit/drift-map.html.

Every published how-to / reference / tutorial doc was sliced into line-anchored content atoms, each
prose atom decomposed into individual factual claims, and every claim checked against the real
repositories.

Results

64 docs -> 2,560 content atoms. Verifiable subset (how-to / reference / tutorial): 34 files -> 960 claims.
ADR/design docs parked (historical record, not drift).

Verdict Claims Share Meaning
true 763 79% reuse as-is
misleading 119 12% right fact, drifted context - fix in place
incorrect 56 6% false as written - rewrite or bin
unverifiable 22 2% needs a live system

79% of documented claims are true - the rework is a re-organise-and-fill exercise, not a blank page.

Key findings

  • Repo-split drift dominates. Most non-true claims are misleading, not wrong: tooling moved
    smartem-decisions -> smartem-devtools, so documented commands/paths point at the wrong repo.
  • Dead docs. The two athena pages document the athena_api package deleted under ADR 0015 - remove/rewrite.
  • Real code bugs surfaced (not just doc rot): agent validate exits 0 when docs promise 1 (Typer
    ignores the handler return); --log-file is a no-op; -v/--verbose is inconsistent across subcommands.

Scope

This audit answers only "is what is written true?" It does not measure coverage gaps (undocumented
surface) or structure/consistency - the other two drivers of the rework. Those are tracked below.

Rework workstreams

  • Drift audit (this PR) - what is true / drifted / wrong
  • Fix drifted docs - 119 misleading claims, mostly path re-homing to smartem-devtools
  • Remove/rewrite dead docs - athena pages + 56 incorrect claims
  • Coverage-gap analysis - diff the real system surface (CLI, OpenAPI, env vars, k8s, routes) against the docs
  • Restructure and consistency - regroup by audience/task, unify terminology, reusing the true atoms
  • File code bugs - validate exit-code, --log-file no-op, verbose-flag mismatches

Notes

Draft, not for merge - the primary purpose is to track the rework. Analysis data (verdicts.jsonl) and the
re-runnable slicer (atomize.py) travel with the branch so findings are auditable and refreshable.

The docs under docs/ have drifted from the code since the workspace tooling
was split into smartem-devtools. Before rewriting blind, measure the drift:
slice each how-to/reference/tutorial doc into content atoms, decompose prose
into factual claims, and verify each against the repositories.

Lands the analysis (verdicts.jsonl, REPORT.md, interactive drift-map.html)
and the re-runnable slicer (atomize.py) as the tracking point for the rework.
@vredchenko vredchenko added the documentation Improvements or additions to project documentation label Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to project documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant