feat(generators): add --deterministic flag with hybrid RDFC-1.0 + rdflib serialization

[jdsika]

Summary

Add a --deterministic flag to OWL, SHACL, and JSON-LD generators that produces byte-identical output across invocations, eliminating spurious diffs in version-controlled artifacts.

This is a review-ready fork of the approach discussed in upstream linkml/linkml#3295, rebuilt to address maintainer feedback.

Problem

Generated OWL and SHACL artifacts contain blank nodes whose identifiers change between runs due to Python dict ordering and rdflib serialization non-determinism. This makes version-controlled artifacts show massive diffs even when the underlying schema change is trivial.

Solution

Three-Phase Hybrid Pipeline (deterministic_turtle())

1. RDFC-1.0 canonicalization (W3C Recommendation) via pyoxigraph ensures isomorphic inputs produce identical triple sets.
2. Weisfeiler-Lehman structural hashing replaces sequential _:c14nN identifiers with content-based hashes. These depend only on predicate IRIs, literal values, and named-node IRIs — not on blank-node numbering — so adding or removing a triple only affects directly involved blank nodes.
3. Hybrid rdflib re-serialization parses the canonicalized, WL-hashed triples back into an rdflib Graph and serializes with rdflib's native Turtle writer. This recovers idiomatic Turtle features that pyoxigraph cannot emit:
   • Inline blank nodes ([ … ]) for singly-referenced blank nodes (Turtle §2.7)
   • Collection syntax (( … )) for rdf:List chains (Turtle §2.8)
   • Prefix filtering: only prefixes actually used in the graph's IRIs are declared, following the practice of Apache Jena, Eclipse RDF4J, and Raptor

All triples from the source graph are preserved — the hybrid step only changes syntactic form, never semantic content. Plain string literals have their xsd:string datatype stripped per RDF 1.1 §2.5.1 (simple literals are syntactic sugar for xsd:string).

Additional Features

Collection sorting (gated behind --deterministic):

• owl:oneOf, sh:in, sh:ignoredProperties items are sorted when the flag is set
• Preserves existing behaviour by default

deterministic_json():

• Recursive deep-sort for JSON-LD context output

Benchmark Results

Tested on the Gaia-X Trust Framework ontology (~68K OWL / ~165K SHACL triples) and schema.org (~18K triples):

Semantic Equivalence

Artifact	Triples (det)	Triples (non-det)	rdflib.compare.isomorphic()
OWL	68,178	68,178	✅ True
SHACL	165,029	165,029	✅ True
schema.org	17,949	17,949	✅ True

Byte-Level Stability

Test	Deterministic	Non-Deterministic
SHA-256 identical across runs	✅	❌ (~1,400 lines differ)

Diff Quality (Signal-to-Noise Ratio)

Controlled mutations on a LinkML schema:

Mutation	Generator	Deterministic	Non-Deterministic	Noise Reduction
Change 1 description	OWL	1 line	344 lines	344×
Change 1 description	SHACL	13 lines	290 lines	22×
Add 1 new slot (+18 triples)	OWL	16 lines	350 lines	22×
Add 1 new slot (+7 triples)	SHACL	23 lines	305 lines	13×

Output Size (Gaia-X Trust Framework)

Artifact	Before (pyoxigraph-only)	After (hybrid)	Change
OWL	58,397 lines, 34 prefixes	58,291 lines, 11 prefixes	-0.2%
SHACL	163,993 lines, ~34 prefixes	9,118 lines, ~9 prefixes	-94%

The SHACL 18× size reduction comes from replacing 157,552 named _:bHASH blank nodes with inline [ … ] syntax and 77,358 explicit rdf:first/rdf:rest triples with ( … ) collection shorthand — matching the upstream Gaia-X registry convention.

Performance

Graph	Triples	Time	Throughput
schema.org	17,949	1.5s	~12,000 triples/s
Gaia-X OWL	68,178	~5s	~14,000 triples/s

Dependency

pyoxigraph >= 0.4.0 is imported lazily only when --deterministic is used. It is not a core dependency, avoiding conflict with morph-kgc's pin on pyoxigraph < 0.4.0. Tests skip gracefully when pyoxigraph >= 0.4.0 is unavailable.

Relationship to upstream linkml#3295

The original PR was closed after maintainer feedback requesting an established canonicalization standard. This PR:

1. Replaces custom WL canonicalization with W3C RDFC-1.0 via pyoxigraph
2. Retains WL only for blank node ID assignment (post-canonicalization remapping for diff stability — not a canonicalization algorithm)
3. Adds hybrid rdflib re-serialization for idiomatic Turtle output (inline blank nodes, collection syntax, prefix filtering)
4. Makes pyoxigraph an optional lazy import

How was this tested?

• 37 tests covering idempotency, isomorphism, performance, diff quality, and edge cases
  • test_deterministic_output.py: 27 tests (stability, sorting, prefix format, enum ordering, kitchen_sink)
  • test_deterministic_benchmark.py: 10 local + 4 network tests (schema.org equivalence, mutation diff quality, signal-to-noise assertions)
• Tests skip cleanly when pyoxigraph >= 0.4.0 is not available
• All existing tests pass (9500+ across the full matrix)

Benchmark Test Assertions

The benchmark enforces quantitative properties:

• Deterministic diff ≤ 20 lines for a single description change
• Signal-to-noise ratio ≥ 5× (actual: 13–344×)
• Diff proportional to new triples (≤ 5× margin)
• SHA-256 byte-identical across 3 consecutive runs
• Every declared prefix has at least one IRI using it
• schema.org (17,949 triples) serializes in < 60s

References

• W3C RDFC-1.0 — RDF Dataset Canonicalization
• W3C Turtle 1.1 — Terse RDF Triple Language (§2.7 inline blank nodes, §2.8 collections)
• Weisfeiler & Leman (1968) — graph canonical form
• schema.org — benchmark ontology

Areas of uncertainty

• Fork main now ships linkml_runtime.utils.rdf_canonicalize (upstreamed via feat(generators): add --default-language flag for language-tagged literals linkml/linkml#3449); this branch still carries its own linkml.utils.rdf_canonicalize copy and should be rebased to reuse the runtime module before upstreaming.
• pyoxigraph >= 0.4.0 stays a lazy, optional import to avoid the morph-kgc pin conflict; behaviour with future pyoxigraph majors is untested.
• Benchmark thresholds (diff-size / SNR / wall-clock) are machine-sensitive; the network-marked tests are excluded from CI by default.

Checklist

• My code follows the contributor guidelines
• I have added tests that prove my fix/feature works
• Existing tests pass locally with my changes

AI Assistance

If you used AI tools while preparing this PR, you are still the author and responsible for understanding, verifying, and defending your submission. Please engage with reviewers personally rather than through your agent during feedback and revisions. See our AI Covenant for details.

[jdsika]

🔍 Adversarial Review — PR #1

Summary

A well-engineered feature with strong documentation and benchmark data. The three-phase pipeline (RDFC-1.0 → WL → rdflib) is architecturally sound but introduces significant complexity. I found 2 bugs (dead code shipped as functional features), 1 algorithmic concern in collision handling, and several design/test gaps worth addressing before merge.

---

🐛 Bugs & Issues

1. Dead code: normalize_prefixes field and CLI option do nothing

The normalize_prefixes: bool = False field is added to Generator (line ~429) and a --normalize-prefixes/--no-normalize-prefixes CLI option is registered — but self.normalize_prefixes is never read in any generator code in this PR. Users who pass --normalize-prefixes get silent no-op behavior. This should either be removed from this PR (it belongs in PR #4) or wired up.

# Added to Generator dataclass but never checked anywhere:
normalize_prefixes: bool = False

2. Dead code: well_known_prefix_map() defined but never called

The function is defined in generator.py but is not imported or called anywhere in this PR. It returns {str(ns): str(pfx) for pfx, ns in Graph().namespaces() if str(pfx)} — a dynamic map that changes across rdflib versions (see Concern #3). PR #4 redefines this with a static frozen map, creating a direct merge conflict.

3. WL collision counter assignment depends on c14n ordering, not structure

In _wl_signatures(), when two structurally identical blank nodes produce the same WL hash, the counter suffix (_0 vs _1) is assigned by iterating sorted(bnode_ids) — which sorts by canonical c14n ID (e.g., "c14n0" < "c14n42"):

for bid in sorted(bnode_ids):  # sorted by c14n ID, NOT by structure
    digest = hashlib.sha256(sig[bid].encode("utf-8")).hexdigest()[:12]
    count = seen_hashes.get(digest, 0)
    seen_hashes[digest] = count + 1
    label = f"b{digest}" if count == 0 else f"b{digest}_{count}"

Adding an unrelated triple can change RDFC-1.0 numbering, which changes which colliding node gets the base label vs _1 suffix. This defeats WL's core promise of diff-stable IDs in the (admittedly rare) collision case. Fix: use the full WL signature string as a secondary sort key before assigning counters:

for bid in sorted(bnode_ids, key=lambda b: (sig[b], b)):

---

⚠️ Concerns

1. well_known_prefix_map() is rdflib-version-dependent (direct conflict with PR #4)

Graph().namespaces() returns different prefix sets across rdflib 6.x vs 7.x (e.g., brick, csvw, geo were added/changed). This means "deterministic" output can change on dependency upgrade. PR #4 fixes this with _WELL_KNOWN_PREFIX_MAP: MappingProxyType — a static frozen map. Both PRs also add normalize_prefixes to Generator. These are two direct merge conflicts.

Recommendation: Remove well_known_prefix_map() and normalize_prefixes from this PR entirely. They're unused dead code here and belong exclusively in PR #4.

2. Sorting OWL expression members by repr is fragile

members = sorted(members, key=repr)

repr() for LinkML model objects depends on dataclass field ordering and internal representation. While stable within a single Python version, it can change across:

• Python versions (if dataclass repr format changes)
• linkml-runtime versions (if fields are reordered or types change)
• Any field containing objects with id()-based repr

A more robust approach: define an explicit sort key using stable, semantic fields (e.g., key=lambda x: (x.range or "", x.minimum_value or "")) or serialize to a canonical string.

3. _mutate_kitchen_sink writes temp files to the test input directory

out_path = ks_path.parent / f"_benchmark_mutated_{os.getpid()}_kitchen_sink.yaml"

This writes to tests/linkml/test_generators/input/ which may be read-only in containerized CI. Also, yaml.dump(yaml.safe_load(...)) reformats the entire schema (quoting styles, flow/block, comment stripping), so the benchmark diffs measure YAML reformatting noise in addition to the intended mutation. Consider using tmpdir fixture or string manipulation instead of full YAML round-trip.

4. No RDFC-1.0 timeout for pathological graphs

The W3C RDFC-1.0 spec acknowledges exponential worst-case complexity for certain graph topologies. pyoxigraph.Dataset.canonicalize() has no timeout parameter exposed. A signal.alarm() or thread-based timeout would protect CI from hangs on adversarial input.

5. _deep_sort doesn't propagate parent_key into list items

sorted_items = [_deep_sort(item) for item in value]  # parent_key defaults to ""

If a list item is itself a list, it will be sorted regardless of whether the grandparent key is in _JSONLD_ORDERED_KEYS. Not a practical JSON-LD issue today, but a latent correctness gap.

6. O(n×m) prefix filtering

if pfx_s and any(iri.startswith(ns_s) for iri in used_iris):

With ~30 prefixes × thousands of IRIs, this is quadratic. A trie or pre-sorted binary search would scale better for large ontologies (though the current 68K-triple benchmarks likely absorb this).

7. xsd:string stripping is correct but lossy

The code strips explicit xsd:string datatype annotations during pyoxigraph→rdflib conversion. This is correct per RDF 1.1 §2.5.1 (simple literals = xsd:string), but users who intentionally annotated xsd:string for tooling compatibility lose the annotation after round-trip. Worth documenting this in the docstring.

---

🧪 Test Coverage Assessment

Strong coverage (✅):

• Byte-level idempotency across runs (4 generators)
• Sorted key verification for JSON generators
• Prefix format (@prefix vs PREFIX)
• Large schema stability (kitchen_sink)
• xfail documentation of non-isomorphism trade-off
• Non-deterministic mode regression guard

Gaps (❌):

• No test for WL hash collisions: Need a graph with two structurally identical blank node subgraphs to verify counter-based dedup produces stable output
• No test for deep blank node nesting (>4 levels): 4 WL iterations may be insufficient for chains of 5+ structurally similar nodes
• No test for well_known_prefix_map(): Dead code, but if kept, needs coverage
• No test for normalize_prefixes: Dead code — the CLI option silently does nothing
• xfail tests document non-isomorphism but don't verify semantic equivalence: The assertion "OWL/SHACL interpret these as unordered sets" should be backed by a test that verifies the same classes/constraints exist in both modes (e.g., compare extracted sh:in value sets, not just triple counts)
• Benchmark YAML round-trip may inflate diff counts: yaml.dump(yaml.safe_load(original)) reformats the entire file — diff measurements may include YAML formatting noise, not just the intended mutation
• Network tests (@pytest.mark.network): schema.org download can fail in CI. No conftest.py marker filtering shown — these may run and flake by default
• Performance test (10s limit): Fragile on slow CI runners; consider a more generous threshold or @pytest.mark.slow marker

---

📋 Fix Plan

1. Remove normalize_prefixes and well_known_prefix_map() from this PR — they are dead code, belong in PR fix(generators): add --normalize-prefixes flag for well-known prefix names #4, and create merge conflicts
2. Fix WL collision counter ordering: Sort by (sig[bid], bid) instead of just bid to make counter assignment structure-dependent
3. Add a WL collision test: Create a graph with two identical blank node subgraphs, verify both get stable IDs across runs
4. Replace repr sorting with explicit key function for OWL expression members
5. Use tmp_path fixture for _mutate_kitchen_sink output (create a symlink or copy imports alongside)
6. Add @pytest.mark.slow to performance tests and document CI threshold expectations
7. Document xsd:string stripping in deterministic_turtle() docstring explicitly as a known behavior

✅ What's Good

• Excellent documentation: Thorough docstrings with W3C spec references, clear parameter docs, and well-written PR description with benchmark tables
• Lazy import pattern: pyoxigraph is only loaded when --deterministic is used; graceful ImportError with actionable message
• Defensive test design: pytestmark = pytest.mark.skipif(not _has_pyoxigraph, ...) and fixture-level pytest.skip() for network tests
• The hybrid pipeline is clever: RDFC-1.0 for correctness, WL for diff stability, rdflib for Turtle readability — each phase has a clear purpose
• The 94% SHACL size reduction (inline blank nodes + collection syntax) is a compelling result
• xfail tests properly document trade-offs rather than hiding them
• Prefix filtering removes the ~27 rdflib default bindings that leak into output — real quality-of-life improvement