feat(gen-shacl): translate LinkML rules to SHACL-SPARQL constraints by rmessaou · Pull Request #18 · ASCS-eV/linkml

rmessaou · 2026-07-08T14:49:14Z

Summary

Adds a rules → SHACL-SPARQL converter to the SHACL generator: LinkML
rules: blocks (cross-parameter / conditional validation) are now translated
into sh:SPARQLConstraint (sh:sparql) nodes on the corresponding
sh:NodeShape, gated behind a new --emit-rules / --no-emit-rules CLI flag
(default on). Every generated query pre-binds $this per
W3C SHACL §5.3.1
and is validated by pyshacl with advanced=True.

Rule patterns supported

Three named patterns (matched first, output unchanged):

Pattern	Precondition → Postcondition	Meaning
Boolean guard	`value_presence: PRESENT` → `equals_string: "true"`	if value present, flag must be `true`
Presence-implies-value	`value_presence: PRESENT` → `equals_string` / `equals_string_in`	if value present, target must hold one of the allowed enum values
Exclusive value	`equals_string` → `maximum_cardinality: N` (same slot)	if value V present, slot has at most N values

…plus a compositional fallback for combinations outside the named patterns
(tried only after them, so it never changes named-pattern output):

M1 – conditional-required: precondition holds → target slot required
(violation = FILTER NOT EXISTS).
M2 – conditional-absent: precondition holds → target slot
value_presence: ABSENT.
M3 – numeric threshold precondition: minimum_value / maximum_value
bound on the trigger slot.
M4 – nested precondition: one hop into an inlined child via
range_expression.slot_conditions (e.g. sun_position.elevation <= 0).
M5 – has_member list-membership: a multivalued slot must contain a member
matching a nested range_expression (violation = FILTER NOT EXISTS over
the members).

Unrecognised patterns are logged at DEBUG and skipped — the converter
never emits a mis-translated constraint. deactivated rules are skipped;
bidirectional / open_world rules warn (forward direction emitted).

Supporting generator changes carried on this branch

--deterministic flag: diff-stable output via Weisfeiler-Lehman structural
hashing layered on the upstream RDFC-1.0 canonicalization (OWL / SHACL /
JSON-LD / JSON-LD-context), plus deterministic collection / expression /
permissible-value ordering; trailing-newline normalization.
--normalize-prefixes flag: map non-standard prefix aliases to well-known
names from a static, version-independent map (OWL / SHACL / context).
shaclgen: emit sh:minCount/maxCount 0 for zero cardinality
(is not None checks, matching owlgen); emit sh:pattern for pattern
constraints inside any_of.
gen-json-schema: --include-null / --no-include-null; emit propertyNames
from an inlined-dict key slot's string-applicable constraints.

Review-hardening pass

A final adversarial review commit fixes three latent defects in the rule
converters, each with a regression test that fails before the fix:

Induced-slot parity (silent false negative). _slot_uri and
_resolve_enum_value_ref resolved the base slot, so a slot_usage
override of slot_uri (or a narrowed enum range) made the SPARQL body
query a property / enum IRI the data never uses, while sh:path used the
induced IRI. The constraint then silently never fired. Both now resolve the
induced slot for the class, matching the sh:path logic in the main slot
loop.
Combined operators (silent under-constraint / false positives). A single
precondition or member condition dispatched on the first matching operator,
so a bounded range {minimum_value: X, maximum_value: Y} dropped the lower
bound. A shared _scalar_filters helper now emits every recognised
operator (and still returns None — skip, never mis-translate — when none is
recognised).
SPARQL string escaping (invalid / injectable output). An equals_string
or permissible-value name containing a ", \, or newline produced
unparseable SPARQL. New _sparql_string_literal escapes per
SPARQL 1.1 §19.7.

How was this tested?

Per pattern: structural triple assertions, prepareQuery syntax
validation, and pyshacl end-to-end (conforming instances pass, crafted
violations fail) with advanced=True.
6 new regression tests for the review-hardening pass (including the
below-threshold case that proves the lower bound is enforced, and an
end-to-end proof that a slot_usage-overridden constraint actually fires).
Each was confirmed to fail on the pre-fix source.
Full test_shaclgen.py suite green (111 passed); the PR's other touched
test files (test_deterministic_output, test_normalize_prefixes,
test_jsonschemagen, test_jsonldcontextgen, test_gen_json_schema) green
(281 passed, 2 skipped, 7 xfailed, 261 subtests).
ruff check and ruff format --check clean on the changed files.

Areas of uncertainty

Known, deliberately-scoped limitations (candidates for follow-ups, not
regressions):

Boolean-guard flag typing: the guard compares ?flag != true (an
xsd:boolean); a flag modeled as an xsd:string "true" yields a SPARQL
type-error under three-valued logic. The modeled use case uses boolean flags.
equals → equals postconditions: an "if A=x then B=y" rule (equals
precondition + equals postcondition) is not yet handled by the compositional
fallback and is skipped (logged at DEBUG), not mis-translated.
Combined min+max on a scalar precondition now emits both bounds; a
condition mixing a range_expression with scalar operators still resolves via
the nested branch only.

Checklist

My code follows the contributor guidelines
I have added tests that prove my fix/feature works
Existing tests pass locally with my changes

AI Assistance

AI tooling assisted with the final adversarial-review pass (defect discovery
and the accompanying regression tests). The findings were reproduced and the
changes verified against the pyshacl end-to-end suite by the author, who
remains responsible for the submission and will engage with reviewers
personally. See our AI Covenant for details.

Add a --deterministic / --no-deterministic CLI flag (default off) to OWL, SHACL, JSON-LD Context, and JSON-LD generators that produces diff-stable output using Weisfeiler-Lehman structural hashing on top of the RDFC-1.0 canonicalization from upstream (linkml#3407). Three-phase hybrid pipeline (when --deterministic is set): 1. RDFC-1.0 canonicalization (upstream) produces sequential _:c14nN IDs 2. Weisfeiler-Lehman structural hashing replaces sequential IDs with content-based _:b<sha256> hashes that remain stable when unrelated triples are added/removed 3. rdflib re-serialization recovers idiomatic Turtle (inline blank nodes, collection syntax, filtered prefixes, preserved xsd:string) Without --deterministic, upstream's always-on RDFC-1.0 canonicalization is used directly (via canonicalize_rdf_graph). Additional features gated behind --deterministic: - Expression sorting (any_of/all_of/none_of/exactly_one_of) in owlgen - Collection sorting (sh:in, sh:ignoredProperties) in shaclgen - Permissible value sorting in owlgen and shaclgen - JSON-LD deterministic key ordering (deterministic_json) - JSON-LD context structured ordering (jsonldcontextgen) Rebased on top of upstream linkml#3407 (pyoxigraph RDFC-1.0). Refs: linkml#1847, linkml#3407 Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>

rdflib's Turtle serializer always emits a trailing double newline. Normalize to single newline in deterministic_turtle() and the rdflib fallback path in canonicalize_rdf_graph() for consistent file endings. Note: CLI print() still adds a newline after serialize()'s trailing newline. Callers capturing stdout should strip trailing blank lines (e.g. via sed). Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>

… names Add an opt-in --normalize-prefixes flag to OWL, SHACL, and JSON-LD Context generators that normalises non-standard prefix aliases to well-known names from a static prefix map (derived from rdflib 7.x defaults, cross-checked against prefix.cc consensus). Key design decisions: - Static frozen map (MappingProxyType) instead of runtime Graph().namespaces() lookup eliminates rdflib version dependency - Both http://schema.org/ and https://schema.org/ map to 'schema' - Shared normalize_graph_prefixes() helper used by OWL and SHACL - Two-phase graph normalisation: Phase 1 normalises schema-declared prefixes, Phase 2 cleans up runtime-injected bindings - Collision detection: skip with warning when standard prefix name is already user-declared for a different namespace - Phase 2 guard prevents overwriting HTTPS bindings with HTTP variants The flag defaults to off, preserving existing behaviour. Tests cover OWL, SHACL, and context generators with sdo->schema, dce->dc, http/https edge case, custom prefix preservation, flag-off backward compatibility, cross-generator consistency, prefix collision detection, schema1 regression prevention, Phase 2 HTTPS guard, empty schema edge case, and static map integrity. Signed-off-by: jdsika <carlo.van-driesten@bmw.de> Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>

Implement SHACL-SPARQL constraint generation for the boolean-guard pattern commonly used in conditional validation rules. When a LinkML class has rules: blocks with preconditions (value_presence: PRESENT) and postconditions (equals_string: true), the generator now emits sh:SPARQLConstraint nodes on the corresponding sh:NodeShape. Features: - New _add_rules() method translates recognised rule patterns to SPARQL - Boolean-guard pattern: if value present then flag must be true - Rule description mapped to sh:message on the constraint - Deactivated rules are skipped - Warnings emitted for bidirectional/open_world rule flags - New --emit-rules/--no-emit-rules CLI flag (default: enabled) - Full URI references in SPARQL (no PREFIX declarations needed) The generated SPARQL follows W3C SHACL Section 5 and uses the pre-bound \ variable per Section 5.3.1. Constraints are validated by pyshacl with advanced=True. Refs: linkml#2464 Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>

Python truthiness (`if s.maximum_cardinality:`) treats 0 as falsy, so `maximum_cardinality: 0`, `minimum_cardinality: 0` and `exact_cardinality: 0` emitted no constraint at all. `maximum_cardinality: 0` (SHACL `sh:maxCount 0`, "property must not appear") is the idiomatic way to suppress an inherited slot on a subclass via slot_usage, and owlgen already emits `owl:maxCardinality 0` for it -- so the SHACL and OWL output silently diverged. Use explicit `is not None` checks for minimum_cardinality, maximum_cardinality and exact_cardinality, matching the pattern already used in owlgen.py and docgen.py. Precedence: an explicit minimum_cardinality wins over the `required` fallback in the elif cascade, consistent with owlgen.py (which uses the same `if minimum_cardinality is not None ... elif required` order), so `required: true` + `minimum_cardinality: 0` yields `sh:minCount 0`. That combination is a schema-authoring contradiction (the metamodel documents minimum_cardinality as a multivalued-slot count); the explicit, more specific constraint is emitted. Tests cover maximum_cardinality: 0, exact_cardinality: 0, minimum_cardinality: 0, and the required + minimum_cardinality: 0 precedence case. Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>

The SHACL generator translated any_of branches by dispatching solely on `any.range` (class, type, enum, or simple datatype). If a branch specified `pattern:` — either alone or combined with a range — the constraint was silently dropped, producing an empty blank node `[ ]` (trivially satisfied) instead of the intended `[ sh:pattern "..." ]`. This is a problem for schemas that use pattern alternatives in `any_of`, such as the SPDX license field where valid values are either members of a fixed enum (SPDX identifiers), IRIs, or custom identifiers matching the LicenseRef- pattern defined in SPDX Specification v2.3 Annex D (ABNF: license-ref = ["DocumentRef-"(idstring)":"]"LicenseRef-"(idstring)). The fix adds a single check after the range dispatch: if any.pattern: g.add((range_list[-1], SH.pattern, Literal(any.pattern))) This correctly handles: - Pattern-only branches (no range): node gets only sh:pattern - Range + pattern branches: node gets both sh:datatype and sh:pattern - Range-only branches (no pattern): unchanged behaviour The test suite now includes a dedicated schema exercising all three cases, with assertions on both the generated RDF triples and pyshacl validation of conforming/non-conforming data. Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>

Expose the existing JsonSchemaGenerator.include_null field on the gen-json-schema CLI. --no-include-null forbids explicit JSON null in optional slots so optionality is expressed only via absence from required (JSON Schema Validation 6.5.3), keeping the bare value type (6.1.1) -- needed for strict parity with reference schemas that forbid null. Default unchanged (include_null=True). Tested at the CLI surface via CliRunner over scalar, multivalued, and required slots; the standards rationale lives in the include_null field docstring. Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>

…nstraints For an inlined-as-dict slot whose range class has an identifier/key slot, render the key slot's string-applicable constraints onto JSON Schema propertyNames (draft-06+) instead of dropping them. In the inlined-dict form the mapping key is the identifier value, so the key slot's constraints constrain the keys. JSON object keys are always strings, so only pattern, enum (equals_string_in) and a string const (equals_string) are emitted; numeric minimum/maximum, numeric const (equals_number) and allOf are excluded -- a numeric const would otherwise reject every key. structured_pattern is honored when materialize_patterns is enabled, consistent with value patterns. Backward compatible: emitted only when a string-applicable key constraint applies. Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>

Generalise the boolean-guard SHACL-SPARQL pattern to enum-valued targets. A rule whose precondition is `value_presence: PRESENT` on a value slot and whose postcondition is `equals_string` / `equals_string_in` on a target slot now emits an `sh:sparql` constraint requiring the target slot to be present and hold one of the allowed values. Each allowed value resolves to its enum `meaning` IRI, with a string-literal fallback. Motivating case (aiSim environment): - "if texture_sky_color is set, sky_model must be TextureSky" - "if overcast_sky_illuminance is set, sky_model must be OvercastSky or MeasuredOvercastSky" The existing boolean-guard (`equals_string: "true"`) and exclusive-value patterns are unchanged; boolean guard keeps priority over the new branch. Adds focused unit tests (enum IRI vs. literal fallback, single value vs. set membership, message emission, SPARQL syntax) plus pyshacl end-to-end validation for the new pattern.

Add a compositional fallback in _rule_to_sparql that handles operator combinations outside the three named patterns. First operator: equals_string precondition + required postcondition, emitting a SHACL-SPARQL constraint whose SELECT flags focus nodes that satisfy the precondition but lack the required slot (FILTER NOT EXISTS). Tests: structural, SPARQL syntax (prepareQuery) and pyshacl end-to-end.

Handle a value_presence: ABSENT postcondition in the compositional fallback: when the precondition holds, the target slot must not be present. Violation SPARQL matches focus nodes where the precondition holds and the forbidden slot is present. Tests: structural, SPARQL syntax and pyshacl end-to-end.

Support minimum_value / maximum_value in rule preconditions: emit a numeric SPARQL FILTER (inclusive bound) on the trigger slot, combinable with any supported postcondition. Adds _sparql_number to render the bound. Tests: structural, SPARQL syntax and pyshacl end-to-end.

Support a precondition that reaches one hop into an inlined child object via range_expression.slot_conditions (e.g. sun_position.elevation <= 0). Adds _member_conditions (shared inner-condition emitter) and _resolve_member_enum_ref, which resolves inner enum values against the container slot's range class (handles slot_usage-specialised enums). Tests: structural, SPARQL syntax and pyshacl end-to-end.

Support a has_member postcondition with a nested range_expression: the violation is a FILTER NOT EXISTS over the multivalued slot's members, requiring at least one member that matches the inner conditions (e.g. the light-group list must contain {group: Vehicle, type: front_fog_light}). Reuses _member_conditions so inner enum values resolve against the member class. Tests: structural, SPARQL syntax and pyshacl end-to-end.

…and SPARQL escaping Review hardening for the SHACL-SPARQL rule converters. Three defects, each with a regression test that fails before this change: - Induced-slot parity: _slot_uri and _resolve_enum_value_ref resolved the *base* slot, so a slot_usage override of slot_uri (or a narrowed enum range) made the generated SPARQL query a property/enum the data never uses while sh:path used the induced IRI. The constraint then silently never fired (false negative). Both now resolve the induced slot for the class, matching the sh:path logic in the main slot loop. - Combined operators: a single precondition / member condition dispatched on the first matching operator, so {minimum_value: X, maximum_value: Y} dropped the lower bound and under-constrained the trigger (false positives). A shared _scalar_filters helper now emits every recognised operator, and still returns None -- skip, never mis-translate -- when none is recognised. - SPARQL string escaping: an equals_string / permissible-value name containing a double quote, backslash or newline produced invalid, unparseable SPARQL. New _sparql_string_literal escapes per SPARQL 1.1 section 19.7. Tests: 6 new regression tests (structural, prepareQuery syntax, and pyshacl end-to-end) covering all three defects; full shaclgen suite green (111). Signed-off-by: Carlo van Driesten <carlo.van-driesten@vdl.digital> Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>

jdsika · 2026-07-10T20:23:26Z

The new work on this branch (relative to the already-open atomic PRs #1/#4/#11/#12/#13/#15/#16) has been extracted into a reviewable stacked series, per the one-feature-one-PR convention:

main
└─ feat/shaclgen-rules-sparql                      #11 (green)
   └─ feat/shaclgen-presence-implies-value-stacked   #19 (supersedes #17)
      └─ feat/shaclgen-compositional-rule-fallback   #20 (M1–M5, squashed, authorship preserved)
         └─ fix/shaclgen-rule-converter-hardening    #21 (e263def7 cherry-pick)

Commit mapping verified by patch-id: 0ec311cb+d286fbe5=#1, 985e6500=#12, 442b5132=#15, ef7ad851=#16 are byte-identical to their atomic PRs; abed0245≈#4, 3f68eb8d≈#11, 4f757fba≈#13, ea5cf575≈#17 differ only by stacking context. This branch remains the integration candidate.

Two follow-ups surfaced by the audit (details in the #19/#20/#21 audit comments — all findings reproduce on this branch's tip):

Real defects in the rule converters worth fixing here and in the stack before upstreaming: inner-slot resolution against the outer class (M4/M5), operator mixes silently dropping conjuncts (incl. expression-level any_of), non-numeric minimum_value/maximum_value producing unparseable or silently-wrong SPARQL, elseconditions dropped without warning.
Rebase note: this branch carries linkml.utils.rdf_canonicalize while upstream main now ships linkml_runtime.utils.rdf_canonicalize (merged via feat(generators): add --default-language flag for language-tagged literals linkml/linkml#3449) — the --deterministic commit should be rebased to reuse the runtime module and drop the duplicate.

jdsika · 2026-07-11T06:14:30Z

Follow-up: the audit findings are fixed in #22; cherry-pick d0a7316e onto this consolidated branch to align it.

jdsika · 2026-07-11T10:47:45Z

Closing as superseded: feat/envited-x-pipeline (#14) is now the single consolidated feature branch. It carries this branch's first 8 commits verbatim (identical SHAs) and replaces the rule-converter commits with the audited, hardened stack (#19 → #20 → #21 → #22 review lineage, all CI-green and GPG-signed, original authorship preserved) plus the generator docs (#23). Combined-branch test run: 132 shaclgen + 109 other-suite tests green. Nothing from this branch is lost — every commit either lives in #14 by SHA or was superseded by a reviewed equivalent.

jdsika and others added 14 commits July 8, 2026 10:54

rmessaou self-assigned this Jul 8, 2026

jdsika changed the title ~~Feat/shacl rule converters~~ feat(gen-shacl): translate LinkML rules to SHACL-SPARQL constraints Jul 9, 2026

jdsika force-pushed the feat/shacl-rule-converters branch from c6415c7 to e263def Compare July 9, 2026 06:27

This was referenced Jul 10, 2026

feat(gen-shacl): add presence-implies-value rule pattern (stacked on #11) #19

Open

feat(gen-shacl): add compositional fallback rule converters (M1-M5) #20

Open

fix(gen-shacl): correct rule-converter path parity, combined bounds, and SPARQL escaping #21

Open

This was referenced Jul 10, 2026

feat(gen-shacl): add presence-implies-value rule pattern #17

Closed

fix(gen-shacl): enforce operator exactness, nested-slot parity, numeric bounds #22

Open

This was referenced Jul 11, 2026

fix(generators): add --normalize-prefixes flag for well-known prefix names #4

Open

feat(gen-shacl): generate sh:sparql constraints from LinkML rules #11

Open

chore: consolidated feature branch for ENVITED-X pipeline #14

Draft

jdsika closed this Jul 11, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(gen-shacl): translate LinkML rules to SHACL-SPARQL constraints#18

feat(gen-shacl): translate LinkML rules to SHACL-SPARQL constraints#18
rmessaou wants to merge 15 commits into
mainfrom
feat/shacl-rule-converters

rmessaou commented Jul 8, 2026 •

edited by jdsika

Loading

Uh oh!

jdsika commented Jul 10, 2026

Uh oh!

jdsika commented Jul 11, 2026 •

edited

Loading

Uh oh!

jdsika commented Jul 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rmessaou commented Jul 8, 2026 • edited by jdsika Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Rule patterns supported

Supporting generator changes carried on this branch

Review-hardening pass

How was this tested?

Areas of uncertainty

Checklist

AI Assistance

Uh oh!

jdsika commented Jul 10, 2026

Uh oh!

jdsika commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jdsika commented Jul 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rmessaou commented Jul 8, 2026 •

edited by jdsika

Loading

jdsika commented Jul 11, 2026 •

edited

Loading