chore: consolidated feature branch for ENVITED-X pipeline#14
Draft
jdsika wants to merge 17 commits into
Draft
Conversation
jdsika
force-pushed
the
feat/envited-x-pipeline
branch
9 times, most recently
from
a705c35 to
3d3a52a
Compare
jdsika
force-pushed
the
feat/envited-x-pipeline
branch
3 times, most recently
from
cab84ad to
b2b3dba
Compare
Add a --deterministic / --no-deterministic CLI flag (default off) to OWL, SHACL, JSON-LD Context, and JSON-LD generators that produces diff-stable output using Weisfeiler-Lehman structural hashing on top of the RDFC-1.0 canonicalization from upstream (linkml#3407). Three-phase hybrid pipeline (when --deterministic is set): 1. RDFC-1.0 canonicalization (upstream) produces sequential _:c14nN IDs 2. Weisfeiler-Lehman structural hashing replaces sequential IDs with content-based _:b<sha256> hashes that remain stable when unrelated triples are added/removed 3. rdflib re-serialization recovers idiomatic Turtle (inline blank nodes, collection syntax, filtered prefixes, preserved xsd:string) Without --deterministic, upstream's always-on RDFC-1.0 canonicalization is used directly (via canonicalize_rdf_graph). Additional features gated behind --deterministic: - Expression sorting (any_of/all_of/none_of/exactly_one_of) in owlgen - Collection sorting (sh:in, sh:ignoredProperties) in shaclgen - Permissible value sorting in owlgen and shaclgen - JSON-LD deterministic key ordering (deterministic_json) - JSON-LD context structured ordering (jsonldcontextgen) Rebased on top of upstream linkml#3407 (pyoxigraph RDFC-1.0). Refs: linkml#1847, linkml#3407 Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>
rdflib's Turtle serializer always emits a trailing double newline. Normalize to single newline in deterministic_turtle() and the rdflib fallback path in canonicalize_rdf_graph() for consistent file endings. Note: CLI print() still adds a newline after serialize()'s trailing newline. Callers capturing stdout should strip trailing blank lines (e.g. via sed). Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>
… names Add an opt-in --normalize-prefixes flag to OWL, SHACL, and JSON-LD Context generators that normalises non-standard prefix aliases to well-known names from a static prefix map (derived from rdflib 7.x defaults, cross-checked against prefix.cc consensus). Key design decisions: - Static frozen map (MappingProxyType) instead of runtime Graph().namespaces() lookup eliminates rdflib version dependency - Both http://schema.org/ and https://schema.org/ map to 'schema' - Shared normalize_graph_prefixes() helper used by OWL and SHACL - Two-phase graph normalisation: Phase 1 normalises schema-declared prefixes, Phase 2 cleans up runtime-injected bindings - Collision detection: skip with warning when standard prefix name is already user-declared for a different namespace - Phase 2 guard prevents overwriting HTTPS bindings with HTTP variants The flag defaults to off, preserving existing behaviour. Tests cover OWL, SHACL, and context generators with sdo->schema, dce->dc, http/https edge case, custom prefix preservation, flag-off backward compatibility, cross-generator consistency, prefix collision detection, schema1 regression prevention, Phase 2 HTTPS guard, empty schema edge case, and static map integrity. Signed-off-by: jdsika <carlo.van-driesten@bmw.de> Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>
Implement SHACL-SPARQL constraint generation for the boolean-guard pattern commonly used in conditional validation rules. When a LinkML class has rules: blocks with preconditions (value_presence: PRESENT) and postconditions (equals_string: true), the generator now emits sh:SPARQLConstraint nodes on the corresponding sh:NodeShape. Features: - New _add_rules() method translates recognised rule patterns to SPARQL - Boolean-guard pattern: if value present then flag must be true - Rule description mapped to sh:message on the constraint - Deactivated rules are skipped - Warnings emitted for bidirectional/open_world rule flags - New --emit-rules/--no-emit-rules CLI flag (default: enabled) - Full URI references in SPARQL (no PREFIX declarations needed) The generated SPARQL follows W3C SHACL Section 5 and uses the pre-bound \ variable per Section 5.3.1. Constraints are validated by pyshacl with advanced=True. Refs: linkml#2464 Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>
Python truthiness (`if s.maximum_cardinality:`) treats 0 as falsy, so `maximum_cardinality: 0`, `minimum_cardinality: 0` and `exact_cardinality: 0` emitted no constraint at all. `maximum_cardinality: 0` (SHACL `sh:maxCount 0`, "property must not appear") is the idiomatic way to suppress an inherited slot on a subclass via slot_usage, and owlgen already emits `owl:maxCardinality 0` for it -- so the SHACL and OWL output silently diverged. Use explicit `is not None` checks for minimum_cardinality, maximum_cardinality and exact_cardinality, matching the pattern already used in owlgen.py and docgen.py. Precedence: an explicit minimum_cardinality wins over the `required` fallback in the elif cascade, consistent with owlgen.py (which uses the same `if minimum_cardinality is not None ... elif required` order), so `required: true` + `minimum_cardinality: 0` yields `sh:minCount 0`. That combination is a schema-authoring contradiction (the metamodel documents minimum_cardinality as a multivalued-slot count); the explicit, more specific constraint is emitted. Tests cover maximum_cardinality: 0, exact_cardinality: 0, minimum_cardinality: 0, and the required + minimum_cardinality: 0 precedence case. Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>
The SHACL generator translated any_of branches by dispatching
solely on `any.range` (class, type, enum, or simple datatype).
If a branch specified `pattern:` — either alone or combined
with a range — the constraint was silently dropped, producing
an empty blank node `[ ]` (trivially satisfied) instead of the
intended `[ sh:pattern "..." ]`.
This is a problem for schemas that use pattern alternatives in
`any_of`, such as the SPDX license field where valid values are
either members of a fixed enum (SPDX identifiers), IRIs, or
custom identifiers matching the LicenseRef- pattern defined in
SPDX Specification v2.3 Annex D (ABNF: license-ref =
["DocumentRef-"(idstring)":"]"LicenseRef-"(idstring)).
The fix adds a single check after the range dispatch:
if any.pattern:
g.add((range_list[-1], SH.pattern, Literal(any.pattern)))
This correctly handles:
- Pattern-only branches (no range): node gets only sh:pattern
- Range + pattern branches: node gets both sh:datatype and sh:pattern
- Range-only branches (no pattern): unchanged behaviour
The test suite now includes a dedicated schema exercising all
three cases, with assertions on both the generated RDF triples
and pyshacl validation of conforming/non-conforming data.
Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>
Expose the existing JsonSchemaGenerator.include_null field on the gen-json-schema CLI. --no-include-null forbids explicit JSON null in optional slots so optionality is expressed only via absence from required (JSON Schema Validation 6.5.3), keeping the bare value type (6.1.1) -- needed for strict parity with reference schemas that forbid null. Default unchanged (include_null=True). Tested at the CLI surface via CliRunner over scalar, multivalued, and required slots; the standards rationale lives in the include_null field docstring. Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>
…nstraints For an inlined-as-dict slot whose range class has an identifier/key slot, render the key slot's string-applicable constraints onto JSON Schema propertyNames (draft-06+) instead of dropping them. In the inlined-dict form the mapping key is the identifier value, so the key slot's constraints constrain the keys. JSON object keys are always strings, so only pattern, enum (equals_string_in) and a string const (equals_string) are emitted; numeric minimum/maximum, numeric const (equals_number) and allOf are excluded -- a numeric const would otherwise reject every key. structured_pattern is honored when materialize_patterns is enabled, consistent with value patterns. Backward compatible: emitted only when a string-applicable key constraint applies. Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>
rmessaou
force-pushed
the
feat/envited-x-pipeline
branch
from
97e73d0 to
ef7ad85
Compare
3 tasks
Generalise the boolean-guard SHACL-SPARQL pattern to enum-valued targets. A rule whose precondition is `value_presence: PRESENT` on a value slot and whose postcondition is `equals_string` / `equals_string_in` on a target slot now emits an `sh:sparql` constraint requiring the target slot to be present and hold one of the allowed values. Each allowed value resolves to its enum `meaning` IRI, with a string-literal fallback. Motivating case (aiSim environment): - "if texture_sky_color is set, sky_model must be TextureSky" - "if overcast_sky_illuminance is set, sky_model must be OvercastSky or MeasuredOvercastSky" The existing boolean-guard (`equals_string: "true"`) and exclusive-value patterns are unchanged; boolean guard keeps priority over the new branch. Adds focused unit tests (enum IRI vs. literal fallback, single value vs. set membership, message emission, SPARQL syntax) plus pyshacl end-to-end validation for the new pattern. (cherry picked from commit ea5cf57)
Add a compositional fallback in _rule_to_sparql for rule-operator combinations outside the three named patterns (boolean guard, presence-implies-value, exclusive value). Tried only after the named patterns, so their output is unchanged. The fallback translates a conjunction of precondition slot conditions plus a single postcondition into one SELECT $this violation query; any unsupported operator makes the converter return None -- skip, never mis-translate. Supported combinations: - M1 conditional-required: equals_string / value_presence: PRESENT precondition + required: true postcondition (violation = FILTER NOT EXISTS on the target slot). - M2 conditional-absent: value_presence: ABSENT postcondition (violation = the forbidden slot is present). - M3 numeric threshold preconditions: minimum_value / maximum_value inclusive bounds on the trigger slot, rendered via _sparql_number. - M4 nested precondition: one hop into an inlined child object via range_expression.slot_conditions (e.g. sun_position.elevation <= 0); adds _member_conditions (shared inner-condition emitter) and _resolve_member_enum_ref, which resolves inner enum values against the container slot's range class (handles slot_usage-specialised enums). - M5 has_member list-membership: a multivalued slot must contain a member matching a nested range_expression (violation = FILTER NOT EXISTS over the members); reuses _member_conditions. Tests per converter: structural triple assertions, prepareQuery syntax validation, and pyshacl end-to-end (conforming instances pass, crafted violations fail) with advanced=True. Squashed from the five M1-M5 commits on feat/shacl-rule-converters (7566e30, fac681e, f8ea709, 25f8cd2, 4776977). Signed-off-by: Carlo van Driesten <carlo.van-driesten@vdl.digital>
…and SPARQL escaping
Review hardening for the SHACL-SPARQL rule converters. Three defects, each
with a regression test that fails before this change:
- Induced-slot parity: _slot_uri and _resolve_enum_value_ref resolved the
*base* slot, so a slot_usage override of slot_uri (or a narrowed enum range)
made the generated SPARQL query a property/enum the data never uses while
sh:path used the induced IRI. The constraint then silently never fired
(false negative). Both now resolve the induced slot for the class, matching
the sh:path logic in the main slot loop.
- Combined operators: a single precondition / member condition dispatched on
the first matching operator, so {minimum_value: X, maximum_value: Y} dropped
the lower bound and under-constrained the trigger (false positives). A shared
_scalar_filters helper now emits every recognised operator, and still returns
None -- skip, never mis-translate -- when none is recognised.
- SPARQL string escaping: an equals_string / permissible-value name containing
a double quote, backslash or newline produced invalid, unparseable SPARQL.
New _sparql_string_literal escapes per SPARQL 1.1 section 19.7.
Tests: 6 new regression tests (structural, prepareQuery syntax, and pyshacl
end-to-end) covering all three defects; full shaclgen suite green (111).
Signed-off-by: Carlo van Driesten <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
(cherry picked from commit e263def)
…ic bounds Resolve the adversarial-audit findings on the rule-converter stack. Every fix carries regression tests; 18 of the 19 new tests fail on the pre-fix source (the 19th locks in already-correct zero-member has_member semantics that had no coverage). - Operator exactness (audit A1/B2/C2): named patterns and the compositional fallback now require their conditions to set EXACTLY the operators they translate. A new _set_operator_fields helper enumerates the constraint-bearing fields actually set on a condition or class expression (metadata excluded; scalars never judged by truthiness, so minimum_value: 0 still counts). Previously a condition mixing a recognized operator with an unrecognized one (equals_string + pattern), expression-level any_of/all_of/none_of/exactly_one_of, co-set equals_string + equals_string_in, or value_presence: ABSENT combined with a bound was partially translated, silently dropping conjuncts and producing demonstrated false positives. All such rules are now skipped (skip, never mis-translate). - Boolean-guard range gate (audit A2): the boolean-guard pattern only dispatches when the target slot's induced range is boolean. An equals_string "true" postcondition on a string-range slot previously hijacked the boolean comparison and flagged conforming data; it now dispatches to presence-implies-value, which compares the string. - Nested inner-slot parity (audit B1/C1): _member_conditions resolves the container slot in the induced context of the outer class, requires its range to be a class, and resolves inner slot URIs and enum values against that range class. Previously inner slots resolved against the OUTER class: a slot_usage slot_uri override on the member class made sh:path and the SPARQL body diverge (constraint silently never fired), and an inner slot name colliding with an outer slot_usage override queried a predicate members never carry, making has_member's FILTER NOT EXISTS vacuously true (false positives on conforming data). The _resolve_member_enum_ref special case is subsumed by passing the range class to _resolve_enum_value_ref. - Numeric bound gate (audit B3/C3): _sparql_number returns None for anything but int / finite float (bool excluded), and callers skip the rule. The metamodel range of minimum_value/maximum_value is Anything: a string bound produced unparsable SPARQL that poisoned the entire shapes graph at validation time, and a YAML date parsed as arithmetic (2020-01-01 = 2018), silently never firing. Docstring corrected. - elseconditions warning (audit B4): rules with an else branch now log a warning that only the forward direction is enforced, consistent with the bidirectional/open_world warnings. - Test hygiene: the vacuous "<Manual> not in query" assertion now checks the full IRI form; new zero-member has_member end-to-end test. Signed-off-by: Carlo van Driesten <carlo.van-driesten@vdl.digital> - Alias-form and unknown rule keys (audit C4): _rule_slot resolves a rule condition key via the class's induced slots, then the underscored alias form (my_slot for a slot named "my slot"), then the base slot. _slot_uri returns None for names that resolve to no slot and every converter skips the rule -- previously an unknown or alias-form key fabricated a default_prefix predicate no shape uses, silently emitting a vacuous constraint (or an always-firing has_member).
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
…key constraints Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The single consolidated feature branch for the ENVITED-X pipeline. This
branch integrates every feature developed on this fork, each of which has its
own atomic, CI-green review PR. It is the branch downstream repositories pin
(
ontology-management-baseinstalls linkml from a SHA on this branch).Based on fork
main(mirroringlinkml/linkmlmainat7414ab98), itcarries, in order:
0ec311cb,d286fbe5--deterministicoutput (RDFC-1.0 + WL hashing + hybrid rdflib serialization), trailing-newline normalizationabed0245--normalize-prefixeswell-known prefix names3f68eb8d--emit-rules)985e6500sh:minCount/maxCount 0for zero cardinality4f757fbash:patterninsideany_of442b5132--include-null/--no-include-null(gen-json-schema)ef7ad851propertyNamesfrom inlined-dict key constraints43a45841a0a33fac48b7b4c0b7889d6eelseconditionswarning)e78014be,cbeb443d,57a4ecf7,29933b5f,6aa7702cSupersedes #18 (
feat/shacl-rule-converters), whose rule-converter commitsare replaced here by the audited, hardened stack (#19–#22 review lineage).
How was this tested?
test_shaclgen.py132 passed;test_jsonschemagen.py+test_deterministic_output.py+test_normalize_prefixes.py109 passed, 2 skipped, 4 xfailed,261 subtests — the features are green in combination.
across cherry-picks.
Areas of uncertainty
(e.g.
ontology-management-basepyproject.toml+ submodule) to the newtip in a coordinated change. The previously pinned
97e73d0fremainsreachable via
feat/shaclgen-presence-implies-value-pipeline— do notdelete that branch until the pin is bumped.
--deterministiccommit still carrieslinkml.utils.rdf_canonicalizealthough upstream
mainnow shipslinkml_runtime.utils.rdf_canonicalize(feat(generators): add --default-language flag for language-tagged literals linkml/linkml#3449); deduplicate when rebasing for upstream submission.
Checklist
AI Assistance
If you used AI tools while preparing this PR, you are still the author and responsible for understanding, verifying, and defending your submission. Please engage with reviewers personally rather than through your agent during feedback and revisions. See our AI Covenant for details.