Implement TypedDict extra_items= (PEP 728)#21712
Open
boogerlad wants to merge 11 commits into
Open
Conversation
Replace TypedDictType.is_closed with a first-class PEP 728 pseudo-item: extra_items: Type | None plus extra_items_readonly: bool. None means default-open; closed=True is stored as extra_items=UninhabitedType() per the PEP's closed=True == extra_items=Never equivalence, and is_closed remains available as a derived property so all existing readers are unchanged. This is a pure representation change with no user-visible behavior difference: item() resolves missing keys through the pseudo-item, every type visitor (translator, queries, traverser, expandtype, fixup, indirection, deps, astmerge, astdiff snapshots) now visits extra_items, constraint inference covers the pseudo-item, and both JSON and binary cache formats serialize it (CACHE_VERSION bumped). join/meet keep their boolean closed logic behind TODOs until non-Never extra_items becomes constructible. Groundwork for TypedDict extra_items= support (python#18176).
Accept the extra_items= class argument in both the class and functional TypedDict syntaxes, with any ReadOnly[] qualifier unwrapped into extra_items_readonly (Required[]/NotRequired[] are rejected, and combining closed= with extra_items= is an error, per the PEP). Inheritance treats the pseudo-item like a regular item: it is inherited as-is, may only be redeclared over a read-only base pseudo-item (with a narrower type), and new fields must be compatible with each base's pseudo-item (non-required and equivalent for mutable extra_items, assignable for read-only). Requiredness/closedness violations are reported in semantic analysis; checks that need subtyping are deferred to checker.check_typeddict_inheritance via TypedDictData, whose base entries now carry the base pseudo-item mapped through type arguments. Explicit closed=False errors if any base restricts extra items. The extra_items keyword value is a type expression, so class keyword analysis no longer rejects type variables in it. Assignability, join, and meet now handle the pseudo-item through the generalized item()/extra_item() machinery: read-only pseudo-items are covariant, mutable ones require equivalence (subsuming "a closed type must remain closed"), a join of incompatible pseudo-items widens to read-only, and a meet of incompatible mutable pseudo-items is uninhabited. Generic TypedDicts support extra_items referencing their type variables in both syntaxes. Use-site support (construction with extra keys, subscripting, plugin methods, Unpack kwargs, narrowing) and documentation are the next phase of python#18176.
Rename the extra_items_from field to merged_from: it holds the TypedDicts merged in with ** in the experimental inline TypedDict syntax and is unrelated to PEP 728 extra_items, which now lives on the same class. Also teach stubgen's functional-to-class TypedDict conversion about the extra_items= argument; previously such definitions degraded to "Incomplete" in generated stubs.
Per PEP 728's assignability rules, a TypedDict B is assignable to A even when B lacks one of A's keys, provided A's item is read-only, not required, and of top value type (ReadOnly[NotRequired[object]]): that is exactly the implicit pseudo-item of a default-open TypedDict. Previously this case was rejected unconditionally, which also made it inconsistent with the equivalent extra_items=ReadOnly[object] spelling, which is accepted through the pseudo-item comparison. Treat the missing key in a default-open subtype as ReadOnly[NotRequired[object]] instead, which only changes behavior when the supertype's item is exactly the top type.
Implement the use-site semantics for the extra_items pseudo-item: Construction: TypedDicts with a non-Never extra_items accept arbitrary extra keys in dict literals and constructor calls, with values checked against the extra_items type (including through the ReadOnly qualifier, which does not restrict construction). The synthesized constructor callable gains a **kwargs parameter of the extra_items type, so type variables in extra_items are inferred from extra keyword arguments. With --extra-checks, ** unpacking a TypedDict source is accepted when the source pseudo-item is provably absorbed by the callee's (closed sources always are; otherwise the source extra_items must be assignable to the callee's). Access and mutation: subscript reads of unknown keys have the extra_items type; writes and TypedDict method calls treat extra keys as non-required items, so get/pop/setdefault/del/update work on them, with read-only extra_items rejecting all mutation. Unpack: **kwargs: Unpack[TD] where TD has extra_items now accepts arbitrary keyword arguments of that type, making it equivalent to the corresponding explicit signature with typed **kwargs. ParamSpec P.kwargs synthesis maps a callable's **kwargs type to extra_items instead of dropping it, making the bound kwargs type more precise (resolves an old TODO). Also document closed= and extra_items= interactions in the TypedDict docs. Completes the core of python#18176; Mapping/dict assignability and values()/items() precision remain as follow-ups.
Implement the Mapping- and dict-interaction rules for TypedDicts with closed=True or extra_items= set, whose complete value types are statically known: Assignability: such a TypedDict is assignable to Mapping[str, VT] when all of its value types, including the extra_items pseudo-item, are assignable to VT. It is additionally assignable to dict[str, VT] (and MutableMapping[str, VT]) when every item is non-required, mutable, and consistent with VT; dict[str, VT] remains not assignable to TypedDicts. Overlap checks for Mapping types now consider the pseudo-item as well. Precise views: values() and items() return views with the union of all value types instead of object (via new default-plugin hooks), and subscript reads accept arbitrary str keys, yielding that union. dict-like TypedDicts: when every item is non-required, mutable, and consistent with extra_items (tracked by the new subtypes.typed_dict_dict_value_type helper), operations that are normally unsafe become available with dict[str, VT] semantics: clear(), popitem(), and __setitem__/__delitem__ with arbitrary str keys. The _TypedDict test fixture gains an items() method mirroring typeshed's shape, and the dict fixture gains clear()/popitem(). This completes PEP 728 support (python#18176).
Two behaviors that already work but had no test pinning them: extra_items may reference the TypedDict being defined (the pseudo-item is checked recursively during construction), and a callable taking **kwargs: Unpack[TD] with extra_items satisfies a Protocol declaring the equivalent explicit signature, in both directions (PEP 728's Unpack equivalence, at the protocol-member level).
…ll sites Previously a **td actual mapped only its named items to formals, so the extra keys a PEP 728 TypedDict may contain were silently dropped: passing a TypedDict with extra_items=str to a function taking **kwargs: int went unflagged. This leniency was load-bearing for default-open TypedDicts, whose possible extras are untyped, but for a TypedDict with extra_items= the extras type is declared and checking it has no false positives (pyright already flags this). Map the pseudo-item of a non-closed extra_items TypedDict to the callee's **kwargs formal in map_actuals_to_formals, and have ArgTypeExpander yield the extra_items type once the named items are exhausted. When the callee has no **kwargs the extras stay ignored, since they may all be absent (matching pyright). This also feeds extra_items into generic **kwargs: T inference.
The redeclaration machinery resolved the requiredness of a read-only field inherited from multiple sources with any(), silently promoting it to required. This masked two invalid redeclarations and one base-class conflict (typing conformance suite, typeddicts_readonly_inheritance lines 94/106/132): - redeclaring a mutable field as read-only (unsound: the subclass would no longer be assignable to the base, which permits mutation); - redeclaring a ReadOnly[Required[...]] field as non-required; - inheriting from two bases whose read-only fields disagree on requiredness, when the base declaring it non-required comes first in MRO (the reverse order is fine: a required redeclaration may strengthen a non-required read-only item). Take requiredness from the primary source like the mutable path does, so verify_requiredness_compatibility sees the mismatches, and add the missing mutable-as-read-only check; the analogous rule already existed for the extra_items pseudo-item. The 'not in base class' message variant covers read-only fields, where 'can be deleted' would be wrong. testTypedDictReadOnlyMixinTotality asserted the silent promotion for C(B, A); both the conformance suite and pyright reject that order and accept D(A, B), so the test now expects the error. This makes typeddicts_readonly_inheritance.py fully conformant.
Extract TypedDictType.value_types_with_extra() to replace the three
copies of 'all value types plus the non-Never pseudo-item' in
subtypes.py, checkexpr.py, and plugins/default.py.
Also generalize TypedDict constraint inference from zip() to zipall(),
so a key named on only one side is matched against the other side's
extra_items pseudo-item. Previously such keys produced no constraints,
so e.g. passing a TypedDict with extra_items=int to a parameter of
TypedDict type {x: NotRequired[T]} left T unsolved (inferred Never with
a spurious arg-type error); it now infers T=int. Keys missing from a
default-open side still infer nothing.
This comment has been minimized.
This comment has been minimized.
Author
|
Hm, looking into the parallel tests with py314-ubuntu now |
The native parser (forced on with --mypy-num-workers>0) reports a repeated keyword argument as a blocking syntax error, which aborts semantic analysis and suppresses the other errors in the file. Split the repeated-keyword case out of testTypedDictExtraItemsInvalidArgument into a _no_parallel test, mirroring testTypedDictWithDuplicateKeywordArguments_no_parallel.
Contributor
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #18176
I'm going to be completely honest. This is 100% AI generated. I've read the contributing guidelines:
So of course, feel free to close it. However, at the very least the test cases are useful, and I know mypy will eventually have full PEP 728 support, so in the meantime, I made this to prototype the type system in a HTML-as-data library. It works well.
Feel free to cherry pick individual commits (57cab29 is independent and pre-existing issue on master), claim authorship, reuse the code in some way, or do nothing with it. Consider this as a donation in Fable xhigh tokens. I've attached the session transcripts for full transparency. It can even be resumed if desired. Summarized "thinking" traces are included.
Archive.zip
3f014f7a-b87f-4fdd-b1ae-2548b61bf640.jsonlis the implementation session.262f1ec4-275b-466f-98df-6633f95c124e.jsonlis the review session.Basically, the prompts are:
and