From a1ac47c9adab7e81ec78b09fd08b16f1578232dd Mon Sep 17 00:00:00 2001 From: igerber Date: Tue, 7 Jul 2026 17:46:37 -0400 Subject: [PATCH] test(se-audit): G2 hetero + cluster machine-precision fixest locks on unbalanced heteroskedastic DGP MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds two appended scenarios to the fixest golden (error sd varying by arm/period, ~15% rows dropped; the original balanced scenarios' RNG draws precede them and reproduce value-identically). On the plain-OLS DiD path, hetero (HC1) no longer collapses to iid and is locked at machine precision — and the CR1 cluster SE matches fixest EXACTLY on plain OLS (balanced + unbalanced), so the former DiD cluster band-pin is tightened to a machine lock: the documented ~0.25% fixest-CR1 DOF-convention deviation is absorbed-FE-only. The TWFE cluster band is retained and re-scoped to that documented non-nested-FE ssc deviation; TWFE hetero has no public unclustered surface (auto-cluster-at-unit), so its scenario locks iid — pinning the D4 full-K rescale on an UNBALANCED panel for the first time. Required golden blocks assert (not skip) when missing. Closes SE-audit G2(a) in the TODO. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 13 +++ TODO.md | 2 +- .../R/generate_fixest_did_twfe_golden.R | 98 ++++++++++++++-- benchmarks/data/fixest_did_twfe_golden.json | 67 +++++++++++ tests/test_fixest_did_twfe_parity.py | 110 +++++++++++++++--- 5 files changed, 264 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d04bde5c..c16e65b8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -49,6 +49,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `TestHonestFLCIParityR`); the M=0 result and all existing behaviour are unchanged. ### Testing +- **fixest hetero + cluster SE machine-precision locks on an unbalanced, heteroskedastic + DGP (SE-audit G2 completion).** The committed `fixest_did_twfe_golden.json` gains two + appended scenarios (error sd varying by arm/period, ~15% rows dropped; the original + balanced scenarios' RNG draws precede them and reproduce value-identically): on the + plain-OLS DiD path, `hetero` (HC1) no longer collapses to iid and is locked against + `fixest` at machine precision — and the cluster-robust CR1 SE turns out to match fixest + **exactly** on plain OLS (balanced and unbalanced), so the former ~0.5% DiD cluster + band-pin is tightened to a machine-precision lock: the documented ~0.25% fixest-CR1 + DOF-convention deviation is an absorbed-FE (within-transform) phenomenon only. The TWFE + cluster band-pin is retained and re-scoped to that documented non-nested-FE ssc + deviation (~0.3% unbalanced); TWFE `hetero` has no public unclustered surface + (auto-cluster-at-unit convention), so its scenario locks iid — which also pins the D4 + full-K rescale on an UNBALANCED panel for the first time. - **`CallawaySantAnna` ipw R-parity yardsticks folded into the golden fixture + no-covariate ipw structural-parity decision recorded.** `csdid_golden_values.json` regenerated (R 4.5.2, did 2.5.1, DRDID 1.3.0): all pre-existing data and result blocks reproduced byte-identically; diff --git a/TODO.md b/TODO.md index 2927bb44..7d091d29 100644 --- a/TODO.md +++ b/TODO.md @@ -62,7 +62,7 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m | Issue | Location | Origin | Effort | Priority | |-------|----------|--------|--------|----------| -| SE-audit CI-lock — remaining "fiddly bits" after the second coverage batch landed (that batch pinned C2 `dof_hc2_bm`/`dof_per_coef` via CI-inversion, C3 LOO `df`, C4 estimatr HC1/CR1 intercept SE, C5 Yatchew `p`/`sigma2_lin`/`sigma2_diff`, and the G2 fixest cluster-SE band). Still deferred, each needing a golden regeneration, new computation, or a documented-deviation call: **(a) G2 machine-precision hetero/cluster SE** — needs an unbalanced/heteroskedastic-DGP regen of `fixest_did_twfe_golden.json` (hetero collapses to iid on the current balanced design; the cluster SE is the documented ~0.25% fixest-CR1 DOF-convention deviation, currently band-pinned only); **(b) PlaceboTests `boundary_gap`** — a permutation randomization-inference margin NOT computed anywhere in code (a new feature + result field, not a coverage lock); **(c) StackedDiD intercept SEs** (`se_cr1/cr2_intercept`, C1) — MEASURED to diverge ~0.3% from R: a nuisance-parameter reference-cell/parameterization gap, NOT machine-precision lockable (the event-study interaction SEs already match ~2e-13; surfacing it would add an unasserted, R-divergent public field); **(d) estimatr `classical` intercept SE** — same documented `O(1/n)` projection/DOF deviation as the slope (reference-only, excluded from parity). Plus the tolerance-tightenings C6-C8. | `benchmarks/R/generate_fixest_did_twfe_golden.R`, `tests/test_fixest_did_twfe_parity.py`, `tests/test_methodology_stacked_did.py`, `tests/test_methodology_placebo.py` | SE-audit | Mid | Low | +| SE-audit CI-lock — remaining "fiddly bits" after the second coverage batch landed (that batch pinned C2 `dof_hc2_bm`/`dof_per_coef` via CI-inversion, C3 LOO `df`, C4 estimatr HC1/CR1 intercept SE, C5 Yatchew `p`/`sigma2_lin`/`sigma2_diff`, and the G2 fixest cluster-SE band). Still deferred, each needing a golden regeneration, new computation, or a documented-deviation call — **(a) G2 hetero/cluster is DONE (2026-07-07)**: the unbalanced/heteroskedastic-DGP regen landed, the DiD-path hetero AND cluster CR1 SEs are machine-precision-locked (the plain-OLS CR1 matches fixest exactly — the DOF-convention deviation is absorbed-FE-only), and the TWFE cluster band-pin is retained for the documented non-nested-FE ssc deviation (tracked under "Needs external reference"); TWFE has no public unclustered-hetero surface (auto-cluster convention). Remaining: **(b) PlaceboTests `boundary_gap`** — a permutation randomization-inference margin NOT computed anywhere in code (a new feature + result field, not a coverage lock); **(c) StackedDiD intercept SEs** (`se_cr1/cr2_intercept`, C1) — MEASURED to diverge ~0.3% from R: a nuisance-parameter reference-cell/parameterization gap, NOT machine-precision lockable (the event-study interaction SEs already match ~2e-13; surfacing it would add an unasserted, R-divergent public field); **(d) estimatr `classical` intercept SE** — same documented `O(1/n)` projection/DOF deviation as the slope (reference-only, excluded from parity). Plus the tolerance-tightenings C6-C8. | `benchmarks/R/generate_fixest_did_twfe_golden.R`, `tests/test_fixest_did_twfe_parity.py`, `tests/test_methodology_stacked_did.py`, `tests/test_methodology_placebo.py` | SE-audit | Mid | Low | | Render `docs/methodology/REPORTING.md` and `REGISTRY.md` as in-site Sphinx pages so cross-refs can use `:doc:` instead of off-site `blob/main` URLs (stable-docs readers can otherwise land on a different revision than their package version). Two paths: (a) add `myst-parser` to `conf.py` + docs extras and link with `:doc:`, or (b) convert both to `.rst`. **Note:** REGISTRY.md is ~4.5k lines of LaTeX-heavy markdown — high risk under the `-W` (warnings-as-errors) Sphinx build; budget multiple rounds. | `docs/conf.py`, `docs/api/business_report.rst`, `docs/api/diagnostic_report.rst`, tutorials 18 & 19 | follow-up | Mid | Low | | `ImputationDiD` covariate-path variance lacks a dedicated parity anchor — only the no-covariate staggered panel is R-parity'd, though the covariate path shares the same validated projection code. Add a small dense-design **hand-calc** for the covariate projection (no external tooling), or a covariate (time-varying X) R `didimputation` golden asserting overall/ES SE parity (the golden variant needs local R). | `tests/test_methodology_imputation.py`, `benchmarks/R/generate_didimputation_golden.R` | imputation-validation | Mid | Low | | Add true half-sample BRR replicate-weight regressions per estimator family (current tests use Fay-like 0.5/1.5 perturbations; `test_survey_phase6.py` covers true BRR at the helper level). | `tests/test_replicate_weight_expansion.py` | #253 | Mid | Low | diff --git a/benchmarks/R/generate_fixest_did_twfe_golden.R b/benchmarks/R/generate_fixest_did_twfe_golden.R index 7dd5e87b..712c17ec 100644 --- a/benchmarks/R/generate_fixest_did_twfe_golden.R +++ b/benchmarks/R/generate_fixest_did_twfe_golden.R @@ -7,12 +7,16 @@ # panels and fixest's feols() ATT + SE so tests can assert machine-precision SE # parity WITHOUT R at test time. # -# Scope (this golden): the classical / iid SE, which Python matches to machine -# precision on both the 2x2 DiD path and the within-transform TWFE path (the -# latter also locks the SE-audit D4 full-K rescale). The cluster-robust ATT is -# stored too; its SE carries the documented CR1 small-sample DOF-convention -# difference vs fixest and is left to a follow-up. (`hetero`/HC1 collapses to iid -# on these balanced 2-group designs, so it is not a distinct target here.) +# Scope (this golden): scenarios 1-2 are the original balanced designs — the +# classical / iid SE locks (the TWFE one also pins the SE-audit D4 full-K +# rescale) plus the cluster blocks. Scenarios 3-4 (G2 completion, 2026-07) are +# heteroskedastic + unbalanced so `hetero` (HC1) is a distinct target: the +# plain-OLS DiD path locks hetero AND cluster CR1 at machine precision (the +# CR1 DOF-convention difference vs fixest is absorbed-FE-only); the TWFE +# cluster SE stays band-pinned for that documented non-nested-FE ssc +# deviation, and TWFE hetero has no public unclustered Python surface +# (auto-cluster-at-unit convention), so scenario 4 locks iid on an +# UNBALANCED panel. # # Regenerate: Rscript benchmarks/R/generate_fixest_did_twfe_golden.R # Output: benchmarks/data/fixest_did_twfe_golden.json @@ -97,6 +101,84 @@ twfe_golden <- list( cluster_unit = fit_att(twfe_m, ~unit) ) +# --------------------------------------------------------------------------- +# Scenario 3: heteroskedastic + unbalanced 2x2 DiD (SE-audit G2 hetero lock). +# Error sd depends on treatment arm and period (so HC1 'hetero' does NOT +# collapse to iid) and ~15% of rows are dropped deterministically-by-draw +# (unbalanced groups). Appended AFTER scenarios 1-2 so their RNG draws (and +# the committed scenario 1-2 golden values) are unchanged on regeneration. +# --------------------------------------------------------------------------- +n_units_h <- 120 +did_h_rows <- list() +i <- 1 +for (unit in 0:(n_units_h - 1)) { + is_treated <- as.integer(unit < 45) # unequal arms: 45 treated / 75 control + for (period in c(0, 1)) { + sd_it <- 0.5 + 1.5 * is_treated + 0.8 * period # heteroskedastic + y <- 10.0 + period * 2.0 + if (is_treated == 1 && period == 1) y <- y + 3.0 + y <- y + rnorm(1, 0, sd_it) + keep <- runif(1) > 0.15 # unbalanced: drop ~15% + if (keep) { + did_h_rows[[i]] <- data.frame(unit = unit, outcome = y, treated = is_treated, post = period) + i <- i + 1 + } + } +} +did_h <- do.call(rbind, did_h_rows) +did_h_m <- feols(outcome ~ treated * post, data = did_h) + +did_hetero_golden <- list( + data = list(unit = did_h$unit, outcome = did_h$outcome, treated = did_h$treated, post = did_h$post), + n_obs = unbox(nrow(did_h)), + iid = fit_att(did_h_m, "iid"), + hetero = fit_att(did_h_m, "hetero"), + cluster_unit = fit_att(did_h_m, ~unit) +) + +# --------------------------------------------------------------------------- +# Scenario 4: heteroskedastic + unbalanced TWFE. The fixest `hetero` block is +# stored for reference only — Python's TwoWayFixedEffects auto-clusters at +# unit on hc1 (no public unclustered-hetero surface), so the public locks are +# the unbalanced iid/full-K rescale plus the clustered ATT (exact) and SE +# (band, documented non-nested-FE ssc deviation). +# --------------------------------------------------------------------------- +n_units_th <- 40 +n_periods_th <- 5 +twfe_h_rows <- list() +i <- 1 +for (unit in 0:(n_units_th - 1)) { + is_treated <- as.integer(unit < 15) # unequal arms + unit_effect <- unit * 0.2 + for (period in 0:(n_periods_th - 1)) { + post <- as.integer(period >= 3) + sd_it <- 0.4 + 1.2 * is_treated + 0.3 * post + y <- 5.0 + unit_effect + period * 1.5 + if (is_treated == 1 && post == 1) y <- y + 2.5 + y <- y + rnorm(1, 0, sd_it) + keep <- runif(1) > 0.12 + if (keep) { + twfe_h_rows[[i]] <- data.frame( + unit = unit, period = period, outcome = y, treated = is_treated, post = post + ) + i <- i + 1 + } + } +} +twfe_h <- do.call(rbind, twfe_h_rows) +twfe_h_m <- feols(outcome ~ treated:post | unit + post, data = twfe_h) + +twfe_hetero_golden <- list( + data = list( + unit = twfe_h$unit, period = twfe_h$period, outcome = twfe_h$outcome, + treated = twfe_h$treated, post = twfe_h$post + ), + n_obs = unbox(nrow(twfe_h)), + iid = fit_att(twfe_h_m, "iid"), + hetero = fit_att(twfe_h_m, "hetero"), + cluster_unit = fit_att(twfe_h_m, ~unit) +) + # --------------------------------------------------------------------------- golden <- list( meta = list( @@ -109,7 +191,9 @@ golden <- list( )) ), did = did_golden, - twfe = twfe_golden + twfe = twfe_golden, + did_hetero = did_hetero_golden, + twfe_hetero = twfe_hetero_golden ) out <- "benchmarks/data/fixest_did_twfe_golden.json" diff --git a/benchmarks/data/fixest_did_twfe_golden.json b/benchmarks/data/fixest_did_twfe_golden.json index 949abfeb..d1dbe362 100644 --- a/benchmarks/data/fixest_did_twfe_golden.json +++ b/benchmarks/data/fixest_did_twfe_golden.json @@ -55,5 +55,72 @@ "ci_lower": 1.9861794154923553, "ci_upper": 3.1561311339780804 } + }, + "did_hetero": { + "data": { + "unit": [0, 0, 1, 1, 2, 2, 3, 3, 4, 5, 5, 6, 7, 8, 8, 9, 9, 10, 10, 11, 11, 12, 12, 13, 13, 14, 15, 15, 16, 16, 17, 17, 18, 19, 20, 21, 22, 22, 23, 23, 24, 24, 25, 25, 26, 26, 27, 27, 28, 28, 29, 29, 30, 31, 31, 32, 32, 33, 33, 34, 34, 35, 35, 36, 36, 37, 38, 38, 39, 39, 40, 40, 41, 41, 42, 42, 43, 43, 44, 44, 45, 46, 46, 47, 47, 48, 48, 49, 49, 50, 50, 51, 51, 52, 52, 53, 53, 54, 54, 55, 55, 56, 56, 57, 57, 58, 58, 59, 59, 60, 60, 61, 61, 62, 62, 63, 63, 64, 64, 65, 66, 66, 67, 67, 68, 68, 69, 69, 70, 70, 71, 71, 72, 72, 73, 73, 74, 75, 75, 76, 76, 77, 77, 78, 79, 79, 80, 81, 82, 82, 83, 83, 84, 84, 85, 85, 86, 88, 88, 89, 89, 90, 90, 91, 91, 92, 92, 93, 93, 94, 94, 95, 95, 96, 97, 97, 98, 98, 99, 99, 100, 101, 101, 102, 103, 103, 104, 104, 105, 106, 107, 107, 108, 108, 109, 109, 110, 111, 111, 112, 112, 113, 113, 114, 115, 115, 116, 116, 117, 117, 118, 119, 119], + "outcome": [6.7267418053608399, 15.604635424548732, 8.6752855760876582, 16.779054859168163, 10.15338556268644, 20.138191522574857, 10.043372543889575, 12.141230756341606, 7.2071847430164571, 11.605781684439133, 17.766168169300794, 10.802622752952701, 10.411020943718876, 11.43556199751492, 20.515294320024964, 10.069430169108179, 12.944259544598921, 7.9145536941482071, 15.510323800762315, 12.921881723655812, 13.473162098347327, 10.991813560881209, 15.98536733070183, 7.4991211637716102, 16.723654205162109, 12.168155875236259, 12.671042946412271, 12.532079940819422, 11.778491376537543, 12.570518594878573, 7.5263542775914249, 15.17571296988061, 17.441434577367399, 16.590142123753701, 7.2627968792809021, 12.049915625666138, 8.1086227594625235, 12.231878359999225, 9.3691153671872769, 14.988753826082005, 11.926835378472823, 9.7104591980296657, 6.3859106808725139, 16.343758075965312, 10.329589582940631, 18.292247749353194, 12.272099302979182, 17.468465529271075, 7.7945478050572481, 18.400267567445031, 10.682623121251265, 16.052624857879497, 15.760654383924773, 7.2584813323541812, 14.62466429869834, 8.8454779049833423, 13.689015819121931, 11.122565427301309, 17.069724790859166, 9.8842429625203518, 17.91570640547355, 9.9295591434922468, 20.780858249320442, 10.934345361632854, 14.754180439580638, 8.9339934259501117, 10.871866680082322, 17.291224352196039, 13.361787675470126, 19.559355386861572, 9.4169087914054117, 16.666168898178348, 8.1965646486378887, 18.361694777460389, 5.7010081101485657, 16.973090730943305, 8.9610556887810908, 19.977651390508527, 10.176462707627365, 15.671507481639026, 12.79565648957119, 9.7643475067694343, 11.323177648027778, 9.5975876946856893, 11.794052677612976, 9.6725077748661228, 14.177792950775569, 10.950412815618359, 11.926407453806807, 10.230882693123981, 11.032660499587141, 9.6256147293942753, 15.091760233367072, 9.3614322822334763, 11.574456145576718, 9.535185272189949, 12.031374842259698, 9.9578971491163237, 10.402400662792349, 10.431653771133343, 10.144564680152067, 11.24845925340362, 12.140551691295929, 9.8812212716576937, 9.966818894370606, 10.244467133858539, 8.9971702101060611, 10.471577576067567, 11.129827611435292, 10.109912710167826, 12.679965530779324, 10.234695784564996, 11.143589308108862, 9.8130852900438015, 9.5933985821530676, 10.546329079628615, 12.697597242000976, 10.435308534611922, 13.826803418971032, 11.778363310463384, 10.252294934485969, 12.848847579714413, 9.8753670641578495, 10.620115652448774, 10.244084942658429, 12.740716411184094, 9.1889383615604654, 8.9472321418691045, 9.6297075154650553, 11.391829005592287, 10.474019801275855, 11.633911265519398, 10.039195784101642, 10.686639080945788, 10.069255325093589, 13.036849541127498, 9.578793883678653, 10.325214935026889, 10.846278822339469, 9.6404562914753242, 12.167976821065114, 9.6801282570421918, 12.64758487015059, 13.871160650190543, 9.0279273180993957, 11.311523573165694, 11.891780421375037, 14.813732178964035, 8.9131307817551022, 13.136754201129639, 10.138536351311039, 12.27989847127377, 9.7341541890440819, 12.17946304458567, 10.402534834707705, 11.713670437745138, 10.003091844797989, 9.422011881618559, 10.268721926820591, 9.9609758793997774, 11.275783568191541, 10.102834805556796, 11.814640132441816, 9.6361592817718371, 13.209874424090732, 10.155015874683416, 9.762521754060721, 10.256745292246974, 13.541488285843634, 10.470763703404664, 9.5322512130453525, 10.386188450936205, 12.289008703117814, 12.889342678587678, 10.120152086945412, 8.8458157820573096, 9.8131698899303963, 15.312987375276755, 10.007604682114852, 14.007100454957797, 8.8658726316545327, 10.375121788033205, 13.401602315614506, 11.436466237526089, 9.8430675380335106, 12.928049541920757, 9.7235833759964407, 10.552382479947189, 10.005980524108526, 10.50793742441228, 9.8711273159859942, 13.040176982992088, 9.5157795117361381, 13.5982034897932, 9.5157467340318167, 11.186785961789116, 11.399839430776794, 10.575892492333347, 13.658797361719015, 9.1283260925905143, 12.399767431394581, 9.7429972295316958, 12.030783622838065, 11.413578488726037, 9.6556894073906125, 11.731302989253376, 9.2185329099956821, 11.888590864048737, 10.489016234067845, 12.670719527690672, 10.211165165210293, 9.8675288072691369, 11.891137958394099], + "treated": [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], + "post": [0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1] + }, + "n_obs": 213, + "iid": { + "att": 4.4172208913011985, + "se": 0.46637796741333454, + "t_stat": 9.4713326956682096, + "p_value": 6.1682246041857892e-18, + "ci_lower": 3.4978129450242617, + "ci_upper": 5.3366288375781359 + }, + "hetero": { + "att": 4.4172208913011985, + "se": 0.54491557314193617, + "t_stat": 8.1062482135202778, + "p_value": 4.3485432823401043e-14, + "ci_lower": 3.3429855221317775, + "ci_upper": 5.4914562604706196 + }, + "cluster_unit": { + "att": 4.4172208913011985, + "se": 0.55898948172024987, + "t_stat": 7.9021538611201043, + "p_value": 1.6059825457968944e-12, + "ci_lower": 3.3102695330151053, + "ci_upper": 5.5241722495872914 + } + }, + "twfe_hetero": { + "data": { + "unit": [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 7, 7, 7, 7, 8, 8, 8, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 11, 11, 11, 12, 12, 12, 12, 13, 13, 13, 13, 13, 14, 14, 14, 15, 15, 15, 15, 16, 16, 16, 16, 17, 17, 17, 17, 18, 18, 18, 18, 18, 19, 19, 19, 19, 20, 20, 20, 21, 21, 21, 22, 22, 22, 23, 23, 23, 24, 24, 24, 24, 24, 25, 25, 25, 25, 26, 26, 26, 26, 27, 27, 27, 27, 27, 28, 28, 28, 28, 28, 29, 29, 29, 30, 30, 30, 31, 31, 31, 31, 31, 32, 32, 32, 32, 32, 33, 33, 33, 33, 34, 34, 34, 34, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36, 37, 37, 37, 37, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39], + "period": [0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 1, 2, 3, 0, 1, 3, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 3, 4, 1, 2, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 3, 4, 0, 2, 3, 4, 0, 1, 2, 3, 4, 0, 3, 4, 0, 1, 2, 3, 0, 1, 2, 3, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 4, 1, 3, 4, 0, 3, 4, 0, 1, 2, 0, 1, 2, 0, 1, 2, 3, 4, 0, 1, 2, 3, 0, 1, 2, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 2, 3, 4, 1, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 4, 0, 1, 2, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4], + "outcome": [5.520138624458311, 8.010719422422552, 11.020576830933432, 14.073091037026884, 10.501654352261189, 2.6651526650933577, 5.5199515190539721, 8.7845996900975365, 13.320866993795057, 12.748294619300513, 7.1384989059540764, 6.9717152923277101, 11.366882161920037, 13.771555644054237, 12.219770494513632, 8.0370254490090307, 6.6446488927200402, 12.003352247712231, 5.9747266010154911, 9.9018641802989755, 13.761220132375531, 6.2894227189308776, 7.8903740670806659, 9.4962859041357834, 11.372787756427817, 13.908601537203952, 3.1642266564387822, 6.0411367504675457, 10.273112762384967, 17.142147413012196, 13.472808226708908, 7.2253597317829605, 9.5166045255144649, 10.983596672870714, 10.362808480546608, 10.656466707378396, 8.8622085780362028, 15.860980912515473, 8.1662508675892926, 5.6150919943558266, 9.8425531797509347, 16.944389413810462, 16.656949204137327, 7.3593808494189901, 10.267070796359095, 10.455727363151691, 12.623325457202778, 12.778311432156379, 5.0266732815465218, 15.237992729108807, 17.492556182547705, 6.1355376783854734, 10.42682476542725, 13.689343135951724, 14.700304402955515, 12.959600776940057, 5.2556642692803184, 10.050610835110945, 16.432884535907611, 17.67467704136175, 6.1048793860227901, 13.138470718758606, 13.599330184243382, 8.3008060435995326, 9.701629165052589, 11.254076524690053, 11.74094496497243, 7.8697164496450398, 9.5060849944913812, 11.841019956570477, 11.792472434539516, 9.6520391521126694, 11.100396347944065, 12.699852069697471, 14.272967422704843, 9.08251915169099, 9.8599993544631364, 11.421252074443846, 12.899238490989312, 14.91850961052997, 8.6356866491423752, 10.941586136250429, 12.051718427622403, 15.777168618388295, 10.315938076507136, 13.590008260136189, 14.705680870136174, 9.3038765838512614, 13.85208212764676, 15.693068956150123, 9.9072522612784137, 10.530560541056808, 12.661595860965651, 9.4629863499959868, 11.065169142347568, 12.113523202785135, 8.9201761520533349, 10.657467091294484, 13.467027221990632, 13.32283505325934, 15.79502714034146, 10.445782092705523, 11.938390662139801, 12.392449946339445, 13.732702317639705, 10.412323968017574, 12.028358196759852, 13.27391797127737, 16.070151242305268, 10.149650646945986, 11.791150219830509, 14.357379807853514, 15.227313694760833, 17.544840854550067, 10.875123476008822, 12.218764059938694, 12.722383819724508, 14.640902207267382, 16.857005196349228, 14.153870803302956, 15.737257922266778, 17.778659867006837, 11.928811210304318, 16.063446439462044, 17.137115568071213, 10.883696714245783, 12.764872273195408, 13.564428814084645, 16.354100598959807, 18.242596448794487, 11.292847316852905, 12.632011804045783, 14.497263545510977, 15.767278870491815, 17.64364073127928, 11.745904808314156, 12.745783298009624, 14.411086620063548, 16.995545393303956, 11.541598976489023, 13.747438985986511, 14.747176453073122, 18.481094011883513, 10.761742442805387, 13.864347055023043, 15.0771823734141, 16.369392584090964, 16.842320566344974, 11.785894783784386, 13.575902602271825, 14.519709622628653, 17.29111060564254, 18.033620846218017, 12.417754680869258, 13.329936290756232, 16.801269495776474, 18.552396303399096, 13.072760837979583, 13.518379072150481, 14.983210056382473, 16.66436456032438, 19.096087886541707, 12.968776816321052, 14.313750624635281, 15.990767480214831, 16.725323861755797, 19.125976237425515], + "treated": [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], + "post": [0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1] + }, + "n_obs": 167, + "iid": { + "att": 2.2205379446299731, + "se": 0.58888995059447868, + "t_stat": 3.7707180134223068, + "p_value": 0.00024989668471548737, + "ci_lower": 1.0550516456110601, + "ci_upper": 3.3860242436488859 + }, + "hetero": { + "att": 2.2205379446299731, + "se": 0.62517777596617641, + "t_stat": 3.5518504175844368, + "p_value": 0.00054027306568574717, + "ci_lower": 0.98323353553937309, + "ci_upper": 3.4578423537205731 + }, + "cluster_unit": { + "att": 2.2205379446299731, + "se": 0.63107014693636854, + "t_stat": 3.5186864018365176, + "p_value": 0.0011193849930118225, + "ci_lower": 0.94407808851551622, + "ci_upper": 3.4969978007444302 + } } } diff --git a/tests/test_fixest_did_twfe_parity.py b/tests/test_fixest_did_twfe_parity.py index a2a074fa..8382ae08 100644 --- a/tests/test_fixest_did_twfe_parity.py +++ b/tests/test_fixest_did_twfe_parity.py @@ -76,24 +76,98 @@ def test_twfe_classical_se_matches_fixest_iid(self): np.testing.assert_allclose(res.att, exp["att"], atol=1e-10, rtol=0) np.testing.assert_allclose(res.se, exp["se"], atol=1e-10, rtol=0) - def test_cluster_att_matches_fixest(self): - """The cluster-robust ATT matches fixest exactly (the SE carries the - documented CR1 DOF-convention difference and is deferred).""" + def test_did_cluster_se_matches_fixest_exactly(self): + """Plain-OLS CR1 == fixest CR1 to machine precision on the DiD path + (balanced AND heteroskedastic/unbalanced scenarios). With no absorbed + FE, both apply the same (G/(G-1))*((n-1)/(n-k)) small-sample factor — + the documented fixest-CR1 DOF-convention deviation is an absorbed-FE + (within-transform) phenomenon only, so the former DiD band-pin is + tightened to a machine-precision lock (SE-audit G2).""" golden = _load_golden() - for key, est in ( - ("did", DifferenceInDifferences(vcov_type="hc1", cluster="unit")), - ("twfe", TwoWayFixedEffects(vcov_type="hc1", cluster="unit")), - ): + for key in ("did", "did_hetero"): + assert key in golden, f"required golden block {key!r} missing — regenerate the fixture" df = _build_df(golden[key]) - res = est.fit(df, outcome="outcome", treatment="treated", time="post", unit="unit") - np.testing.assert_allclose( - res.att, golden[key]["cluster_unit"]["att"], atol=1e-10, rtol=0 + res = DifferenceInDifferences(vcov_type="hc1", cluster="unit").fit( + df, outcome="outcome", treatment="treated", time="post", unit="unit" ) - # SE-audit G2: ratio-band pin on the cluster-robust SE. The exact - # value carries the documented ~0.25% fixest-CR1 small-sample - # DOF-convention deviation (SE_AUDIT.md), so it is not machine- - # precision lockable here; this pins that we never regress BEYOND - # the known band (catches an unintended CR1 SE-formula change). The - # machine-precision hetero/cluster lock is the deferred G2 golden - # regeneration (needs an unbalanced/heteroskedastic DGP). - assert res.se == pytest.approx(golden[key]["cluster_unit"]["se"], rel=0.005) + exp = golden[key]["cluster_unit"] + np.testing.assert_allclose(res.att, exp["att"], atol=1e-10, rtol=0) + np.testing.assert_allclose(res.se, exp["se"], atol=1e-10, rtol=0) + + def test_twfe_cluster_att_matches_fixest(self): + """The TWFE cluster-robust ATT matches fixest exactly; the SE stays + band-pinned. The residual gap is the documented fixest-CR1 ssc + convention for absorbed FE that are NOT nested in the cluster: with + ``absorb=[unit, time]`` and ``cluster=unit``, fixest counts the + non-nested time FE in the (n-1)/(n-k) denominator while the + within-transform path uses k_visible (measured ~0.25% balanced / + ~0.3% unbalanced; tracked in TODO under "Needs external reference"). + The band pins that we never regress BEYOND the known deviation.""" + golden = _load_golden() + for key in ("twfe", "twfe_hetero"): + assert key in golden, f"required golden block {key!r} missing — regenerate the fixture" + df = _build_df(golden[key]) + res = TwoWayFixedEffects(vcov_type="hc1", cluster="unit").fit( + df, outcome="outcome", treatment="treated", time="post", unit="unit" + ) + exp = golden[key]["cluster_unit"] + np.testing.assert_allclose(res.att, exp["att"], atol=1e-10, rtol=0) + assert res.se == pytest.approx(exp["se"], rel=0.005) + + +@_SKIP +class TestFixestHeteroskedasticParity: + """SE-audit G2 machine-precision hetero lock on an unbalanced, + heteroskedastic DGP (error sd varies by arm and period; ~15% of rows + dropped), where fixest's ``hetero`` (HC1) no longer collapses to iid. + + Scope: the plain-OLS DiD path, where Python exposes an unclustered HC1 + (``DifferenceInDifferences(vcov_type="hc1")``). ``TwoWayFixedEffects`` + deliberately auto-clusters at unit on hc1 (documented convention), so it + has no public unclustered-hetero surface to lock — its scenario locks + iid (which also exercises the D4 full-K rescale on an UNBALANCED panel) + and the cluster ATT via the class above. + """ + + def test_did_hetero_hc1_matches_fixest_machine_precision(self): + golden = _load_golden() + assert ( + "did_hetero" in golden + ), "required golden block 'did_hetero' missing — regenerate the fixture" + df = _build_df(golden["did_hetero"]) + res = DifferenceInDifferences(vcov_type="hc1").fit( + df, outcome="outcome", treatment="treated", time="post", unit="unit" + ) + exp = golden["did_hetero"]["hetero"] + np.testing.assert_allclose(res.att, exp["att"], atol=1e-10, rtol=0) + np.testing.assert_allclose(res.se, exp["se"], atol=1e-10, rtol=0) + # Discriminating: hetero must NOT collapse to iid on this DGP. + assert abs(exp["se"] - golden["did_hetero"]["iid"]["se"]) > 0.01 + + def test_did_hetero_iid_matches_fixest_machine_precision(self): + golden = _load_golden() + assert ( + "did_hetero" in golden + ), "required golden block 'did_hetero' missing — regenerate the fixture" + df = _build_df(golden["did_hetero"]) + res = DifferenceInDifferences(vcov_type="classical").fit( + df, outcome="outcome", treatment="treated", time="post", unit="unit" + ) + exp = golden["did_hetero"]["iid"] + np.testing.assert_allclose(res.att, exp["att"], atol=1e-10, rtol=0) + np.testing.assert_allclose(res.se, exp["se"], atol=1e-10, rtol=0) + + def test_twfe_hetero_iid_matches_fixest_machine_precision(self): + """Unbalanced-panel iid lock — the D4 full-K within-transform rescale + must hold off the balanced design too.""" + golden = _load_golden() + assert ( + "twfe_hetero" in golden + ), "required golden block 'twfe_hetero' missing — regenerate the fixture" + df = _build_df(golden["twfe_hetero"]) + res = TwoWayFixedEffects(vcov_type="classical").fit( + df, outcome="outcome", treatment="treated", time="post", unit="unit" + ) + exp = golden["twfe_hetero"]["iid"] + np.testing.assert_allclose(res.att, exp["att"], atol=1e-10, rtol=0) + np.testing.assert_allclose(res.se, exp["se"], atol=1e-10, rtol=0)