perf(query-engine): ClickStack-style skip indexes + token-aware log search#170
Open
Makisuo wants to merge 2 commits into
Open
perf(query-engine): ClickStack-style skip indexes + token-aware log search#170Makisuo wants to merge 2 commits into
Makisuo wants to merge 2 commits into
Conversation
…earch Apply optimizations from ClickHouse's "Making ClickStack 5x faster" to Maple's telemetry read path: - logs.Body gains a tokenbf_v1(32768,3,0) index on lower(Body); body search AND-s an interior-token `hasToken(lower(Body), t)` pre-filter ahead of the substring ILIKE. Only separator-bounded, ASCII, interior tokens qualify, so the result set is byte-for-byte unchanged — the index just prunes granules. - logs gains the four attribute-map bloom filters that traces already had. - traces gains ClickStack "Items" bloom indexes over arrayMap((k,v)->concat(k,'=',v), ...) so attribute-equality filters prune via has(items,'k=v') (AND-ed with the exact map equality; enabled only on the raw traces table, not trace_detail_spans or metrics). - Index expressions live in one shared module (index-exprs.ts) imported by the datasource, the migration, and the query predicate so they can't drift. New committed benchmark suite (bench suite): 13 cases compiled from the real query-engine DSL, plus `--table-map` for A/B against shadow tables and `compare --max-regression-pct` as a CI regression gate. BYO migration 0005 adds every index with ADD INDEX (no MATERIALIZE — new parts only; TTL churn backfills), bumping SCHEMA_VERSION to 5. Tests: builder 72, domain 174, query-engine 727, apps/api bench 11; repo typecheck 24/24. Pre-merge gates (need a live warehouse): - Tinybird `tb --cloud deploy --check` on a branch to confirm managed Tinybird accepts tokenbf_v1 + the arrayMap expression index as an in-place ADD INDEX. - Live `EXPLAIN indexes=1` granule-drop check on a dev ClickHouse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Your LLM provider API key was rejected. Rotate the key in your provider dashboard, then update the matching GitHub Actions secret. Update repo secret → · Model settings → · Setup docs → · Ask in Discord →
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
🚩 Log timeseries raw path does not use bodySearchConditions, but this is pre-existing
The bodySearchConditions integration was applied to logsCountQuery (packages/query-engine/src/ch/queries/logs.ts:360) and logsListQuery (line 455), but the timeseries query's raw path was not updated. This is NOT a regression: the timeseries raw path never had a search filter — canUseLogsAggregateInterior (line 81) returns false when opts.search is set, routing to the raw-only scan, but the raw timeseries scan itself doesn't filter by search term. The timeseries query counts all logs in the window for the volume chart, while search filtering is only applied to the list and count queries. If search-filtered timeseries is ever needed, bodySearchConditions should be added there too.
Was this helpful? React with 👍 or 👎 to provide feedback.
Ingest Rust Test + Benchmark ResultsCommit: Load Benchmark —
|
| Metric | main (median) | PR (median) | Delta |
|---|---|---|---|
| Requests/sec | 2090.05 | 2268.59 | +8.5% better |
| Rows/sec | 20900.50 | 22685.86 | +8.5% better |
| p50 latency | 29.84 ms | 27.58 ms | -7.6% better |
| p95 latency | 35.23 ms | 31.59 ms | -10.3% better |
| p99 latency | 39.72 ms | 34.25 ms | -13.8% better |
| Export catch-up | 0.026 s | 0.026 s | +2.0% worse |
| Max RSS | 101.37 MiB | 100.47 MiB | -0.9% better |
| Failures | 0 | 0 | same |
Same code path on both sides (same LOAD_TEST_INGEST_MODE), so the delta column is meaningful. Numbers come from ubuntu-latest, which is noisy — treat single-digit-percent deltas as noise.
PR load benchmark JSON (per-iteration)
[
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 25,
"duration_seconds": 1.080888458,
"export_catchup_seconds": 0.026426321,
"request_rps": 1850.329684989568,
"row_rps": 18503.29684989568,
"p50_ms": 29.266,
"p95_ms": 50.781,
"p99_ms": 140.999,
"max_rss_mb": 100.46875,
"max_cpu_percent": 64.2,
"avg_cpu_percent": 50.6
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 27,
"duration_seconds": 0.849835962,
"export_catchup_seconds": 0.025862109,
"request_rps": 2353.395348548453,
"row_rps": 23533.953485484533,
"p50_ms": 26.186,
"p95_ms": 31.589,
"p99_ms": 32.603,
"max_rss_mb": 99.87109375,
"max_cpu_percent": 80.3,
"avg_cpu_percent": 56.8
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 25,
"duration_seconds": 0.881606332,
"export_catchup_seconds": 0.026644017,
"request_rps": 2268.586246950867,
"row_rps": 22685.86246950867,
"p50_ms": 27.578,
"p95_ms": 30.522,
"p99_ms": 34.248,
"max_rss_mb": 101.80078125,
"max_cpu_percent": 80.3,
"avg_cpu_percent": 56.8
}
]main load benchmark JSON (per-iteration)
[
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 25,
"duration_seconds": 0.98045698,
"export_catchup_seconds": 0.025451398,
"request_rps": 2039.8651249338855,
"row_rps": 20398.651249338855,
"p50_ms": 30.047,
"p95_ms": 37.033,
"p99_ms": 39.722,
"max_rss_mb": 103.5078125,
"max_cpu_percent": 73.2,
"avg_cpu_percent": 44.900000000000006
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 25,
"duration_seconds": 0.956915124,
"export_catchup_seconds": 0.02671958,
"request_rps": 2090.049524601306,
"row_rps": 20900.495246013063,
"p50_ms": 29.839,
"p95_ms": 35.232,
"p99_ms": 42.583,
"max_rss_mb": 99.42578125,
"max_cpu_percent": 73.2,
"avg_cpu_percent": 44.900000000000006
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 24,
"duration_seconds": 0.851577095,
"export_catchup_seconds": 0.025917336,
"request_rps": 2348.583600642758,
"row_rps": 23485.83600642758,
"p50_ms": 26.424,
"p95_ms": 29.983,
"p99_ms": 32.672,
"max_rss_mb": 101.37109375,
"max_cpu_percent": 82.1,
"avg_cpu_percent": 49.349999999999994
}
]WAL-acked microbench (cargo bench --bench ingest_bench)
Compiling maple-ingest v0.1.0 (/home/runner/work/maple/maple/apps/ingest)
Finished `bench` profile [optimized] target(s) in 40.30s
Running benches/ingest_bench.rs (target/release/deps/ingest_bench-581d2100de893627)
Gnuplot not found, using plotters backend
test ingest_accept/logs_10_rows_wal_ack ... bench: 508338 ns/iter (+/- 43976)
test ingest_accept/traces_10_spans_wal_ack ... bench: 534728 ns/iter (+/- 26045)
cargo test
test telemetry::tests::metrics_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::metrics_summary_data_points_are_dropped ... ok
test telemetry::tests::migrate_legacy_shard_relocates_frames_into_lanes ... ok
test telemetry::tests::pipeline_can_start_for_clickhouse_only_without_tinybird_credentials ... ok
test telemetry::tests::clickhouse_export_drops_passworded_non_https_endpoint_without_sending ... ok
test telemetry::tests::pipeline_e2e_exports_gzip_ndjson_to_fake_tinybird ... ok
test telemetry::tests::pipeline_e2e_exports_metrics_to_fake_tinybird ... ok
test telemetry::tests::sampling_keeps_errors_even_when_ratio_low ... ok
test telemetry::tests::scraper_contract::scraper_otlp_json_decodes_with_gateway_serde_and_encodes_to_rows ... ok
test telemetry::tests::signal_tag_round_trips_all_variants ... ok
test telemetry::tests::pipeline_e2e_exports_traces_to_fake_tinybird ... ok
test telemetry::tests::telemetry_signal_as_str_is_canonical_lowercase ... ok
test telemetry::tests::timestamp_has_nano_precision ... ok
test telemetry::tests::timestamps_match_clickhouse_datetime64_nine_format ... ok
test telemetry::tests::trace_encoder_matches_tinybird_row_shape ... ok
test telemetry::tests::traces_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::wal_partial_drain_advances_cursor_without_truncating ... ok
test telemetry::tests::wal_round_trips_frame ... ok
test telemetry::tests::wal_truncates_after_full_drain_allowing_further_appends ... ok
test telemetry::tests::pipeline_exports_ready_org_to_clickhouse_without_tinybird_calls ... ok
test telemetry::tests::slow_clickhouse_lane_does_not_block_cosharded_tinybird_org ... ok
test telemetry::tests::clickhouse_breaker_sheds_after_threshold_failures ... ok
test result: ok. 36 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.77s
Running unittests src/bin/load_test.rs (target/debug/deps/load_test-661a0aa1eb3f6d6d)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running unittests src/main.rs (target/debug/deps/maple_ingest-c33bf80c577edb95)
running 38 tests
test autumn::tests::allowed_only_no_balance_field ... ok
test autumn::tests::flat_hardcap_with_remaining_allows ... ok
test autumn::tests::flat_hardcap_depleted_blocks ... ok
test autumn::tests::flat_sub_one_gb_remaining_still_allows ... ok
test autumn::tests::flat_overage_allows ... ok
test autumn::tests::flat_unlimited_allows ... ok
test autumn::tests::nested_balance_object_depleted_blocks ... ok
test autumn::tests::nested_overage_allows ... ok
test autumn::tests::nested_balance_object_with_remaining_allows ... ok
test autumn::tests::null_balance_no_subscription_blocks ... ok
test autumn::tests::unrecognized_shape_returns_none ... ok
test tests::api_error_from_pipeline_maps_variants_to_status ... ok
test tests::clickhouse_destination_is_terminal_in_dual_mode ... ok
test tests::clickhouse_destination_uses_native_pipeline_even_in_forward_mode ... ok
test tests::api_error_kind_maps_status_to_stable_label ... ok
test tests::clickhouse_target_resolver_requires_current_schema ... ok
test tests::clickhouse_target_resolver_rejects_password_over_http ... ok
test tests::clickhouse_target_resolver_decrypts_current_schema_password ... ok
test tests::cloudflare_timestamps_support_rfc3339_unix_and_unix_nano ... ok
test tests::cloudflare_log_record_maps_body_severity_and_attributes ... ok
test tests::cloudflare_validation_payload_is_detected ... ok
test tests::cloudflare_ndjson_payload_parses_multiple_records ... ok
test tests::decrypt_aes256_gcm_matches_node_crypto_fixture ... ok
test tests::enrichment_overwrites_tenant_fields ... ok
test tests::extract_ingest_key_returns_sentinel_literal_unchanged ... ok
test tests::hash_is_deterministic ... ok
test tests::rejection_span_status_is_error_only_for_5xx ... ok
test tests::resolve_ingest_key_keeps_stale_schema_on_managed_native_path ... ok
test tests::resolve_connector_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_returns_none_when_hash_missing ... ok
test tests::resolve_ingest_key_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_returns_self_managed_true_when_active_settings_row ... ok
test tests::resolve_ingest_key_returns_self_managed_false_when_no_settings_row ... ok
test tests::sentinel_token_matches_only_exact_literal ... ok
test tests::tinybird_destination_keeps_forward_mode_on_forward_path ... ok
test tests::resolve_ingest_key_serves_last_known_routing_when_refresh_fails ... ok
test autumn::tests::fails_open_on_transport_error ... ok
test tests::forward_mode_switches_ready_org_to_clickhouse_without_forwarding_again ... ok
test result: ok. 38 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s
Doc-tests maple_ingest
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
…-inserts Two generated/barrel artifacts missed in the first commit: - tinybird/index.ts now re-exports ./index-exprs like its sibling modules. - local-inserts.json regenerated to the new schema revision (was stale, would have failed clickhouse:schema:check in CI). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Applies the schema/query optimizations from ClickHouse's Making ClickStack 5x faster to Maple's telemetry read path. The blog's biggest wins map directly onto gaps Maple had:
logs.Bodyhad no index — search was a full-partitionILIKEscan behind amax_block_size=512OOM guardrailtokenbf_v1onlower(Body)+ a semantics-safehasTokenpre-filterlogshad zero attribute-map indexes (traces had 4)bloom_filterindexes onlogsidx_*_attr_itemsbloom overarrayMap((k,v)->concat(k,'=',v),…)ontracesbench suitecompiled from the real DSL + A/B + regression gateChanges
Indexes (
packages/domain/src/tinybird/datasources.ts)logs:idx_body_tokens(tokenbf_v1(32768,3,0)onlower(Body)) + blooms on mapKeys/mapValues ofLogAttributesandResourceAttributes.traces:idx_span_attr_items+idx_resource_attr_itemsover the concatenatedkey=value"Items" expression.index-exprs.tsand imported by the datasource, the migration, and the query predicate so they can't drift (a mismatch silently disables the index).Query engine
body-search.ts:bodySearchConditionsAND-shasToken(lower(Body), t)for interior, separator-bounded, ASCII tokens only, keeping theILIKEas the final predicate. A single word or edge token yields no pre-filter → today's behavior. Whole-token semantics ≠ substring, so this is the only sound derivation; result sets are unchanged.buildAttrFilterCondition(…, { itemsIndex }): equality filters emithas(items,'k=v')(OR-ed across HTTP semconv aliases) AND-ed with the exact map equality. Enabled only on the rawtracestable (nottrace_detail_spansor metrics, which lack the index).CH.hasToken/CH.has.Migration & rollout
0005_body_and_attr_indexes.tsadds every index withADD INDEX IF NOT EXISTSand noMATERIALIZE(a materialize over billion-row parts exceeds the Worker subrequest budget; indexes cover new parts and backfill via the 30-day TTL).SCHEMA_VERSIONbumps 4→5.Benchmark harness (
apps/api/scripts)bench suitecompiles 13 representative cases from the real DSL;--table-map from=torewritesFROM/JOINfor A/B against shadow tables;bench compare --max-regression-pct Nexits non-zero as a CI gate. SeeBENCH.md.Testing
tb build+ branchtb --cloud deploy --checkto confirm managed Tinybird acceptstokenbf_v1and thearrayMapexpression index as an in-placeADD INDEX(noFORWARD_QUERY/rebuild). Fallback if rejected: keep those indexes BYO-migration-only.EXPLAIN indexes=1granule-drop check on a dev ClickHouse (the localclickhousebinary is macOS-Gatekeeper-blocked in the dev box used here).SCHEMA_VERSIONbump un-readies BYO-ClickHouse ingest untilapplySchemastamps v5; deploy API + applySchema + Rust gateway together.Follow-up (separate PR)
The time-bucketed sort-key /
read_in_orderspike (Phase 3) — thebench suite --table-mapA/B harness in this PR is built to drive it against a dedicated ClickHouse.🤖 Generated with Claude Code