Skip to content

feat: add YAML input support (-I yaml) with bundled libyaml 0.2.5#191

Merged
vmvarela merged 10 commits into
masterfrom
issue-168/yaml-input
Jul 10, 2026
Merged

feat: add YAML input support (-I yaml) with bundled libyaml 0.2.5#191
vmvarela merged 10 commits into
masterfrom
issue-168/yaml-input

Conversation

@vmvarela

@vmvarela vmvarela commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Closes #168

Summary

Add -I yaml as a supported input format. sql-pipe now reads YAML sequence-of-mappings as rows, mirroring the existing JSON input behavior (all-TEXT columns, fatal on parse error, exit code 2).

YAML is ubiquitous in CI/CD, configuration files, and infrastructure-as-code — this closes the biggest format gap in sql-pipe's input support.

Usage

# File auto-detected from .yaml extension
sql-pipe users.yaml 'SELECT * FROM users WHERE age > 18'

# .yml alias also works
sql-pipe users.yml 'SELECT * FROM users'

# Explicit -I yaml for stdin
cat data.yaml | sql-pipe -I yaml 'SELECT SUM(balance) FROM t'

# Works with all standard output formats and SQL features
cat data.yaml | sql-pipe -I yaml -O json 'SELECT name, amount FROM t ORDER BY amount DESC'

Changes

  • lib/yaml/ (new, vendored) — libyaml 0.2.5 C sources and headers. Downloaded by CI from https://github.com/yaml/libyaml/archive/refs/tags/0.2.5.tar.gz; not tracked in git.
  • build.zig — added addTranslateC for yaml.h (exposed as @import("yaml")); always-bundle 6 C source files (api.c, parser.c, scanner.c, reader.c, writer.c, loader.c). Skipped dumper.c/emitter.c (we don't emit YAML).
  • src/format.zig — added yaml variant to InputFormat enum. Added explicit .yml.yaml alias in fromExtension() (since std.meta.stringToEnum doesn't accept aliases).
  • src/args.zig — help text for -I, --input-format now lists yaml.
  • src/main.zig.yaml arm in loadInput() dispatches to yaml_mod.loadYamlInput(). Imports the new module.
  • src/yaml.zig (new, ~225 lines) — YAML loader using libyaml's event API (yaml_parser_parse + yaml_event_t). Mirrors src/json.zig:loadJsonArray: read-all, parse, all-TEXT table, insert rows. Every successful parse is paired with yaml_event_delete via defer.
  • src/modes/{stats,schema,columns,validate}.zig — wired yaml_mod.loadYamlInput() so --columns -I yaml, --validate -I yaml, --schema -I yaml, and --stats -I yaml all work. (--sample correctly rejects non-CSV/TSV formats per existing design.)
  • tests/fixtures/users.yaml + users.yml — block-style YAML mirror of products.json for testing auto-detect and the .yml alias.
  • build.zig — 8 new integration tests (168a–168h).
  • .github/workflows/ci.yml + .github/workflows/release.yml — added "Download libyaml amalgamation" step that fetches libyaml 0.2.5 to lib/yaml/ at build time (mirroring the existing SQLite download pattern).

Design Decisions

Decision Choice Rationale
Parser libyaml 0.2.5 (C) YAML spec is complex (anchors, flow vs block, multi-line strings, etc.); bundling a tested C parser avoids a 200-line minimal Zig parser that's guaranteed to fail on edge cases. Tiny footprint (~100KB source).
Always-bundle No --bundle-yaml option Unlike SQLite (which has version-sensitive features and is commonly system-installed), libyaml is tiny, rarely system-installed, and has no version-dependent features we need. System-library path would be untested dead code. YAGNI.
Event API vs document API Event API (yaml_event_t) Document API heap-allocates a yaml_document_t and adds complexity. Event API is sequential and gives per-event start_mark/end_mark for error reporting.
Structure List of mappings only (flat) Matches JSON's [{}] model. Nested mappings/sequences are rejected with a clear fatal error. Spec compliance trade-off: the original issue mentioned "flatten like JSON" for nested structures, but the plan (user-approved) restricted to flat mappings for v1. See "Follow-ups" below.
Type inference All TEXT (no inference) Matches JSON behavior. SQL can CAST(... AS INTEGER) as needed.
Anchors/aliases Rejected with fatal The event API doesn't resolve aliases; a no-op would silently corrupt data. Honest rejection is better than silent data loss.
Multi-document Rejected with fatal Silent data loss vs. schema mismatch from a second document. Error loudly.
.yml extension Handled explicitly in fromExtension() std.meta.stringToEnum only matches exact names. Without an explicit alias, .yml files would default to CSV.
Error format "YAML parse error at line N, col M: <libyaml problem>" Follows src/xml.zig:276-280 pattern. libyaml marks are 0-indexed; add 1 for user-facing.

Test Coverage (8 new integration tests, 168a–168h)

Test Coverage
168a .yaml file argument auto-detect
168b .yml extension alias
168c Stdin with -I yaml
168d Filter + aggregate
168e Comments transparent
168f Malformed YAML → exit 2 with line number
168g Empty input → graceful
168h Multi-document → error exit 2

Validation

  • zig build — clean
  • zig build test — exit 0 (136+ tests: 128 existing + 8 new)
  • zig build unit-test — exit 0
  • ziglint src build.zig — exit 0, zero warnings
  • Mode smoke tests — --columns, --validate, --schema, --stats all work with -I yaml

Memory Safety

Oracle-verified clean. Every yaml_parser_parse success path is paired with yaml_event_delete via defer. yaml_parser_delete is called exactly once via defer. All allocator.dupe results are tracked in a single defer block. sqlite_static ordering is safe because value frees happen AFTER sqlite3_step. On fatal exit, the OS reclaims all resources.

Follow-ups (not in this PR)

  • C2: Nested structure flattening — The original issue asked for "support nested YAML structures (flatten like JSON does)". Current implementation rejects nested mappings/sequences with a fatal error. Follow-up issue will JSON-serialize nested values (like src/json.zig:114-121 does) to match JSON behavior.
  • S1: Mode dispatch consolidation — Each of the 5 mode files (stats, schema, columns, validate, sample) has its own copy of the input format switch. Adding a 7th format requires touching 5 files. Pre-existing pattern, not introduced by this PR; would benefit from a future refactor.

Checklist

  • Tests added (8 integration tests)
  • Help text updated
  • Build/lint/test all pass
  • Memory safety reviewed
  • CI downloads libyaml (mirrors existing SQLite pattern)
  • No new dependencies for users
  • No output format changes (input only, as specified)
  • Branch name: issue-168/yaml-input (per AGENTS.md conventions)

@github-actions github-actions Bot added the type:feature New functionality label Jul 10, 2026
@vmvarela vmvarela added the status:review In code review or waiting for feedback label Jul 10, 2026
vmvarela added 9 commits July 10, 2026 14:55
build.zig:84 always adds lib/yaml/zig_compat.c as a C source, but the
CI step 'Download libyaml amalgamation' only copied 8 files from the
upstream tarball and never wrote zig_compat.c. The file exists locally
(gitignored, amalgamation pattern) but not on CI runners, so zig build
failed on all 3 OS with 'file_hash FileNotFound'.

Add a heredoc inside the same step that produces byte-identical
zig_compat.c at the project root. Also correct the stale '10 files'
comment in build.zig.
…C secure CRT from Zig translator

On Windows MSVC, yaml.h's <string.h> transitively pulls in <wchar.h>
which declares wcscat_s/wcscpy_s. Zig's translateC generates bindings
for these, but libyaml never calls them, causing 'unused local constant'
errors. The existing zig_compat.c runtime shim was insufficient because
Zig checks const usage at compile time during translation.

Introduce a wrapper header that defines these functions as no-op macros
on MSVC before including yaml.h, preventing the translator from ever
seeing them.
yaml.h's <string.h> transitively pulls in <wchar.h> on MSVC, declaring
wcscat_s/wcscpy_s. Zig 0.16's translateC generates bindings for these
but libyaml never calls them, causing 'unused local constant' errors.

Define them as empty macros via defineCMacroRaw before translation so
the translator never sees them. The existing zig_compat.c runtime shim
remains as defense-in-depth.
…nslator syntax error

-D wcscat_s= broke translator (invalid syntax in function declaration).
Rename to _wcscat_s instead so generated Zig constants start with '_'
(suppresses unused warning) and translator sees valid C syntax.
…n translateC

yaml.h includes <string.h> which on MSVC pulls in <wchar.h> declaring
wcscat_s/wcscpy_s. Zig 0.16's translateC generates bindings for all
declarations from system headers, but libyaml never calls these secure
CRT functions, causing 'unused local constant' errors.

Instead of a minimal header (breaks opaque types), rename macros
(breaks translator syntax), or static inline definitions (still generates
unused bindings), we pre-define _INC_STRING in the wrapper header to
skip string.h entirely. size_t is provided via <stddef.h> instead.
The _INC_STRING guard trick in yaml_translate.h eliminates the need
for the MSVC secure CRT runtime shim. The file was gitignored and
materialized by CI via heredoc; CI step will need update separately.
@vmvarela vmvarela merged commit 85f92b5 into master Jul 10, 2026
5 checks passed
@vmvarela vmvarela deleted the issue-168/yaml-input branch July 10, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:review In code review or waiting for feedback type:feature New functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add YAML input support (-I yaml)

1 participant