feat: add YAML input support (-I yaml) with bundled libyaml 0.2.5#191
Merged
Conversation
build.zig:84 always adds lib/yaml/zig_compat.c as a C source, but the CI step 'Download libyaml amalgamation' only copied 8 files from the upstream tarball and never wrote zig_compat.c. The file exists locally (gitignored, amalgamation pattern) but not on CI runners, so zig build failed on all 3 OS with 'file_hash FileNotFound'. Add a heredoc inside the same step that produces byte-identical zig_compat.c at the project root. Also correct the stale '10 files' comment in build.zig.
…C secure CRT from Zig translator On Windows MSVC, yaml.h's <string.h> transitively pulls in <wchar.h> which declares wcscat_s/wcscpy_s. Zig's translateC generates bindings for these, but libyaml never calls them, causing 'unused local constant' errors. The existing zig_compat.c runtime shim was insufficient because Zig checks const usage at compile time during translation. Introduce a wrapper header that defines these functions as no-op macros on MSVC before including yaml.h, preventing the translator from ever seeing them.
yaml.h's <string.h> transitively pulls in <wchar.h> on MSVC, declaring wcscat_s/wcscpy_s. Zig 0.16's translateC generates bindings for these but libyaml never calls them, causing 'unused local constant' errors. Define them as empty macros via defineCMacroRaw before translation so the translator never sees them. The existing zig_compat.c runtime shim remains as defense-in-depth.
…nslator syntax error -D wcscat_s= broke translator (invalid syntax in function declaration). Rename to _wcscat_s instead so generated Zig constants start with '_' (suppresses unused warning) and translator sees valid C syntax.
…n translateC yaml.h includes <string.h> which on MSVC pulls in <wchar.h> declaring wcscat_s/wcscpy_s. Zig 0.16's translateC generates bindings for all declarations from system headers, but libyaml never calls these secure CRT functions, causing 'unused local constant' errors. Instead of a minimal header (breaks opaque types), rename macros (breaks translator syntax), or static inline definitions (still generates unused bindings), we pre-define _INC_STRING in the wrapper header to skip string.h entirely. size_t is provided via <stddef.h> instead.
The _INC_STRING guard trick in yaml_translate.h eliminates the need for the MSVC secure CRT runtime shim. The file was gitignored and materialized by CI via heredoc; CI step will need update separately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #168
Summary
Add
-I yamlas a supported input format. sql-pipe now reads YAML sequence-of-mappings as rows, mirroring the existing JSON input behavior (all-TEXT columns, fatal on parse error, exit code 2).YAML is ubiquitous in CI/CD, configuration files, and infrastructure-as-code — this closes the biggest format gap in sql-pipe's input support.
Usage
Changes
lib/yaml/(new, vendored) — libyaml 0.2.5 C sources and headers. Downloaded by CI fromhttps://github.com/yaml/libyaml/archive/refs/tags/0.2.5.tar.gz; not tracked in git.build.zig— addedaddTranslateCforyaml.h(exposed as@import("yaml")); always-bundle 6 C source files (api.c, parser.c, scanner.c, reader.c, writer.c, loader.c). Skipped dumper.c/emitter.c (we don't emit YAML).src/format.zig— addedyamlvariant toInputFormatenum. Added explicit.yml→.yamlalias infromExtension()(sincestd.meta.stringToEnumdoesn't accept aliases).src/args.zig— help text for-I, --input-formatnow listsyaml.src/main.zig—.yamlarm inloadInput()dispatches toyaml_mod.loadYamlInput(). Imports the new module.src/yaml.zig(new, ~225 lines) — YAML loader using libyaml's event API (yaml_parser_parse+yaml_event_t). Mirrorssrc/json.zig:loadJsonArray: read-all, parse, all-TEXT table, insert rows. Every successful parse is paired withyaml_event_deleteviadefer.src/modes/{stats,schema,columns,validate}.zig— wiredyaml_mod.loadYamlInput()so--columns -I yaml,--validate -I yaml,--schema -I yaml, and--stats -I yamlall work. (--samplecorrectly rejects non-CSV/TSV formats per existing design.)tests/fixtures/users.yaml+users.yml— block-style YAML mirror ofproducts.jsonfor testing auto-detect and the.ymlalias.build.zig— 8 new integration tests (168a–168h)..github/workflows/ci.yml+.github/workflows/release.yml— added "Download libyaml amalgamation" step that fetches libyaml 0.2.5 tolib/yaml/at build time (mirroring the existing SQLite download pattern).Design Decisions
--bundle-yamloptionyaml_event_t)yaml_document_tand adds complexity. Event API is sequential and gives per-eventstart_mark/end_markfor error reporting.[{}]model. Nested mappings/sequences are rejected with a clear fatal error. Spec compliance trade-off: the original issue mentioned "flatten like JSON" for nested structures, but the plan (user-approved) restricted to flat mappings for v1. See "Follow-ups" below.CAST(... AS INTEGER)as needed..ymlextensionfromExtension()std.meta.stringToEnumonly matches exact names. Without an explicit alias,.ymlfiles would default to CSV."YAML parse error at line N, col M: <libyaml problem>"src/xml.zig:276-280pattern. libyaml marks are 0-indexed; add 1 for user-facing.Test Coverage (8 new integration tests, 168a–168h)
.yamlfile argument auto-detect.ymlextension alias-I yamlValidation
zig build— cleanzig build test— exit 0 (136+ tests: 128 existing + 8 new)zig build unit-test— exit 0ziglint src build.zig— exit 0, zero warnings--columns,--validate,--schema,--statsall work with-I yamlMemory Safety
Oracle-verified clean. Every
yaml_parser_parsesuccess path is paired withyaml_event_deleteviadefer.yaml_parser_deleteis called exactly once viadefer. Allallocator.duperesults are tracked in a singledeferblock.sqlite_staticordering is safe because value frees happen AFTERsqlite3_step. On fatal exit, the OS reclaims all resources.Follow-ups (not in this PR)
src/json.zig:114-121does) to match JSON behavior.stats,schema,columns,validate,sample) has its own copy of the input format switch. Adding a 7th format requires touching 5 files. Pre-existing pattern, not introduced by this PR; would benefit from a future refactor.Checklist
issue-168/yaml-input(per AGENTS.md conventions)