Skip to content

fix(export): keep every value through an export and its import - #3707

Merged
datlechin merged 2 commits into
mainfrom
test/3666-export-import-roundtrip
Oct 20, 2026
Merged

datlechin merged 2 commits into
mainfrom
test/3666-export-import-roundtrip

Conversation

@datlechin

Copy link
Copy Markdown
Member

Summary

  • FormatPluginTests/ExportImportRoundTripTests: a SQLite table with every storage class, NULL, quotes, delimiters, LF/CRLF/CR line breaks, Unicode, a control character and binary goes out through each export plugin and back in through its import plugin. CSV (every delimiter and line ending, BOM, NULL spelled out), JSON and NDJSON, XLSX, SQL (every insert mode), SQL gzipped, SQL split into 1 MB parts restored in order. FormatLimits names what each format cannot carry.
  • The tests link the vendored SQLite the driver ships (SQLiteLinkage on FormatPluginTests), read rows the way the SQLite driver does, and insert them the way ImportDataSinkAdapter does.
  • scripts/probes/check-export-import-roundtrip.sh: the same round trip on MySQL/MariaDB or PostgreSQL with 200,000 rows (110 to 172 MB per file), through the built driver and format plugins. Row count and a server-side checksum on both sides. SQL dumps restore through the engine's own client.
  • CSV: formula sanitizing no longer prefixes a plain number. -5 was written '-5, and PostgreSQL refused it on import. Same rule as the Linux app's is_plain_number.
  • CSV: quoting checks Unicode scalars. \r\n is one Character, so a value with a Windows line break was written unquoted and split its row, which also put the text after the break, a formula included, at the start of a new row. Fixed for the CSV export, the MCP export, Copy as CSV, the structure grid copy and the activity log export, which now share CsvRowConverter.encode and PluginRowWriters.csvField.
  • JSON: true and false are written bare only in a boolean column. Text TRUE in a text column came back true.
  • XLSX: a carriage return is written 
. A raw one read back as a line feed.
  • PluginRowWritersTests: -1 and +1 now stay as they are, changed on purpose.

Test plan

  • verify.sh test, 16 suites, 180 cases: pass. Without the fixes, 6 of the 10 round-trip tests fail (CSV x3, JSON x2, XLSX).
  • verify.sh ios ClipboardExporterPasteboardTests: pass (PluginKit compiles in the iOS app).
  • Probe, 200,000 rows: PostgreSQL 17.11 and MariaDB 13.0.2, every format OK. Before the fix, the CSV import stopped at '-49998.999999 and XLSX came back with a different checksum.

Not in this PR

  • iOS ClipboardExporter has the same contains("\n") check.
  • XLSX: text such as NaN or Infinity is written as a number cell, <t> has no xml:space="preserve", control characters are dropped rather than written _xHHHH_. Excel's reading of each could not be checked here.
  • XLSX import reads only the first sheet, so an export split across sheets past 1,048,576 rows imports partly.
  • CSV and XLSX read '' and NULL back as NULL; binary goes out as hex text (CSV, XLSX) or base64 (JSON). A decimal-comma CSV export has no import option to read it back.
  • The macOS formula leads lack the Linux app's LF and full-width leads.
  • SQLFileParser, the app's SQL import reader, is app code neither the plugin test bundle nor the probe reaches.

Fixes #3666

@datlechin
datlechin enabled auto-merge (squash) October 20, 2026 16:37
@datlechin
datlechin merged commit 31d54f7 into main Oct 20, 2026
8 checks passed
@datlechin
datlechin deleted the test/3666-export-import-roundtrip branch October 21, 2026 03:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test that every export format imports back with the same rows

1 participant