Repository navigation
Conversation
Replace the self-contained driver (benchmark.py / run.sh / query.py) with
the lifecycle scripts the shared driver (lib/benchmark-common.sh) expects:
install, start, stop, check, load, query, data-size, plus template.json
and a thin benchmark.sh that sets the BENCH_* knobs and exec's the lib.
MotherDuck is a managed, serverless cloud service, so:
- BENCH_DOWNLOAD_SCRIPT="" — load CTAS's directly from S3 into the cloud;
nothing is fetched onto the client VM.
- BENCH_RESTARTABLE=no — the engine is remote; restarting the local client
and dropping its page cache does nothing to MotherDuck (inherently
"no-cold"), and would otherwise burn bench_wait_stopped's 60s timeout.
- BENCH_DURABLE=yes — data persists in the cloud across client restarts.
- BENCH_CONCURRENT_DURATION=0 — each query opens a fresh md: connection,
so the QPS test would mostly measure connection setup; off by default.
load/data-size/query reuse the original SQL (create.sql, the epoch_ms /
EventDate coercions, pragma_database_size -> bytes). query passes its program
via `python3 -c` so stdin stays free for the piped query, and uses
con.execute() to tolerate the trailing ";" on each queries.sql line.
Verified end to end against MotherDuck: full load (99,997,497 rows), data
size ~17.2 GiB, and all 43 queries run with 0 failures.
Known follow-ups (outside this directory, needed for the official AWS
automation): cloud-init labels results with the client EC2 instance type
rather than the MotherDuck warehouse size, and has no generic secret
forwarding for motherduck_token — both want a dedicated workflow like
clickhouse-cloud.yml.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Duckling (compute) size can't be set via SQL or the md: connection string; it
is an account setting changed through the MotherDuck REST API. Add helpers:
set-instance-size <pulse|standard|jumbo|mega|giga>
GET the current config, PUT read_write.instance_size (preserving the
read_scaling block the API requires), then poll md_live_duckling_size()
until the new size reports live. That function flips before the compute is
stable, so it then settles for STABILIZE_SECONDS (default 120; >=2 min is
required). Username is auto-discovered via MD_USER_INFO().
run-all-sizes.sh [size ...]
Install the client once, then for each size: set-instance-size -> run the
full benchmark -> make-result. Runs proceed largest size first; the default
set is mega/jumbo/standard/pulse (giga excluded by default since some trial
accounts lack it; opt in with INCLUDE_GIGA=1 or by naming it). A size the
account can't use is skipped, not fatal. RESTORE_SIZE resets at the end.
make-result <size> [log] [date]
Convert a per-size log into results/<date>/<size>.json labeled
"Motherduck: <size>". The shared cloud-init derives a result's machine from
the client EC2 type, which is wrong for a remote service, so we commit
results directly (as the existing MotherDuck results do) rather than rely on
the sink path.
Also add the "DuckDB derivative" tag to template.json to match the published
MotherDuck entry.
Verified end to end on EC2 (c6a.4xlarge, us-east-1) against MotherDuck: the full
largest-first sweep ran mega/jumbo/standard/pulse with 43/43 queries and no
failures, set-instance-size performed real PUTs across all four sizes, and the
generated results JSONs pass validate-results.py and match the format/numbers of
the published PR ClickHouse#2319 (load times within a few percent).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
set-instance-size now, in order: issues SHUTDOWN NORMAL so the current duckling shuts down cleanly, PUTs the new read_write size, waits SETTLE_SECONDS (default 120, min 120) WITHOUT querying MotherDuck so the duckling can finish shutting down on its own, then double-checks md_live_duckling_size() reports the new size (polling up to SET_SIZE_TIMEOUT if propagation lags). This replaces the previous "PUT then poll, then stabilize" approach. The fixed 2-minute wait is required — md_live_duckling_size() flips before the compute is settled — and nothing may query MotherDuck during it or the old duckling is kept alive and never recycles. duckdb is now a hard requirement (used for SHUTDOWN NORMAL and the size check); the STABILIZE_SECONDS knob is renamed SETTLE_SECONDS. SHUTDOWN NORMAL returns (True,) and leaves the connection usable; verified live along with username discovery and the GET. The PUT is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each query try opens a fresh md: connection. Connect with
config={"motherduck_attach_mode": "single"} so it attaches only the database the
benchmark uses instead of enumerating the whole workspace catalog on every
connect. Verified via the config dict (the connection-string form is silently
ignored); queries still return correct results.
Context: the geomean comparison vs PR ClickHouse#2319 showed the migrated entry carries a
small, systematic per-query overhead from the one-connection-per-try model
(worst on fast queries / cold runs). Single-attach trims the catalog work of
that per-try connect. A local A/B was inconclusive (in-process connection
pooling + cross-country latency mask the fresh-process cost); the real effect
should be measured on the co-located EC2 client.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jtigani
requested a deployment
to
benchmark-approval
October 20, 2026 03:03 — with 
GitHub Actions
Waiting
jtigani
commented
Oct 20, 2026
jtigani
left a comment
Contributor
Author
There was a problem hiding this comment.
I (or, Claude actually) migrated MotherDuck to use the new shared runner infrastructure. Wanted to make sure we following the correct protocols. Also included a couple of scripts that can be used to run all motherduck production instance sizes. Happy to remove those if is preferable to keep the repo clean.
jtigani
marked this pull request as ready for review
October 20, 2026 03:14
This branch is waiting to be deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Migrates the
motherduck/entry from its self-contained driver(
benchmark.py/run.sh/query.py) to the sharedlib/benchmark-common.shrunner that the rest of the catalog now uses.Core migration
Replaces the old driver with the lifecycle scripts the shared runner expects —
install,start,stop,check,load,query,data-size,template.json, and a thinbenchmark.sh. The SQL (create.sql,queries.sql, the load CTAS) is unchanged.MotherDuck is a managed, serverless service, so the driver settings are:
BENCH_DOWNLOAD_SCRIPT=""— nothing is downloaded to the client;loadCTAS's directly from the ClickBench parquet on S3 into MotherDuck.
BENCH_RESTARTABLE=no— the engine is remote; there's no local daemon tostop/start, and it's inherently "no-cold".
BENCH_DURABLE=yes— data persists across client restarts.measure connection setup rather than throughput).
MotherDuck-specific helpers
These are isolated to
motherduck/and are not part of the shared contract:set-instance-size— change the account's read_write Duckling (compute) sizevia the MotherDuck REST API (it can't be set via SQL). Shuts the duckling down
cleanly, PUTs the new size, waits for stabilization, verifies via
md_live_duckling_size().run-all-sizes.sh— run the full benchmark once per Duckling size(largest-first;
gigaopt-in), emitting a committable result per size.make-result— convert a per-size log intoresults/<date>/<size>.jsonlabeled
Motherduck: <size>. The shared cloud-init derivesmachinefrom theclient EC2 instance type, which is meaningless for a remote service, so we
commit results directly (as the existing MotherDuck results do).
Notes for reviewers
prefer the bare migration to land on its own.
motherduck_tokenandlabels
machineby the client EC2 type, so running this under the officialAWS automation would need a dedicated workflow (like
clickhouse-cloud.yml).Flagging for guidance.
🤖 Generated with Claude Code