cmsDumpExporter requires lots of memory #1715

Activity

1. pxsit commented on Sep 3, 2026

   

   pxsit

   on Sep 3, 2026

   Contributor

   More actions

   This issue also occurs with cmsDumpImporter
2. prandla commented on Sep 3, 2026

   

   prandla

   on Sep 3, 2026

   Member

   More actions

   There's (at least) two separate problems here. One is that DumpExporter creates a bunch of temporary files, so if /tmp is on a tmpfs, this will inflate memory use. This should be relatively simple to fix.

   The other problem is that basically the entire database gets serialized into one json object, so it needs enough memory to store that object. This seems really difficult to fix with the current dump format. If we make a new format, then I guess using something like jsonl (i.e. just one json object per line) could alleviate the need to parse that much json at once, but then we still have to basically rewrite DumpExporter entirely (it relies on loading a lot of sqlalchemy objects in memory at once).

   At that point I wonder how much utility DumpExporter/Importer even have, as it is also possible (and much faster and more memory-efficient) to dump the database directly with pg_dump.

   How big was the IOI contest.json, by the way?
3. gollux commented on Sep 4, 2026

   

   gollux

   on Sep 4, 2026

   ContributorAuthor

   More actions

   One advantage of using CMS dumps over database dumps is that you can dump and restore a single contest.

   This IOI's contest.json has about 10 GB.