Repository navigation
out of memory #568
Description
First off, my JavaScript environment is very custom and different than the web or Node.js. We are running in a wasm32-wasi environment with QuickJS, using wasmedge-quickjs, and Wasmtime on the Internet Computer.
That being said, the strange thing is that I had this working about a week ago, but now it doesn't work. We've been doing a lot of low-level changes to our JS environment (we're working at the engine/platform level), but I just don't understand what could have caused this.
The simple problem is that we get an error just trying to initialize the database: out of memory
Here's our code:
import { Server } from 'azle';
import express from 'express';
import initSqlJs from 'sql.js/dist/sql-asm.js';
export default Server(async () => {
const SQL = await initSqlJs({
locateFile: (filename) => `/dist/${filename}`
});
const db = new SQL.Database();
const app = express();
app.get('/', (req, res) => {
res.send('Hello world');
});
return app.listen();
});We have to use asm as our environment does not have a Wasm interpreter. I've tried sql-asm-memory-growth.js as well and I get the same error. I've even gone back multiple versions to maybe 1.7.0 and the error is always there.
Any ideas on what could cause this? I've also tried adding:
TOTAL_STACK: 4_000_000_000,
TOTAL_MEMORY: 4_000_000_000,
FAST_MEMORY: 4_000_000_000
to the initSqlJs options.
Activity
lovasoa commentedon Jan 27, 2024
Do you have an easy and minimal way to reproduce your issue?
@kripken , any idea?
lovasoa commentedon Jan 28, 2024
Are you sure
/dist/${filename}is what you need? Did you check if the files were loaded correctly in your custom environment.lovasoa commentedon Jan 28, 2024
Compiling sqlite to js and quickjs to wasm to run one on the other strikes me as bizarre and inefficient. Why not compile sqlite to wasm directly if you have access to a wasm runtime ?
lastmjs commentedon Jan 29, 2024
I could get you a reproduction maybe after our next release, but it's not going to be "easy" because you would have to setup our dev environment.
lastmjs commentedon Jan 29, 2024
I don't think this matters in our case at all since we are using
sql-asm.js, am I correct? I've tried changing that path and even removing it entirely and the behavior is the same.Though I'm not really sure what that path is supposed to do, when I logged it it was looking for a
.wasmfile I believe, so seemed not applicable.lastmjs commentedon Jan 29, 2024
We're basically building a Node/Deno for the Internet Computer which is a Wasm runtime, and we are trying to allow developers to npm install any package and just use it. So we would like SQL.js to work without modifications.
It was working...alas this bug that I can't seem to get around now.
kripken commentedon Jan 29, 2024
Sorry, no - an internal out of memory error is too general.
I'd recommend the usual ideas, though, as before:
-sASSERTIONS)-fsanitize=undefined,address)lovasoa commentedon Jan 30, 2024
@lastmjs , what does the debug build output for you ?
lastmjs commentedon Aug 6, 2024
I think I tracked down the source of the issue, though I don't fully understand yet, my understanding is deepening. This seems to have to do with the order of some of our imports, particularly our importing of a polyfill of TextEncoding and TextDecoding. It's relatively project-specific, so not sure if this has anything to do with sql.js itself.
It would be nice if the error could be better...I'm still investigating. I was able to reproduce and fix is the good part.