Skip to content

Intermittent crashes in CI #1022

Closed
Closed
Intermittent crashes in CI#1022

Description

@mhdawson
mhdawson
opened on Jul 14, 2021

There have been a few recent crashes (maybe 3 in last 15 or so days) in the CI https://ci.nodejs.org/view/x%20-%20Abi%20stable%20module%20API/job/node-test-node-addon-api-LTS%20versions/

A number of the failures look like the following just before the crash

Running test 'objectwrap_constructor_exception'
Running test 'objectwrap_multiple_inheritance'
Running test 'objectwrap_worker_thread'
Running test 'promise'
Running test 'reference'
Running test 'run_script'
Running test 'symbol'
Running test 'threadsafe_function/threadsafe_function'
Running test 'threadsafe_function/threadsafe_function_ctx'
Tests aborted with SIGSEGV
napiVersion:8
Testing with Node-API Version '8'.
Starting test suite

Activity

  1. mhdawson commented on Jul 14, 2021

    @mhdawson
    mhdawson
    on Jul 14, 2021
    MemberAuthor

    I think I've managed to recreate a similar instance and the stack trace looks like:

    sing host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/midawson/newpull/land/node/out/Release/node --expose-gc --no-concurrent-a'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000e9ffe1 in v8::internal::GlobalHandles::Destroy(unsigned long*) ()
[Current thread is 1 (Thread 0x7fda22e537c0 (LWP 2218825))]
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-127.el8.x86_64 libgcc-8.3.1-5.1.el8.x86_64 libstdc++-8.3.1-5.1.el8.x86_64
(gdb) bt
#0  0x0000000000e9ffe1 in v8::internal::GlobalHandles::Destroy(unsigned long*) ()
#1  0x0000000000ad98dd in napi_resolve_deferred ()
#2  0x00007fda20085ebd in Napi::Promise::Deferred::Resolve (this=0x5935dc0, value=0x5945090) at ../../napi-inl.h:2170
#3  0x00007fda200b272c in (anonymous namespace)::TSFNWrap::<lambda(Napi::Env, Napi::Reference<Napi::Value>*)>::operator()(Napi::Env, Napi::Reference<Napi::Value> *) const (__closure=0x5939dc8, env=..., ctx=0x5a09dc0)
    at ../threadsafe_function/threadsafe_function_ctx.cc:51
#4  0x00007fda200b386b in Napi::details::ThreadSafeFinalize<Napi::Reference<Napi::Value>, (anonymous namespace)::TSFNWrap::TSFNWrap(const Napi::CallbackInfo&)::<lambda(Napi::Env, Napi::Reference<Napi::Value>*)>, void>::FinalizeWrapperWithContext(napi_env, void *, void *) (env=0x5979870, rawFinalizeData=0x5939dc0, rawContext=0x5a09dc0) at ../../napi-inl.h:245
#5  0x0000000000aeb97d in node::Environment::CloseHandle<uv_handle_s, v8impl::(anonymous namespace)::ThreadSafeFunction::CloseHandlesAndMaybeDelete(bool)::{lambda(uv_handle_s*)#1}>(uv_handle_s*, v8impl::(anonymous namespace)::ThreadSafeFunction::CloseHandlesAndMaybeDelete(bool)::{lambda(uv_handle_s*)#1})::{lambda(uv_handle_s*)#1}::_FUN(uv_handle_s*) ()
#6  0x00000000015b4155 in uv__finish_close (handle=0x596ed78) at ../deps/uv/src/unix/core.c:303
#7  uv__run_closing_handles (loop=0x487e080 <default_loop_struct>) at ../deps/uv/src/unix/core.c:317
#8  uv_run (loop=0x487e080 <default_loop_struct>, mode=UV_RUN_DEFAULT) at ../deps/uv/src/unix/core.c:395
#9  0x0000000000a63ed5 in node::SpinEventLoop(node::Environment*) () at ../deps/uv/src/unix/core.c:854
#10 0x0000000000b64546 in node::NodeMainInstance::Run(node::EnvSerializeInfo const*) () at ../deps/uv/src/unix/core.c:854
#11 0x0000000000ae9152 in node::Start(int, char**) () at ../deps/uv/src/unix/core.c:854
#12 0x00007fda21b4a7b3 in __libc_start_main () from /lib64/libc.so.6
#13 0x0000000000a619ce in _start () at ../deps/uv/src/unix/core.c:854
(gdb) Quit

  2. mhdawson commented on Jul 14, 2021

    @mhdawson
    mhdawson
    on Jul 14, 2021
    MemberAuthor

    And another one:

    warning: Loadable section ".note.gnu.property" outside of ELF segments
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/midawson/newpull/land/node/out/Release/node --expose-gc --no-concurrent-a'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000ea21e8 in v8::internal::GlobalHandles::Create(v8::internal::Object) ()
[Current thread is 1 (Thread 0x7f7c0579a7c0 (LWP 374115))]
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-127.el8.x86_64 libgcc-8.3.1-5.1.el8.x86_64 libstdc++-8.3.1-5.1.el8.x86_64
(gdb) bt
#0  0x0000000000ea21e8 in v8::internal::GlobalHandles::Create(v8::internal::Object) ()
#1  0x0000000000d0c2d4 in v8::V8::GlobalizeReference(v8::internal::Isolate*, unsigned long*) ()
#2  0x0000000000aeed03 in napi_create_threadsafe_function ()
#3  0x00007f7c01703ead in Napi::ThreadSafeFunction::New<char const*, Napi::Reference<Napi::Value>, (anonymous namespace)::TSFNWrap::TSFNWrap(const Napi::CallbackInfo&)::<lambda(Napi::Env, Napi::Reference<Napi::Value>*)>, void>(napi_env, const Napi::Function &, const Napi::Object &, const char *, size_t, size_t, Napi::Reference<Napi::Value> *, (anonymous namespace)::TSFNWrap::<lambda(Napi::Env, Napi::Reference<Napi::Value>*)>, void *, napi_finalize) (env=0x5378ef0, 
    callback=..., resource=..., resourceName=0x7f7c017380c3 "Test", maxQueueSize=1, initialThreadCount=1, context=0x532ffa0, finalizeCallback=..., data=0x0, 
    wrapper=0x7f7c01703d45 <Napi::details::ThreadSafeFinalize<Napi::Reference<Napi::Value>, (anonymous namespace)::TSFNWrap::TSFNWrap(const Napi::CallbackInfo&)::<lambda(Napi::Env, Napi::Reference<Napi::Value>*)>, void>::FinalizeWrapperWithContext(napi_env, void *, void *)>) at ../../napi-inl.h:5261
#4  0x00007f7c017033b9 in Napi::ThreadSafeFunction::New<char const*, Napi::Reference<Napi::Value>, (anonymous namespace)::TSFNWrap::TSFNWrap(const Napi::CallbackInfo&)::<lambda(Napi::Env, Napi::Reference<Napi::Value>*)> >(napi_env, const Napi::Function &, const Napi::Object &, const char *, size_t, size_t, Napi::Reference<Napi::Value> *, (anonymous namespace)::TSFNWrap::<lambda(Napi::Env, Napi::Reference<Napi::Value>*)>) (env=0x5378ef0, callback=..., resource=..., 
    resourceName=0x7f7c017380c3 "Test", maxQueueSize=1, initialThreadCount=1, context=0x532ffa0, finalizeCallback=...) at ../../napi-inl.h:5119
#5  0x00007f7c01702e78 in (anonymous namespace)::TSFNWrap::TSFNWrap (this=0x53fc120, info=...) at ../threadsafe_function/threadsafe_function_ctx.cc:54
#6  0x00007f7c01704335 in Napi::ObjectWrap<(anonymous namespace)::TSFNWrap>::<lambda()>::operator()(void) const (this=0x7ffdd8d6e370)
    at ../../napi-inl.h:4034
#7  0x00007f7c01704c86 in Napi::details::WrapCallback<Napi::ObjectWrap<T>::ConstructorCallbackWrapper(napi_env, napi_callback_info) [with T = (anonymous namespace)::TSFNWrap]::<lambda()> >(Napi::ObjectWrap<(anonymous namespace)::TSFNWrap>::<lambda()>) (callback=...) at ../../napi-inl.h:82
#8  0x00007f7c0170449f in Napi::ObjectWrap<(anonymous namespace)::TSFNWrap>::ConstructorCallbackWrapper (env=0x5378ef0, info=0x7ffdd8d6e400)
    at ../../napi-inl.h:4032
#9  0x0000000000acfdaf in v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) ()
#10 0x0000000000d72ac5 in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<true>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) ()
#11 0x0000000000d730af in v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) ()
#12 0x0000000000d736d6 in v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) ()
#13 0x0000000001638539 in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit ()
#14 0x00000000015ceff8 in Builtins_JSBuiltinsConstructStub ()

  3. mhdawson commented on Jul 14, 2021

    @mhdawson
    mhdawson
    on Jul 14, 2021
    MemberAuthor

    Related to the last crash

    41 TSFNWrap::TSFNWrap(const CallbackInfo &info) : ObjectWrap<TSFNWrap>(info) {
 42   Napi::Env env = info.Env();
 43 
 44   Reference<Napi::Value> *_ctx = new Reference<Napi::Value>;
 45   *_ctx = Persistent(info[0]);
 46 
 47   _tsfn = ThreadSafeFunction::New(
 48       info.Env(), Function::New(env, [](const CallbackInfo & /*info*/) {}),
 49       Object::New(env), "Test", 1, 1, _ctx,
 50       [this](Napi::Env env, Reference<Napi::Value> *ctx) {
 51         _deferred->Resolve(env.Undefined());
 52         ctx->Reset();
 53         delete ctx;
 54       });
 55 }
 56 
 57 } // namespace

  4. mhdawson commented on Jul 14, 2021

    @mhdawson
    mhdawson
    on Jul 14, 2021
    MemberAuthor

    And

    #0  0x0000000000ea21e8 in v8::internal::GlobalHandles::Create(v8::internal::Object) ()
[Current thread is 1 (Thread 0x7fd9b2a657c0 (LWP 900741))]
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-127.el8.x86_64 libgcc-8.3.1-5.1.el8.x86_64 libstdc++-8.3.1-5.1.el8.x86_64
(gdb) bt
#0  0x0000000000ea21e8 in v8::internal::GlobalHandles::Create(v8::internal::Object) ()
#1  0x0000000000d0c2d4 in v8::V8::GlobalizeReference(v8::internal::Isolate*, unsigned long*) ()
#2  0x0000000000ad96d3 in napi_create_promise ()
#3  0x00007fd9b01a8e41 in Napi::Promise::Deferred::Deferred (this=0x51c1790, env=0x5208ef0) at ../../napi-inl.h:2157
#4  0x00007fd9b01d0adc in (anonymous namespace)::TSFNWrap::Release (this=0x534da80, info=...) at ../threadsafe_function/threadsafe_function_ctx.cc:21
#5  0x00007fd9b01d14ce in Napi::InstanceWrap<(anonymous namespace)::TSFNWrap>::<lambda()>::operator()(void) const (this=0x7fffef3ba650)
    at ../../napi-inl.h:3617
#6  0x00007fd9b01d2001 in Napi::details::WrapCallback<Napi::InstanceWrap<T>::InstanceMethodCallbackWrapper(napi_env, napi_callback_info) [with T = (anonymous namespace)::TSFNWrap]::<lambda()> >(Napi::InstanceWrap<(anonymous namespace)::TSFNWrap>::<lambda()>) (callback=...) at ../../napi-inl.h:82
#7  0x00007fd9b01d152b in Napi::InstanceWrap<(anonymous namespace)::TSFNWrap>::InstanceMethodCallbackWrapper (env=0x5208ef0, info=0x7fffef3ba6b0)
    at ../../napi-inl.h:3610
#8  0x0000000000acfdaf in v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) ()
#9  0x0000000000d71bab in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) ()
#10 0x0000000000d7305c in v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) ()
#11 0x0000000000d736d6 in v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) ()
#12 0x0000000001638539 in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit ()
#13 0x00000000015d17eb in Builtins_InterpreterEntryTrampoline ()

  5. mhdawson commented on Jul 14, 2021

    @mhdawson
    mhdawson
    on Jul 14, 2021
    MemberAuthor

    Going to recompile node in debug mode to see if we get any more info as to what is going on in the GlobalHandles code.

  6. mhdawson commented on Jul 14, 2021

    @mhdawson
    mhdawson
    on Jul 14, 2021
    MemberAuthor

    Of note I am testing with current master, we've seen the similar failure in master. From the CI this is what we have seen lately:

  7. mhdawson commented on Jul 14, 2021

    @mhdawson
    mhdawson
    on Jul 14, 2021
    MemberAuthor
    Jul 4 - AIX - Node.js 14.x - crash after

Running test 'threadsafe_function/threadsafe_function'
Running test 'threadsafe_function/threadsafe_function_ctx'
Tests aborted with SIGSEGV 

June 28 - OSX 15 - master - crash after

Running test 'threadsafe_function/threadsafe_function'
Running test 'threadsafe_function/threadsafe_function_ctx'
Tests aborted with SIGSEGV

Jun 25 - OSX 14 - 16.x - 
Running test 'threadsafe_function/threadsafe_function'
Running test 'threadsafe_function/threadsafe_function_ctx'
Mismatched noop function calls. Expected exactly 1, actual 0.
    at Object.exports.mustCall (/Users/iojs/build/workspace/node-test-node-addon-api-new/no

From other issues believe this might have been OSX specific issue

June 22 - Fedora - 14.x - Crash after

Running test 'threadsafe_function/threadsafe_function'
Running test 'threadsafe_function/threadsafe_function_ctx'
Tests aborted with SIGSEGV
npm ERR! Test failed.  See

  8. mhdawson commented on Jul 15, 2021

    @mhdawson
    mhdawson
    on Jul 15, 2021
    MemberAuthor

    Seems like a debug check fires in the same place

    warning: Loadable section ".note.gnu.property" outside of ELF segments
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/midawson/newpull/land/node/out/Debug/node --expose-gc --no-concurrent-arr'.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  v8::base::OS::<lambda()>::operator() (__closure=<optimized out>) at ../deps/v8/src/base/platform/platform-posix.cc:502
502	    IMMEDIATE_CRASH();
[Current thread is 1 (Thread 0x7efe8ee2f7c0 (LWP 1991247))]
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-127.el8.x86_64 libgcc-8.3.1-5.1.el8.x86_64 libstdc++-8.3.1-5.1.el8.x86_64
(gdb) bt
#0  v8::base::OS::<lambda()>::operator() (__closure=<optimized out>) at ../deps/v8/src/base/platform/platform-posix.cc:502
#1  v8::base::OS::Abort () at ../deps/v8/src/base/platform/platform-posix.cc:502
#2  0x0000000002d81368 in V8_Fatal (file=0x36a6a20 "../deps/v8/src/handles/global-handles.cc", line=497, 
    format=format@entry=0x59240d2 "Debug check failed: %s.") at ../deps/v8/src/base/logging.cc:167
#3  0x0000000002d81383 in v8::base::(anonymous namespace)::DefaultDcheckHandler (file=<optimized out>, line=<optimized out>, 
    message=<optimized out>) at ../deps/v8/src/base/logging.cc:57
#4  0x000000000166cbfe in v8::internal::GlobalHandles::Node::next_free (this=<optimized out>) at /usr/include/c++/8/bits/basic_string.h:2290
#5  v8::internal::GlobalHandles::NodeSpace<v8::internal::GlobalHandles::Node>::Acquire (object=..., this=0x748d4f0)
    at ../deps/v8/src/handles/global-handles.cc:224
#6  v8::internal::GlobalHandles::Create (this=0x748d440, value=...) at ../deps/v8/src/handles/global-handles.cc:937
#7  0x000000000166cd65 in v8::internal::GlobalHandles::Create (this=<optimized out>, value=<optimized out>)
    at ../deps/v8/src/objects/tagged-impl.h:38
#8  0x00000000013ca5fd in v8::V8::GlobalizeReference (isolate=0x7467280, obj=0x74d7ef8) at ../deps/v8/src/execution/isolate.h:1125
#9  0x00000000010be9c5 in v8::PersistentBase<v8::Value>::New (isolate=0x7467280, that=0x74d7ef8) at ../deps/v8/include/v8.h:10906
#10 0x00000000010d81d4 in v8::PersistentBase<v8::Value>::Reset<v8::Promise::Resolver> (this=0x74576a0, isolate=0x7467280, other=...)
    at ../deps/v8/include/v8.h:10945
#11 0x00000000010d5b40 in napi_create_promise (env=0x75e63c0, deferred=0x74587d8, promise=0x74587e0) at ../src/js_native_api_v8.cc:3042
#12 0x00007efe7ed5fdf2 in Napi::Promise::Deferred::Deferred (this=0x74587d0, env=0x75e63c0) at ../../napi-inl.h:2157
#13 0x00007efe7ed8c520 in (anonymous namespace)::TSFNWrap::Release (this=0x75dc050, info=...)
    at ../threadsafe_function/threadsafe_function_ctx.cc:21
#14 0x00007efe7ed8cfac in Napi::InstanceWrap<(anonymous namespace)::TSFNWrap>::<lambda()>::operator()(void) const (this=0x7ffe1c6fbd30)
    at ../../napi-inl.h:3617
#15 0x00007efe7ed8dadb in Napi::details::WrapCallback<Napi::InstanceWrap<T>::InstanceMethodCallbackWrapper(napi_env, napi_callback_info) [with T = (anonymous namespace)::TSFNWrap]::<lambda()> >(Napi::InstanceWrap<(anonymous namespace)::TSFNWrap>::<lambda()>) (callback=...)
    at ../../napi-inl.h:74
#16 0x00007efe7ed8d029 in Napi::InstanceWrap<(anonymous namespace)::TSFNWrap>::InstanceMethodCallbackWrapper (env=0x75e63c0, 
    info=0x7ffe1c6fbe90) at ../../napi-inl.h:3610
#17 0x00000000010c9662 in v8impl::(anonymous namespace)::CallbackWrapperBase::<lambda(napi_env)>::operator()(napi_env) const (
    __closure=0x7ffe1c6fbe50, env=0x75e63c0) at ../src/js_native_api_v8.cc:311
#18 0x00000000010d67c9 in napi_env__::CallIntoModule<v8impl::(anonymous namespace)::CallbackWrapperBase::InvokeCallback()::<lambda(napi_env)> >(v8impl::(anonymous namespace)::CallbackWrapperBase::<lambda(napi_env)> &&, void (&&)(napi_env__ *, v8::Local<v8::Value>)) (this=0x75e63c0, 
    call=..., handle_exception=
    @0x10d73da: {void (napi_env__ *, v8::Local<v8::Value>)} 0x10d73da <napi_env__::HandleThrow(napi_env__*, v8::Local<v8::Value>)>)
    at ../src/js_native_api_v8.h:95
#19 0x00000000010c96d1 in v8impl::(anonymous namespace)::CallbackWrapperBase::InvokeCallback (this=0x7ffe1c6fbe90)
    at ../src/js_native_api_v8.cc:310
#20 0x00000000010c9726 in v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke (info=...) at ../src/js_native_api_v8.cc:328
--Type <RET> for more, q to quit, c to continue without paging-- 
#21 0x000000000146bce0 in v8::internal::FunctionCallbackArguments::Call (this=this@entry=0x7ffe1c6fc000, handler=handler@entry=...)
    at ../deps/v8/src/api/api-arguments-inl.h:158
#22 0x000000000146cb07 in v8::internal::(anonymous namespace)::HandleApiCallHelper<false> (isolate=isolate@entry=0x7467280, function=..., 
    function@entry=..., new_target=..., new_target@entry=..., fun_data=..., receiver=..., receiver@entry=..., args=...)
    at ../deps/v8/src/base/platform/wrappers.h:37
#23 0x0000000001471762 in v8::internal::Builtin_Impl_HandleApiCall (args=..., isolate=isolate@entry=0x7467280)
    at ../deps/v8/src/handles/handles.h:132
#24 0x00000000014724e7 in v8::internal::Builtin_HandleApiCall (args_length=5, args_object=0x7ffe1c6fc1a8, isolate=0x7467280)
    at ../deps/v8/src/builtins/builtins-api.cc:131
#25 0x00000000023156a0 in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit () at ../../deps/v8/src/builtins/torque-internal.tq:84
#26 0x000000000209102b in Builtins_InterpreterEntryTrampoline () at ../../deps/v8/src/objects/string.tq:183

  9. mhdawson commented on Jul 15, 2021

    @mhdawson
    mhdawson
    on Jul 15, 2021
    MemberAuthor

    I'm wondering if this is related to the issue we see in the ci or specific to main. Going to run with 16.x to see if we get the same debug checks there.

  10. mhdawson commented on Jul 15, 2021

    @mhdawson
    mhdawson
    on Jul 15, 2021
    MemberAuthor

    16.x seems to have less, just 2 in over 10000 runs so far. This is one of the stack traces:

    sing host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/midawson/newpull/land/node/out/Debug/node --expose-gc --no-concurrent-arr'.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  v8::base::OS::<lambda()>::operator() (__closure=<optimized out>) at ../deps/v8/src/base/platform/platform-posix.cc:502
502	    IMMEDIATE_CRASH();
[Current thread is 1 (Thread 0x7f79e4f687c0 (LWP 3797863))]
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-127.el8.x86_64 libgcc-8.3.1-5.1.el8.x86_64 libstdc++-8.3.1-5.1.el8.x86_64
(gdb) bt
#0  v8::base::OS::<lambda()>::operator() (__closure=<optimized out>) at ../deps/v8/src/base/platform/platform-posix.cc:502
#1  v8::base::OS::Abort () at ../deps/v8/src/base/platform/platform-posix.cc:502
#2  0x0000000002d82068 in V8_Fatal (file=0x3698260 "../deps/v8/src/handles/global-handles.cc", line=497, 
    format=format@entry=0x590eef2 "Debug check failed: %s.") at ../deps/v8/src/base/logging.cc:167
#3  0x0000000002d82083 in v8::base::(anonymous namespace)::DefaultDcheckHandler (file=<optimized out>, line=<optimized out>, 
    message=<optimized out>) at ../deps/v8/src/base/logging.cc:57
#4  0x000000000166e0ae in v8::internal::GlobalHandles::Node::next_free (this=<optimized out>) at /usr/include/c++/8/bits/basic_string.h:2290
#5  v8::internal::GlobalHandles::NodeSpace<v8::internal::GlobalHandles::Node>::Acquire (object=..., this=0x7be9170)
    at ../deps/v8/src/handles/global-handles.cc:224
#6  v8::internal::GlobalHandles::Create (this=0x7be90c0, value=...) at ../deps/v8/src/handles/global-handles.cc:937
#7  0x000000000166e215 in v8::internal::GlobalHandles::Create (this=<optimized out>, value=<optimized out>)
    at ../deps/v8/src/objects/tagged-impl.h:38
#8  0x00000000013ca55d in v8::V8::GlobalizeReference (isolate=0x7bc2fb0, obj=0x7c33b78) at ../deps/v8/src/execution/isolate.h:1125
#9  0x00000000010be995 in v8::PersistentBase<v8::Value>::New (isolate=0x7bc2fb0, that=0x7c33b78) at ../deps/v8/include/v8.h:11146
#10 0x00000000010d81d4 in v8::PersistentBase<v8::Value>::Reset<v8::Promise::Resolver> (this=0x7c7a0e0, isolate=0x7bc2fb0, other=...)
    at ../deps/v8/include/v8.h:11185
#11 0x00000000010d5b38 in napi_create_promise (env=0x7d30b10, deferred=0x7bb4628, promise=0x7bb4630) at ../src/js_native_api_v8.cc:3045
#12 0x00007f79e1198df2 in Napi::Promise::Deferred::Deferred (this=0x7bb4620, env=0x7d30b10) at ../../napi-inl.h:2157
#13 0x00007f79e11c5520 in (anonymous namespace)::TSFNWrap::Release (this=0x7cc87a0, info=...)
    at ../threadsafe_function/threadsafe_function_ctx.cc:21
#14 0x00007f79e11c5fac in Napi::InstanceWrap<(anonymous namespace)::TSFNWrap>::<lambda()>::operator()(void) const (this=0x7ffcbde96e40)
    at ../../napi-inl.h:3617
#15 0x00007f79e11c6adb in Napi::details::WrapCallback<Napi::InstanceWrap<T>::InstanceMethodCallbackWrapper(napi_env, napi_callback_info) [with T = (anonymous namespace)::TSFNWrap]::<lambda()> >(Napi::InstanceWrap<(anonymous namespace)::TSFNWrap>::<lambda()>) (callback=...)
    at ../../napi-inl.h:74
#16 0x00007f79e11c6029 in Napi::InstanceWrap<(anonymous namespace)::TSFNWrap>::InstanceMethodCallbackWrapper (env=0x7d30b10, 
    info=0x7ffcbde96fa0) at ../../napi-inl.h:3610
#17 0x00000000010c9632 in v8impl::(anonymous namespace)::CallbackWrapperBase::<lambda(napi_env)>::operator()(napi_env) const (
    __closure=0x7ffcbde96f60, env=0x7d30b10) at ../src/js_native_api_v8.cc:311
#18 0x00000000010d67c1 in napi_env__::CallIntoModule<v8impl::(anonymous namespace)::CallbackWrapperBase::InvokeCallback()::<lambda(napi_env)> >(v8impl::(anonymous namespace)::CallbackWrapperBase::<lambda(napi_env)> &&, void (&&)(napi_env__ *, v8::Local<v8::Value>)) (this=0x7d30b10, 
    call=..., handle_exception=
    @0x10d73d2: {void (napi_env__ *, v8::Local<v8::Value>)} 0x10d73d2 <napi_env__::HandleThrow(napi_env__*, v8::Local<v8::Value>)>)
    at ../src/js_native_api_v8.h:95
#19 0x00000000010c96a1 in v8impl::(anonymous namespace)::CallbackWrapperBase::InvokeCallback (this=0x7ffcbde96fa0)
    at ../src/js_native_api_v8.cc:310
#20 0x00000000010c96f6 in v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke (info=...) at ../src/js_native_api_v8.cc:328
#21 0x000000000146d190 in v8::internal::FunctionCallbackArguments::Call (this=this@entry=0x7ffcbde97110, handler=handler@entry=...)
    at ../deps/v8/src/api/api-arguments-inl.h:158
#22 0x000000000146dfb7 in v8::internal::(anonymous namespace)::HandleApiCallHelper<false> (isolate=isolate@entry=0x7bc2fb0, function=..., 
    function@entry=..., new_target=..., new_target@entry=..., fun_data=..., receiver=..., receiver@entry=..., args=...)
    at ../deps/v8/src/base/platform/wrappers.h:37
#23 0x0000000001472c12 in v8::internal::Builtin_Impl_HandleApiCall (args=..., isolate=isolate@entry=0x7bc2fb0)
    at ../deps/v8/src/handles/handles.h:132
#24 0x0000000001473997 in v8::internal::Builtin_HandleApiCall (args_length=5, args_object=0x7ffcbde972b8, isolate=0x7bc2fb0)
    at ../deps/v8/src/builtins/builtins-api.cc:131
#25 0x0000000002317020 in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit () at ../../deps/v8/src/builtins/torque-internal.tq:84
#26 0x00000000020929ab in Builtins_InterpreterEntryTrampoline () at ../../deps/v8/src/objects/string.tq:183

  11. KevinEady commented on Jul 15, 2021

    @KevinEady
    KevinEady
    on Jul 15, 2021
    Contributor

    I'm not sure but all of the stack traces have GlobalHandles Create and Destroy involved... Is there something wrong with the test's code? Should I create handles? For example, I don't create a handle in Finalize because there wasn't one done in the original Node-API tsfn test https://github.com/nodejs/node/blob/4b0776ab64596d1bbd98a93e3de7cee82a7e8130/test/node-api/test_threadsafe_function/binding.c#L167-L188

    But this one is odd:

    And another one:

    warning: Loadable section ".note.gnu.property" outside of ELF segments
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/midawson/newpull/land/node/out/Release/node --expose-gc --no-concurrent-a'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000ea21e8 in v8::internal::GlobalHandles::Create(v8::internal::Object) ()
[Current thread is 1 (Thread 0x7f7c0579a7c0 (LWP 374115))]
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-127.el8.x86_64 libgcc-8.3.1-5.1.el8.x86_64 libstdc++-8.3.1-5.1.el8.x86_64
(gdb) bt
#0  0x0000000000ea21e8 in v8::internal::GlobalHandles::Create(v8::internal::Object) ()
#1  0x0000000000d0c2d4 in v8::V8::GlobalizeReference(v8::internal::Isolate*, unsigned long*) ()
#2  0x0000000000aeed03 in napi_create_threadsafe_function ()
#3  0x00007f7c01703ead in Napi::ThreadSafeFunction::New<char const*, Napi::Reference<Napi::Value>, (anonymous namespace)::TSFNWrap::TSFNWrap(const Napi::CallbackInfo&)::<lambda(Napi::Env, Napi::Reference<Napi::Value>*)>, void>(napi_env, const Napi::Function &, const Napi::Object &, const char *, size_t, size_t, Napi::Reference<Napi::Value> *, (anonymous namespace)::TSFNWrap::<lambda(Napi::Env, Napi::Reference<Napi::Value>*)>, void *, napi_finalize) (env=0x5378ef0, 
    callback=..., resource=..., resourceName=0x7f7c017380c3 "Test", maxQueueSize=1, initialThreadCount=1, context=0x532ffa0, finalizeCallback=..., data=0x0, 
    wrapper=0x7f7c01703d45 <Napi::details::ThreadSafeFinalize<Napi::Reference<Napi::Value>, (anonymous namespace)::TSFNWrap::TSFNWrap(const Napi::CallbackInfo&)::<lambda(Napi::Env, Napi::Reference<Napi::Value>*)>, void>::FinalizeWrapperWithContext(napi_env, void *, void *)>) at ../../napi-inl.h:5261
#4  0x00007f7c017033b9 in Napi::ThreadSafeFunction::New<char const*, Napi::Reference<Napi::Value>, (anonymous namespace)::TSFNWrap::TSFNWrap(const Napi::CallbackInfo&)::<lambda(Napi::Env, Napi::Reference<Napi::Value>*)> >(napi_env, const Napi::Function &, const Napi::Object &, const char *, size_t, size_t, Napi::Reference<Napi::Value> *, (anonymous namespace)::TSFNWrap::<lambda(Napi::Env, Napi::Reference<Napi::Value>*)>) (env=0x5378ef0, callback=..., resource=..., 
    resourceName=0x7f7c017380c3 "Test", maxQueueSize=1, initialThreadCount=1, context=0x532ffa0, finalizeCallback=...) at ../../napi-inl.h:5119
#5  0x00007f7c01702e78 in (anonymous namespace)::TSFNWrap::TSFNWrap (this=0x53fc120, info=...) at ../threadsafe_function/threadsafe_function_ctx.cc:54
#6  0x00007f7c01704335 in Napi::ObjectWrap<(anonymous namespace)::TSFNWrap>::<lambda()>::operator()(void) const (this=0x7ffdd8d6e370)
    at ../../napi-inl.h:4034
#7  0x00007f7c01704c86 in Napi::details::WrapCallback<Napi::ObjectWrap<T>::ConstructorCallbackWrapper(napi_env, napi_callback_info) [with T = (anonymous namespace)::TSFNWrap]::<lambda()> >(Napi::ObjectWrap<(anonymous namespace)::TSFNWrap>::<lambda()>) (callback=...) at ../../napi-inl.h:82
#8  0x00007f7c0170449f in Napi::ObjectWrap<(anonymous namespace)::TSFNWrap>::ConstructorCallbackWrapper (env=0x5378ef0, info=0x7ffdd8d6e400)
    at ../../napi-inl.h:4032
#9  0x0000000000acfdaf in v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) ()
#10 0x0000000000d72ac5 in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<true>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) ()
#11 0x0000000000d730af in v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) ()
#12 0x0000000000d736d6 in v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) ()
#13 0x0000000001638539 in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit ()
#14 0x00000000015ceff8 in Builtins_JSBuiltinsConstructStub ()

    This one is part of the synchronous stack from JavaScript -> C++ binding

    node-addon-api/test/threadsafe_function/threadsafe_function_ctx.cc

    Line 47 in e02e8a4

    _tsfn = ThreadSafeFunction::New(
    No clue how this could fail...?

  12. mhdawson commented on Jul 15, 2021

    @mhdawson
    mhdawson
    on Jul 15, 2021
    MemberAuthor

    I tried running just the one test and 30000 iterations led to no recreates so trying with larger set of tests.

  13. mhdawson
    closed this as completedon Jul 15, 2021
  14. mhdawson
    reopened this on Jul 15, 2021

  15. mhdawson commented on Jul 15, 2021

    @mhdawson
    mhdawson
    on Jul 15, 2021
    MemberAuthor

    @KevinEady still trying to narrow down, I agree it's not obvious why some of those failures could/would happen. I'm thinking maybe we are getting memory corruption which then leads to those asserts. Trying to narrow down recreate but seems like it running a single test does not result in the problem.

  16. mhdawson commented on Jul 16, 2021

    @mhdawson
    mhdawson
    on Jul 16, 2021
    MemberAuthor

    Have been working through removing just a subset of tests. Seems like if I remove just

    objectwrap_worker_thread then I got no failures in 26300 runs. Typically I would have gotten failures long before then.

  17. mhdawson commented on Jul 16, 2021

    @mhdawson
    mhdawson
    on Jul 16, 2021
    MemberAuthor

    objectwrap_worker_thread:

    'use strict';
const { Worker, isMainThread, workerData } = require('worker_threads');

if (isMainThread) {
    const buildType = process.config.target_defaults.default_configuration;
    new Worker(__filename, { workerData: buildType });
} else {
    const test = binding => {
        new binding.objectwrap.Test();
    };

    const buildType = workerData;
    require('./common').runTest(test, buildType);
}

  18. mhdawson commented on Jul 16, 2021

    @mhdawson
    mhdawson
    on Jul 16, 2021
    MemberAuthor

    Turns out the worker in objectwrap_worker_thread was running in parallel with other tests, not sure if that is related to the crash but have a refactored version that makes sure worker has completed before running the rest of the tests to test out.

  19. mhdawson commented on Jul 16, 2021

    @mhdawson
    mhdawson
    on Jul 16, 2021
    MemberAuthor

    15000 runs with the refactor and looking good. Heading out for a long weekend so I don't want to leave running but I'll kickoff longer run on Tuesday and then follow up with PR if its still looks good.

  20. mhdawson commented on Jul 20, 2021

    @mhdawson
    mhdawson
    on Jul 20, 2021
    MemberAuthor

    Ok so 41595 more successful runs with the change, will submit a PR. There is an unrelated issue I found along the way but will open a different issue fort that.

  21. mhdawson
    added a commit that references this issue on Jul 20, 2021

    doc: update tests to avoid running in parallel

    8858f5d

  22. mhdawson commented on Jul 20, 2021

    @mhdawson
    mhdawson
    on Jul 20, 2021
    MemberAuthor

    Submitted #1024

  23. mhdawson
    closed this as completedin a459f5con Jul 27, 2021
  24. deepakrkris
    added a commit that references this issue on Sep 23, 2021

    doc: update tests to avoid running in parallel

    6ff72ac
  25. deepakrkris
    added a commit that references this issue on Oct 15, 2021

    doc: update tests to avoid running in parallel

    e7b7409
  26. kevindavies8
    added a commit that references this issue on Aug 24, 2022

    doc: update tests to avoid running in parallel

    5f07939
  27. Marlyfleitas
    added a commit that references this issue on Aug 26, 2022

    doc: update tests to avoid running in parallel

    af9274f
  28. wroy7860
    added a commit that references this issue on Sep 19, 2022

    doc: update tests to avoid running in parallel

    a9b9ee6
  29. johnfrench3
    added a commit that references this issue on Aug 11, 2023

    doc: update tests to avoid running in parallel

    c208c66
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions