Mochawesome/Testomatio/junit reporter: run-workers: Running of test with --suite doesn't aggregate results from all suites per browser

The issue is even in older versions, e.g. in 3.5.3. Retested with the latest 3.7.6 and the issue persists.

npm i mochawesome
npm i mocha-junit-reporter
npm i mocha-multi
Run the tests codeceptjs run-workers 3 dev --by suite --reporter mocha-multi (parallelization per browser and per suite). Another variant is --suites instead of --by suite, but it behaves the same way.

$ codeceptjs run-workers 3 dev --by suite --reporter mocha-multi
CodeceptJS v3.7.6 #StandWithUkraine
Running tests in 3 workers...

[Worker 03]   ✔ SuiteB › Scenario 1 in 4ms
[Worker 03]   ✔ SuiteB › Scenario 2 in 1ms
[Worker 01]   ✔ SuiteA › Scenario 1 in 5ms
[Worker 01]   ✔ SuiteA › Scenario 2 in 1ms
[Worker 02]   ✔ SuiteA › Scenario 1 in 3ms
[Worker 02]   ✔ SuiteA › Scenario 2 in 1ms
[Worker 04]   ✔ SuiteB › Scenario 1 in 4ms
[Worker 04]   ✔ SuiteB › Scenario 2 in 1ms

  OK  | 8 passed   // 3s

Expected result:

output/console.log contains data from both test suites
output/dev_chromium1/report.html contains data from both test suites
output/dev_chromium1/report.json contains data from both test suites
output/dev_chromium1/result.xml contains data from both test suites
output/dev_firefox1/report.html contains data from both test suites
output/dev_firefox1/report.json contains data from both test suites
output/dev_firefox1/result.xml contains data from both test suites

This way it works with run-multiple (parallelization per browser, but test suites are run in sequence) ✔️

Actual result:

All the mentioned files contains data from the SuiteA only. The SuiteB is missing there 🐞 . A problem seems to be that test suites overwrite each other's results when running in threads.

Config codecept.conf.ts:

import { setHeadlessWhen, setCommonPlugins } from "@codeceptjs/configure";
// turn on headless mode when running with HEADLESS=true environment variable
// export HEADLESS=true && npx codeceptjs run
setHeadlessWhen(process.env.HEADLESS);

// enable all common plugins https://github.com/codeceptjs/configure#setcommonplugins
setCommonPlugins();

export const config: CodeceptJS.MainConfig = {
  tests: "./*_test.ts",
  output: "./output",
  helpers: {
    Playwright: {
      browser: "chromium",
      url: "http://localhost",
      show: true,
    },
    Mochawesome: {
      uniqueScreenshotNames: true,
    },
  },
  include: {
    I: "./steps_file",
  },
  multiple: {
    dev: {
      browsers: [
        {
          browser: "chromium",
          outputName: "chromium",
        },
        {
          browser: "firefox",
          outputName: "firefox",
        },
      ],
    },
  },
  mocha: {
    reporterOptions: {
      "codeceptjs-cli-reporter": {
        stdout: "-",
        options: {
          verbose: true,
          debug: true,
          steps: true,
        },
      },
      mochawesome: {
        stdout: "./output/console.log",
        options: {
          reportDir: "./output",
          reportFilename: "report",
        },
      },
      "mocha-junit-reporter": {
        stdout: "./output/console.log",
        options: {
          mochaFile: "./output/result.xml",
          jenkinsMode: true,
          attachments: true, //add screenshot for a failed test
        },
      },
    },
  },

  name: "my",
};

Test suites:

SuiteA_test.ts

Feature("SuiteA");

Scenario("Scenario 1", ({ I }) => {
  I.say("Executing Scenario 1 in Suite A");
});

Scenario("Scenario 2", ({ I }) => {
  I.say("Executing Scenario 2 in Suite A");
});

SuiteB_test.ts

Feature("SuiteB");

Scenario("Scenario 1", ({ I }) => {
  I.say("Executing Scenario 1 in Suite B");
});

Scenario("Scenario 2", ({ I }) => {
  I.say("Executing Scenario 2 in Suite B");
});

Used SW:

CodeceptJS 3.7.6
mochawesome 7.1.4
mocha-junit-reporter 2.2.1
mocha-multi 1.1.7

fix: aggregate custom reporter results in run-workers --by suite (cod…

This issue is stale because it has been open for 90 days with no activity.

@DavertMik

Mochawesome is deprecated in 4.x and JUnit XML was rewritten into a plugin:

https://codecept.io/reports#custom-reporter:

So I retested it with the Testomatio reporter and the new JUnit XML plugin.

The JUnit XML works well, I'm getting extra XML for Chromium and Webkit and they don't overwrite themselves 👍 .

mirao@rog:/tmp/runmultiple401$ ls -l output/dev_chromium1/report.xml 
-rw-rw-r-- 1 mirao mirao 754 May 24 09:49 output/dev_chromium1/report.xml
mirao@rog:/tmp/runmultiple401$ cat output/dev_chromium1/report.xml 
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="CodeceptJS" tests="1" failures="0" skipped="0" errors="0" time="0.294" timestamp="2026-05-24T07:49:46.546Z"><testsuite name="My" id="0" tests="1" failures="0" skipped="0" errors="0" time="0.294" timestamp="2026-05-24T07:49:47.033Z" hostname="rog" file="/tmp/runmultiple401/tests/My_test.js"><testcase name="test something" classname="My" time="0.294" file="/tmp/runmultiple401/tests/My_test.js"><properties><property name="browser" value="chromium"/><property name="browserVersion" value="148.0.7778.96"/><property name="windowSize" value="1280x720"/></properties><system-out>I am on page "https://example.com" (287ms)
I see "Example Domain" (3ms)</system-out></testcase></testsuite></testsuites>

mirao@rog:/tmp/runmultiple401$ ls -l output/dev_webkit1/report.xml 
-rw-rw-r-- 1 mirao mirao 743 May 24 09:49 output/dev_webkit1/report.xml
mirao@rog:/tmp/runmultiple401$ cat output/dev_webkit1/report.xml 
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="CodeceptJS" tests="1" failures="0" skipped="0" errors="0" time="0.323" timestamp="2026-05-24T07:49:48.570Z"><testsuite name="My" id="0" tests="1" failures="0" skipped="0" errors="0" time="0.323" timestamp="2026-05-24T07:49:49.125Z" hostname="rog" file="/tmp/runmultiple401/tests/My_test.js"><testcase name="test something" classname="My" time="0.323" file="/tmp/runmultiple401/tests/My_test.js"><properties><property name="browser" value="webkit"/><property name="browserVersion" value="26.4"/><property name="windowSize" value="1280x720"/></properties><system-out>I am on page "https://example.com" (316ms)
I see "Example Domain" (3ms)</system-out></testcase></testsuite></testsuites>

But the issue with overwritten HTML persists 👎 : Only one report is created and it contains data from the worker that wrote there last time.

mirao@rog:/tmp/runmultiple401$ codeceptjs run-workers 1 dev --by suite
CodeceptJS v4.0.1 #StandWithUkraine
Running tests in 1 workers...
[TESTOMATIO] Testomatio Reporter v2.8.2

Test distribution:
  Worker 1: 1 test
  Worker 2: 1 test

[TESTOMATIO] Pipes: HTML Reporter
[Worker 01] CodeceptJS v4.0.1 #StandWithUkraine
[Worker 01] Using test root "/tmp/runmultiple401"
[Worker 01] My --
[Worker 01]   ✔ My › test something in 180ms
[Worker 01]   OK  | 1 passed   // 362ms
[Worker 02] CodeceptJS v4.0.1 #StandWithUkraine
[Worker 02] Using test root "/tmp/runmultiple401"
[Worker 02] My --
[Worker 02]   ✔ My › test something in 228ms
[Worker 02]   OK  | 1 passed   // 467ms
[Worker 01]   ✔ My › test something in 228ms

  OK  | 2 passed   // 5s
⏳ The test results will be added to the HTML report. It will take some time...
📊 The HTML report was successfully generated. Full filepath: file:///tmp/runmultiple401/output/report/testomatio-report.html

The report contains data from WebKit only (the data for Chromium were overwritten) 🐞 .

CC: @DavertMik

🤖 AI Analysis

Why the HTML report shows only one browser's results

When running run-workers N dev --by suite with multiple browsers configured (e.g. browsers: ['chromium', 'webkit']), CodeceptJS creates one worker per browser per suite. Each browser run executes the same tests, so their test UIDs are identical (UID = SHA-256 hash of test.fullTitle()).

The Testomatio reporter's aggregateTestRetries() in html.js groups test results by rid (= UID):

const key = rid ? `rid:${rid}` : `ft:${t?.file || ''}|${t?.title || ''}`

Since Chromium and WebKit runs of the same test produce identical UIDs, they end up in the same group. Only one browser's result survives (the last writer wins).

Part 1 — What needs to change in CodeceptJS

File: lib/workers.js

When building workers for multiple browser runs (selectedRuns), store each run's name (e.g. dev:chromium1, dev:webkit1) on the WorkerObject. Then, when dispatching event.test.after back to the main process, append the run name to the test UID before passing it to the event dispatcher:

_applyRunNameToTest(workerIndex, test) {
  const workerObj = this.workers[workerIndex - 1]
  if (!workerObj || !workerObj.runName) return
  const runSuffix = workerObj.runName.replace(/[^a-zA-Z0-9]/g, '_')
  if (test.uid) test.uid = `${test.uid}_${runSuffix}`
  if (!test.meta) test.meta = {}
  test.meta.env = workerObj.runName  // e.g. 'dev:chromium1'
}

Effect: Tests from different browser runs have distinct UIDs (e.g. abc123_dev_chromium1 vs abc123_dev_webkit1), so aggregateTestRetries() no longer merges them. The HTML report shows all results per browser (e.g. 4 tests × 2 browsers = 8 entries) instead of 4. This approach was tested and verified by the issue reporter.

Part 2 — What needs to change in Testomatio reporter

Each test would carry meta.env = 'dev:chromium1' (or 'dev:webkit1'). The data would be there, but the HTML template doesn't expose it as a filter.

What would be needed in @testomatio/reporter:

Read meta.env (or a dedicated browser field) on each test result
Add a Browser / Environment filter pill to the HTML report UI (similar to the existing passed/failed/skipped status filter)
Optionally group or label each test row with its browser name

This is a pure UI/template change inside the HTML pipe — the data plumbing from CodeceptJS to the reporter would be handled by the Part 1 fix above.

Retest with 4.2.1 and @testomatio/reporter 2.18.0, plus a working fix

I retested this with CodeceptJS 4.2.1, @testomatio/reporter 2.18.0 and the built-in junitReporter plugin, then fixed it locally. Both are verified on a real project: 3 suites × 2 browsers (Chromium + WebKit) × 2 workers.

Current state

Command	JUnit XML	Testomatio HTML
`run-workers 2 --by suite` (one browser)	✅ one `output/report.xml` with all tests	✅ one report with all tests
`run-workers 2 dev --by suite` (two browsers)	🐞 `output/dev_chromium1/report.xml` keeps only the last worker's suites; the main process's `output/report.xml` has one browser only	🐞 one report, one browser only

The single-browser case works: the main process writes the merged report after all workers finish. Workers also write their own partial report.xml to the same path, but the main process writes last.

Why the earlier AI analysis is not enough

The earlier comment correctly identified the cause: the same test has the same uid in every browser. Its fix has three gaps:

The uid is changed only on event.test.after. In the main process, uid is also the key in Result.addTest() (fed by the workers' event.all.result), in _testStates (event.test.finished, used for the final passed/failed events) and in skipped tests. The JUnit plugin builds its XML from Result.tests, so it still drops one browser.
Per-browser JUnit files are still overwritten. Each worker writes its partial report.xml into its run's folder, and two Chromium workers share output/dev_chromium1/. This is the original bug of this issue, and the earlier fix doesn't touch it.
The proposed Testomatio UI change (a browser filter) isn't needed. Once the uids are distinct, the combined HTML already lists each browser's result separately. To get one report per browser, as run-multiple produces and existing CI jobs expect, the reporter needs only a small change that writes one file per run folder. It doesn't need a UI change.

Proposed fix

The result should be the same layout as run-multiple: one XML and one HTML per browser, each containing every suite from all of that browser's workers.

output/dev_chromium1/report.xml
output/dev_chromium1/report/testomatio-report.html
output/dev_webkit1/report.xml
output/dev_webkit1/report/testomatio-report.html

CodeceptJS lib/workers.js: in the main process, tag every incoming message from a multi-run worker at the single entry point, instead of tagging individual events:

// createWorkerObjects(), multi-run branch, after workerObj.addOptions(options)
workerObj.runOutputDir = path.resolve(testRoot, finalConfig.output)

// createWorker(), before return
worker.runOutputDir = workerObject.runOutputDir

// The same test runs once per browser in a multi-run, so make its uid unique per run and remember the run's output folder
const tagWithRun = (message, runOutputDir) => {
  if (!message.event || !message.data || typeof message.data !== 'object') return
  const runName = path.basename(runOutputDir)
  const tag = obj => {
    if (!obj || typeof obj !== 'object' || obj.outputDir) return
    obj.outputDir = runOutputDir
    if (obj.uid) obj.uid = `${obj.uid}@${runName}`
  }
  tag(message.data)
  if (Array.isArray(message.data.tests)) message.data.tests.forEach(tag)
}

// _listenWorkerEvents(), first line of worker.on('message', ...)
if (worker.runOutputDir) tagWithRun(message, worker.runOutputDir)

Every uid-keyed structure in the main process then keeps the browsers apart without further changes. Each deserialized test also carries outputDir, so reporters know which run it belongs to. Single-browser run-workers and plain run are not affected, because runOutputDir is set only in the multi-run branch.

CodeceptJS lib/plugin/junitReporter.js: workers no longer write partial files, and the main process writes one file per run:

import { isMainThread } from 'worker_threads'

// in hookFailures.push({...}) add: outputDir: hook.outputDir

const writeReport = result => {
  if (written) return
  if (!result || !Array.isArray(result.tests)) return
  // A worker only has its own part of the results, the main process writes the whole report
  if (!isMainThread) return
  written = true

  // Results of a multi-run carry the output folder of their run, write one report per run
  const runDirs = [...new Set(result.tests.map(test => test.outputDir).filter(Boolean))]
  if (runDirs.length) {
    for (const dir of runDirs) {
      const runResult = { tests: result.tests.filter(test => test.outputDir === dir), stats: result.stats }
      saveXml(dir, buildXml(runResult, config, hookFailures.filter(hf => hf.outputDir === dir)))
    }
    return
  }
  // ...unchanged single-run path, writing via saveXml()
}

With only these two CodeceptJS changes, the Testomatio HTML is already correct. It is one combined output/report/testomatio-report.html with all results from both browsers (16 entries in the repro below), and each entry shows its browser in the metadata. No Testomatio UI change is needed.

@testomatio/reporter (optional, a separate PR there; only needed for one HTML per browser, like run-multiple):

lib/adapter/codecept.js: pass outputDir: test.outputDir (hook.outputDir for hook failures) in every client.addTestRun() call.
lib/client.js: take outputDir from testData and put it into the data object sent to the pipes.
lib/pipe/html.js, finishRun(): if tests carry outputDir, build one report per run into <outputDir>/report/<reportName>; otherwise keep the current behaviour.

Side note: isSameTest() in the reporter's utils.js compares Object.values(a.example) === Object.values(b.example), i.e. two new arrays by reference. It is therefore always false, so HtmlPipe.addTest() never merges entries.

Verification

Real project, run-workers 2 personal --by suite --grep "^(ChangePassword|AddUserRole|AddTenant) @" with browsers: [chromium, webkit], 4 workers in parallel:

  OK  | 38 passed   // 2m
[TESTOMATIO] 📊 ... output/personal_chromium1/report/testomatio-report.html
[TESTOMATIO] 📊 ... output/personal_webkit1/report/testomatio-report.html

File	Tests	Browser
`personal_chromium1/report.xml`	19 (ChangePassword 8, AddTenant 7, AddUserRole 4)	chromium only
`personal_webkit1/report.xml`	19	webkit only
`personal_chromium1/report/testomatio-report.html`	19	chromium only
`personal_webkit1/report/testomatio-report.html`	19	webkit only

A minimal repro (4 suites, one failing scenario) gives 8 tests / 1 failure in each browser's XML and HTML. Without the optional reporter change, the repro gives one combined HTML with all 16 entries. Single-browser run-workers 2 --by suite and plain run produce the same single output/report.xml and HTML as before.

I deliberately didn't change two behaviours, because existing run-workers users may depend on them:

The run folder names (dev_chromium1) differ from run-multiple's (dev_chromium_1, dev_webkit_2).
The console prefix is [Worker NN] without the browser name.

A non-breaking improvement would be to add the run label to the "Test distribution" header, e.g. Worker 1: 8 tests (dev:chromium1).

Created by AI agent (Claude Code)

Navigation Menu

Repository navigation

Mochawesome/Testomatio/junit reporter: run-workers: Running of test with --suite doesn't aggregate results from all suites per browser #5411

🤖 AI Analysis

Why the HTML report shows only one browser's results

Part 1 — What needs to change in CodeceptJS

Part 2 — What needs to change in Testomatio reporter

Retest with 4.2.1 and @testomatio/reporter 2.18.0, plus a working fix

Current state

Why the earlier AI analysis is not enough

Proposed fix

Verification

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Mochawesome/Testomatio/junit reporter: run-workers: Running of test with --suite doesn't aggregate results from all suites per browser #5411

Description

Activity

github-actions commented on Apr 27, 2026

mirao commented on May 24, 2026

mirao commented on May 24, 2026

🤖 AI Analysis

Why the HTML report shows only one browser's results

Part 1 — What needs to change in CodeceptJS

Part 2 — What needs to change in Testomatio reporter

mirao commented on Oct 21, 2026

Retest with 4.2.1 and @testomatio/reporter 2.18.0, plus a working fix

Current state

Why the earlier AI analysis is not enough

Proposed fix

Verification

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions