ContentDetector merge filter: one short scene early on changes where (or whether) a much later cut is reported

[sungbin1015]

I'm using ContentDetector with threshold=15 and min_scene_len of about 1s (default filter_mode) to split long sports broadcasts into shots. While testing on synthetic clips I noticed that a single short scene early in the video changes how a cut hundreds of frames later gets reported.

import cv2, numpy as np
from scenedetect import open_video, SceneManager, ContentDetector

def make(path, cuts, n=600):
    # 30 fps, alternating black/white shots, a hard cut at every frame in `cuts`
    w = cv2.VideoWriter(path, cv2.VideoWriter_fourcc(*"mp4v"), 30.0, (320, 180))
    for i in range(n):
        w.write(np.full((180, 320, 3), 255 * (sum(i >= c for c in cuts) % 2), np.uint8))
    w.release()

def detect(path):
    sm = SceneManager()
    sm.add_detector(ContentDetector(threshold=15, min_scene_len=30))
    sm.detect_scenes(open_video(path))
    return [start.frame_num for start, _ in sm.get_scene_list()[1:]]

for cuts in ([100, 400, 415], [100, 110, 400, 415], [100, 580], [100, 110, 580]):
    make("t.mp4", cuts)
    print(cuts, "->", detect("t.mp4"))

On 0.7.2 (same on 0.6.7 and 0.7.0, OpenCV 5.0.0):

[100, 400, 415] -> [100, 400]
[100, 110, 400, 415] -> [100, 415]
[100, 580] -> [100, 580]
[100, 110, 580] -> [100]

The 2nd and 4th clips only add a cut at frame 110, which is 290 / 470 frames before the cut I care about. With it, the 400/415 pair comes out at 415 instead of 400, and the cut at 580 (20 frames before the end) is gone entirely. With filter_mode=FlashFilter.Mode.SUPPRESS all four give 400 / 580 as I'd expect.

From reading _filter_merge, the 100→110 short scene sets _merge_triggered, and that only clears once _last_above - _merge_start >= min_scene_len, which a lone short scene never satisfies. So the filter stays armed through the quiet stretch, then holds the next real cut until another min_scene_len of quiet frames, and loses it if the video ends first. Your comment in #35 describes cuts being suppressed "until min-scene-len frames pass without a cut", so I expected it to reset somewhere in those 290 frames. #469 looks related, but there the difference came from changing min_scene_len, not from an earlier short scene with the same settings.

Is this the intended merge behaviour, or should the merge state clear after min_scene_len frames without a cut?

[akintaha2010-cloud]

I can reproduce all four results on 0.7.2 (OpenCV 5.0.0). _filter_merge is the same on main today.

Your reading is right. Walking through [100, 110, 400, 415] with min_scene_len=30:

1. Frame 100: 100 frames since the last cut, so it is emitted and _merge_enabled becomes True.
2. Frame 110: only 10 frames later, so _merge_triggered = True and _merge_start = 110.
3. Frames 111-399: the only way to disarm is min_length_met and not above_threshold and _min_length_met(_merge_start, _last_above). The last check compares 110 with 110, so it is 0 >= 30 and never true. The filter stays armed.
4. Frame 400 is swallowed because the filter is still merging. Frame 415 becomes _last_above. At frame 445 (30 quiet frames, and 415 - 110 >= 30) the filter finally disarms and emits 415.
5. In [100, 110, 580] the same thing happens to 580, but the 30 quiet frames after it never arrive before the video ends, so it is lost.

So a merge group only closes once the group itself spans min_scene_len. A lone short scene (100 -> 110) never does, and the "armed" state leaks into whatever cut comes next. Compare [100, 400, 415]: there the first cut of the pair (400) is reported. In your case it's the last (415). That inconsistency suggests this isn't intended.

A small change that disarms the filter, without emitting anything, once min_scene_len quiet frames have passed after a short merged group gives the results you expected. You can monkeypatch it until there's a fix upstream:

from scenedetect.detector import FlashFilter

_orig_filter_merge = FlashFilter._filter_merge

def _filter_merge(self, timecode, above_threshold):
    # A lone short scene was merged into the previous one: once min_scene_len frames pass
    # without a cut, disarm without emitting anything.
    if (self._merge_triggered and not above_threshold
            and self._min_length_met(self._last_above, timecode)
            and not self._min_length_met(self._merge_start, self._last_above)):
        self._merge_triggered = False
        return []
    return _orig_filter_merge(self, timecode, above_threshold)

FlashFilter._filter_merge = _filter_merge

With this applied, using your make/detect:

[100, 400, 415]                     -> [100, 400]
[100, 110, 400, 415]                -> [100, 400]
[100, 580]                          -> [100, 580]
[100, 110, 580]                     -> [100, 580]
[100, 110]                          -> [100]
[100, 200, 210, 220, 230, 240, 400] -> [100, 200, 240, 400]   (unchanged from stock: long flash groups still emit their end)

If you don't want to patch, filter_mode=FlashFilter.Mode.SUPPRESS (as you found) avoids the state entirely. The trade-off is that it reports the first cut of a burst rather than merging it.

[sungbin1015]

Thanks for walking through it, that matches what I saw. I ran your _filter_merge patch against the script from the post and got the same results as your table: [100, 110, 400, 415] → [100, 400], [100, 110, 580] → [100, 580], and the long flash group is unchanged. Happy to turn it into a PR with tests if @Breakthrough is fine with this approach.

[Breakthrough]

Sure, happy to take this route. Please feel free to file a bug for this as well.

[Breakthrough]

Is this the intended merge behaviour, or should the merge state clear after min_scene_len frames without a cut?

The current behaviour sounds inconsistent, and indeed I may have overlooked something here. Feel free to file a bug report for this.