Add EdenLive by skygoo2000 · Pull Request #15066 · huggingface/diffusers

skygoo2000 · 2026-10-16T17:14:56Z

What does this PR do?

Adds support for EdenLive, a generative rendering model that produces video from a reference image, an RGB proxy video, and a text prompt.

This integration includes:

EdenLivePipeline and EdenLiveCausalPipeline classes for bidirectional and causal generation.
The EdenLive transformer, DINOv3 proxy conditioning, flow-matching scheduler, and LightTAE decoder.
Documentation, tests, CPU offloading, and inference examples using context parallelism and FSDP2 on multiple GPUs.

Diffusers checkpoints are available on the Hub:

Basic usage

The example below uses the 5B bidirectional model. Prepare a reference image and a proxy video containing at least 49 frames sampled at 24 fps.

import torch
from diffusers import EdenLivePipeline
from diffusers.utils import export_to_video, load_image, load_video

pipe = EdenLivePipeline.from_pretrained(
    "VAST-AI/EdenLive-5B-Diffusers",
    torch_dtype={
        "default": torch.bfloat16,
        "proxy_encoder": torch.float32,
        "proxy_projection": torch.float32,
    },
).to("cuda")

frames = pipe(
    image=load_image("reference.png"),
    video=load_video("proxy-24fps.mp4"),
    prompt="A character explores a snowy landscape.",
    num_frames=49,
).frames[0]

export_to_video(frames, "edenlive.mp4", fps=24)

See the pipeline documentation and examples/edenlive for causal generation, changing prompts between chunks, LightTAE decoding, and multi-GPU inference.

Demo

comparison.1.mp4

Validation

Local refactor validation included:

Standard model and pipeline tests covering loading, saving, offloading, compilation, dynamic prompts, and cache cleanup.
Context parallelism and FSDP2 tests on 2/4/8 A800 GPUs.
A self-review was completed against the Diffusers contribution guidelines.

github-actions · 2026-10-17T07:37:33Z

Hi @skygoo2000, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Fixes #1234) to the PR description so the issue is linked. See the contribution guide for more details. If this PR intentionally does not fix a tracked issue, a maintainer can add the no-issue-needed label to silence this reminder.

Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice.

Once the PR links an issue (or gets the no-issue-needed label), you can ignore this message — it stays here as a comment, but it no longer applies.

skygoo2000 · 2026-10-18T07:43:36Z

Hi @yiyixuxu, thanks for adding the label! Could you take a look at this PR, or help suggest the right reviewer for the EdenLive integration? Happy to make any changes needed.

…ain-recipe-sync

yiyixuxu

i left some initial comments
additionally, is it possible to remove the streaming out of the official pipeline?

yiyixuxu · 2026-10-18T19:20:54Z

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Read the four original EdenLive releases without writing converted checkpoints."""


can we remove this?
the checkpoint should be converted into diffusers format and then our from_pretrained() should be able to work out of box

yiyixuxu · 2026-10-18T19:22:52Z

+        super().__init__()
+        if chunk_size < 1 or sink_frames < 0 or max_context_frames < sink_frames + chunk_size:
+            raise ValueError("The context must fit the sink and a positive-size chunk.")
+        utils.initialize_pipeline(


can we remove this?

yiyixuxu · 2026-10-18T19:23:21Z

+        self.register_to_config(chunk_size=chunk_size, max_context_frames=max_context_frames, sink_frames=sink_frames)
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, *, profile=None, **kwargs):


can we not override from_pretrained?

yiyixuxu · 2026-10-18T19:29:01Z

@skygoo2000
the PR deviates a lot from our standard and requirement, are you able to run a self-review? (see guide here https://huggingface.co/docs/diffusers/v0.41.0/en/conceptual/contribution#coding-with-ai-agents)

skygoo2000 · 2026-10-19T16:49:33Z

@skygoo2000 the PR deviates a lot from our standard and requirement, are you able to run a self-review? (see guide here https://huggingface.co/docs/diffusers/v0.41.0/en/conceptual/contribution#coding-with-ai-agents)

Thanks @yiyixuxu!

We've updated the PR to address your comments: removed streaming, additional initialization helper, and the from_pretrained override. Checkpoint conversion is now handled by a standalone script, and both pipelines use standard Diffusers loading.

We also ran the self-review and addressed the integration issues you highlighted. Please take another look when you have a chance!

HuggingFaceDocBuilderDev · 2026-10-19T23:24:16Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yiyixuxu

thanks!
i left some comments

yiyixuxu · 2026-10-20T07:36:57Z

+four fixed denoising steps per chunk.
+
+```python
+from diffusers import EdenLiveCausalPipeline, EdenLiveTransformer3DModel, EdenLiveFlowMatchEulerDiscreteScheduler


we recommend one repo per checkpoint, so probably would work better if casual has it own repo

yiyixuxu · 2026-10-20T07:43:03Z

+        return attn.to_out[1](attn.to_out[0](output))
+
+
+class EdenLiveCausalAttnProcessor(EdenLiveAttnProcessor):


Suggested change

class EdenLiveCausalAttnProcessor(EdenLiveAttnProcessor):

class EdenLiveCausalAttnProcessor:

yiyixuxu · 2026-10-20T07:47:37Z

+        return attn.to_out[1](attn.to_out[0](output))
+
+
+class EdenLiveProxyProjection(ModelMixin, ConfigMixin):


why is this not part of the transformer?

yiyixuxu · 2026-10-20T08:06:36Z

+        self.scale_shift_table = nn.Parameter(torch.randn(1, 2, inner_dim) / inner_dim**0.5)
+        self.gradient_checkpointing = False
+
+    def _context_parallel_info(self):


can we remove this?

yiyixuxu · 2026-10-21T04:57:10Z

+            raise ValueError("The context must contain the current chunk and be larger than its nonnegative sink.")
+        if max_context_frames % chunk_size or sink_frames % chunk_size:
+            raise ValueError("The fixed context and sink must contain whole chunks.")
+        cp_size, cp_rank = self._context_parallel_info()


so our model implementations should not be aware of complex parallelism. cp support should come in with _cp_plan

I think our cp does not yet support kv cache,, we are still working on adding that in a general way. but individual models shouild not implement their own version. could you remove all the cp stuff from the PR?

yiyixuxu · 2026-10-21T05:00:31Z

+    """Per-rollout, per-guidance-branch KV storage. Positions count latent frames."""
+
+    layers: list[_EdenLiveLayerCache]
+    batch_size: int


why does the cache container need all of these info like height/width etc?

yiyixuxu · 2026-10-21T05:04:15Z

+        )
+        if batch_size < 1:
+            raise ValueError("Provide a positive prompt or prompt embeddings.")
+        try:


can we remove this?

bugfix: device_map, offload, and BF16 handling.

skygoo2000 · 2026-10-21T12:11:08Z

Thanks again for the detailed feedback @yiyixuxu.

We've updated the implementation for these issues:

remove custom cp and simplify the KV cache state.
move the projection module into the transformer.
remove 'try' wrappers in pipeline.
split 5B/14B bidirectional and causal models.

We also reran the repository self-review, and found no remaining code issues.
I'd appreciate another look at the code changes. Thanks again!

skygoo2000 added 3 commits October 16, 2026 05:22

feat: add EdenLive v2v bidir&causal inference pipelines

0426240

feat: add streaming v2v pipeline

845d853

update readme

6f56be2

github-actions Bot added documentation Improvements or additions to documentation models tests utils pipelines examples schedulers size/L PR with diff > 200 LOC labels Oct 16, 2026

Merge branch 'main' into edenlive

0f5e164

yiyixuxu added the no-issue-needed for PRs that do not require link to an issue label Oct 17, 2026

skygoo2000 added 2 commits October 18, 2026 16:40

Merge remote-tracking branch 'huggingface/main' into codex/edenlive-m…

56d8ecb

…ain-recipe-sync

fix: align EdenLive defaults and cache commits

c9b2d0e

yiyixuxu reviewed Oct 18, 2026

View reviewed changes

skygoo2000 added 2 commits October 19, 2026 23:58

Merge branch 'huggingface/main'

750c1a5

Refactor pipelines to follow conventions

ea77dec

yiyixuxu reviewed Oct 21, 2026

View reviewed changes

feat: remove custom cp; split causal ckpt; simplify KV cache state;

d4fdc86

bugfix: device_map, offload, and BF16 handling.

		return attn.to_out[1](attn.to_out[0](output))


		class EdenLiveCausalAttnProcessor(EdenLiveAttnProcessor):

	class EdenLiveCausalAttnProcessor(EdenLiveAttnProcessor):
	class EdenLiveCausalAttnProcessor:

		return attn.to_out[1](attn.to_out[0](output))


		class EdenLiveProxyProjection(ModelMixin, ConfigMixin):

Conversation

skygoo2000 commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Basic usage

Demo

Validation

Uh oh!

github-actions Bot commented Oct 17, 2026

Uh oh!

skygoo2000 commented Oct 18, 2026

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yiyixuxu commented Oct 18, 2026

Uh oh!

skygoo2000 commented Oct 19, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Oct 19, 2026

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skygoo2000 commented Oct 21, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

skygoo2000 commented Oct 16, 2026 •

edited

Loading