Skip to content

sqlmesh run keeps reporting "environment is being updated" after the updating plan has failed #6157

Open

Description

@lafirm

I found this issue when I accidentally did sqlmesh plan --auto-apply to update PROD environment while sqlmesh run is running, the run detects the environment change and waits for the plan to finish.

However, if the plan fails due to some reason before the environment finalization, the run continues firing warnings like below:

[WARNING] Environment 'prod' is being updated by plan '<plan_id>'. Retrying in 30 seconds...

The plan process has already exited with an error, but the run command warning message implies that plan process is still updating the environment. The run does not resume quickly or prompts that manual recovery is required.

I understand that running against a partially applied environment is unsafe. My concern is to properly handle the failure and help the developers debug the issue, not bypassing environment finalization.

Environment

  • SQLMesh: 0.236.3
  • Execution: CLI
  • Engine: Trino
  • State: PostgreSQL
  • Target SQLMesh environment: prod

Observed sequence

  1. Start sqlmesh run.
  2. While it is executing, run sqlmesh plan --auto-apply against the same environment.
  3. Fail or abort the plan.
  4. The run continues emitting the warning above even though the plan process has exited.
  5. Restarting sqlmesh run does not resolve this issue (or the underlying unfinalized environment state).

Expected behavior

A failed plan should leave the environment protected, but the waiting run should provide a clear actionable outcome.

For example:

  • If plan failure is recorded, run should report that the updating plan failed and exit with a nonzero status if possible.
  • Otherwise, make the warning clear that SQLMesh is waiting for environment finalization, not necessarily an active process.
  • Include the timeout and recovery guidance such as inspecting the failed plan and successfully reapplying it.

Actual behavior

The run repeatedly reports that the environment "is being updated" without distinguishing an active plan from state left behind by a failed plan.

The default timeout is six hours, which makes this appear to be an indefinite hang (firing warnings for every 30s).

Initial investigation

In the installed SQLMesh 0.236.3 pkg:

  • Context.run() waits for a true value of environment_state.finalized_ts.
  • The warning is based on the stored environment record, not the current live plan-process.
  • Plan application updates the environment record before virtual-layer promotion.
  • Environment finalization happens afterward.
  • A failure between those stages can therefore leave subsequent runs waiting (for the default 6 hours).

Is retaining this unfinalized state the intended recovery mechanism? If so, could the waiting message and failed-plan recovery guidance be improved?

Activity

  1. lafirm commented on Oct 21, 2026

    @lafirm
    ContributorAuthor

    MRE

    The following standalone DuckDB example intentionally fails during virtual-layer promotion, after the environment record is updated but before finalization. So that we can avoid needing a long-running concurrent query: a subsequent run command should encounter the same persisted unfinalized state.

    Use a fresh, disposable directory with SQLMesh 0.236.3 and DuckDB support installed.

    Create config.yaml:

    gateways:
      local:
        connection:
          type: duckdb
          database: repro.duckdb
    
    default_gateway: local
    
    model_defaults:
      dialect: duckdb
    
    disable_anonymized_analytics: true
    
    run:
      environment_check_interval: 2
      environment_check_max_wait: 20

    Create models/example.sql:

    MODEL (
      name repro.example,
      kind FULL
    );
    
    SELECT 1::INT AS id;

    Apply the initial successful plan:

    sqlmesh plan prod --auto-apply

    Then replace models/example.sql with:

    MODEL (
      name repro.example,
      kind FULL
    );
    
    SELECT 2::INT AS id;
    
    ON_VIRTUAL_UPDATE_BEGIN;
    SELECT CAST('intentional_promotion_failure' AS INTEGER);
    ON_VIRTUAL_UPDATE_END;

    Apply the changed plan:

    sqlmesh plan prod --auto-apply

    The virtual-update statement intentionally raises a conversion error.

    After the plan exits unsuccessfully, execute:

    sqlmesh run prod --skip-janitor

    Observe repeated warnings referencing the failed plan:

    Environment 'prod' is being updated by plan '<failed_plan_id>'. Retrying in 2 seconds...
    

    With this config, the run should eventually fail after approximately 20 seconds. Without the override, the default maximum wait is 6 hours.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ImprovementImproves existing functionality

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions