Skip to content

Built-in z_score audit ignores condition for the rows it checks, and valid_http_method ignores condition entirely #6155

Open

Description

@DG47

Most built-in audits accept an optional condition argument that restricts the rows being audited. Two audits do not honor it correctly:

  1. z_score applies condition only inside the stats CTE that computes the mean and standard deviation. The outer query reads every row of the model, so rows that the condition excludes are still compared against statistics computed from a different subset and are reported as failures.
  2. valid_http_method has no condition default and its query does not reference @condition, so a condition passed to it is silently ignored and the whole model (or time range) is audited.

Reproduction (DuckDB)

import duckdb
from sqlglot import exp, parse_one
from sqlmesh.core.audit import builtin
from sqlmesh.core.model import FullKind, create_sql_model

model = create_sql_model(
    "db.t", parse_one("SELECT region, v, method FROM db.src"), kind=FullKind(), dialect="duckdb"
)
con = duckdb.connect()
con.execute(
    "CREATE SCHEMA db; CREATE TABLE db.t AS SELECT * FROM (VALUES "
    "('us', 10, 'GET'), ('us', 10, 'POST'), ('us', 11, 'PUT'), "
    "('eu', 1000, 'get'), ('eu', 1000, 'post')) AS t(region, v, method)"
)
condition = parse_one("region = 'us'")

q = model.render_audit_query(
    builtin.z_score_audit, column=exp.to_column("v"), threshold=exp.convert(1), condition=condition
)
print(con.execute(q.sql("duckdb")).fetchall())

q = model.render_audit_query(
    builtin.valid_http_method_audit, column=exp.to_column("method"), condition=condition
)
print(q.sql("duckdb"))
print(con.execute(q.sql("duckdb")).fetchall())

The same applies when the audits are declared on a model, e.g. z_score(column := v, threshold := 1, condition := region = 'us').

Expected

  • z_score returns only (11, 1.1547...), the one us row whose z-score exceeds the threshold.
  • valid_http_method adds the condition to its WHERE clause and returns no rows, because every us row has a valid method.

Actual

  • z_score returns [(11, 1.1547...), (1000, 1714.15...), (1000, 1714.15...)]: the two eu rows are flagged even though the condition excludes them.
  • valid_http_method renders SELECT * FROM "db"."t" AS "t" WHERE NOT "method" IN ('GET', 'POST', ...) with no condition, and the two eu rows are flagged.

So the audit fails (or warns, for the _non_blocking variants) on rows the user excluded on purpose. It also scans the full model instead of the filtered subset.

Environment

  • sqlmesh main at c0026dc (0.236.4.dev13)
  • sqlglot 30.8.0
  • duckdb (bundled dev dependency), Python 3.12, macOS

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions