[raghavyadav01]

Which issue does this PR close?

Closes #26542.

Rationale for this change

Under filter semantics only rows for which the predicate is TRUE are kept, so NULL rejects a row exactly like FALSE. That makes two rewrites sound for IN lists that hold NULL literals:

• x IN (a, NULL) is TRUE exactly when x matches a. A NULL member can only turn FALSE into NULL, and both reject. The NULL members can be dropped, and a list of nothing but NULLs rejects every row.
• x NOT IN (a, NULL) is FALSE when x matches a and NULL otherwise, so it is never TRUE and rejects every row.

Leaving the NULLs in place keeps a filter that PruningPredicate does not handle well, which is the pruning degradation described in the issue. #24711 fixed this only for the compact path (lists over 20 members) and #25187 made short lists on several types reach the same path.

What changes are included in this PR?

An Expr::InList arm in simplify_filter_predicate, plus a simplify_filter_in_list helper and an is_null_literal predicate, in datafusion/optimizer/src/eliminate_filter.rs.

The rewrite stays in EliminateFilter deliberately. It is not sound in ExprSimplifier: in a general expression context SELECT 3 IN (1, NULL) must remain NULL, and dropping the NULL would make it FALSE.

The negated case keeps the original InList inside FilterPredicate::RejectsAll rather than substituting a literal, so nothing is discarded that the AND/OR reconstruction may still need.

Are these changes tested?

Yes — five tests in eliminate_filter.rs:

test	asserts
filter_in_list_with_null_drops_null	a IN (1, 2, NULL) → a IN (1, 2)
filter_in_list_of_only_nulls_rejects_all	a IN (NULL) → EmptyRelation
filter_not_in_list_with_null_rejects_all	a NOT IN (1, NULL) → EmptyRelation
filter_in_list_without_null_is_untouched	no NULL → unchanged
filter_not_in_list_without_null_is_untouched	no NULL → unchanged

I wrote the tests first and confirmed the three NULL cases fail on main before the change and pass after, while the two control tests pass in both states.

Also run: the full datafusion-optimizer suite (948 + 29 + 5 passing), cargo fmt --all -- --check, and cargo clippy -p datafusion-optimizer --all-targets -- -D warnings, all clean. in_list.slt and the null_aware_* slt files pass. expr, predicates, scalar and aggregate slt files each fail identically with and without this change on my machine (a local test-data issue), so they are unaffected.

Are there any user-facing changes?

Explain output for filters whose IN list contains NULL literals: the NULL members no longer appear, and a filter that can never be TRUE becomes an empty relation. Query results are unchanged.