Skip to content

refactor(workflows): let the evaluator report its own leaves (#4274) - #4460

Open
ntdatt812 wants to merge 1 commit into
github:mainfrom
ntdatt812:refactor/condition-gate-leaf-sink
Open

refactor(workflows): let the evaluator report its own leaves (#4274)#4460
ntdatt812 wants to merge 1 commit into
github:mainfrom
ntdatt812:refactor/condition-gate-leaf-sink

Conversation

@ntdatt812

Copy link
Copy Markdown
Contributor

Implements #4274, which was the write-up you asked for at the end of #4230.

The problem

_unresolvable_term answers one question — does every operand in this condition resolve to something? — by walking the expression itself: filters, then or/and/not, then comparisons, then list literals, down to the leaves.

That walk is a second implementation of the parsing in _evaluate_simple_expression, kept in step with it by hand. Two helpers exist purely to restate rules the evaluator already has, and both say so:

  • _looks_numeric"Mirror the evaluator's numeric literal test exactly." A bare float() accepts 1e3; the evaluator does not.
  • _is_literal"Mirror the evaluator's literal tests exactly." startswith/endswith accepts 'a' 'b'; the evaluator does not.

When the two drift, nothing breaks loudly. The gate keeps answering, just wrongly, and the wrong answer is a paste-ready correction that silently inverts a condition. Seven of the nine findings in #4230 were that same defect in different clothes — the gate disagreeing with the evaluator about where the operands are. Each round fixed one shape; nothing stopped a tenth.

The change

_evaluate_simple_expression has exactly one place where a substring stops being grammar and becomes a name to resolve — its final line, _resolve_dot_path. Literals return before it; operands, filter arguments and list elements all arrive there by construction. So let the evaluator report what it reaches:

    # Variable reference (dot-path)
    sink = _leaf_sink.get()
    if sink is not None:
        sink.append(expr)
    return _resolve_dot_path(namespace, expr)

_collect_leaves runs the probe _evaluator_rejects already uses, with the sink armed. _unresolvable_leaf keeps only the namespace rules — root membership, path-segment shape, and the item index narrowing from the last round of #4230. The gate now contains no grammar at all.

A ContextVar rather than a module global, so concurrent probes cannot append into each other's list; it is None outside a probe, so a normal evaluation costs one .get().

Two properties this rests on

Both are asserted rather than assumed, because if either changed the gate would go quietly blind rather than fail:

or/and are not short-circuited. _evaluate_simple_expression evaluates both sides and only then combines them, so a leaf is recorded whatever the other side is worth. test_both_sides_of_a_boolean_are_reported pins it.

A probe run can raise on its own placeholder values, which is what _evaluator_rejects sorts out. The leaves seen before that point are real — the evaluator reached them — so they are kept rather than discarded. Discarding them would lose bogus in inputs.tags | join(bogus), which is the filter-argument case an earlier round of #4230 had to add by hand. test_leaves_seen_before_a_probe_error_are_kept pins it.

Verification

expressions.py: 109 lines removed, 84 added.

All 336 existing tests pass unchanged, including the 20 cases of test_operands_must_be_literals_or_known_paths that took eight rounds to get right. That is the main evidence: the new gate agrees with the old one on every shape review found, without knowing about any of them.

One test changed rather than passed: test_literal_test_mirrors_the_evaluator tested the mirror, and the mirror is gone. It becomes test_literal_handling_comes_from_the_evaluator and asserts the same knowledge — 1e3 is not a number to the evaluator, 'a' 'b' is not one literal — through _unresolvable_term. That is the property that actually mattered; the old test could pass while the two had drifted.

Five tests added for the mechanism itself. Checked by breaking it:

reverted killed
evaluator stops reporting its leaves 38 tests, including the whole _unresolvable_term suite
discard the leaves seen before a probe error the filter-argument tests
drop the item[0] narrowing test_an_indexed_item_root_keeps_the_correction
leave the sink armed after a probe the two sink-hygiene tests

uvx ruff@0.15.0 check src tests — clean. Wider run (tests/unit, test_workflows.py, test_extensions.py): 1988 passed, and the set of failing test names is identical before and after — 24 symlink tests that cannot run unprivileged on Windows.

…4274)

_unresolvable_term answered one question -- does every operand in this
condition resolve to something? -- by walking the expression itself:
filters, then or/and/not, then comparisons, then list literals, down to
the leaves. That walk was a second implementation of the parsing in
_evaluate_simple_expression, kept in step with it by hand.

Two helpers existed only to restate rules the evaluator already had.
_looks_numeric mirrored the float()-only-when-a-dot-is-present rule
because a bare float() accepts 1e3 and the evaluator does not.
_is_literal mirrored the matching-close-is-the-final-character string
test because startswith/endswith accepts 'a' 'b' and the evaluator does
not. Both docstrings said "mirror the evaluator exactly", which is the
tell: when the two drift nothing breaks loudly, the gate just answers
wrongly, and the wrong answer is a paste-ready correction that inverts a
condition.

Seven of the nine findings in github#4230 were the same defect wearing
different clothes -- the gate disagreeing with the evaluator about where
the operands are. Each round fixed one shape. Nothing stopped a tenth.

_evaluate_simple_expression has exactly one place where a substring stops
being grammar and becomes a name to resolve: its final line,
_resolve_dot_path. Literals return before it; operands, filter arguments
and list elements all arrive there by construction. Record the leaf
there, behind a ContextVar that is None outside a probe, and the gate
applies namespace rules to that list instead of re-deriving it. It now
contains no grammar at all.

Two properties this rests on, both asserted rather than assumed:

  * or/and are not short-circuited -- both sides are evaluated and only
    then combined -- so a leaf is recorded whatever the other side is
    worth. If that ever changes the gate would go quietly blind, so
    there is a test for it.

  * A probe run can raise on its own placeholder values. The leaves seen
    before that point are real, so they are kept rather than discarded;
    discarding them would lose `bogus` in `inputs.tags | join(bogus)`,
    which an earlier round of github#4230 had to add by hand.

expressions.py is 109 lines lighter and 84 heavier. All 336 existing
tests pass unchanged, including the 20 cases of
test_operands_must_be_literals_or_known_paths that took eight rounds to
get right. test_literal_test_mirrors_the_evaluator tested the mirror, so
it becomes test_literal_handling_comes_from_the_evaluator and asserts the
same knowledge about 1e3 and 'a' 'b' through the gate instead.

Four mutations, each killed by the tests that should kill it -- removing
the leaf report alone turns 38 red. ruff 0.15.0 clean.
@ntdatt812
ntdatt812 requested a review from mnriem as a code owner September 7, 2026 09:00
@mnriem

mnriem commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Thanks — this is the right root-cause fix: having the evaluator report its own leaves instead of _unresolvable_term re-implementing the grammar is exactly what removes the drift. Two things: (1) please add the AI-disclosure per CONTRIBUTING — the body has none. (2) Coordination: #4416 and #4417 (by [@NgoQuocViet2001](https://github.com/NgoQuocViet2001)) are point-fixes to the same _unresolvable_term grammar duplication this PR deletes, touching the same file. I'd like to land this refactor first and then have those two rebase/verify on top (their specific cases should be covered once the gate stops re-implementing the grammar) — could you confirm your refactor handles the negative-index and parenthesised-group cases they fixed, ideally with tests mirroring theirs? I'll sequence the merges accordingly.

@mnriem mnriem added author-needs-disclosure AI assistance not disclosed — disclose AI use per CONTRIBUTING triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review labels Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author-needs-disclosure AI assistance not disclosed — disclose AI use per CONTRIBUTING triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants