refactor(workflows): let the evaluator report its own leaves (#4274) - #4460
refactor(workflows): let the evaluator report its own leaves (#4274)#4460ntdatt812 wants to merge 1 commit into
Conversation
…4274) _unresolvable_term answered one question -- does every operand in this condition resolve to something? -- by walking the expression itself: filters, then or/and/not, then comparisons, then list literals, down to the leaves. That walk was a second implementation of the parsing in _evaluate_simple_expression, kept in step with it by hand. Two helpers existed only to restate rules the evaluator already had. _looks_numeric mirrored the float()-only-when-a-dot-is-present rule because a bare float() accepts 1e3 and the evaluator does not. _is_literal mirrored the matching-close-is-the-final-character string test because startswith/endswith accepts 'a' 'b' and the evaluator does not. Both docstrings said "mirror the evaluator exactly", which is the tell: when the two drift nothing breaks loudly, the gate just answers wrongly, and the wrong answer is a paste-ready correction that inverts a condition. Seven of the nine findings in github#4230 were the same defect wearing different clothes -- the gate disagreeing with the evaluator about where the operands are. Each round fixed one shape. Nothing stopped a tenth. _evaluate_simple_expression has exactly one place where a substring stops being grammar and becomes a name to resolve: its final line, _resolve_dot_path. Literals return before it; operands, filter arguments and list elements all arrive there by construction. Record the leaf there, behind a ContextVar that is None outside a probe, and the gate applies namespace rules to that list instead of re-deriving it. It now contains no grammar at all. Two properties this rests on, both asserted rather than assumed: * or/and are not short-circuited -- both sides are evaluated and only then combined -- so a leaf is recorded whatever the other side is worth. If that ever changes the gate would go quietly blind, so there is a test for it. * A probe run can raise on its own placeholder values. The leaves seen before that point are real, so they are kept rather than discarded; discarding them would lose `bogus` in `inputs.tags | join(bogus)`, which an earlier round of github#4230 had to add by hand. expressions.py is 109 lines lighter and 84 heavier. All 336 existing tests pass unchanged, including the 20 cases of test_operands_must_be_literals_or_known_paths that took eight rounds to get right. test_literal_test_mirrors_the_evaluator tested the mirror, so it becomes test_literal_handling_comes_from_the_evaluator and asserts the same knowledge about 1e3 and 'a' 'b' through the gate instead. Four mutations, each killed by the tests that should kill it -- removing the leaf report alone turns 38 red. ruff 0.15.0 clean.
|
Thanks — this is the right root-cause fix: having the evaluator report its own leaves instead of |
Implements #4274, which was the write-up you asked for at the end of #4230.
The problem
_unresolvable_termanswers one question — does every operand in this condition resolve to something? — by walking the expression itself: filters, thenor/and/not, then comparisons, then list literals, down to the leaves.That walk is a second implementation of the parsing in
_evaluate_simple_expression, kept in step with it by hand. Two helpers exist purely to restate rules the evaluator already has, and both say so:_looks_numeric— "Mirror the evaluator's numeric literal test exactly." A barefloat()accepts1e3; the evaluator does not._is_literal— "Mirror the evaluator's literal tests exactly."startswith/endswithaccepts'a' 'b'; the evaluator does not.When the two drift, nothing breaks loudly. The gate keeps answering, just wrongly, and the wrong answer is a paste-ready correction that silently inverts a condition. Seven of the nine findings in #4230 were that same defect in different clothes — the gate disagreeing with the evaluator about where the operands are. Each round fixed one shape; nothing stopped a tenth.
The change
_evaluate_simple_expressionhas exactly one place where a substring stops being grammar and becomes a name to resolve — its final line,_resolve_dot_path. Literals return before it; operands, filter arguments and list elements all arrive there by construction. So let the evaluator report what it reaches:_collect_leavesruns the probe_evaluator_rejectsalready uses, with the sink armed._unresolvable_leafkeeps only the namespace rules — root membership, path-segment shape, and theitemindex narrowing from the last round of #4230. The gate now contains no grammar at all.A
ContextVarrather than a module global, so concurrent probes cannot append into each other's list; it isNoneoutside a probe, so a normal evaluation costs one.get().Two properties this rests on
Both are asserted rather than assumed, because if either changed the gate would go quietly blind rather than fail:
or/andare not short-circuited._evaluate_simple_expressionevaluates both sides and only then combines them, so a leaf is recorded whatever the other side is worth.test_both_sides_of_a_boolean_are_reportedpins it.A probe run can raise on its own placeholder values, which is what
_evaluator_rejectssorts out. The leaves seen before that point are real — the evaluator reached them — so they are kept rather than discarded. Discarding them would losebogusininputs.tags | join(bogus), which is the filter-argument case an earlier round of #4230 had to add by hand.test_leaves_seen_before_a_probe_error_are_keptpins it.Verification
expressions.py: 109 lines removed, 84 added.All 336 existing tests pass unchanged, including the 20 cases of
test_operands_must_be_literals_or_known_pathsthat took eight rounds to get right. That is the main evidence: the new gate agrees with the old one on every shape review found, without knowing about any of them.One test changed rather than passed:
test_literal_test_mirrors_the_evaluatortested the mirror, and the mirror is gone. It becomestest_literal_handling_comes_from_the_evaluatorand asserts the same knowledge —1e3is not a number to the evaluator,'a' 'b'is not one literal — through_unresolvable_term. That is the property that actually mattered; the old test could pass while the two had drifted.Five tests added for the mechanism itself. Checked by breaking it:
_unresolvable_termsuiteitem[0]narrowingtest_an_indexed_item_root_keeps_the_correctionuvx ruff@0.15.0 check src tests— clean. Wider run (tests/unit,test_workflows.py,test_extensions.py): 1988 passed, and the set of failing test names is identical before and after — 24 symlink tests that cannot run unprivileged on Windows.