Record the producing `tau` version on generated workloads, and stamp payload-digest consistently

[chokevin]

User problem

When a run fails or behaves unexpectedly days later, you cannot reconstruct what produced it. Two specific gaps, both verified against cb7e255:

1. No workload records the tau version that rendered it. I traced every version path in the repo:

• app.kubernetes.io/version — cli/internal/runhistory/runhistory.go:622 reads it; no writer exists anywhere in the repo. In practice it carries the researcher's own git SHA from their manifest labels, which is easily mistaken for a TauGrid version.
• tau.azure.com/controller-version — declared at core/workloadmeta/metadata.go:68; the only reference is that same read. No writer.
• core/version.Version — used by tau version, cobra --version, reposcaffold, the portal API, and run_connection.go:183 (a client-side check against the connection descriptor's minTauVersion). None of it reaches the cluster.

So a generated Job or RayJob carries rich provenance — tau.azure.com/image, image-digest, code-sha, config-hash, tau-command, capture-version, submission-id — but nothing that answers "which tau built this?"

This is not hypothetical. A recent review of a live cluster had to diagnose two separate symptoms (orphaned per-run ConfigMaps, and Jobs without ttlSecondsAfterFinished) by deducing that the submitting CLI predated a particular refactor, because no artifact on the cluster stated it. The deduction was only possible because the relevant code paths happen to have observable side effects; the conclusion still cannot be cited, only inferred. A single label would have made it a lookup.

Note this is genuinely a CLI version question, not a platform one: the binary that submits a run lives on the researcher's machine and is installed independently of the cluster's charts, so no chart or image version on the cluster answers it either.

2. tau.azure.com/payload-digest is computed and then discarded by the managed-workflow renderer. This one is an inconsistency between renderers rather than a missing feature:

• cli/internal/rayjobrender/render.go:373 stamps payload.AnnotationDigest.
• cli/internal/manifest (the managed-workflow renderer used for both the Job and RayJob templates) already computes the digests — payloadEmbeds.ScriptDigest and payloadEmbeds.ManifestDigest at render.go:1064-1073 — and uses them for in-pod verification via TAU_PAYLOAD_DIGEST, but never surfaces them as a workload annotation.
• cli/internal/jobrender embeds no payloads, so it is legitimately N/A.

The value exists at render time and is deliberately dropped at exactly the point where it would be useful. Since payload embeds are what make a workload self-contained, the digest is the only thing that identifies which script bundle a given workload actually ran.

Desired outcome

A workload's own metadata is sufficient to identify what produced it:

• The producing tau version is recorded on every generated Job and RayJob.
• tau.azure.com/payload-digest is stamped consistently by every renderer that embeds a payload, not just rayjobrender.

Proposed approach

1. Stamp the CLI version in experiment.Metadata.KubernetesMetadata() (core/experiment/metadata.go:91). That is the single point all three run paths already funnel through — run_managed_workflow.go:387, run_job.go:321, and run_ray.go:173 all call experiment.MergeMetadata with the same capture metadata — so one addAnnotation there covers every workload kind uniformly, with no per-renderer changes and no asymmetry risk.

   An annotation (not a label) is the right shape: version strings like v0.3.0-12-gabc1234 and dev are not guaranteed to satisfy label-value constraints. Suggested key tau.azure.com/tau-version, alongside the existing capture-version.

   Worth deciding explicitly: what to record for a locally built binary, where core/version.Version defaults to dev. dev is honest and still more useful than nothing, but a commit SHA would be better — core/version.Commit is already wired for -ldflags.

2. Stamp payload.AnnotationDigest in cli/internal/manifest's workload annotations from the already-computed payloadEmbeds digests, matching rayjobrender. Decide whether the script and manifest digests are reported separately or combined, since payloadEmbeds carries two.

Compatibility: both are additive annotations. runhistory already tolerates absent annotations (text(annotations[...])), so older workloads keep working and nothing needs a migration.

Alternatives considered

• Rely on app.kubernetes.io/version. Doesn't work — nothing writes it, and in practice it holds the researcher's SHA, so reusing it would overwrite user data and conflate two different things.
• Rely on the chart or image version. Answers a different question. The submitting CLI is installed on the researcher's machine (make install-tau-cli) and nothing reconciles it with the cluster's charts, so platform version is not a proxy for CLI version.
• Infer from behaviour. What the review actually had to do. It works only when a version difference happens to have observable side effects, and yields an inference rather than a fact.
• Stamp per-renderer. Rejected for item 1 — that is exactly how item 2 became inconsistent. MergeMetadata is the choke point and should be used.

Additional context

This is the concrete cost behind the "improve environment provenance" recommendation in the East US 2 review, which had previously been recorded as a low-priority nice-to-have with no demonstrated impact. It has one now.

One correction that came out of the same investigation and is worth recording here, since it is easy to reach the opposite conclusion from cluster observation alone: batch Jobs and RayJobs are not asymmetric in their tau.azure.com/* provenance at HEAD. All three run paths share experiment.MergeMetadata, so both kinds receive the same annotation set. Live clusters running older builds may show batch Jobs carrying markedly less than RayJobs, but that is version skew, not current behaviour. The only genuine provenance asymmetry at HEAD is the payload-digest gap in item 2, and it is a renderer difference rather than a workload-kind difference.

Separately, and deliberately out of scope here: batch Jobs really are less diagnosable than RayJobs at terminal state, but for an unrelated reason — a Kubernetes Job's status carries only BackoffLimitExceeded while KubeRay writes the actual cause onto the RayJob object. That is upstream API behaviour, not a TauGrid annotation gap, and #110 addressed it by synthesizing a failure summary from pod state. Mentioned only so the two are not conflated.

Related: #110 (durable failure summaries, workspace TTL default), #111 (scratchMount CRD gap).