Own immutable Tau workload lifecycle - #31
Conversation
| if !isRunWorkloadNotFound(err) { | ||
| return err | ||
| } | ||
| deleted, cleanupErr := deleteOwnedRunRayClusters(cmd.Context(), newKubernetesRunSubmissionRunner(r), ns, args[0]) |
There was a problem hiding this comment.
P1 This passes the original selector as a physical RayJob name, but orphan lookup filters ray.io/originated-from-cr-name. After the RayJob is gone, tau run delete <run-id> or a unique logical name therefore finds no RayCluster. Resolve orphan clusters by physical origin, run-ID, or logical name with the same ambiguity rules before deletion.
| {resource: "jobs.batch", kind: "Job"}, | ||
| {resource: "rayjobs.ray.io", kind: "RayJob"}, | ||
| } { | ||
| out, err := runner.Raw(ctx, []string{ |
There was a problem hiding this comment.
P1 Exact-name operations now require list permission for both Jobs and RayJobs before matching starts. A Job-only principal fails on the RayJob list (and vice versa), so it cannot operate on a workload it is authorized to read. Try exact named GETs first, and reserve cross-kind listing for run-ID or logical-name resolution.
| return err | ||
| } | ||
| defer restore() | ||
| workloadRef, err := resolveRunWorkload(cmd.Context(), kube.New(resolvedContext), ns, name) |
There was a problem hiding this comment.
P1 This unconditional live-workload lookup breaks the documented recovery path after metadata deletion. Even with both --path and --pvc, a deleted Job or RayJob returns no-match before the supplied durable location is read. Skip workload resolution when both overrides are present.
| service := runSubmission{Resource: "service", Name: run.Name + "-headless", Namespace: run.Namespace} | ||
| metadata, err := existingRunMetadata(ctx, runner, service) | ||
| if err == nil { | ||
| if metadata.Labels[workloadmeta.LabelManagedBy] != workloadmeta.ManagedByValue || |
There was a problem hiding this comment.
P1 Pre-upgrade multi-node Jobs have a run-ID label on the Job, but their headless Service only has managed-by=tau; ownership was recorded by its controller ownerReference. This new check rejects that Service before deleting the Job, so cancel/delete cannot clean up existing runs after upgrade. Accept the legacy shape only when the controller ownerReference matches the resolved Job name and UID.
2 participants
Why
Completed Tau workloads with fixed Kubernetes names could block later submissions of the same logical run, forcing users to preserve evidence manually and use
kubectl deleteor replacement. Tau should own both immutable execution identity and safe workload lifecycle without overwriting prior runs.What changed
cancel, terminal-onlydelete/remove, and terminal-onlyarchivebehavior.Compatibility and safety
The existing config name remains the public logical run name. Physical names are capped at 47 characters for KubeRay compatibility. Client dry-run output remains deterministic, while server dry-run and real submissions use immutable identities. Cleanup never performs namespace-wide or label-only broad deletion and refuses foreign or ambiguous resources.
Validation
cd cli && go test ./... -count=1cd core && go test ./runs ./workloadmeta -count=1git diff --checkAI tools assisted with implementation and review; the resulting changes were independently reviewed and tested before submission.