fix(api,watcher): cascade delete Results on namespace deletion

[pratap0007]

fix(api,watcher): cascade delete Results on namespace deletion and map Loki/Splunk HTTP errors to proper gRPC codes

When a namespace is deleted, Results API records persisted in Postgres but log retrieval failed with a generic 500 Internal Error. The root cause was twofold:

1. The watcher had no Namespace informer, so Results/Records were never cleaned up when a namespace was deleted, leaving orphaned data.

2. The log plugin mapped ALL non-200 HTTP responses from Loki/Splunk to codes.Internal with a generic error message. When Loki returned 403 Forbidden for a deleted namespace tenant, this surfaced as an unhelpful "Error fetching log data (code: 500)" to the user.

This commit addresses both issues:

Namespace cascade deletion (Fix 1):

• Add a new Namespace reconciler (pkg/watcher/reconciler/namespace/) that watches for namespace deletion events via the Kubernetes informer
• On namespace deletion or DeletionTimestamp being set, the reconciler paginates through ListResults and calls DeleteResult for each result in that namespace. DB-level cascade (OnDelete:CASCADE) handles Records
• Register the new controller in the watcher's main.go

HTTP-to-gRPC error mapping (Fix 2):

• Add httpStatusToGRPCCode() helper mapping HTTP 400->InvalidArgument, 401->Unauthenticated, 403->PermissionDenied, 404->NotFound, 429->ResourceExhausted, and default->Internal
• Apply mapping to Loki and both Splunk error paths, including the HTTP status code in the error message for debuggability
• Fix response body leak in getLokiLogs (missing defer resp.Body.Close)

Resolves: SRVKP-12883

/kind bug

Changes

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you review them:

• Has Docs included if any changes are user facing
• Has Tests included if any functionality added or changed
• Tested your changes locally (if this is a code change)
• Follows the commit message standard
• Meets the Tekton contributor standards (including functionality, content, code)
• Has a kind label. You can add a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
• Release notes block below has been updated with any user-facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
• Release notes contain the string "action required" if the change requires additional action from users switching to the new release

Release Notes

Added a namespace reconciler to the watcher to handle namespace deletions. When a namespace is deleted, all associated Results are removed through the API, and Records are automatically cleaned up by database foreign key constraints, preventing orphaned data.

[tekton-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign enarha after the PR has been reviewed.
You can assign the PR to them by writing /assign @enarha in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[khrm]

Tbh, it makes sense to persist data for deleted namespace. It's an issue from Loki side that it's not persisting data.

[khrm]

Are we sure that this issue exist with Splunk. Also, does this issue exist with only OpenShift Loki or normal Loki also.

[enarha]

Tbh, it makes sense to persist data for deleted namespace. It's an issue from Loki side that it's not persisting data.

I agree. We should not simply purge the data for deleted namespace. The namespace could be deleted by mistake and easily re-created, but by that time the archived data would be gone. Furthermore, I think this should not be the watcher responsibility and I don't like this extra delete permission added to it. Instead that could be made responsibility of the retention-policy-agent (it already has this delete permission and that's its primary job) which can be enabled by the user with some flag e.g. purge-data-for-deleted-namespace and probably make that opt-in initially . As currently implemented, this is also a breaking change.

Providing proper error message is important though.

[pratap0007]

Are we sure that this issue exist with Splunk. Also, does this issue exist with only OpenShift Loki or normal Loki also.

It has an issue with Loki, and I haven't checked it with Splunk

[tekton-robot]

@pratap0007: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.