Cannot access NERC S3 OpenStack dashboard (OIDC login) and S3 credentials failing since Aug 4; possibly related to 7/31 suspension notice

[schwesig]

What I tried:
Logging into https://stack.nerc.mghpcc.org/ with Google account (OpenID Connect).
Tried normal and also over Harvard VPN, same result both times.

Exact error on login page:
"Login failed: You are not authorized for any projects or domains."

The login page also shows this banner:
"IMPORTANT: All NERC services will be suspended on 7/31/2026."

Screenshot: will add in a comment.

Related issue, probably same root cause:
Our NERC S3 credentials stopped working at the same time.
We run nightly backups (metrics + loki) to Wasabi via rclone (endpoint https://stack.nerc.mghpcc.org:13808, access key id 63...62cd).
Last successful run was the night of Aug 3, 2026.
Every run since Aug 4 fails with 403 SignatureDoesNotMatch on the S3 endpoint.
Wasabi side (destination) is fine, we checked.

My suspicion:
Both symptoms showed up at the same time, so I think this is not just a rotated key.
Looks more like the project/account itself got suspended or deauthorized, maybe connected to the 7/31 suspension notice on the login page.
Would be good to get this confirmed either way.

Impact:

• No dashboard access at all right now.
• Nightly observability backups to Wasabi failing since Aug 4 (metrics + loki).
• No access to any historical metrics on the S3 storage right now; only the local caches are left (a few days Thanos side, ~15 days per managed cluster).
• Data through Aug 3 is safe in Wasabi already (rclone copy, no delete), only the deltas since Aug 4 are missing.
• Should catch up fine once access is restored, as long as source objects still exist on NERC side.

Ask:

• Can someone confirm the status of accounts / keys?
• Was it suspended as part of the 7/31 notice?

---

Memo:

https://coldfront.mss.mghpcc.org/project/563/

NERC
OpenStack
Allocated Project Name: OBS-bb3d72

https://coldfront.mss.mghpcc.org/resource/1/

https://coldfront.mss.mghpcc.org/allocation/745/

[schwesig]

[schwesig]

One more impact detail:
Independent of the current obs cluster node outage (2 worker nodes down), we cannot query any metrics data that lives on the S3 storage right now.
What is left is only the local cache on the Thanos side (a few days) and the ~15 days each managed cluster keeps locally itself.
So until S3 access is restored, there is no access to any historical metrics beyond that window.

[schwesig]

Update: the project got reactivated on the NERC side, and I can confirm S3 access works again.
Same credentials as before, so it was the project suspension, not a rotated key.
All buckets are visible again and the older data (July) is intact.
The Loki compactor recovered on its own already.
I will check tonight's backup run went green and the metrics deltas since Aug 4 caught up.

[schwesig]

Thanks to @jtriley

[schwesig]

some background info/sources

https://coldfront.mss.mghpcc.org/project/563/

NERC
OpenStack
Allocated Project Name: OBS-bb3d72

https://coldfront.mss.mghpcc.org/resource/1/

https://coldfront.mss.mghpcc.org/allocation/745/

[hpdempsey]

@jtriley why was this project suspended though, when we still had OpenShift customers running? Did we just disable all project in prod except for a list of "still runnings" that we should have added our infra tools to? Asking so we know what to do next time.