load cluster token from Den instead of generating client side

[jlewitt1]

• generate cluster tokens in Den instead of client side. tokens are now OIDC compliant, and required for enabling telemetry
• prevent regenerating the cluster tokens unnecessarily on the client side
• ping new den API on clusters in order to validate the token received

[sentry-io]

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: runhouse/resources/hardware/cluster.py

Function	Unhandled Issue
restart_server	ValueError: Error installing runhouse on cluster node <3.145.45.24> ... Event Count: 1
restart_server	ValueError: Error installing runhouse on cluster node <34.229.108.213> ... Event Count: 1
restart_server	ValueError: Failed to restart server llama-cluster main in <modul... Event Count: 1

---

_{Did you find this useful? React with a 👍 or 👎}

[jlewitt1]

• update env vars for den accounts in CI #1243
• updated tokens for dev ci testing #1236
• load cluster token from Den instead of generating client side #1228 👈
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @jlewitt1 and the rest of your teammates on Graphite

[mkandler]

👍