[Core] Admin policy enforcement plugin

[Michaelvll]

This PR allows the admin to bring a customized policy enforcement script in Python, which can apply any mutation to all user tasks based on customized policy requirements.

Usage:

User-side

In ~/.sky/config.yaml:

policy: my_package.skypilot_policy_fn

Admin-side

Create the skypilot_policy_fn function with the following signature:

def skypilot_policy_fn(user_task: sky.UserTask) -> sky.MutatedUserTask:
    pass

See policy.py for the definition of these two types.

TODO:

• Check when to apply policy, e.g. skip the policy apply for exec? (Let's apply for both for now)
• Upload policy package to controllers (We now apply the policy locally before uploading to controller, so this should be fine)
• Add to config.rst
• Add to some admin doc page for how to set this up
• Policy name/hash label (blocked by [UX] Make labels work in task.yaml without the requirement for specifying cloud #3965)

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • normal launch with customized labels
  • normal launch/exec with autostop enforcement
  • jobs launch with customized labels
  • jobs launch with autostop enforcement
  • service launch with customized labels

export SKYPILOT_CONFIG='/home/gcpuser/skypilot-dev/examples/admin_policy/config_label_config.yaml'
sky serve up -n test-global-label examples/serve/http_server/task.yaml --cloud gcp

• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[concretevitamin]

A bit unclear if it's in-place since it also returns the task. How about "apply_in_place"?

[Michaelvll]

It is not in-place. Either way is fine, but I found not in-place is more commonly seen operations.

[concretevitamin]

LGTM @Michaelvll. I think two TODOs before merging:

(1) Smoke tests, since this touches config / backend
(2) Update PR description (policy)

[concretevitamin]

Q: what's the latency impact of this policy on sky launch/exec? Is it small? Do we expect users to use this example policy and be ok with the latency?

[Michaelvll]

Tested:

• pytest tests/test_smoke.py (failed [Azure] test_cancel_azure failed due to error of resnet user program #3974 , [GCP] test_gcp_disk_tier for best tier failed due to best tier not mapping to pd-extreme #3973; fixed test_skyserve_fast_update)