[Azure] Support fractional A10 instance types #3877

cblmemo · 2024-08-26T22:30:20Z

This PR support fractional A10 instance types from instance_type=xxx and accelerators=A10:{0.25,0.5,0.75}.

Tested (run the relevant ones):

Code formatting: bash format.sh
Any manual or new tests for this PR (please specify below)

# Launch cluster with
$ sky launch --instance-type Standard_NV6ads_A10_v5 -c sky-59bd-memory
# or
$ sky launch --gpus A10:0.25 -c sky-59bd-memory
# then
$ sky launch --gpus A10:0.24 -c sky-59bd-memory
sky.exceptions.ResourcesMismatchError: Task requested resources with fractional accelerator counts. For fractional counts, the required count must match the existing cluster. Got required accelerator A10:0.24 but the existing cluster has A10:0.25.
$ sky launch --gpus A10:1 -c sky-59bd-memory
sky.exceptions.ResourcesMismatchError: Requested resources do not match the existing cluster.
  Requested:    {1x <Cloud>({'A10': 1})}
  Existing:     1x Azure(Standard_NV6ads_A10_v5, {'A10': 0.25})
To fix: specify a new cluster name, or down the existing cluster first: sky down sky-59bd-memory
$ sky launch --gpus A10:0.25 -c sky-59bd-memory nvidia-smi
Task from command: nvidia-smi
Running task on cluster sky-59bd-memory...
W 08-28 11:16:57 cloud_vm_ray_backend.py:1937] Trying to launch an A10 cluster on Azure. This may take ~20 minutes due to driver installation.
I 08-28 11:16:57 cloud_vm_ray_backend.py:1314] To view detailed progress: tail -n100 -f /home/memory/sky_logs/sky-2024-08-28-11-16-56-005018/provision.log
I 08-28 11:16:58 provisioner.py:65] Launching on Azure eastus (all zones)
I 08-28 11:17:07 provisioner.py:450] Successfully provisioned or found existing instance.
I 08-28 11:17:19 provisioner.py:552] Successfully provisioned cluster: sky-59bd-memory
I 08-28 11:17:21 cloud_vm_ray_backend.py:3294] Job submitted with Job ID: 3
I 08-28 18:17:22 log_lib.py:412] Start streaming logs for job 3.
INFO: Tip: use Ctrl-C to exit log streaming (task will not be killed).
INFO: Waiting for task resources on 1 node. This will block if the cluster is full.
INFO: All task resources reserved.
INFO: Reserved IPs: ['10.8.0.4']
(sky-cmd, pid=15884) Wed Aug 28 18:17:23 2024       
(sky-cmd, pid=15884) +---------------------------------------------------------------------------------------+
(sky-cmd, pid=15884) | NVIDIA-SMI 535.161.08             Driver Version: 535.161.08   CUDA Version: 12.2     |
(sky-cmd, pid=15884) |-----------------------------------------+----------------------+----------------------+
(sky-cmd, pid=15884) | GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
(sky-cmd, pid=15884) | Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
(sky-cmd, pid=15884) |                                         |                      |               MIG M. |
(sky-cmd, pid=15884) |=========================================+======================+======================|
(sky-cmd, pid=15884) |   0  NVIDIA A10-4Q                  On  | 00000002:00:00.0 Off |                    0 |
(sky-cmd, pid=15884) | N/A   N/A    P0              N/A /  N/A |      0MiB /  4096MiB |      0%      Default |
(sky-cmd, pid=15884) |                                         |                      |             Disabled |
(sky-cmd, pid=15884) +-----------------------------------------+----------------------+----------------------+
(sky-cmd, pid=15884)                                                                                          
(sky-cmd, pid=15884) +---------------------------------------------------------------------------------------+
(sky-cmd, pid=15884) | Processes:                                                                            |
(sky-cmd, pid=15884) |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
(sky-cmd, pid=15884) |        ID   ID                                                             Usage      |
(sky-cmd, pid=15884) |=======================================================================================|
(sky-cmd, pid=15884) |  No running processes found                                                           |
(sky-cmd, pid=15884) +---------------------------------------------------------------------------------------+
INFO: Job finished (status: SUCCEEDED).
Shared connection to 23.101.130.81 closed.
I 08-28 11:17:24 cloud_vm_ray_backend.py:3329] Job ID: 3
I 08-28 11:17:24 cloud_vm_ray_backend.py:3329] To cancel the job:       sky cancel sky-59bd-memory 3
I 08-28 11:17:24 cloud_vm_ray_backend.py:3329] To stream job logs:      sky logs sky-59bd-memory 3
I 08-28 11:17:24 cloud_vm_ray_backend.py:3329] To view the job queue:   sky queue sky-59bd-memory
I 08-28 11:17:24 cloud_vm_ray_backend.py:3425] 
I 08-28 11:17:24 cloud_vm_ray_backend.py:3425] Cluster name: sky-59bd-memory
I 08-28 11:17:24 cloud_vm_ray_backend.py:3425] To log into the head VM: ssh sky-59bd-memory
I 08-28 11:17:24 cloud_vm_ray_backend.py:3425] To submit a job:         sky exec sky-59bd-memory yaml_file
I 08-28 11:17:24 cloud_vm_ray_backend.py:3425] To stop the cluster:     sky stop sky-59bd-memory
I 08-28 11:17:24 cloud_vm_ray_backend.py:3425] To teardown the cluster: sky down sky-59bd-memory
Clusters
NAME             LAUNCHED        RESOURCES                                        STATUS  AUTOSTOP  COMMAND                       
sky-59bd-memory  a few secs ago  1x Azure(Standard_NV6ads_A10_v5, {'A10': 0.25})  UP      -         sky launch -c sky-59bd-me...

All smoke tests: pytest tests/test_smoke.py
Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

Michaelvll · 2024-08-27T06:14:28Z

sky/clouds/service_catalog/azure_catalog.py

+    # Filter out instance types that only contain a fractional of GPU.
+    df_filtered = _df.loc[~_df['InstanceType'].isin(_FILTERED_A10_INSTANCE_TYPES
+                                                   )]


Instead of excluding the instances directly, can we print out some hints like the one when we specify sky launch --gpus L4:

Multiple AWS instances satisfy L4:1. The cheapest AWS(g6.xlarge, {'L4': 1}) is considered among: I 08-27 06:09:54 optimizer.py:922] ['g6.xlarge', 'g6.2xlarge', 'g6.4xlarge', 'gr6.4xlarge', 'g6.8xlarge', 'gr6.8xlarge', 'g6.16xlarge'].

This hint is used to print instances w/ same accelerator number. I'm thinking if we should do this to fractional GPUs..

…ployment group works for fractional one

cblmemo · 2024-08-28T18:21:33Z

I support launching from --gpus A10:0.25 and only allow strict equal on fractional GPU requirements. Also updated the test I've done in the PR description. PTAL!

Michaelvll

Thanks for the update @cblmemo! Mostly looks good to me with some slight issue.

Michaelvll · 2024-09-02T22:16:58Z

sky/clouds/service_catalog/data_fetchers/fetch_azure.py

+    # Manually update the GPU count for fractional A10 instance types.
+    df_ret['AcceleratorCount'] = df_ret.apply(_upd_a10_gpu_count,
+                                              axis='columns')


Could we say more in the comment for why we need to do it manually?

Good point! Added. PTAL

Michaelvll · 2024-09-02T22:17:48Z

sky/clouds/service_catalog/scp_catalog.py

@@ -67,7 +67,7 @@ def get_default_instance_type(


 def get_accelerators_from_instance_type(
-        instance_type: str) -> Optional[Dict[str, int]]:
+        instance_type: str) -> Optional[Dict[str, Union[int, float]]]:


For the clouds that won't return float, we probably don't need to update it to Union[int, float]?

Same, here is what I got:

./format.sh SkyPilot Black: All done! ✨ 🍰 ✨ 4 files left unchanged. SkyPilot yapf: Done SkyPilot isort: Skipped 5 files SkyPilot mypy: sky/clouds/service_catalog/aws_catalog.py:248: error: Incompatible return value type (got "Optional[Dict[str, Union[int, float]]]", expected "Optional[Dict[str, int]]") [return-value] Found 1 error in 1 file (checked 225 source files)

Michaelvll · 2024-09-02T22:19:13Z

sky/clouds/azure.py

@@ -252,7 +251,7 @@ def zones_provision_loop(
    def get_accelerators_from_instance_type(
        cls,
        instance_type: str,
-    ) -> Optional[Dict[str, int]]:
+    ) -> Optional[Dict[str, Union[int, float]]]:


For the clouds that will not return float, we can avoid using Union, as mypy should support a inherited method that only uses a subset of the type.

I tried and this is what I got:

./format.sh SkyPilot Black: All done! ✨ 🍰 ✨ 4 files left unchanged. SkyPilot yapf: Done SkyPilot isort: Skipped 5 files SkyPilot mypy: sky/clouds/aws.py:365: error: Return type "Optional[Dict[str, int]]" of "get_accelerators_from_instance_type" incompatible with return type "Optional[Dict[str, Union[int, float]]]" in supertype "Cloud" [override] sky/clouds/aws.py:369: error: Incompatible return value type (got "Optional[Dict[str, Union[int, float]]]", expected "Optional[Dict[str, int]]") [return-value] sky/clouds/aws.py:397: error: Argument 1 to "make_ray_custom_resources_str" has incompatible type "Optional[Dict[str, int]]"; expected "Optional[Dict[str, Union[int, float]]]" [arg-type] Found 3 errors in 1 file (checked 225 source files)

Michaelvll · 2024-09-02T22:23:42Z

sky/resources.py

+                if isinstance(self.accelerators[acc], float) or isinstance(
+                        other_accelerators[acc], float):
+                    # If the requested accelerator count is a float, we only
+                    # allow strictly equal counts since all of the float point
+                    # accelerator counts are less than 1 (e.g., 0.1, 0.5), and
+                    # we want to avoid semantic ambiguity (e.g. launching
+                    # with --gpus A10:0.25 on a A10:0.75 cluster).
+                    if not math.isclose(self.accelerators[acc],
+                                        other_accelerators[acc]):
+                        return False


We should allow requested resources to be a float, while the existing accelerators to be a int, as long as requested resources is <= existing resources.

That said, the

Suggested change

if isinstance(self.accelerators[acc], float) or isinstance(

other_accelerators[acc], float):

# If the requested accelerator count is a float, we only

# allow strictly equal counts since all of the float point

# accelerator counts are less than 1 (e.g., 0.1, 0.5), and

# we want to avoid semantic ambiguity (e.g. launching

# with --gpus A10:0.25 on a A10:0.75 cluster).

if not math.isclose(self.accelerators[acc],

other_accelerators[acc]):

return False

if isinstance(other_accelerators[acc], float) and not other_accelerators[acc].is_integer():

# If the requested accelerator count is a float, we only

# allow strictly equal counts since all of the float point

# accelerator counts are less than 1 (e.g., 0.1, 0.5), and

# we want to avoid semantic ambiguity (e.g. launching

# with --gpus A10:0.25 on a A10:0.75 cluster).

if not math.isclose(self.accelerators[acc],

other_accelerators[acc]):

return False

Good point! Updated. Thanks!

Actually, after a second thought, I think we should still keep the original isinstance(self.accelerators[acc], float) or isinstance(other_accelerators[acc], float) condition. Considering the following case: user submit the jobs with --gpus A10:0.5 and the cluster has A10:1. In fact the requirements 0.5 will be translated to 1 and thus the user can only have one A10:0.5 job instead of 2, which is confusing. The original condition capture such case but the updated one (isinstance(other_accelerators[acc], float) and not other_accelerators[acc].is_integer()) does not.

We did allow having two A10:0.5 running on a single cluster with A10:1. Do you know when did we change the behavior of this? or did we ever change this behavior before this PR?

We need to fix this before we merge the PR

Michaelvll

Thanks for adding the support @cblmemo! It seems good to me. Please do some tests to make sure the changes do not cause issues with other clouds and other ACC types (considering we have changed significant amount of places)

Michaelvll · 2024-09-06T22:30:46Z

sky/backends/cloud_vm_ray_backend.py

+                                    'Task requested resources with fractional '
+                                    'accelerator counts. For fractional '
+                                    'counts, the required count must match the '
+                                    'existing cluster. Got required accelerator'
+                                    f' {acc}:{self_count} but the existing '
+                                    f'cluster has {acc}:{existing_count}.')


This error message is not accurate? Our check is for ACC count of existing cluster instead of the task requested resources?

Please see the above comments 🤔

Michaelvll · 2024-09-09T23:39:11Z

sky/resources.py

+                if isinstance(self.accelerators[acc], float) or isinstance(
+                        other_accelerators[acc], float):
+                    # If the requested accelerator count is a float, we only
+                    # allow strictly equal counts since all of the float point
+                    # accelerator counts are less than 1 (e.g., 0.1, 0.5), and
+                    # we want to avoid semantic ambiguity (e.g. launching
+                    # with --gpus A10:0.25 on a A10:0.75 cluster).
+                    if not math.isclose(self.accelerators[acc],
+                                        other_accelerators[acc]):
+                        return False


We need to fix this before we merge the PR

Michaelvll · 2024-09-11T06:50:24Z

sky/resources.py

+                    # If the existing accelerator count is a float, we only
+                    # allow strictly equal counts since all of the float point
+                    # accelerator counts are less than 1 (e.g., 0.167, 0.5), and
+                    # we want to avoid semantic ambiguity (e.g. launching
+                    # with --gpus A10:0.25 on a A10:0.75 cluster).


Could we rephrase this comment? Not sure what we are talking about here.

sky/backends/cloud_vm_ray_backend.py

cblmemo · 2024-09-11T07:04:20Z

Just identified another bug: for a A10:0.5 cluster, previous implementation would force using --gpus A10:0.5 when sky exec, which could actually have 2 jobs simultaneous running as the ray cluster has resources A10:1. Just fixed by if we found a fractional cluster, then we set the gpu demand to its ceiling value (which is essentially 1).

Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com>

cblmemo · 2024-09-11T07:16:01Z

Actually, after a second thought, I think we could even allow requiring --gpus A10:0.25 for a A10:0.5 cluster - we just need to convert it. The actual required num of gpus can be calculated by {required_count} / {cluster_acc_count} * 1, as we set the remote ray cluster's custom resources to A10:1. For this example it should require A10:0.5, so that two --gpus A10:0.25 can running simultaneously for a A10:0.5 cluster.

cblmemo · 2024-09-11T07:27:19Z

Another TODO: I just found that there are 1/6 and 1/3 A10 instance types. We need to figure out a precision to display such decimals.

cblmemo · 2024-09-12T06:13:50Z

All todos is done. After smoke test it should be able to merge ;)

Michaelvll · 2024-09-13T01:18:54Z

sky/clouds/service_catalog/common.py

+            return int(value)
+        return float(value)
+
+    return {acc_name: _convert(acc_count)}


 def get_instance_type_for_accelerator_impl(


Should we update the acc_count type here? Also, when we are comparing the acc_count should we make sure every number within math.abs(df['AcceleratorCount'] - acc_count) <= 0.01 should work. Otherwise, a user running sky launch --gpus A10:0.16 or sky launch --gpus A10:0.1666 would fail?

Should we update the acc_count type here?

Sry could you elaborate on this...? Are you saying there are a better place to update the type or what..?

Also, when we are comparing the acc_count should we make sure every number within math.abs(df['AcceleratorCount'] - acc_count) <= 0.01 should work.

For this, I'm slightly concerned about the case when user:

sky launch -c a10-frac --gpus A10:0.16 # detected as 0.167 in catalog, then launch cluster with 0.167 gpu sky exec a10-frac --gpus A10:0.16 sleep 100000 # user would think the cluster is full sky exec a10-frac --gpus A10:0.007 sleep 100000 # however, this still running as the cluster is actually launched with 0.167 gpu.

To deal with the failing, we currently shows all valid instance type as fuzzy candidate and user could then modify their acc count:

sky launch --gpus A10:0.16 I 09-12 22:34:50 optimizer.py:1301] No resource satisfying <Cloud>({'A10': 0.16}) on [AWS, GCP, Azure, RunPod]. I 09-12 22:34:50 optimizer.py:1305] Did you mean: ['A100-80GB-SXM:1', 'A100-80GB-SXM:2', 'A100-80GB-SXM:4', 'A100-80GB-SXM:8', 'A100-80GB:1', 'A100-80GB:2', 'A100-80GB:4', 'A100-80GB:8', 'A100:1', 'A100:16', 'A100:2', 'A100:4', 'A100:8', 'A10:0.167', 'A10:0.333', 'A10:0.5', 'A10:1', 'A10:2', 'A10G:1', 'A10G:4', 'A10G:8'] sky.exceptions.ResourcesUnavailableError: Catalog does not contain any instances satisfying the request: Task<name=sky-cmd>(run=<empty>) resources: <Cloud>({'A10': 0.16}). To fix: relax or change the resource requirements. Try one of these offered accelerators: ['A100-80GB-SXM:1', 'A100-80GB-SXM:2', 'A100-80GB-SXM:4', 'A100-80GB-SXM:8', 'A100-80GB:1', 'A100-80GB:2', 'A100-80GB:4', 'A100-80GB:8', 'A100:1', 'A100:16', 'A100:2', 'A100:4', 'A100:8', 'A10:0.167', 'A10:0.333', 'A10:0.5', 'A10:1', 'A10:2', 'A10G:1', 'A10G:4', 'A10G:8'] Hint: sky show-gpus to list available accelerators. sky check to check the enabled clouds.

Does that sounds good to you?

fix

de35be4

cblmemo mentioned this pull request Aug 26, 2024

[Azure] Add back fractional A10 instance types skypilot-org/skypilot-catalog#79

Open

Michaelvll reviewed Aug 27, 2024

View reviewed changes

cblmemo added 7 commits August 27, 2024 14:06

change catalog to float gpu num

39d6c15

support print float point gpu in sky launch. TODO: test if the ray de…

7324504

…ployment group works for fractional one

fix unittest

347ad62

format

71af06e

patch ray resources to ceil value

d419442

support launch from --gpus A10

f529689

only allow strictly match fractional gpu counts

2031a50

cblmemo changed the title ~~[Azure] Support fractional A10 instance types only from instance_type=xxx~~ [Azure] Support fractional A10 instance types Aug 28, 2024

cblmemo requested a review from Michaelvll August 28, 2024 18:19

Michaelvll reviewed Sep 2, 2024

View reviewed changes

cblmemo added 2 commits September 3, 2024 14:54

address comment

07e47d6

Merge remote-tracking branch 'origin/master' into support-fractional-a10

639c686

Michaelvll approved these changes Sep 6, 2024

View reviewed changes

cblmemo added 3 commits September 6, 2024 16:54

Merge remote-tracking branch 'origin/master' into support-fractional-a10

4c45ff7

change back condition

84d6d0d

fix

eca7033

Michaelvll requested changes Sep 9, 2024

View reviewed changes

apply suggestions from code review

0055fc1

Michaelvll reviewed Sep 11, 2024

View reviewed changes

fix

9652119

cblmemo and others added 2 commits September 11, 2024 00:04

Update sky/backends/cloud_vm_ray_backend.py

a5c5b15

Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com>

format

d2cff96

fix display of fuzzy candidates

e8e9954

cblmemo added 2 commits September 11, 2024 22:48

fix precision issue

db607fa

fix num gpu required

e98ecdc

Michaelvll reviewed Sep 13, 2024

View reviewed changes

cblmemo requested a review from Michaelvll September 19, 2024 19:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Azure] Support fractional A10 instance types #3877

[Azure] Support fractional A10 instance types #3877

cblmemo commented Aug 26, 2024 •

edited

Loading

Michaelvll Aug 27, 2024

cblmemo Aug 27, 2024

cblmemo commented Aug 28, 2024

Michaelvll left a comment

Michaelvll Sep 2, 2024

cblmemo Sep 3, 2024

Michaelvll Sep 2, 2024

cblmemo Sep 3, 2024

Michaelvll Sep 2, 2024

cblmemo Sep 3, 2024

Michaelvll Sep 2, 2024

cblmemo Sep 3, 2024

cblmemo Sep 7, 2024

Michaelvll Sep 9, 2024

Michaelvll Sep 9, 2024

Michaelvll left a comment

Michaelvll Sep 6, 2024

cblmemo Sep 7, 2024

Michaelvll Sep 9, 2024

Michaelvll Sep 11, 2024

cblmemo commented Sep 11, 2024

cblmemo commented Sep 11, 2024

cblmemo commented Sep 11, 2024

cblmemo commented Sep 12, 2024

Michaelvll Sep 13, 2024

cblmemo Sep 13, 2024

[Azure] Support fractional A10 instance types #3877

Are you sure you want to change the base?

[Azure] Support fractional A10 instance types #3877

Conversation

cblmemo commented Aug 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cblmemo commented Aug 28, 2024

Michaelvll left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Michaelvll left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cblmemo commented Sep 11, 2024

cblmemo commented Sep 11, 2024

cblmemo commented Sep 11, 2024

cblmemo commented Sep 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cblmemo commented Aug 26, 2024 •

edited

Loading