Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[k8s] on-demand TPU support on GKE #3947

Draft
wants to merge 31 commits into
base: master
Choose a base branch
from
Draft

[k8s] on-demand TPU support on GKE #3947

wants to merge 31 commits into from
+461 −191

Commits on Sep 16, 2024

  1. initial version of TPU support on GKE

    @landscapepainter
    landscapepainter committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    a929474 View commit details
    Browse the repository at this point in the history

  2. revert unnecesary change

    @landscapepainter
    landscapepainter committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    80e1877 View commit details
    Browse the repository at this point in the history

  3. revert

    @landscapepainter
    landscapepainter committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    70a07ab View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. use TPU_LABEL_KEY constant

    @landscapepainter
    landscapepainter committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    0cba9a5 View commit details
    Browse the repository at this point in the history

  2. nit

    @landscapepainter
    landscapepainter committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    17bcbd8 View commit details
    Browse the repository at this point in the history

  3. nit

    @landscapepainter
    landscapepainter committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    9233bf5 View commit details
    Browse the repository at this point in the history

  4. update detect_gpu_label_formatter() to use match_label_key()

    @landscapepainter
    landscapepainter committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    12e62c0 View commit details
    Browse the repository at this point in the history

  5. tidy get_gpu_label_key_value

    @landscapepainter
    landscapepainter committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    c795fe7 View commit details
    Browse the repository at this point in the history

  6. nit

    @landscapepainter
    landscapepainter committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    1c895f0 View commit details
    Browse the repository at this point in the history

  7. update method name

    @landscapepainter
    landscapepainter committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    a8f5b6b View commit details
    Browse the repository at this point in the history

  8. update get_gke_accelerator_name to support TPU

    @landscapepainter
    landscapepainter committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    bdb3469 View commit details
    Browse the repository at this point in the history

  9. add support for get_label_keys method due to TPU label key

    @landscapepainter
    landscapepainter committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    1d2d243 View commit details
    Browse the repository at this point in the history

  10. syntax

    @landscapepainter
    landscapepainter committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    92f4f38 View commit details
    Browse the repository at this point in the history

  11. update get_tpu_topology_label_key_value

    @landscapepainter
    landscapepainter committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    2662ec8 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. nit

    @landscapepainter
    landscapepainter committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    58f8ad6 View commit details
    Browse the repository at this point in the history

  2. refactor error surfacing methods to have it work with TPU support

    @landscapepainter
    landscapepainter committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    1cf82b6 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2024

  1. update toleration comment

    @landscapepainter
    landscapepainter committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    7b551c9 View commit details
    Browse the repository at this point in the history

  2. support listing available TPUs and show-gpus for TPUs

    @landscapepainter
    landscapepainter committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    81a05ee View commit details
    Browse the repository at this point in the history

  3. nit

    @landscapepainter
    landscapepainter committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    e8764f1 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2024

  1. Update /tmp/tpu_logs dir's write permission

    @landscapepainter
    landscapepainter committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    724806a View commit details
    Browse the repository at this point in the history

  2. nit

    @landscapepainter
    landscapepainter committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    e8d73fe View commit details
    Browse the repository at this point in the history

  3. nit

    @landscapepainter
    landscapepainter committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    7ac5036 View commit details
    Browse the repository at this point in the history

  4. comment update on TPU resource lackage error handling

    @landscapepainter
    landscapepainter committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    4470dbe View commit details
    Browse the repository at this point in the history

  5. Update to use global constant instead of hard coded string of nvidia.… 

    …com/gpu and google.com/tpu
    @landscapepainter
    landscapepainter committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    0860e45 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. add smoke test and make exec work on TPU pods

    @landscapepainter
    landscapepainter committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    35f3c80 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. update smoke test to check if TPU is reachable.

    @landscapepainter
    landscapepainter committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    2b56a9e View commit details
    Browse the repository at this point in the history

  2. add comment

    @landscapepainter
    landscapepainter committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    305705c View commit details
    Browse the repository at this point in the history

  3. nit

    @landscapepainter
    landscapepainter committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    c2b5bfc View commit details
    Browse the repository at this point in the history

  4. Comment on number of requested TPU chips for multi- and single- host … 

    …TPU slice.
    @landscapepainter
    landscapepainter committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    2ba5537 View commit details
    Browse the repository at this point in the history

  5. update method to check GKE supported TPU name

    @landscapepainter
    landscapepainter committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    92cd77d View commit details
    Browse the repository at this point in the history

  6. nit

    @landscapepainter
    landscapepainter committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    d085a5b View commit details
    Browse the repository at this point in the history