Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support chain dag in example_app #267

Closed
wants to merge 18 commits into from
Closed

Support chain dag in example_app #267

wants to merge 18 commits into from

Conversation

Michaelvll
Copy link
Collaborator

@Michaelvll Michaelvll commented Jan 30, 2022

This PR supports the chain dag in the example_app.py. It is now available to run the chain dag.

Tested:

  • examples/example_app.py. It now can handle the input/output automatically.
  • Mocked training and inference with Inferentia.

Note: This is experimental support for chain dag. Need a lot of refactorization.

Future TODO:

  • Real training, inference pipeline.

  • GCS file mounts Cloud storage file mounts for gs:// #269

  • Multiple input/outputs

  • Do not download the cloud storage on remote

    • Proposed API:
    # train.py will read from the CLOUD storage (CLOUD may be decided by the optimizer) directly
    task.run = "python train.py --input SKY_INPUTS[0]"
    task.set_inputs('CLOUD://sky-example-test', estimated_size_gigabytes=150)
    
    # train.py will read from local disk
    task.run = "python train.py --input local_path
    task.set_inputs('CLOUD://sky-example-test', estimated_size_gigabytes=150)
    task.set_filemounts({local_path: 'SKY_INPUTS[0]'})

@Michaelvll Michaelvll changed the title [WIP] Support chain dag in example_app Support chain dag in example_app Jan 31, 2022
@Michaelvll Michaelvll added the P1 label Feb 16, 2022
@Michaelvll Michaelvll closed this Jun 3, 2022
@Michaelvll Michaelvll deleted the chain-dag branch June 3, 2022 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant