Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workflow parameters #159

Merged
merged 24 commits into from
Aug 25, 2023
Merged

Workflow parameters #159

merged 24 commits into from
Aug 25, 2023

Conversation

riga
Copy link
Owner

@riga riga commented Aug 25, 2023

This PR adds workflow parameters, a way to parametrize workflows in a way that makes the branch lookup dynamic, based on values in the branch map.

From the README:


Example code:

import law
import luigi


class MyWorkflow(law.LocalWorkflow):

    option_a = law.WorkflowParameter()
    option_b = law.WorkflowParameter(cls=luigi.IntParameter)

    @classmethod
    def create_branch_map(cls, params):
        return {
            0: {"option_a": "foo", "option_b": 123},
            1: {"option_a": "bar", "option_b": 456},
            2: {"option_a": "foo", "option_b": 456},
            # ... you could add more branches here
        }

    def run(self):
        # this run method is only called by *branches*, i.e.,
        #   - self.branch will be 0, 1 or 2, and
        #   - self.branch_data will refer to the corresponding dictionary in the branch map

        pass  # ... implementation

option_a and option_b are law.WorkflowParameters that can optionally define a cls (or inst) of a other parameter object that is used for parameter parsing and serialization (as always).

create_branch_map becomes a @classmethod and receives all task parameters in a dictionary params.
This is necessary since the automatic lookup of branches based on the values of workflow parameters, internally, must happen before the task is actually instantiated.

As a result, parameters can be defined verbosely on the command line, translate to branch values, and configure which branches are run by the workflow:

  • --option-a foo --option-b 123 → finds branch=0 (but for this, you probably wouldn't need a workflow in the first place)
  • --option-a foo → finds branches=[0, 2] and runs them as a workflow
  • --option-b 456 → finds branches=[1, 2] and runs them as a workflow
  • --option-b 123 → finds branches=[0] and runs it in a workflow; in particular, this is not a branch as --option-a is not specified, meaning we're "not interested in any particular value of option_a, just run all that are found".

Closes #131, #137.

@riga riga self-assigned this Aug 25, 2023
@riga riga merged commit cec32d9 into master Aug 25, 2023
12 checks passed
@riga riga deleted the feature/workflow_parameters branch August 25, 2023 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Proposal: high-level workflows
1 participant