Simple Entry Point PipeLines (seppl). Python library for parsing pipeline components with their own options.
seppl takes a very light-weight approach to avoid encroaching too much on your code. If you want to, you can add some compatibility checks between the pipeline components (see further down). However, the execution of the pipeline (and potentially moving data between components) is left to you and your code.
pip install seppl
pip install git+https://github.com/waikato-datamining/seppl.git
seppl can operate in three different modes:
- class lister registry - a single function (aka class lister) returns a dictionary
of superclasses and their associated modules (slower, due to analyzing code) - dynamic registry - only superclass and module need to be supplied for initializing a plugin class hierarchy (slower, due to analyzing code)
- explicit registry - every single plugin needs to be listed (fast)
Your plugins need to be derived from the seppl.Plugin
superclass, for the
magic to work. Such a plugin needs to implement the following two methods:
name(self)
: returns the name/identifier to use in the pipeline command string (potentially followed by options for the plugin to parse)description(self)
: returns a description (how long is up to you), which is used for the argparse parser; the same parser that can be used to generate a help screen for a plugin
In order to have custom options parsed by your plugin, you need to override the following two methods:
_create_argparser(self)
: this method assembles the argparse parser_apply_args(self, ns: argparse.Namespace)
: in this method, you need to transfer the parsed options into member variables of your plugin
If you want to enable checks between pipeline components, then your plugins need to implement the appropriate mixins and implement the relevant methods:
seppl.OutputProducer
: for plugins that generate data; thegenerates
method returns a list of classes that the plugin can output (typically only one).seppl.InputConsumer
: for plugins that consume data; theaccepts
method returns a list of classes that the plugin can process.
E.g., reader plugins would implement OutputProducer
, filters would
implement both, OutputProducer
and InputConsumer
, and writers only
InputConsumer
.
You can then run the check_compatibility
method against your pipeline,
i.e., a list of plugins, to see whether they are compatible. This method will
raise an exception if plugins are not implementing the mixins or if there is
no overlap between the generated/accepted classes.
For general purpose plugins, you can use AnyData
dummy class in the
accepts
/generates
methods, as this class is used by default as a
match all by the check_compatibility
method.
See the plugins in plugins_comp.py and some example pipeline checks done with them.
The following methods can be used for escaping/unescaping unicode characters to make them copyable into remote ssh sessions:
seppl.escape_args
seppl.unescape_args
The following command-line tools are available:
seppl-escape
: for escaping unicode charactersseppl-unescape
: for unescaping unicode characters