v0.1.8

Features and improvements

Breaking

• Fully remove --start-branch and --end-branch from workflows. (a03e7c8)

Major

• Speedup SiblingFileCollection removal. (be1317c)
• Add mtime patience feature for remote caches. (4a2dac9)
• Allow csv-style double-quote quoting in (Multi)CSVParameter. Close #123. (eb8536b)
• Refactor target inheritance structure. (44e0f0e)
• Disentangle copying and rendering of job input files. (fe378a6)
• Add ParquetTableFormatter to pyarrow contrib. (57ad33c)

Minor

• Unified bash variable usage. (22cf9fe)
• Support output placeholders in ForestMerge contrib task. (9dc0c53)
• Hook logger setup into get_logger(). (7c9b604)
• Yield DynamicRequirements in local wf if available. (0ac39c7)
• Show exit code in failed sandbox env commands. (0c62aa2)
• Allow returning keys in target collections. (99e47dc)
• Generalize forest merge inputs. (72d9f18)
• Allow sanbox and workflow objects to be setup lazily. (f8a4fef)
• Favor perf_counter() over time() where appropriate. (409f431)
• Allow skipping run method decorators. (b460f42)
• Consistent handling of executable in all job interfaces. (fee7202)
• Allow setting root task manually. (7463653, 8376034)
• Add inplace option to util.merge_dicts. (1cda443)
• Add JobInputFile. (1a3a6de)
• Use JobInputFile in arc, glite, slurm and lsf workflows. (e13d369)
• Add rendering error. (35ab3e0)
• Update docker images 39 and 310. (81c9afb)
• Allow control over merge forest caching in ForestMerge. (8188f8f)
• Refine workflow completion hook. (21cdb59)
• Filter unknown parameters after modify_param_values. (5798f34)
• Refine proxy setup in sandbox and workflow tasks. (7f0cb24)
• Safe map -> list conversion. (936c1d1)
• Properly log sandbox commands. (01ef2b9)
• Add split_csv option to all config getters. (e5c52cf)
• Make the coffea contrib formatter understand parquet and root. (ad32e33)

Fixes

• Support htcondor cluster submission. (313f31d)
• Support law3 executable. (f42a07b, 344361f)
• Fix namespaces in py3. (5018a76)
• Fix sandbox command concat. (69fb5b0)
• Make slurm job file executable. (e56a761)
• Take submodules into account for calculating repo checksums. (2e646b7)
• Fix improved target collection removal. (c4ec1b8)
• Refine command joining in sandboxes. (9b13f91)
• Fix base names of non-copied input files in remote job interfaces. (3b84293)
• Drop run_decorators feature from ForestMerge. (5c5ddc9)
• Proper float format. (90ab576)
• Fix workflow proxy instantiation. (14cd58e)
• Fix slurm executable. (006a6c6)
• Fix executable paths, adjust slurm peculiarity (#118). (3fed891)
• Fix ambiguity between submission-relatve paths and actual basenames. (8b848d7)
• Fix branch repr. (1ac0032)
• Fix input files in render variables. (f24e837)
• Fix LocalTarget.del for legacy 2.7 support. (a1457b3)
• Fix GFAL2 URL (7f1bcd9)
• Favor task id in TaskParameter serialization. (6a650ad)
• Add modify_param_args and modify_param_values to BaseTask for correct super mro's. (69a2c2b)
• Fix range_expand and range_join utils. (ecdc0a3)
• Fix branch setting in contrib ForestMerge task. (732da23)
• Fix cache cleanup. (11f6824)
• Fix remote target cache cache cleanup. (b10c4fb)
• Fix parquet writer version in pyarrow contrib. (d40b533)

v0.1.7

Features and Improvements

• Add VenvSandbox. (5321b50)
• Slurm workflows 🎉 (5b05715)
• Refactor job file handling in existing remote workflows (39cc5a7, 6dab50c, a400ff6, 9251c3a, 38bae13, 81ddfa6)
• Add awkward contrib package. (9f44202)
• Add dump() to uproot formatter. (8da8e6d)
• Add util.InsertableDict. (f88eeb6)
• Add iter_progress to base task. (e17fa05)
• Refactor human time parsing. (8625955)
• Add BytesParameter. (402201f)
• Optionally add file size to FileSystemTarget (#114, 5e6153d, 7399a3b)
• Update docker files to latest deps (22faf15, a06d888)
• Add default set of prefered cli parameters to BaseTask. (c8f4589)

Fixes

• Fix missing fetch_output support for target collections. (010f379)
• Fix env encoding in bash sandbox. (87203fd)
• Fix log flushing in sandboxes. (b20629f)

v0.1.6

Features and improvements

• Add python 3.10 docker image. (7a2b5ba, 4a6aea2)
• Update docker images. (03befc2)
• Identify finished jobs with missing task outputs in remote base workflow. (cce0ef0, 7494add)
• Use task logger in timeit decorator. (250aab4)

Fixes

• Fix race conditions during directory creation. (f51adad)
• Fix remote target path setter. (be56987)
• Fix stat call in interactive fetch_task_output. (ab5232c)

v0.1.5

Features and improvements

• Update docker images. (04d96f5)
• Minor update to hadd task in root contrib. (ebeb08a)
• Safe yaml loading (#111, @harrypuuter)
• Remove unused parameter job_ids (#110, @meliache)
• Make job manager and file factory classes configurable in remote workflows. (06d8902)
• Add additional hook to modify task parameters before instantiation. (09eee5e)
• Use new logger class across law. (be1f595)
• Add custom logger class with *_once() methods. (ec76a89)
• Add option to use unexpanded paths in file target repr. (e4bc244)
• Add 'return_all' to targets uri(). (06f3283)
• Rename get and put functions in contrib wlcg tools. (9c90b70)
• Implement mkdir_rec in gfal file interface. (c0fce22)
• Colorize messages of warning and error logs. (9e488ce)

Fixes

• Fix bash stderr pipes. (65f523a)
• Catch special cases in gfal file interface for gsiftp protocol. (bed428f)

(Minor) breaking changes

• Make FileSystemTarget.stat a method rather than a property. (e992b9c)

v0.1.4

Tasks

• Add law_run() method to all tasks. (a6e796e)
• Remove deprecated task.colored_repr(). (369b789)
• Move task logger to instance level and forward to scheduler. (a176211)

Targets

• Refactor target API, separate remote FS from underlying interface. (c675e54)
• Add NestedSiblingFileCollection. (a070308, 8e3ecd4)
• Speed up some target collection methods. (6a5b01c)
• Catch certain SRM messages in gfal file interface. (029625e)
• Refactor target hashes and uris of targets. (6d50c7e, 16268a3, 1f8b207, 38c399a)
• Make all repr helpers instance methods for better control. (9bf351e)
• Allow all targets to declare themselves external. (3c0967c)
• Avoid race condition in local directory creation. (1f28eb4, 6986368)
• Update usage of _child_args() in file targets. (d2ddd22)
• Replace mktemp patterns in FileSystemDirectoryTarget.child(). (6ec0eda)

Workflows

• Refactor branch_map handling of workflows. (7323b99, d8bd244)
• Fix initial sleep interval after remote workflow submission. (3f85825)
• Use MultiRangeParameter in --branches parameter of workflows. (1299841)

Jobs

• Fix exit codes in remote job script. (6e24fea)
• Log env variables in job script. (fa1533e)
• Cleanup iof remote job script. (a93ab91, f4cb963)

Misc.

• Add RangeParameter and MultiRangeParameter. (1c81abb)
• Escape separators in (Multi)CSVParameter and util.brace_expand. (34ce543)
• Fix parsing of CSV and Range parameters. (bcd989a)
• Improve encoding and decoloring of notifications. (cc32ae5, e8ab578)
• Colorize standard luigi logs. (0f8fd2b, e9699d0, b4195c1)
• Add scheduler destination to common completion hints. (f6be4ac)
• Remove deprecated contrib package access. (7163894)
• Remove CascadeMerge in contrib tasks in favor of ForestMerge. (2e50db0)
• Remove law db command. (3a6f5fb)
• Update six version requirement. (9b8371b)
• Update docker 3.8 and 3.9 images, update software. (c80fd7b, e37e92a)
• Use pyenv in docker files. (cc18cc2)
• Update to new sphinx. (8b728f4)
• Minor update to module style. (172c331)

v0.1.3

• Fix type of branches parameter in workflow. (a95626d)
• Add option to skip output removal on tasks. (4197826)
• Refactor branch map handling in workflow, allow control over caching. (7aff903)
• Make uri abstract for file targets, allow toggling fs schemes. (3d15203, d729ba0)
• Filter existing files in git repo bundling. (8469554)
• Update copyright. (65fe1ec)
• Add option to run task after output removal. (5f1f510)
• Add sort option to CSVParameter, simplify value checks. (63bc32b)
• Add choices option to CSVParameter. (8ec625f)

v0.1.2

This release is mainly intended to fix a bug introduced in 0.1.1.

Fixes

• Partially revert 7cd17b4 (abstract workflow properties). (b9168c0)

Improvements

• Update gfal2 setup in docker images. (bc9e5f4)
• Fix boost version in docker images. (bf7b71b)

v0.1.1

Updates and features

• Add logger property to all tasks. (590d569)
• Streamline interative task output. (0efd55b)
• Add MultiCSVParameter. (31d0fa8, 004801c)
• Add cleaner API to util.readable_popen. (d6204d6)
• Add helper to workflows to get chunks of branches. (0f48f42)
• Add range_expand and range_join functions to util. (258d93f)
• Add mechanism to require abstract workflow methods. (7cd17b4)
• Unify common methods across remote workflows, update repr_params. (a2eaec1)
• Add mechanism to further control excluded parameters in tasks. (5c1767b)
• Remove only_missing parameter from remote workflow, improve task skipping. (d76fa84)
• Update python and boost versions in docker files. (bc4243d)

Improvements and fixes

• Fix parsing of empty CSVParameter. (adbefb8)
• Fix tar gz compression flag in tar formatter, add lzma support. (d226ede, 0f719e7, 9c09b73)
• Improve interactive output fetching. (f74f26a)
• Fix remote job shuffling, add option to append retry jobs to the end. (1a35a99)
• Fix opening uncached remote targets, fix #105. (732950b)
• Fix interactive python shells in docker images. (6b6da61)

v0.1.0

• Fix sphinx version for docs. (0bc91c4)
• Refactor docker setup, add 39 image. (1d15b41)
• Mark empty values in cached_workflow_property. (09bc558)
• Add DotDict to util and use it in workflow requirements and outputs. (1428afe)
• Enable branch map resetting in workflows. (b161466, 42fc289)
• Allow progress publishing with precision > 1. (382a430)
• Optimize interactive task outputs. (76aa896)
• Fix file touching for remote targets with empty content. (0da0e17)
• Show job log files in polling loop on failures. (fdac7a6)
• Disable job dir cleanup by default for htcondor and lsf jobs. (80df7d7)
• Fix parsing of global command line arguments. (01c5543)
• Move Proxy command to task.proxy, use also in workflows for gathering task parameters. (a3f92f3)
• Add ProxyCommand helper for sandboxes. (398ee8d)
• Improve handling of cli args. (c709f4c)
• Add min_len and max_len options to CSVParameter. (eb8d03a)
• Support bundles in tf.keras and keras formatters. (c7f846c)

v0.0.41

Fixes

• Add requirements_docs.txt to releases. (10fdb6d)