-
Notifications
You must be signed in to change notification settings - Fork 4
/
CHANGELOG
232 lines (202 loc) · 14.7 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
Changelog started with version 1.2.0
## [Unreleased]
## [2.0.8] - 2022-04-25
### Added
- [CMD] Added flag 'ignoreParamHashInResume' that allows to disable parameter hash validate in resume mode
### Fixed
- [PLUGIN] Filename in path was not replaced completely during mount point detection when using docker
- [BASH] Excessive memory usage when a command with large stdout/stderr output was executed
- [SGE] Small sge job ids were not matched by the regex pattern
- [] Verify return value names of process tables during XML parsing if disableExistenceCheck is not true (TODO: update documentation)
- [] Status of slave tasks was set to finished prematurely by status update events (-> depending tasks might not be scheduled)
- [] Stdout and stderr files are now not monitored if a task was loaded from a resume file
- [] Attribute 'shebang' of executor was not set correctly
- [CMD] Redirection of stdin was not working in all cases
- [CMD] Tasks with two or more global dependencies on process blocks of type 'processInput' were never scheduled
- [CMD] Fixed rare null pointer exception during startup
- [CMD] Sometimes tasks were prematurely marked as finished and hence not all subtasks were scheduled
- [CMD] Fixed non-reachable condition in error checker
- [SLURM] Information on hold jobs was not obtained by sacct
- [CMD] Fixes in the module create helper script regarding missing newlines, double range checks and wrong XSD parameter naming
- [SLURM] Jobs on hold were not detected when re-attach was used
### Changed
- [] Do not exit if process table contains only a header line (warn instead)
- [CMD] Allow empty values ('\t\t') in process table
- [CMD] Add 'slots' attribute in sge executor only if a value greater than 1 is set
- [PLUGIN] Ensure that podman/docker commands are killed on SIGTERM. Disable with 'disableTerminateWrapper' attribute of docker wrapper.
- [CMD] Updated external libraries to newest available version
- [DOCKER] Mount Watchdog's tmp directory automatically if it is a symbolic link
## [2.0.7] - 2020-05-25
### Fixed
- [GUI] Return values of modules were not loaded when typing '[/(/{' in parameter input field
### Changed
- [PLUGIN] Added 'loadModuleSpecificImage' attribute to docker wrapper to support loading of module specific images
- [PLUGIN] Added '--rm' flag to delete docker/podman containers after an task is completed
## [2.0.6] - 2020-04-24
### Fixed
- [PLUGIN] Watchdog's base directory is only added once when Docker virtualizer is used
- [PLUGIN] Singularity expects different parameters than docker and podman.
- [GUI] Improved layout of process blocks properties
- [GUI] Added missing constant suggestions for conda and docker wrapper
- [GUI] Attribute 'image' of docker wrapper was not loaded correctly from a workflow file
### Changed
- [PLUGIN] Streams instead of constants are now also used to detect path that are automatically mounted when Docker virtualizer is used
- [] Prepared example workflows to work with docker and conda
- [] Updated docker and conda documentation and tutorials
- [PLUGIN] Attribute 'path2conda' must now specify an absolute path to the conda binary
- [GUI] Replaced cancel/ok with no/yes in confirmation dialogs
## [2.0.5] - 2020-04-17
### Added
- [PLUGIN] Used XML plugin system to add support for execution wrappers (package managers / virtualizer)
- [PLUGIN] Implemented support for Conda as package manager execution wrapper (<conda>)
- [PLUGIN] Implemented support for Docker as virtualizer execution wrapper (<docker>)
- [JAR] Added Maven file to build Jar files and download dependencies
- [DOCU] Added replicate analysis and WorkflowDesigner tutorials
- [] Added helper script (formatCondaYaml.py) that formats Conda enironment definition files (*.conda.yml)
- [] Implemented helper script (installCondaYaml.sh) to pre-initialize all conda environments
- [] Added helper script (watchdogLauncher.sh) to make Watchdog's JARs and helper scripts better accessible within the watchdog-wms Conda package
### Fixed
- [CMD] Fixed null pointer exception when no workflow XML file was passed to Watchdog
- [CMD] Don't wait for version query file when task failed.
- [BASH] Added 'lib/' folder to search folder for the javaFX SDK
- [XSD] Numeric range tests did not work as type casting is required
### Changed
- [JAR] Jars are now build with Maven and do not include dependencies
- [README] The readme is now formated using Markdown
- [GUI] JavaFX SDK obtained by Maven is used if not otherwise specified
- [] Templates for writing return variables to a file are added to the bash script created by createNewModule.sh
## [2.0.4] - 2019-11-27
### Fixed
- [BASH] Removed -k flag to make download scripts compatible to busyBox gzip used in docker image
- [GUI] Updated the documentation on how to launch the GUI from the docker image
## [2.0.3] - 2019-11-26
### Fixed
- [BASH] Improve compatibility with macOS
- [GUI] Load XML parser plugins after saving of settings file
### Changed
- [BASH] Download attempts are made now with curl and wget
## [2.0.2] - 2019-11-21
### Fixed
- [CMD] Ignore called command during version detection of Bash-based modules
- [BASH] Corrected wrong installation path for moduleMaker in download script
- [BASH] Modules were listed multiple times during dependency test
- [JAR] Fixed null pointer exception when output file was only a file name during report generation
### Changed
- [EXAMPLE] Adapted example workflow2 to support new versions of featureCounts and DESeq2
## [2.0.1] - 2019-11-19
### Added
- [JAR] Added methodTemplateGenerator.jar that can be used to create templates for module documentation
- [JAR] Added workflowValidator.jar that validates the structure of workflows
- [GUI] Added indentation to workflows written by workflow designer
- [JAR] Allow to add version (%SOFTWARE_VERSION%) and other task parameters (%PARAM_NAME%) in automated generated report
- [JAR] Allow modifiers to be applied on replaced variables in reports (%PARAM_NAME§M%); (N=name only; V=restrict to version number)
- [MOD] Added script to download all community modules (modules/downloadCommunityModules.sh)
- [] Introduced constant ${WF_PARENT} that points to the parent folder of the XML workflow file
- [] Ensure that Watchdog module commands are executable and if not offer user to make it executable
- [JAR] Added url section in detailed view of the module reference book
### Changed
- [GUI] Added indentation to XML files written by workflow designer
- [JAR] Renamed methodTemplateGenerator.jar to reportGenerator.jar
- [JAR] Renamed libraryGenerator.jar to refBookGenerator.jar
- [BASH] Added 'permission denied' to default keywords of bash error checker
- [CMD] Output messages of error checkers on standard output
- [CMD] Reduced mail wait time to 5s
### Fixed
- [JAR] Warnings of missing XSD schema removed from all jars
- [CMD] Software versioning for tasks executed in slave mode was not working
- [CMD] Called command in modules created by the helper script was always ${MODULE_NAME}.sh
- [DOCU] Focus on input field was lost in the module reference book while text search was used
- [CMD] Capture exit code of main command if software versions are used
- [JAR] Fixed null pointer exception when output folder was only a folder name during reference book generation
- [CMD] Kill all children of a process task when local executor is used and Watchdog is terminated
- [JAR] Escaping of HTML entities in the module reference book
- [BASH] Prevent unintended evaluation of globbing patterns in argument parser
- [BASH] Last parameter was missing in called command output (introduced in 2.0.0 with module versions)
## [2.0.0] - 2019-08-12
### Added
- [BASH] Implemented before script that will enforce the requested memory using ulimit (for SGE or Slurm executor).
- [CMD] Workflows can be resumed now; only tasks that were not executed successfully or have changed parameters will be executed
- [CMD] New parameter -tmpFolder, which can be used to set a custom temporary caching and working directory
- [CMD] Detach mode can be requested via STRG+C or by using the webserver
- [CMD] Implemented documentation template extractor plugins for bash and python modules
- [CMD] Implemented detach/attach mode that can be used to stop Watchdog and restart it later while tasks are running on external executors
- [DOCU] PDF version of the documentation is provided in addition to the HTML version
- [GUI] Added -tmpFolder folder parameter to Workflow designer
- [JAR] Added libraryGenerator.jar that generates a module reference book based on a set of modules
- [JAR] Added methodTemplateGenerator.jar that generates XML documentation template files based on XSD modules
- [JAR] Added moduleValidator.jar that validates the structure of modules
- [JAR] Added docuTemplateExtractor.jar that can be used to create templates for module documentation
- [PLUGIN] Implemented binary-based executor (<sge>) to support Sun Grid Engine (tested with v6.1); SGE control binaries must be accessible via the $PATH variable
- [XSD] Custom before and after command scripts can be added to actual command calls using the 'beforeScripts' and 'afterScripts' attributes of executor tags
- [XSD] Modules can now be documented (including basic module information, description and parameters) using a simple XML format
- [XSD] Version of third-party software used in modules can be queried using the 'versionQueryParameter' attribute of tasks
### Changed
- [] Watchdog releases require now Java 11 (and JavaFX SDK for workflow designer)
- [MOD] Modules are no longer shipped together with Watchdog. They can be obtained from https://github.com/watchdog-wms/watchdog-wms-modules
- [PLUGIN] Executor <cluster> is now used for DRMAA cluster implementations; custom attributes (memory, slots, queue) were removed
- [PLUGIN] Moved 'shebang' attribute from environment to executor
### Fixed
- [] Fixed small memory leak caused by not closed ScanResult objects
- [] Do not export environment variables that contain a '.' or '-' in the name
- [BASH] Corrected wrong array access that caused an error in new bash versions
- [BASH] Last parameter of called command was not shown
- [CMD] Removed unnecessary nesting of pre-defined TMP constant
- [CMD] STRG+C signal is now caught by watchdog.sh as otherwise SIGINT is forwarded to all processes forked by the java process
- [CMD] Fixed very rare null pointer exception that could occurr when a process folder is used (e.g. network failure during usage of network storage I/O call after existence checks)
- [EXAMPLE] Fixed wrong dependency in example workflow 1 for task 5
- [GUI] In some cases not all task parameters were loaded from a XML file
- [GUI] When using custom module folders the trailing '/' was missing when the workflow was written
- [PLUGIN] Slurm executor plugin was not using the <cluster> tag
## [1.2.7] - 2018-08-30
### Added
- [] New task action links, part of the summary mail, can control multiple failed subtasks at once
- [XSD] Added concept of module versions that allows the use of different XSD definition versions (of the same module without copy&paste)
- [DOCU] Added documentation on module versions (5.6)
- [MOD] Added version 2 of the featureCount module which supports a newer featureCount release
- [MOD] New parameter 'shortenReads' for cutadapt module that shortens reads to a maximal length after trimming
### Fixed
- [] Constant replacement of ${TMP} failed when used more than once per line
- [] Wrong behavior when attribute 'disableExistenceCheck' attribute was set on process table
- [] Variables were not correctly replaced when a space separated the number of suffixes and the separator
- [] Wrong behavior when a process table file was empty or did not contain the requested column
- [CMD] Prevent sending of mails when workflow execution is ended with STRG+C
- [GUI] Instant change of module library without restart when settings are updated
### Internal
- [PLUGIN] replaced outdated FastClasspathScanner class with its new version named ClassGraph (closed #1)
## [1.2.6] - 2018-01-22
### Added
- [] Added attribute 'pattern' to restrict <copyFolder>/<deleteFolder> actions to files that match that pattern
- [] Added attribute 'keep4Slave' to <depends> in order to make the slave wait with task execution until all sub-tasks with that ID are finished on it
- [CMD] Added flag -useEnvBase that overrides the watchdogBase attribute with the content of the WATCHDOG_HOME environment variable
- [DOCU] Added documentation on how Docker can be used to run Watchdog and how a Docker image can be used within modules
- [MOD] Added Docker example module and an example workflow that use a bowtie2 Docker image
### Fixed
- [GUI] Attribute 'deleteSource' of <copyFile>/<deleteFolder> was loaded incorrectly
- [XSD] Pattern for URI's of task actions was to strict when process block variables should be used in it
- [XSD] Constants are now allowed to contain numbers
## [1.2.5] - 2017-12-08
### Added
- [PLUGIN] Implemented plugin system for XML parsing that allows to extend functionally of Watchdog by community
- [PLUGIN] Applied concept of plugin system to process blocks and executors
- [VFS] Modified task actions to support all virtual file systems that implement the FileProvider interface of the commons vfs project (http://commons.apache.org/proper/commons-vfs/)
- [VFS] Added default support for URI schemes: file, http, https, ftp, ftps, sftp and ram
- [VFS] Implemented interface VFSRegister that can be used to add additional virtual file systems easily
- [] Implemented additional executor to support Slurm Workload Manager (https://slurm.schedmd.com/)
- [GUI] Saving (and loading) of incomplete workflows, not valid according to the XSD format, is supported experimentally
- [XML] Added attribute 'saveResourceUsage' to <stream> that saves used resources to <STDOUT_FILE>.res
- [XSD] Added attribute 'separateFormat' to abstract task that can be used to set the default separator string between multiple occurrences of the same parameter
- [CMD] Added flag -forceLoading (tries to load a workflow even if it is not valid according to the XSD format)
- [CMD] Added flag -disableCheckpoint (ignores all checkpoints that are defined in the workflow)
- [CMD] Added flag -disableMails (does not send mails even if a mail address is set in the workflow)
- [] Added Watchdog to the package manager bioconda; package name: watchdog-wms (https://bioconda.github.io/)
- [] Sources of watchdog are now hosted on github (https://github.com/klugem/watchdog)
### Fixed
- [GUI] workflow designer crashed on windows caused by not escaped backslash in regex
- [CMD] command line version of watchdog crashed when openjdk8 with missing javafx was used
- [] remote executor: not reachable hosts are removed from host list
## [1.1.0] - 2017-08-24
### Added
- [GUI] Implemented workflow designer