Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove unnecessary intermediate outputs #51

Closed
mfoll opened this issue Oct 12, 2015 · 3 comments
Closed

Remove unnecessary intermediate outputs #51

mfoll opened this issue Oct 12, 2015 · 3 comments
Assignees
Milestone

Comments

@mfoll
Copy link
Member

mfoll commented Oct 12, 2015

The only important outputs are: the final VCF and PDF files. All other intermediate files should only be present in nextflow work directory. This will imply:

  • removing storeDir directives in split_bed, samtools_mpileup and mpileup2table processes
  • deleting the intermediate VCF outputs at the end of collect_vcf_result (see how we delete the empty pdf files).
  • renaming the VCF output dir as PDF
  • deleting the PDF folder and the all_variants.vcf file as the first step in the pipeline

It will has the advantage that the pipeline will be restarted from the beginning by default. We should note in the manual that the output is erased when it exists already. Users will still have the option to resume an interrupted pipeline by adding the -resume option to the command line (add that to the manual too).

@mfoll mfoll added this to the v0.3 milestone Oct 12, 2015
@tdelhomme tdelhomme modified the milestones: v0.2, v0.3 Oct 12, 2015
@mfoll
Copy link
Member Author

mfoll commented Oct 19, 2015

See nextflow-io/nextflow#81

@mfoll
Copy link
Member Author

mfoll commented Dec 10, 2015

Now that we use publishDir, my 4th point above is no longer necessary as the default behaviour is (https://github.com/nextflow-io/nextflow/blob/master/docs/process.rst#publishdir):

any existing file in the specified folder will be overridden (default: true during normal pipeline execution and false when pipeline execution is resumed).

@tdelhomme
Copy link
Member

Yes it's true.
Also I modified the script as you mentionned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants