AS module, only run rules after AberrantSplicing_pipeline_FRASER_05_fit_autoencoder_FraseR_R

[wanqingshao]

Hi,

I'm using the DROP pipeline for about 100 samples. Module AS ran successfully all the way till AberrantSplicing_pipeline_FRASER_05_fit_autoencoder_FraseR_R, but was killed at this step due to insufficient memory allocation. Restarting snakemake with "snakemake aberrantSplicing -n" shows that the pipeline ignores all the previous output and starts from the beginning. Is there a way to run only AberrantSplicing_pipeline_FRASER_05_fit_autoencoder_FraseR_R and it's dependent rules?

Have tried -R and --until, but not getting the expected result. -R + --until shows that the pipeline will ignore all previous results, runs all the rules till FRASER_05, -R or --until shows missing input for rule readme_md. Touching the readme_md file causes the pipeline to run but it still ignores all the existing results and start from the beginning.

I'm using version 1.2.2 and everything is run on the HPC with slurm.

Would appreciate any suggestions on this. Thanks in advance!

-Wanqing

[nickhsmith]

Sorry for missing your post, in the future please use https://github.com/gagneurlab/drop/issues for specific problems. As for why snakemake -n seems to want to rerun your jobs that seems like something got deleted or updated. Could you please check to make sure that

1. the files as input for the rule AberrantSplicing_pipeline_FRASER_05_fit_autoencoder_FraseR_R exist?
   a. Output/processed_data/aberrant_splicing/datasets/savedObjects/{dataset}/hyper.done
2. Can you please share the snakemake -n -r output (as a text file)? This shows the reason why each rule is being run

[nickhsmith]

I dug into it a little bit. Because when FRASER is run it updates the internal files, snakemake is recognizing these as different input. As a result it says the input has changed and would like to start at the beginning. We can get around this, by telling snakemake that we don't care if the input changes (generally a bad idea!)

If you are 100% sure that everything is identical, and all you want to do is restart from the rule AberrantSplicing_pipeline_FRASER_05_fit_autoencoder_FraseR_R you can do the following command, where are your cluster or additional snakemake commands. This will say only care about the underlying code that is generating these rules, and snakemake will ignore the input changes it detects.

snakemake -R AberrantSplicing_pipeline_FRASER_05_fit_autoencoder_FraseR_R --rerun-triggers code  <OTHER RUN OPTIONS>

This should trigger the jobs you'd like to run.

[wanqingshao]

Thanks for the reply! i'll try the --return-trigers flag if this happens again. I have overwritten the previous FRASER with the following approach and it seems to work as well.

1. force rerun the first rule
   snakemake -R --until AberrantSplicing_pipeline_Counting_00_define_datasets_from_anno_R --cores 1
2. cleanup the metadata for 2 annotation files
   snakemake --cleanup-metadata processed_data/aberrant_splicing/annotations/fraser.tsv
snakemake --cleanup-metadata html/AberrantSplicing/annotations/fraser.html
3. and if it is still have trouble recognizing the previously generated file refresh the timestamp with
   snakemake aberrantSplicing --touch --cores 1
   -Wanqing