Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AberrantSplicing pipeline error #507

Open
akhilvelluva opened this issue Jan 10, 2024 · 16 comments
Open

AberrantSplicing pipeline error #507

akhilvelluva opened this issue Jan 10, 2024 · 16 comments

Comments

@akhilvelluva
Copy link

I was running the drop demo as like this

drop demo
snakemake aberrantExpression --cores 30

But it got error like this

Finished job 13.
16 of 69 steps (23%) done
Loading required package: rtracklayer
Loading required package: rtracklayer
Loading required package: rtracklayer
Loading required package: rtracklayer
Loading required package: rtracklayer
Wed Jan 10 14:53:11 2024: Count split reads for sample: HG00150
Wed Jan 10 14:53:11 2024: Count split reads for sample: HG00111
Wed Jan 10 14:53:11 2024: Count split reads for sample: HG00096
Wed Jan 10 14:53:12 2024: Count split reads for sample: HG00106
Wed Jan 10 14:53:12 2024: Count split reads for sample: HG00132
Loading required package: rtracklayer
Wed Jan 10 14:53:13 2024: Count split reads for sample: HG00126
Error in find_NCBI_assembly_ftp_dir(assembly_accession, assembly_name = assembly_name) :
unable to find FTP dir for assembly GCF_000001405.25 in
https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/405/
Calls: countSplitReads ... .form_assembly_report_url -> find_NCBI_assembly_ftp_dir
Execution halted
[Wed Jan 10 14:53:15 2024]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
jobid: 20
input: testRNA/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01_1_countRNA_splitReads_samplewise.R
output: testRNA/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser/sample_tmp/splitCounts/sample_HG00096.done
log: testRNA/drop_demo/.drop/tmp/AS/fraser/splitReads/HG00096.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /local/a992a/23555406/cluster_tmp/tmpv8rlvb_b, line 130:
Command 'set -euo pipefail; Rscript --vanilla testRNA/drop_demo/.snakemake/scripts/tmpso0ytzon.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status 1.
File "/local/a992a/23555406/cluster_tmp/tmpv8rlvb_b", line 130, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
File "/home/a992a/micromamba/envs/drop_env_133/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Loading required package: rtracklayer
Loading required package: rtracklayer
Wed Jan 10 14:53:17 2024: Count split reads for sample: HG00149
Wed Jan 10 14:53:17 2024: fraser_external, HG00150 no. splice junctions (split counts) = 472
Wed Jan 10 14:53:17 2024: fraser_external, HG00132 no. splice junctions (split counts) = 469
Wed Jan 10 14:53:17 2024: Count split reads for sample: HG00176
[Wed Jan 10 14:53:17 2024]
Finished job 42.
17 of 69 steps (25%) done
Wed Jan 10 14:53:17 2024: fraser_external, HG00111 no. splice junctions (split counts) = 468
[Wed Jan 10 14:53:17 2024]
Finished job 43.
18 of 69 steps (26%) done
[Wed Jan 10 14:53:18 2024]
Finished job 40.
19 of 69 steps (28%) done
Wed Jan 10 14:53:18 2024: fraser_external, HG00126 no. splice junctions (split counts) = 511
Wed Jan 10 14:53:18 2024: fraser_external, HG00106 no. splice junctions (split counts) = 500
[Wed Jan 10 14:53:18 2024]
Finished job 41.
20 of 69 steps (29%) done
[Wed Jan 10 14:53:18 2024]
Finished job 45.
21 of 69 steps (30%) done
Wed Jan 10 14:53:22 2024: fraser_external, HG00176 no. splice junctions (split counts) = 469
[Wed Jan 10 14:53:22 2024]
Finished job 46.
22 of 69 steps (32%) done
Wed Jan 10 14:53:23 2024: fraser_external, HG00149 no. splice junctions (split counts) = 523
[Wed Jan 10 14:53:23 2024]
Finished job 44.
23 of 69 steps (33%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

Rest pipelines like aberrantExpression and mae running with out any error

What will be the reason for this error ?
@vyepez88
Copy link
Collaborator

Hi, can you share the parameters of your config file? The global ones and the aberrantSplicing ones

@akhilvelluva
Copy link
Author

projectTitle: Detection of RNA Outlier Pipeline
htmlOutputPath: testRNA/drop_demo/Output/html
indexWithFolderName: true
root:testRNA/drop_demo/Output
sampleAnnotation: testRNA/drop_demo/Data/sample_annotation.tsv
geneAnnotation:
v29: testRNA/drop_demo/Data/gencode_annotation_trunc.gtf
genomeAssembly: hg19
genome:
ncbi: testRNA/drop_demo/Data/chr21_ncbi.fa
ucsc: testRNA/drop_demo/Data/chr21.fa
hpoFile: null
random_seed: true
exportCounts:
geneAnnotations:

  • v29
    excludeGroups:
  • mae
  • outrider_external
  • fraser_external
    tools:
    gatkCmd: gatk
    bcftoolsCmd: bcftools
    samtoolsCmd: samtools

aberrantSplicing:
run: true
groups:

  • fraser
  • fraser_external
    recount: true
    longRead: false
    keepNonStandardChrs: false
    filter: false
    minExpressionInOneSample: 20
    quantileMinExpression: 10
    quantileForFiltering: 0.95
    minDeltaPsi: 0.05
    implementation: PCA
    padjCutoff: 1
    maxTestedDimensionProportion: 6
    genesToTest: Data/genes_to_test.yaml
    FRASER_version: FRASER
    deltaPsiCutoff: 0.05

@vyepez88
Copy link
Collaborator

Hi, hard to say, could be that the ncbi webpage was not working on that day. Otherwise, can you change your genomeAssembly param from hg19 to GRCh37?

@vyepez88
Copy link
Collaborator

vyepez88 commented Apr 8, 2024

Hi, were you able to rerun the pipeline?

@geocarvalho
Copy link

Hi @vyepez88, just having the same issue here trying to run the drop demo data:

(drop_env) root@ip-10-34-87-44:/var/snap/amazon-ssm-agent/7993/drop_demo# drop --version
drop, version 1.4.0

(drop_env) root@ip-10-34-87-44:/var/snap/amazon-ssm-agent/7993/drop_demo# snakemake --cores 20 
...
[Sat Jun 22 01:38:44 2024]
Finished job 141.
124 of 196 steps (63%) done
Error in function (type, msg, asError = TRUE)  : server response timeout
Calls: countSplitReads ... .form_assembly_report_url -> find_NCBI_assembly_ftp_dir
Execution halted
[Sat Jun 22 01:40:21 2024]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 60
    input: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01_1_countRNA_splitReads_samplewise.R
    output: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser/sample_tmp/splitCounts/sample_HG00096.done
    log: /var/snap/amazon-ssm-agent/7993/drop_demo/.drop/tmp/AS/fraser/splitReads/HG00096.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmp8sk736zn, line 130:
Command 'set -euo pipefail;  Rscript --vanilla /var/snap/amazon-ssm-agent/7993/drop_demo/.snakemake/scripts/tmp5hni4k91.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status1.
  File "/tmp/tmp8sk736zn", line 130, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/root/miniconda3/envs/drop_env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-06-22T013646.848259.snakemake.log
  • When I use the hs37d5 instead I have issues with the hpo_genes.tsv.gz
Finished job 72.
83 of 116 steps (72%) done
Error in download.file(file, tmpFile, method = method, mode = "wb", quiet = !showProgress) :
  download from 'https://www.cmm.in.tum.de/public/paper/drop_analysis/resource/hpo_genes.tsv.gz' failed
Calls: add_HPO_cols -> fread -> download.file
In addition: Warning messages:
1: In download.file(file, tmpFile, method = method, mode = "wb", quiet = !showProgress) :
  downloaded length 0 != reported length 0
2: In download.file(file, tmpFile, method = method, mode = "wb", quiet = !showProgress) :
  URL 'https://www.cmm.in.tum.de/public/paper/drop_analysis/resource/hpo_genes.tsv.gz': Timeout of 60 seconds was reached
Execution halted
[Sat Jun 22 01:51:09 2024]
Error in rule AberrantExpression_pipeline_OUTRIDER_results_R:
    jobid: 41
    input: /var/snap/amazon-ssm-agent/7993/drop_demo/.drop/helpers/add_HPO_cols.R, /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_results/aberrant_expression/v29/outrider/outrid
er_external/ods.Rds, /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/preprocess/v29/gene_name_mapping_v29.tsv, /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_dat
a/aberrant_expression/v29/params/results/outrider_external_resultParams.csv, Scripts/AberrantExpression/pipeline/OUTRIDER/results.R
    output: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_results/aberrant_expression/v29/outrider/outrider_external/OUTRIDER_results.tsv, /var/snap/amazon-ssm-agent/7993/drop_
demo/Output/processed_results/aberrant_expression/v29/outrider/outrider_external/OUTRIDER_results_all.Rds
    log: /var/snap/amazon-ssm-agent/7993/drop_demo/.drop/tmp/AE/v29/outrider_external/OUTRIDER_results.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmp1jtrzy0p, line 89:
Command 'set -euo pipefail;  Rscript --vanilla /var/snap/amazon-ssm-agent/7993/drop_demo/.snakemake/scripts/tmpz5kcoke1.results.R' returned non-zero exit status 1.
  File "/tmp/tmp1jtrzy0p", line 89, in __rule_AberrantExpression_pipeline_OUTRIDER_results_R
  File "/root/miniconda3/envs/drop_env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Removing output files of failed job AberrantExpression_pipeline_OUTRIDER_results_R since they might be corrupted:
/var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_results/aberrant_expression/v29/outrider/outrider_external/OUTRIDER_results_all.Rds
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-06-22T014423.379072.snakemake.log

I can download it normally using wget

  • Using hg38 and GRCh38 I have the same issue as hg19:
[Sat Jun 22 02:00:40 2024]
Finished job 141.
124 of 196 steps (63%) done
Error in function (type, msg, asError = TRUE)  : server response timeout
Calls: countSplitReads ... .form_assembly_report_url -> find_NCBI_assembly_ftp_dir
Execution halted
[Sat Jun 22 02:01:37 2024]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 56
    input: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01_1_countRNA_splitReads_samplewise.R
    output: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser/sample_tmp/splitCounts/sample_HG00096.done
    log: /var/snap/amazon-ssm-agent/7993/drop_demo/.drop/tmp/AS/fraser/splitReads/HG00096.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmpc2nj6pfj, line 130:
Command 'set -euo pipefail;  Rscript --vanilla /var/snap/amazon-ssm-agent/7993/drop_demo/.snakemake/scripts/tmpa4p5pqb4.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status1.
  File "/tmp/tmpc2nj6pfj", line 130, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/root/miniconda3/envs/drop_env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-06-22T015727.017436.snakemake.log

---

[Sat Jun 22 02:06:38 2024]
Finished job 141.
107 of 196 steps (55%) done
Select jobs to execute...
Error in function (type, msg, asError = TRUE)  : server response timeout
Calls: countSplitReads ... .form_assembly_report_url -> find_NCBI_assembly_ftp_dir
Execution halted
Error in function (type, msg, asError = TRUE)  : server response timeout
Calls: countSplitReads ... .form_assembly_report_url -> find_NCBI_assembly_ftp_dir
Execution halted
[Sat Jun 22 02:07:36 2024]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 87
    input: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser_external/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01
_1_countRNA_splitReads_samplewise.R
    output: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser_external/sample_tmp/splitCounts/sample_HG00116.done
    log: /var/snap/amazon-ssm-agent/7993/drop_demo/.drop/tmp/AS/fraser_external/splitReads/HG00116.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmpiz_fqq2p, line 130:
Command 'set -euo pipefail;  Rscript --vanilla /var/snap/amazon-ssm-agent/7993/drop_demo/.snakemake/scripts/tmpgkbqqkbu.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status
1.
  File "/tmp/tmpiz_fqq2p", line 130, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/root/miniconda3/envs/drop_env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
[Sat Jun 22 02:07:36 2024]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 86
    input: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser_external/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01
_1_countRNA_splitReads_samplewise.R
    output: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser_external/sample_tmp/splitCounts/sample_HG00150.done
    log: /var/snap/amazon-ssm-agent/7993/drop_demo/.drop/tmp/AS/fraser_external/splitReads/HG00150.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmpiz_fqq2p, line 130:
Command 'set -euo pipefail;  Rscript --vanilla /var/snap/amazon-ssm-agent/7993/drop_demo/.snakemake/scripts/tmp3n9m9lo5.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status
1.
  File "/tmp/tmpiz_fqq2p", line 130, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/root/miniconda3/envs/drop_env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Error in function (type, msg, asError = TRUE)  : server response timeout
Calls: countSplitReads ... .form_assembly_report_url -> find_NCBI_assembly_ftp_dir
Execution halted
Error in function (type, msg, asError = TRUE)  : server response timeout
Calls: countSplitReads ... .form_assembly_report_url -> find_NCBI_assembly_ftp_dir
Execution halted
[Sat Jun 22 02:07:37 2024]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 80
    input: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser_external/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01
_1_countRNA_splitReads_samplewise.R
    output: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser_external/sample_tmp/splitCounts/sample_HG00126.done
    log: /var/snap/amazon-ssm-agent/7993/drop_demo/.drop/tmp/AS/fraser_external/splitReads/HG00126.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmpiz_fqq2p, line 130:
Command 'set -euo pipefail;  Rscript --vanilla /var/snap/amazon-ssm-agent/7993/drop_demo/.snakemake/scripts/tmpzl4h6442.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status
1.
  File "/tmp/tmpiz_fqq2p", line 130, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/root/miniconda3/envs/drop_env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
[Sat Jun 22 02:07:37 2024]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 83
    input: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser_external/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01
_1_countRNA_splitReads_samplewise.R
    output: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser_external/sample_tmp/splitCounts/sample_HG00111.done
    log: /var/snap/amazon-ssm-agent/7993/drop_demo/.drop/tmp/AS/fraser_external/splitReads/HG00111.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmpiz_fqq2p, line 130:
Command 'set -euo pipefail;  Rscript --vanilla /var/snap/amazon-ssm-agent/7993/drop_demo/.snakemake/scripts/tmphha1dal_.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status
1.
  File "/tmp/tmpiz_fqq2p", line 130, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/root/miniconda3/envs/drop_env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Error in function (type, msg, asError = TRUE)  : server response timeout
Calls: countSplitReads ... .form_assembly_report_url -> find_NCBI_assembly_ftp_dir
Execution halted
[Sat Jun 22 02:07:37 2024]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 88
    input: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser_external/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01
_1_countRNA_splitReads_samplewise.R
    output: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser_external/sample_tmp/splitCounts/sample_HG00149.done
    log: /var/snap/amazon-ssm-agent/7993/drop_demo/.drop/tmp/AS/fraser_external/splitReads/HG00149.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmpiz_fqq2p, line 130:
Command 'set -euo pipefail;  Rscript --vanilla /var/snap/amazon-ssm-agent/7993/drop_demo/.snakemake/scripts/tmpx66m3mrp.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status
1.
  File "/tmp/tmpiz_fqq2p", line 130, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/root/miniconda3/envs/drop_env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Error in function (type, msg, asError = TRUE)  : server response timeout
Calls: countSplitReads ... .form_assembly_report_url -> find_NCBI_assembly_ftp_dir
Execution halted
[Sat Jun 22 02:08:16 2024]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 60
    input: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01_1_countR
NA_splitReads_samplewise.R
    output: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser/sample_tmp/splitCounts/sample_HG00103.done
    log: /var/snap/amazon-ssm-agent/7993/drop_demo/.drop/tmp/AS/fraser/splitReads/HG00103.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmpiz_fqq2p, line 130:
Command 'set -euo pipefail;  Rscript --vanilla /var/snap/amazon-ssm-agent/7993/drop_demo/.snakemake/scripts/tmpwsj1lwkn.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status
1.
  File "/tmp/tmpiz_fqq2p", line 130, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/root/miniconda3/envs/drop_env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-06-22T020328.753741.snakemake.log

thank you!

@vyepez88
Copy link
Collaborator

Hi, we've seen this error form_assembly_report_url -> find_NCBI_assembly_ftp_dir when the NCBI portal was down. Can you simply rerun it?

@geocarvalho
Copy link

Hey @vyepez88, tried today and I'm having the same error:

[Tue Jun 25 00:22:37 2024]
Finished job 76.
140 of 196 steps (71%) done
Select jobs to execute...
Error in function (type, msg, asError = TRUE)  : server response timeout
Calls: countSplitReads ... .form_assembly_report_url -> find_NCBI_assembly_ftp_dir
Execution halted
[Tue Jun 25 00:22:56 2024]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 52
    input: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01_1_countRNA_splitReads_samplewise.R
    output: /var/snap/amazon-ssm-agent/7993/drop_demo/Output/processed_data/aberrant_splicing/datasets/cache/raw-local-fraser/sample_tmp/splitCounts/sample_HG00096.done
    log: /var/snap/amazon-ssm-agent/7993/drop_demo/.drop/tmp/AS/fraser/splitReads/HG00096.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmpqci163yq, line 130:
Command 'set -euo pipefail;  Rscript --vanilla /var/snap/amazon-ssm-agent/7993/drop_demo/.snakemake/scripts/tmp9z4sxxes.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status 1.
  File "/tmp/tmpqci163yq", line 130, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/root/miniconda3/envs/drop_env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-06-25T001958.851355.snakemake.log

I will try to run with my data and some reference on the server.

@vyepez88
Copy link
Collaborator

vyepez88 commented Jul 1, 2024

ok, let me know if it works

@geocarvalho
Copy link

geocarvalho commented Jul 4, 2024 •
edited
Loading

Hi @vyepez88, I was able to run the outrider module using my data but I am having issues with the Fraser2 module:

[Thu Jul  4 02:29:51 2024]
Finished job 1.
8 of 132 steps (6%) done
Select jobs to execute...
[Thu Jul  4 02:31:44 2024]
Error in rule AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R:
    jobid: 155
    input: /var/snap/amazon-ssm-agent/7993/output/processed_data/aberrant_splicing/datasets/cache/raw-local-fibroblast/fds.done, Scripts/AberrantSplicing/pipeline/Counting/01_1_countRNA_splitReads_samplewise.R
    output: /var/snap/amazon-ssm-agent/7993/output/processed_data/aberrant_splicing/datasets/cache/raw-local-fibroblast/sample_tmp/splitCounts/sample_SAMPLE756970-P-fibroblast.done
    log: /var/snap/amazon-ssm-agent/7993/nl-rna-seq_wf/drop/.drop/tmp/AS/fibroblast/splitReads/UDN756970-P-fibroblast.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmp5pxp_5nb, line 130:
Command 'set -euo pipefail;  Rscript --vanilla /var/snap/amazon-ssm-agent/7993/nl-rna-seq_wf/drop/.snakemake/scripts/tmps928jtxc.01_1_countRNA_splitReads_samplewise.R' returned non-zero exit status 1.
  File "/tmp/tmp5pxp_5nb", line 130, in __rule_AberrantSplicing_pipeline_Counting_01_1_countRNA_splitReads_samplewise_R
  File "/root/miniconda3/envs/drop_env/lib/python3.12/concurrent/futures/thread.py", line 58, in run
[Thu Jul  4 02:31:44 2024]

2024-07-04T022855.468201.snakemake.log

I tried to debug:

snakemake <- readRDS("/var/snap/amazon-ssm-agent/7993/nl-rna-seq_wf/drop/.drop/tmp/AS/fibroblast/splitReads/SAMPLE934476-P-fibroblast.Rds")
source("/var/snap/amazon-ssm-agent/7993/nl-rna-seq_wf/drop/Scripts/AberrantSplicing/pipeline/Counting/01_1_countRNA_splitReads_samplewise.R")

Load packages
Thu Jul  4 02:35:55 2024: Count split reads for sample: SAMPLE934476-P-fibroblast
Error in function (type, msg, asError = TRUE)  : server response timeout

Any idea?

Thank you!

@vyepez88
Copy link
Collaborator

vyepez88 commented Jul 4, 2024

Hi, server response timeout seems to be related to your computational setup. Did other samples get counted or did all of them fail?

@geocarvalho
Copy link

All of them failed, what do you mean by computational setup?

(base) root@ip-10-34-87-23:/var/snap/amazon-ssm-agent/7993# lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  32
  On-line CPU(s) list:   0-31
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
    CPU family:          6
    Model:               106
    Thread(s) per core:  2
    Core(s) per socket:  16
    Socket(s):           1
    Stepping:            6
    BogoMIPS:            5799.91
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtop
                         ology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand h
                         ypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed a
                         dx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves wbnoinvd ida arat avx512vbmi pku ospke avx512_vbmi2 gfni vaes vp
                         clmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid md_clear flush_l1d arch_capabilities
Virtualization features:
  Hypervisor vendor:     KVM
  Virtualization type:   full
Caches (sum of all):
  L1d:                   768 KiB (16 instances)
  L1i:                   512 KiB (16 instances)
  L2:                    20 MiB (16 instances)
  L3:                    54 MiB (1 instance)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-31
Vulnerabilities:
  Gather data sampling:  Unknown: Dependent on hypervisor status
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Mitigation; Clear CPU buffers; SMT Host state unknown
  Retbleed:              Not affected
  Spec rstack overflow:  Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI Syscall hardening, KVM SW loop
  Srbds:                 Not affected
  Tsx async abort:       Not affected

(base) root@ip-10-34-87-23:/var/snap/amazon-ssm-agent/7993# free -m
               total        used        free      shared  buff/cache   available
Mem:          126749         805      122896           5        3047      124719
Swap:              0           0           0

(base) root@ip-10-34-87-23:/var/snap/amazon-ssm-agent/7993# uname -mrs
Linux 6.5.0-1020-aws x86_64

(base) root@ip-10-34-87-23:/var/snap/amazon-ssm-agent/7993# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

@vyepez88
Copy link
Collaborator

vyepez88 commented Jul 4, 2024

were you able to run the aberrantSplicing module with the demo dataset?

@geocarvalho
Copy link

Nop, had the issues with the NCBI error from before.

@geocarvalho
Copy link

I think the reason is similar to the FRASER issue 50, I am using an AWS environment with similar connectivity security.

@geocarvalho
Copy link

geocarvalho commented Jul 16, 2024 •
edited
Loading

Hi @vyepez88, I'm able to have FRASER running outside of DROP. I was trying to create the exportCounts to have the GENE_COUNTS_FILE to reuse but I have this connective issue with the NCBI FTP as mentioned in DROP issue 236.
Is there a way to download the required files (I can download them through wget) and have them in the directory expected by the pipeline?

The issue is happening when I run:

sample_result <- countSplitReads(sampleID = sample_id,
                                 fds = fds,
                                 NcpuPerSample = snakemake@threads,
                                 recount = params$recount,
                                 keepNonStandardChromosomes = params$keepNonStandardChrs,
                                 genome = genome)
Tue Jul 16 17:31:19 2024: Count split reads for sample: SAMPLE-P-fibroblast
Error in download.file(url, destfile, method, quiet = TRUE) :
  cannot open URL 'ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/405/'
In addition: Warning message:
In download.file(url, destfile, method, quiet = TRUE) :
  URL 'ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/405/': Timeout of 60 seconds was reached

I'm not sure why I don't have this issue when I use FRASER outside from DROP running countRNAData, do you have any clue?

@geocarvalho
Copy link

Hi, I was able to run the drop demo without issues in another Linux environment without connectivity restrictions.
I will try to work with my team to change those restrictions given that DROP cannot be run offline.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@geocarvalho @vyepez88 @akhilvelluva