我正在尝试在 SLURM 集群上的管道中使用包装器,其中计算节点无法访问互联网。
我首先使用
--conda-create-envs-only
运行管道,然后更改 wrapper:
指令以指向包含 environment.yaml
文件的本地文件夹。
作业在没有特定错误的情况下失败。具有相同配置但没有包装器的测试规则有效。如果我切换回 wrapper:
指令以在线查看,则带有包装器的规则在登录节点上工作正常。
我在跑步:
snakemake --profile myprofile --cores 40 --use-conda
示例规则:
# Run FastQC on the fastq.gz files
rule fastqc_fastq_gz:
input:
input_dir + "{sample}_{read}_001.fastq.gz",
output:
html = output_dir + "fastqc/{sample}_{read}_fastqc.html",
zip = output_dir + "fastqc/{sample}_{read}_fastqc.zip",
params:
extra = "--quiet",
log:
output_dir + "logs/fastqc/{sample}_{read}.log",
threads: 1,
resources:
mem_mb = 1024,
wrapper:
# "file:///path/envs/v1.31.0/bio/fastqc/"
"v1.31.0/bio/fastqc"
我也尝试过使用更多资源,同样的行为。
我的简介:
cluster:
mkdir -p logs/{rule} &&
sbatch
--partition={resources.partition}
--qos={resources.qos}
--cpus-per-task={threads}
--mem={resources.mem_mb}
--job-name=smk-{rule}-{wildcards}
--output=logs/{rule}/{rule}-{wildcards}-%j.out
--error=logs/{rule}/{rule}-{wildcards}-.%j.err
--account=<account>
--time={resources.time}
--parsable
default-resources:
- partition=<partition>
- qos=sbatch
- mem_mb="490G"
- tmpdir="/path/to/temp/"
- time="0-10:00:00"
max-jobs-per-second: 10
max-status-checks-per-second: 1
latency-wait: 60
jobs: 16
keep-going: True
rerun-incomplete: True
printshellcmds: True
scheduler: greedy
use-conda: True
cluster-status: status-sacct.sh
投稿记录如下:
Error in rule fastqc_fastq_gz:
jobid: 38
input: /path/raw/S9_S9_R2_001.fastq.gz
output: /path/pipeline_out/fastqc/S9_S9_R2_fastqc.html, /path/pipeline_out/fastqc/S9_S9_R2_fastqc.zip
log: /path/pipeline_out/logs/fastqc/S9_S9_R2.log (check log file(s) for error details)
conda-env: /path/.snakemake/conda/a116f377bbaddedd93b228a3d4f74b1d_
cluster_jobid: 1491196
Error executing rule fastqc_fastq_gz on cluster (jobid: 38, external: 1491196, jobscript: /path/.snakemake/tmp.l0raomfw/snakejob.fastqc_fastq_gz.38.sh). For error details see the cluster log and the log files of the involved rule(s).
tmp 文件不存在。 工作日志简单阅读:
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 80
Rules claiming more threads will be scaled down.
Provided resources: mem_mb=1000, mem_mib=954, disk_mb=7876, disk_mib=7512
Select jobs to execute...
[Date Time]
rule fastqc_fastq_gz:
input: /path/raw/S9_S9_R2_001.fastq.gz
output: /path/pipeline_out/fastqc/S9_S9_R2_fastqc.html, /path/pipeline_out/fastqc/S9_S9_R2_fastqc.zip
log: /path/pipeline_out/logs/fastqc/S9_S9_R2.log
jobid: 0
reason: Missing output files: /path/pipeline_out/fastqc/S9_S9_R2_fastqc.zip, /path/pipeline_out/fastqc/S9_S9_R2_fastqc.html
wildcards: sample=S9_S9, read=R2
threads: 2
resources: mem_mb=1000, mem_mib=954, disk_mb=7876, disk_mib=7512, tmpdir=/path/tmp/snakemake, partition=el7taskp, qos=sbatch, time=0-40:00:00
[Date Time]
Error in rule fastqc_fastq_gz:
jobid: 0
input: /path/raw/230319/S9_S9_R2_001.fastq.gz
output: /path/pipeline_out/fastqc/S9_S9_R2_fastqc.html, /path/pipeline_out/fastqc/S9_S9_R2_fastqc.zip
log: /path/pipeline_out/logs/fastqc/S9_S9_R2.log (check log file(s) for error details)
conda-env: /path/.snakemake/conda/a116f377bbaddedd93b228a3d4f74b1d_
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
FastQC 日志为空。