Error with matching genotype files?

[scienception]

Description of the bug

Any idea why am I getting this error?

Command used and terminal output

`${DIR}/software/nextflow-23.04.2-all run ${DIR}/software/pgsc_calc/main.nf -profile singularity --input ${DIR}/data/03_PGS_UKBB/pgen_path.v3.csv --scorefile ${DIR}/data/02.5_PGS_scorefiles.parsed/${pheno}.betas.tsv --target_build GRCh37 --ref ${DIR}/software/pgsc_calc_ref.sqlar --outdir ${DIR}/data/03_PGS_UKBB/PGS.out_of_sample.v3/${pheno}/results/

Ouput:

N E X T F L O W  ~  version 23.04.2
Launching `/data/penetrance_at_scale/software/pgsc_calc/main.nf` [chaotic_mahavira] DSL2 - revision: 5d4bacde78


------------------------------------------------------
  pgscatalog/pgsc_calc v1.3.2
------------------------------------------------------
Core Nextflow options
  runName        : chaotic_mahavira
  containerEngine: singularity
  launchDir      : /data/penetrance_at_scale/scripts
  workDir        : /data/penetrance_at_scale/scripts/work
  projectDir     : /data/penetrance_at_scale/software/pgsc_calc
  userName       : -
  profile        : singularity
  configFiles    : /data/penetrance_at_scale/software/pgsc_calc/nextflow.config

Input/output options
  input          : /data/penetrance_at_scale/data/03_PGS_UKBB/pgen_path.v3.csv
  scorefile      : /data/penetrance_at_scale/data/02.5_PGS_scorefiles.parsed/height.betas.tsv
  pgs_id         : null
  pgp_id         : null
  trait_efo      : null
  target_build   : GRCh37
  ref            : /data/penetrance_at_scale/software/pgsc_calc_ref.sqlar
  genotypes_cache: null
  outdir         : /data/penetrance_at_scale/data/03_PGS_UKBB/PGS.out_of_sample.v3/height/results/

Institutional config options
  hostnames      : [:]

Max job request options
  max_cpus       : 2
  max_memory     : 16.GB

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use pgscatalog/pgsc_calc for your analysis please cite:

* The Polygenic Score Catalog
  https://doi.org/10.1038/s41588-021-00783-5

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/pgscatalog/pgsc_calc/blob/master/CITATIONS.md
------------------------------------------------------
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -

[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... [  0%] 0 of 1
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... [  0%] 0 of 1
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -

executor >  local (2)
[f6/2e3d7b] process > PGSCATALOG_PGSCALC:PGSCALC:... [  0%] 0 of 1
[45/3a95e0] process > PGSCATALOG_PGSCALC:PGSCALC:... [  0%] 0 of 1
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -

executor >  local (2)
[f6/2e3d7b] process > PGSCATALOG_PGSCALC:PGSCALC:... [100%] 1 of 1 ✔
[45/3a95e0] process > PGSCATALOG_PGSCALC:PGSCALC:... [  0%] 0 of 1
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[skipped  ] process > PGSCATALOG_PGSCALC:PGSCALC:... [100%] 8 of 8, stored: 8
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[-        ] process > PGSCATALOG_PGSCALC:PGSCALC:... -
[skipping] Stored process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR (7)
[skipping] Stored process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR (5)
[skipping] Stored process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR (4)
[skipping] Stored process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR (6)
[skipping] Stored process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR (1)
[skipping] Stored process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR (2)
[skipping] Stored process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR (8)
[skipping] Stored process > PGSCATALOG_PGSCALC:PGSCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR (3)
ERROR ~ Cannot compare sun.nio.fs.UnixPath with value '/data/penetrance_at_scale/scripts/work/genomes/ukbb/2/ukbb_2.pvar.zst' and java.util.ArrayList with value '[/data/penetrance_at_scale/scripts/work/genomes/ukbb/2/_001_ukb_imp_chr2_v2.pgen, /data/penetrance_at_scale/scripts/work/genomes/ukbb/2/ukbb_2.pgen]'

 -- Check '.nextflow.log' file for details
-[pgscatalog/pgsc_calc] Pipeline completed with errors-
ERROR ~ ERROR: No scores calculated!

 -- Check '.nextflow.log' file for details
Please remember to cite polygenic score authors if you publish with them!
Check the output report for citation details
WARN: Killing running tasks (1)

### Relevant files

_No response_

### System information

_No response_

[scienception]

I believe the error is in the work directory, is it possible to redirect the output to another directory?
I think it is using the old genome v2 files instead of the new v3 files.

My bash script looks something like this:

#!/bin/bash
#SBATCH --job-name getPGS_outofsample.v3
#SBATCH --output /data/penetrance_at_scale/data/03_PGS_UKBB/PGS.out_of_sample.v3/logs/pgsc_calc.height.out
#SBATCH --error  /data/penetrance_at_scale/data/03_PGS_UKBB/PGS.out_of_sample.v3/logs/pgsc_calc.height.err
#SBATCH --partition urblauna
#SBATCH --nodes 1
#SBATCH --cpus-per-task 8
#SBATCH --mem 50G
#SBATCH --time 6:00:00
#SBATCH --get-user-env=L
#SBATCH --export NONE

module load plink2/2.00a4.3

pheno="height"
DIR=/data/penetrance_at_scale/

${DIR}/software/nextflow-23.04.2-all run ${DIR}/software/pgsc_calc/main.nf -profile singularity --input ${DIR}/data/03_PGS_UKBB/pgen_path.v3.csv --scorefile ${DIR}/data/02.5_PGS_scorefiles.parsed/${pheno}.betas.tsv --target_build GRCh37 --ref ${DIR}/software/pgsc_calc_ref.sqlar --outdir ${DIR}/data/03_PGS_UKBB/PGS.out_of_sample.v3/${pheno}/results/

[scienception]

Update: I think -work-dir does the job, however I still have 2 questions:

• When is it necessary to overwrite the work directory? For every run? Why didn't it automatically overwrite it with the new genotype files?
• Do you have a full list of flags for pgsc_calc? Finding this one was quite difficult as it isn't in the documentation (maybe because it is a nextflow flag(?) but it is still useful).

[smlmbrt]

• When is it necessary to overwrite the work directory? For every run? Why didn't it automatically overwrite it with the new genotype files?

If you're using different sets of genomes you should clear or switch work directories - if the two sets of genomes had the same sampleset ID it probably got confused. The work directory helps restart the pipeline if it fails, or re-uses cached jobs if they don't change in a rerun; however, it's safest to only reuse the genotypes themselves that you store in a cache: https://pgsc-calc.readthedocs.io/en/latest/how-to/cache.html.

• Do you have a full list of flags for pgsc_calc? Finding this one was quite difficult as it isn't in the documentation (maybe because it is a nextflow flag(?) but it is still useful).

The full list of pgsc_calc parameters is here: https://pgsc-calc.readthedocs.io/en/latest/reference/params.html. For some things like work directory those are Nextflow-specific config commands: https://www.nextflow.io/docs/latest/config.html. We try to point to point where relevant in the documentation.

[scienception]

All right, so for jobs using same set of genomes I'm safe using the same work dir for all? They don't overwrite themselves right? It is computed once for one run, and then re-used by the subsequent ones?

[smlmbrt]

Yes, but safest is to make the cache for the genotypes too.

[scienception]

How would I do that? Like this?
--genotypes_cache /some_path/work/genomes/ukbb/

[smlmbrt]

--genotypes_cache /some_path/, don't use work or the sampleset in the name as it will get confusing.

[scienception]

Ok and this would be by dafault the same path where you run your pgsc_calc script right? same directory?

[smlmbrt]

Can be any directory, just as long as the work directory is separate (not nested or within)

[scienception]

If I don't specify any directory (I don't use that flag) it is saved here, right: ./work/genomes?

[smlmbrt]

Yes, but if you delete your work directory it will delete those pre-processing steps (which can take a lot of time depending on data size). They don't change with new scoring files or variant matching parameters, which is why we separate them and treat them as a cache.

[scienception]

Got it, thanks!