Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Docker setup not executed when specifying only a nextflow runner. #731

Open
2 tasks done
DriesSchaumont opened this issue Jun 28, 2024 · 1 comment
Open
2 tasks done
Labels
bug Something isn't working

Comments

@DriesSchaumont
Copy link
Contributor

What happened?

When using viash ns build --setup cb, the docker build is not being executed for a component that has a docker engine specified and only the nextflow runner (executable runner is missing).

Steps to reproduce

First make sure that the docker image does not exist

docker rmi dummycomponent/myscript:latest

Then run the following script to reproduce:

mkdir -p /tmp/test_bug
cd /tmp/test_bug
cat > _viash.yaml << 'HERE'
viash_version: 0.9.0-RC6

source: src
target: target
HERE

mkdir -p src/dummycomponent

cat > src/dummycomponent/config.vsh.yaml << 'HERE'

name: myscript
namespace: "dummycomponent"
arguments:
  - name: "--output"
    type: file
    direction: output
    default: "${id}"
resources:
  - type: python_script
    path: script.py
    text: |
      print("OK!", flush=True)
engines:
  - type: docker
    image: python:latest
runners:
  - type: nextflow
HERE

viash ns build --setup cb
nextflow run . -main-script ./target/nextflow/dummycomponent/myscript/main.nf \
-profile docker \
-c ./target/nextflow/dummycomponent/myscript/nextflow.config \
--publish_dir .

Expected behavior

The docker image should have been build by viash and should be available to nextflow.

Relevant log output

 N E X T F L O W   ~  version 24.04.2

Launching `./target/nextflow/dummycomponent/myscript/main.nf` [silly_goldberg] DSL2 - revision: 5a0941ba36

executor >  local (1)
[51/cb3c78] process > myscript:processWf:myscript_process (run)        [  0%] 0 of 1
[-        ] process > myscript:publishStatesSimpleWf:publishStatesProc -
ERROR ~ Error executing process > 'myscript:processWf:myscript_process (run)'

Caused by:
  Process `myscript:processWf:myscript_process (run)` terminated with an error exit status (125)

Command executed:

  # meta exports
  export VIASH_META_RESOURCES_DIR=".viash_meta_resources"
  export VIASH_META_TEMP_DIR="/tmp"
  export VIASH_META_NAME="myscript"
  # export VIASH_META_EXECUTABLE="$VIASH_META_RESOURCES_DIR/$VIASH_META_NAME"
  export VIASH_META_CONFIG="$VIASH_META_RESOURCES_DIR/.config.vsh.yaml"
  export VIASH_META_CPUS=1

  if [ ! -z ${VIASH_META_MEMORY_B+x} ]; then
    export VIASH_META_MEMORY_KB=$(( ($VIASH_META_MEMORY_B+999) / 1000 ))
    export VIASH_META_MEMORY_MB=$(( ($VIASH_META_MEMORY_KB+999) / 1000 ))
    export VIASH_META_MEMORY_GB=$(( ($VIASH_META_MEMORY_MB+999) / 1000 ))
    export VIASH_META_MEMORY_TB=$(( ($VIASH_META_MEMORY_GB+999) / 1000 ))
    export VIASH_META_MEMORY_PB=$(( ($VIASH_META_MEMORY_TB+999) / 1000 ))
    export VIASH_META_MEMORY_KIB=$(( ($VIASH_META_MEMORY_B+1023) / 1024 ))
    export VIASH_META_MEMORY_MIB=$(( ($VIASH_META_MEMORY_KIB+1023) / 1024 ))
    export VIASH_META_MEMORY_GIB=$(( ($VIASH_META_MEMORY_MIB+1023) / 1024 ))
    export VIASH_META_MEMORY_TIB=$(( ($VIASH_META_MEMORY_GIB+1023) / 1024 ))
    export VIASH_META_MEMORY_PIB=$(( ($VIASH_META_MEMORY_TIB+1023) / 1024 ))
  fi

  # meta synonyms
  export VIASH_TEMP="$VIASH_META_TEMP_DIR"
  export TEMP_DIR="$VIASH_META_TEMP_DIR"

  # create output dirs if need be
  function mkdir_parent {
    for file in "$@"; do
      mkdir -p "$(dirname "$file")"
    done
  }
  mkdir_parent "${id}"

  # argument exports
  export VIASH_PAR_OUTPUT="${id}"

  # process script
  set -e
  tempscript=".viash_script.sh"
  cat > "$tempscript" << VIASHMAIN
  ## VIASH START
  # The following code has been auto-generated by Viash.
  par = {
    'output': $( if [ ! -z ${VIASH_PAR_OUTPUT+x} ]; then echo "r'${VIASH_PAR_OUTPUT//\'/\'\"\'\"r\'}'"; else echo None; fi )
  }
  meta = {
    'name': $( if [ ! -z ${VIASH_META_NAME+x} ]; then echo "r'${VIASH_META_NAME//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'functionality_name': $( if [ ! -z ${VIASH_META_FUNCTIONALITY_NAME+x} ]; then echo "r'${VIASH_META_FUNCTIONALITY_NAME//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'resources_dir': $( if [ ! -z ${VIASH_META_RESOURCES_DIR+x} ]; then echo "r'${VIASH_META_RESOURCES_DIR//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'executable': $( if [ ! -z ${VIASH_META_EXECUTABLE+x} ]; then echo "r'${VIASH_META_EXECUTABLE//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'config': $( if [ ! -z ${VIASH_META_CONFIG+x} ]; then echo "r'${VIASH_META_CONFIG//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'temp_dir': $( if [ ! -z ${VIASH_META_TEMP_DIR+x} ]; then echo "r'${VIASH_META_TEMP_DIR//\'/\'\"\'\"r\'}'"; else echo None; fi ),
    'cpus': $( if [ ! -z ${VIASH_META_CPUS+x} ]; then echo "int(r'${VIASH_META_CPUS//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_b': $( if [ ! -z ${VIASH_META_MEMORY_B+x} ]; then echo "int(r'${VIASH_META_MEMORY_B//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_kb': $( if [ ! -z ${VIASH_META_MEMORY_KB+x} ]; then echo "int(r'${VIASH_META_MEMORY_KB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_mb': $( if [ ! -z ${VIASH_META_MEMORY_MB+x} ]; then echo "int(r'${VIASH_META_MEMORY_MB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_gb': $( if [ ! -z ${VIASH_META_MEMORY_GB+x} ]; then echo "int(r'${VIASH_META_MEMORY_GB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_tb': $( if [ ! -z ${VIASH_META_MEMORY_TB+x} ]; then echo "int(r'${VIASH_META_MEMORY_TB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_pb': $( if [ ! -z ${VIASH_META_MEMORY_PB+x} ]; then echo "int(r'${VIASH_META_MEMORY_PB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_kib': $( if [ ! -z ${VIASH_META_MEMORY_KIB+x} ]; then echo "int(r'${VIASH_META_MEMORY_KIB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_mib': $( if [ ! -z ${VIASH_META_MEMORY_MIB+x} ]; then echo "int(r'${VIASH_META_MEMORY_MIB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_gib': $( if [ ! -z ${VIASH_META_MEMORY_GIB+x} ]; then echo "int(r'${VIASH_META_MEMORY_GIB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_tib': $( if [ ! -z ${VIASH_META_MEMORY_TIB+x} ]; then echo "int(r'${VIASH_META_MEMORY_TIB//\'/\'\"\'\"r\'}')"; else echo None; fi ),
    'memory_pib': $( if [ ! -z ${VIASH_META_MEMORY_PIB+x} ]; then echo "int(r'${VIASH_META_MEMORY_PIB//\'/\'\"\'\"r\'}')"; else echo None; fi )
  }
  dep = {

  }

  ## VIASH END
  print("OK!", flush=True)
  VIASHMAIN
  python -B "$tempscript"

Command exit status:
  125

Command output:
  (empty)

Command error:
  Unable to find image 'dummycomponent/myscript:latest' locally
  docker: Error response from daemon: pull access denied for dummycomponent/myscript, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
  See 'docker run --help'.

Work dir:
  /tmp/test_bug/work/51/cb3c787af167a9c291cf961fa3710d

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

Version

  • OS: Ubuntu 24.04
  • openjdk 21.0.3 2024-04-16
  • Nextflow 24.04.2
  • Viash 0.9.0-RC6

Possible solution

No response

Confirmation

  • I have searched the existing issues to make sure this is not a duplicate.
  • I have provided clear and concise information about the bug.

Additional context

No response

@DriesSchaumont DriesSchaumont added the bug Something isn't working label Jun 28, 2024
@Grifs Grifs self-assigned this Jun 28, 2024
@Grifs
Copy link
Collaborator

Grifs commented Jun 28, 2024

The issue seems to be with line 42 in ViashBuild.scala:

val exec_path = resources.mainScript.map(scr => Paths.get(output, scr.resourcePath).toString)

For Nextflow runners, there is no mainScript, so subsequent code is not executed ... but in itself that makes sense, because the code to generate the docker file would be in the bash wrapper, which we just established isn't there.

@Grifs Grifs removed their assignment Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants