multiprocessing.pool.MaybeEncodingError building reference for fusion-report

[maressyl]

Description of the bug

Following issue #397 I'm still facing problems with fusion-report when building reference from the dev reference (currently 6dfedc2213c806e99ced690fe0fc2e92d6f01102).

Command used and terminal output

./nextflow-23.09.1-edge-all -c custom.conf run nf-core/rnafusion -r dev -profile singularity --outdir build --all --build_references --cosmic_username "REDACTED" --cosmic_passwd "REDACTED"

ERROR ~ Error executing process > 'NFCORE_RNAFUSION:BUILD_REFERENCES:FUSIONREPORT_DOWNLOAD (fusionreport)'

Caused by:
  Process `NFCORE_RNAFUSION:BUILD_REFERENCES:FUSIONREPORT_DOWNLOAD (fusionreport)` terminated with an error exit status (1)

Command executed:

  fusion_report download --cosmic_usr "REDACTED" --cosmic_passwd "REDACTED"  ./

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_RNAFUSION:BUILD_REFERENCES:FUSIONREPORT_DOWNLOAD":
      fusion_report: $(fusion_report --version | sed 's/fusion-report //')
  END_VERSIONS

Command exit status:
  1

Command output:
  Downloading resources...
  Downloading mitelman_db.zip

Command error:
  Downloading resources...
  Downloading mitelman_db.zip
  Traceback (most recent call last):
    File "/usr/local/bin/fusion_report", line 4, in <module>
      __import__('pkg_resources').run_script('fusion-report==2.1.5', 'fusion_report')
    File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 651, in run_script
      self.require(requires)[0].run_script(script_name, ns)
    File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1448, in run_script
      exec(code, namespace, namespace)
    File "/usr/local/lib/python3.9/site-packages/fusion_report-2.1.5-py3.9.egg/EGG-INFO/scripts/fusion_report", line 13, in <module>
      app.run()
    File "/usr/local/lib/python3.9/site-packages/fusion_report-2.1.5-py3.9.egg/fusion_report/app.py", line 71, in run
      Download(params)
    File "/usr/local/lib/python3.9/site-packages/fusion_report-2.1.5-py3.9.egg/fusion_report/download.py", line 22, in __init__
      self.download_all(params)
    File "/usr/local/lib/python3.9/site-packages/fusion_report-2.1.5-py3.9.egg/fusion_report/download.py", line 44, in download_all
      Net.get_fusiongdb(self, return_err)
    File "/usr/local/lib/python3.9/site-packages/fusion_report-2.1.5-py3.9.egg/fusion_report/common/net.py", line 176, in get_fusiongdb
      pool.starmap(Net.get_large_file, pool_params)
    File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 372, in starmap
      return self._map_async(func, iterable, starmapstar, chunksize).get()
    File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 771, in get
      raise self._value
  multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7fed81158f70>'. Reason: 'TypeError("cannot pickle '_io.BufferedReader' object")'

Relevant files

custom.conf.zip

System information

• nextflow-23.09.1-edge-all
• HPC server
• local executor
• singularity-ce version 3.8.0
• Debian 11
• nf-core/rnafusion dev

[rannick]

I can reproduce. It is due to an extra layer of security from the ccsm.uth.edu website that was just added. Can you please try again? I overwrote the container by mistake, so you can actually just use the same branch. I will sort the container situation in the next few days

[maressyl]

Thanks for your help. I still have the same error after cleaning my Nextflow and Singularity caches and resubmitting with the same command, but looking at Docker Hub I suspect you started fixing the containers, should I try later ?

[rannick]

Yes, was just going to write that the container situation should be fixed and I just merged the PR to link rnafusion to the right fusionreport container. So you can try dev again!

[maressyl]

Different error this time (still the same command, after cleaning caches) :

Downloading resources...
Downloading mitelman_db.zip
<urlopen error Tunnel connection failed: 500 Unable to connect>
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/lib/python3.9/http/client.py", line 1279, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1325, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1274, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1034, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.9/http/client.py", line 1441, in connect
    super().connect()
  File "/usr/local/lib/python3.9/http/client.py", line 950, in connect
    self._tunnel()
  File "/usr/local/lib/python3.9/http/client.py", line 929, in _tunnel
    raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 500 Unable to connect

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/fusion_report-2.1.5-py3.9.egg/fusion_report/app.py", line 71, in run
    Download(params)
  File "/usr/local/lib/python3.9/site-packages/fusion_report-2.1.5-py3.9.egg/fusion_report/download.py", line 22, in __init__
    self.download_all(params)
  File "/usr/local/lib/python3.9/site-packages/fusion_report-2.1.5-py3.9.egg/fusion_report/download.py", line 44, in download_all
    Net.get_fusiongdb(self, return_err)
  File "/usr/local/lib/python3.9/site-packages/fusion_report-2.1.5-py3.9.egg/fusion_report/common/net.py", line 177, in get_fusiongdb
    Net.get_large_file(url)
  File "/usr/local/lib/python3.9/site-packages/fusion_report-2.1.5-py3.9.egg/fusion_report/common/net.py", line 103, in get_large_file
    with urllib.request.urlopen(req, context=ctx) as response:
  File "/usr/local/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/local/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/local/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error Tunnel connection failed: 500 Unable to connect>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/fusion_report-2.1.5-py3.9.egg/EGG-INFO/scripts/fusion_report", line 13, in <module>
    app.run()
  File "/usr/local/lib/python3.9/site-packages/fusion_report-2.1.5-py3.9.egg/fusion_report/app.py", line 78, in run
    raise AppException(ex)
fusion_report.common.exceptions.app.AppException: <urlopen error Tunnel connection failed: 500 Unable to connect>
<urlopen error Tunnel connection failed: 500 Unable to connect>

At some point it mentioned it pulled the "p4" image as expected.

[maressyl]

I have to highlight I'm using a proxy (passed through http_proxy and https_proxy environment variables), it works fine with the wget but I don't know how Python would handle it...

[rannick]

This one I cannot reproduce, and most likely is due to the proxy indeed. You could download locally (the download should be lightweight) and copy to HPC after as detailed in

rnafusion/docs/usage.md

Line 69 in 62f78be

#### Issues with building references

[maressyl]

I managed to build it locally following the documentation you provided, thanks a lot for your help !