Click-based recipes on Python 3

[nvictus]

I noticed that all py3k builds for recipes depending on click have been disabled due to what I assume is locale detection or configuration issues.

Maybe it's worth opening a discussion with the Click devs. They seem to be very opinionated about supporting only properly configured utf8 locales. Ideally, it would be great if they provided an explicit way to downgrade the locale runtime error to a warning, because sometimes you really can't control locale. On the other hand, if Click is falsely flagging the container environments as misconfigured, then we should raise an issue there.

It might also be worth pointing out that bioinformatics is so ASCII-oriented that surrogate escapes isn't an issue we are likely to encounter very often.

Btw, in cooler, I circumvented this with a dirty monkey patch that prevents locale detection because I also experienced issues deploying to containers I couldn't configure.

[bgruening]

I can only second that! It is really annoying I tracking a bug in python that should suppose to fix this in python2.7. Part of the problem is that click is not really recommending python3k - actually reading the docu it seems they say - py3k good luck. Thats one of the reasons we disabled it.

However, we provided a workaround for this problem if it is really needed to get it running under py3k by using a different base Docker image.

[nvictus]

To be fair, there seem to be no issues with py3k as long as the environment ships a utf-8 locale like C.UTF-8. Is this not the case in the Dockerfile?

The base image here seems to have it. Maybe it also needs ENV LC_ALL C.UTF-8?\

[EDIT] It seems that you may need to generate the locale first with locale-gen. @golobor has done it successfully with the following:

# Set the locale
RUN locale-gen en_US.UTF-8  
ENV LANG en_US.UTF-8  
ENV LANGUAGE en_US:en  
ENV LC_ALL en_US.UTF-8     

[EDIT] locale-gen is available in the Ubuntu image. The method is different for Debian.

[bgruening]

@nvictus Ubuntu is a huge image with many things preinstalled. We test against busybox with nothing installed and also no UTF-8 available.

Does this help? http://click.pocoo.org/6/python3/#unicode-literals

import click
click.disable_unicode_literals_warning = True

[nvictus]

Looks like C locale coercion to UTF-8 was implemented in Python 3.7 as PEP 538. This works only if the locale is available in the system, so presumably it still won't work on the busybox images.
Btw, @bgruening, I noticed we both gave tutorials at ISMB/ECCB today! Maybe see you in Prague this week...

[bgruening]

@nvictus we have now a workaround for this using a either this:

bioconda-recipes/recipes/biom-format/2.1.5/meta.yaml

Line 48 in 6e6bbfb

- LANG=C.UTF-8 biom --help

or this:

bioconda-recipes/recipes/ggplot/meta.yaml

Line 47 in f575c10

container:

Are you at GCC/BOSC this year?

[SilasK]

Hello, I tried now all possible solutions and no success.
When using

extra:
  container:
    extended-base: true

I get the error

[Feb  7 10:35:07] SERR Additional information: on this system no suitable UTF-8
[Feb  7 10:35:07] SERR locales were discovered.  This most likely requires resolving
[Feb  7 10:35:07] SERR by reconfiguring the locale system.
[Feb  7 10:35:07] ERRO Task processing failed: Unexpected exit co

When using

commands:
   - LANG=C.utf8 atlas --version

I get the error:

This system lists a couple of UTF-8 supporting locales that
you can pick from.  The following suitable locales were
discovered: en_US.utf8


Click discovered that you exported a UTF-8 locale
but the locale system could not pick up from it because
it does not exist.  The exported locale is "C.UTF-8" but it
is not supported

When using

commands:
   - LANG=en_US.utf8 atlas --version

I got the error:

[Feb  7 13:31:24] SERR ' mitigation steps.' + extra
[Feb  7 13:31:24] SERR RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/en/7.x/python3/ for mitigation steps.
[Feb  7 13:31:24] SERR 
[Feb  7 13:31:24] SERR Additional information: on this system no suitable UTF-8
[Feb  7 13:31:24] SERR locales were discovered.  This most likely requires resolving
[Feb  7 13:31:24] SERR by reconfiguring the locale system.
[Feb  7 13:31:24] SERR 
[Feb  7 13:31:24] SERR Click discovered that you exported a UTF-8 locale
[Feb  7 13:31:24] SERR but the locale system could not pick up from it because
[Feb  7 13:31:24] SERR it does not exist.  The exported locale is "en_US.utf8" but it
[Feb  7 13:31:24] SERR is not supported

I now removed the commands from #13532 , but has anybody a better solution?

[SilasK]

I managed to get it work using both solutions:

{% set export_lang = "export LC_ALL=en_US.UTF-8 && export LANG=en_US.UTF-8" %}

test:
  imports:
    - atlas
  commands:
    -  {{ export_lang }} && atlas --version
    - {{ export_lang }} && atlas init --help
    - {{ export_lang }} && atlas run --help


extra:
  container:
    # unicode problem with click
    extended-base: true