Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

netCDF4.rc_set("HTTP.NETRC", ".netrc") has no effect #169

Open
1 task done
magnusuMET opened this issue Oct 29, 2024 · 3 comments
Open
1 task done

netCDF4.rc_set("HTTP.NETRC", ".netrc") has no effect #169

magnusuMET opened this issue Oct 29, 2024 · 3 comments
Labels
bug

Comments

@magnusuMET
Copy link

Solution to issue cannot be found in the documentation.

  • I checked the documentation.

Issue

I am trying to use netCDF4.rc_set which was introduced in python-netcdf4 1.7.2 to set a netrc file to be used. This works when using a virtual environment and system netcdf4. When trying to do the same in a conda environment this has no effect, and does not make curl set the authorization header.

Steps to reproduce:

Server:

python3 -m http.server

Client

main.py:

import netCDF4

netCDF4.rc_set("HTTP.NETRC", ".netrc")
url = "http://localhost:8000/"
d = netCDF4.Dataset(url)
print(d)

if __name__ == "__main__":
    main()

.netrc:

machine localhost
    login foo
    password bar

Run as follows for debug output:

env CURLOPT_VERBOSE=1 python3 main.py

Expected output (using system libnetcdf4 and netcdf=4.9.2 from pip install in a venv)

Output from running the above 
* Host localhost:8000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8000...
* connect to ::1 port 8000 from ::1 port 37164 failed: Connection refused
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000
* Server auth using Basic with user 'foo'
> GET /.dds HTTP/1.1
Host: localhost:8000
Authorization: Basic Zm9vOmJhcg==
User-Agent: oc4.9.4-development
Accept: */*
Accept-Encoding: deflate, gzip

* Request completely sent off
* HTTP 1.0, assume close after body
< HTTP/1.0 404 File not found
< Server: SimpleHTTP/0.6 Python/3.10.12
< Date: Tue, 29 Oct 2024 09:28:19 GMT
< Connection: close
< Content-Type: text/html;charset=utf-8
< Content-Length: 469
<
* Closing connection
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <!DOCTYPE^ HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"><html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"> <title>Error response</title> </head> <body> <h1>Error response</h1> <p>Error code: 404</p> <p>Message: File not found.</p> <p>Error code explanation: HTTPStatus.NOT_FOUND - Nothing matches the given URI.</p> </body></html>
Traceback (most recent call last):
  File "/home/magnusu/Projects/tmp/2024-07-15/public.py", line 7, in <module>
    d = netCDF4.Dataset(url)
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -90] NetCDF: file not found: 'http://localhost:8000/'

Actual output using the conda package

Output from running the above 
* Host localhost:8000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8000...
* connect to ::1 port 8000 from ::1 port 44538 failed: Connection refused
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000
* using HTTP/1.x
> GET /.dds HTTP/1.1
Host: localhost:8000
User-Agent: oc4.9.2
Accept: */*
Accept-Encoding: deflate, gzip, zstd

* Request completely sent off
* HTTP 1.0, assume close after body
< HTTP/1.0 404 File not found
< Server: SimpleHTTP/0.6 Python/3.10.12
< Date: Tue, 29 Oct 2024 09:32:16 GMT
< Connection: close
< Content-Type: text/html;charset=utf-8
< Content-Length: 469
<
* shutting down connection #0
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <!DOCTYPE^ HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"><html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"> <title>Error response</title> </head> <body> <h1>Error response</h1> <p>Error code: 404</p> <p>Message: File not found.</p> <p>Error code explanation: HTTPStatus.NOT_FOUND - Nothing matches the given URI.</p> </body></html>
Traceback (most recent call last):
  File "/home/magnusu/Projects/tmp/2024-07-15/./public.py", line 7, in <module>
    d = netCDF4.Dataset(url)
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -90] NetCDF: file not found: 'http://localhost:8000/'

Notice the lack of * Server auth using Basic with user 'foo' and Authorization: Basic Zm9vOmJhcg== in the output from the conda environment

Installed packages

Name Version Build Channel
────────────────────────────────────────────────────────────────────────
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
blosc 1.21.6 hef167b5_0 conda-forge
bzip2 1.0.8 h4bc722e_7 conda-forge
c-ares 1.34.2 heb4867d_0 conda-forge
ca-certificates 2024.8.30 hbcca054_0 conda-forge
certifi 2024.8.30 pyhd8ed1ab_0 conda-forge
cftime 1.6.4 py313ha014f3b_1 conda-forge
hdf4 4.2.15 h2a13503_7 conda-forge
hdf5 1.14.4 nompi_h2d575fe_101 conda-forge
icu 75.1 he02047a_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.21.3 h659f571_0 conda-forge
ld_impl_linux-64 2.43 h712a8e2_2 conda-forge
libaec 1.1.3 h59595ed_0 conda-forge
libblas 3.9.0 25_linux64_openblas conda-forge
libcblas 3.9.0 25_linux64_openblas conda-forge
libcurl 8.10.1 hbbe4b11_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 hd590300_2 conda-forge
libexpat 2.6.3 h5888daf_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc 14.2.0 h77fa898_1 conda-forge
libgcc-ng 14.2.0 h69a702a_1 conda-forge
libgfortran 14.2.0 h69a702a_1 conda-forge
libgfortran-ng 14.2.0 h69a702a_1 conda-forge
libgfortran5 14.2.0 hd5240d6_1 conda-forge
libgomp 14.2.0 h77fa898_1 conda-forge
libiconv 1.17 hd590300_2 conda-forge
libjpeg-turbo 3.0.0 hd590300_1 conda-forge
liblapack 3.9.0 25_linux64_openblas conda-forge
libmpdec 4.0.0 h4bc722e_0 conda-forge
libnetcdf 4.9.2 nompi_h2564987_115 conda-forge
libnghttp2 1.64.0 h161d5f1_0 conda-forge
libopenblas 0.3.28 pthreads_h94d23a6_0 conda-forge
libsqlite 3.47.0 hadc24fc_0 conda-forge
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx 14.2.0 hc0a3c3a_1 conda-forge
libstdcxx-ng 14.2.0 h4852527_1 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libxml2 2.12.7 he7c6b58_4 conda-forge
libzip 1.11.1 hf83b1b0_0 conda-forge
libzlib 1.3.1 hb9d3cd8_2 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
ncurses 6.5 he02047a_1 conda-forge
netcdf4 1.7.2 nompi_py313h1dd084c_100 conda-forge
numpy 2.1.2 py313h4bf6692_0 conda-forge
openssl 3.3.2 hb9d3cd8_0 conda-forge
packaging 24.1 pyhd8ed1ab_0 conda-forge
pandas 2.2.3 py313ha87cce1_1 conda-forge
pip 24.2 pyh145f28c_1 conda-forge
python 3.13.0 h9ebbce0_100_cp313 conda-forge
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge
python-tzdata 2024.2 pyhd8ed1ab_0 conda-forge
python_abi 3.13 5_cp313 conda-forge
pytz 2024.1 pyhd8ed1ab_0 conda-forge
readline 8.2 h8228510_1 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.2.1 ha2e4443_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
tzdata 2024b hc8b5060_0 conda-forge
xarray 2024.10.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
zlib 1.3.1 hb9d3cd8_2 conda-forge
zstd 1.5.6 ha6fb4c9_0 conda-forge

Environment info

   libmamba version : 1.5.1
 micromamba version : 1.5.1
       curl version : libcurl/7.88.1 OpenSSL/3.1.2 zlib/1.2.13 zstd/1.5.5 libssh2/1.11.0 nghttp2/1.52.0
 libarchive version : libarchive 3.6.2 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.2
   envs directories : /home/magnusu/micromamba/envs
      package cache : /home/magnusu/micromamba/pkgs
                      /home/magnusu/.mamba/pkgs
        environment : /home/magnusu/Projects/tmp/2024-07-15/CONDA (active)
       env location : /home/magnusu/Projects/tmp/2024-07-15/CONDA
  user config files : /home/magnusu/.mambarc

populated config files : /home/magnusu/.condarc
virtual packages : __unix=0=0
__linux=6.8.0=0
__glibc=2.35=0
__archspec=1=x86_64
channels :
base environment : /home/magnusu/micromamba
platform : linux-64
@magnusuMET magnusuMET added the bug label Oct 29, 2024
@xylar
Copy link
Contributor

xylar commented Nov 7, 2024

@magnusuMET, I was able to reproduce your error but didn't have any insight into it at the time you posted it. In the last few days, I have seen this issue come up on the curl feedstock:
conda-forge/curl-feedstock#149
and I have since seen that a specific version of (lib)curl, 8.11.0, has been marked as broken related to netrc issues:
conda-forge/admin-requests#1147
You do not appear to be using that version but I wonder if the problems extend to 8.10.1 as well. Have you tried an older version of curl to see if it happens to solve the issue?

@ocefpaf
Copy link
Member

ocefpaf commented Nov 7, 2024

Is it related to Unidata/netcdf4-python#1380 ?

@magnusuMET
Copy link
Author

@ocefpaf Not directly related, this was to get back a default which curl dropped some time back

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ocefpaf @xylar @magnusuMET