cache everything

[tkelman]

hopefully fixes #44, #36

[tkelman]

@staticfloat any idea why this gives

INFO: Downloading https://cache.e.ip.saba.us/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1/repodata/repomd.xml
===============================[ ERROR: WinRPM ]================================

"Error executing request : Peer certificate cannot be authenticated with given CA certificates"
while loading /home/tkelman/.julia/v0.3/WinRPM/deps/build.jl, in expression starting on line 2

On Ubuntu 14.04?

[staticfloat]

Hmmm, nope. What happens if you do this:

$ curl -v -L https://cache.e.ip.saba.us/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1/repodata/repomd.xml
* Hostname was NOT found in DNS cache
*   Trying 128.52.178.53...
* Connected to cache.e.ip.saba.us (128.52.178.53) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: cache.e.ip.saba.us
* Server certificate: StartCom Class 1 Primary Intermediate Server CA
* Server certificate: StartCom Certification Authority
> GET /http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1/repodata/repomd.xml HTTP/1.1
> User-Agent: curl/7.37.1
> Host: cache.e.ip.saba.us
> Accept: */*
>
< HTTP/1.1 301 MOVED PERMANENTLY
* Server nginx is not blacklisted
< Server: nginx
< Date: Mon, 10 Aug 2015 03:39:43 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 401
< Connection: keep-alive
< Location: http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1/repodata/repomd.xml
< Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
<
* Ignoring the response-body
* Connection #0 to host cache.e.ip.saba.us left intact
* Issue another request to this URL: 'http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1/repodata/repomd.xml'
* Hostname was NOT found in DNS cache
*   Trying 2001:67c:2178:8::13...
* Connected to download.opensuse.org (2001:67c:2178:8::13) port 80 (#1)
> GET /repositories/windows:/mingw:/win32/openSUSE_13.1/repodata/repomd.xml HTTP/1.1
> User-Agent: curl/7.37.1
> Host: download.opensuse.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 10 Aug 2015 03:39:43 GMT
* Server Apache/2.2.12 (Linux/SUSE) is not blacklisted
< Server: Apache/2.2.12 (Linux/SUSE)
< Last-Modified: Sat, 08 Aug 2015 00:14:14 GMT
< ETag: "654-51cc1a1e5a180"
< Accept-Ranges: bytes
< Content-Length: 1620
< Cache-Control: must-revalidate
< Expires: Mon, 10 Aug 2015 03:39:43 GMT
< Content-Type: text/xml
<
<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
 <revision>1438992846</revision>
 <tags>
   <repo>obsrepository://build.opensuse.org/windows:mingw:win32/openSUSE_13.1</repo>
 </tags>
<data type="filelists">
  <checksum type="sha256">9689e8e61c3e280793f7b82772c3d24f674445e493b08a8ebe389b447e046c8a</checksum>
  <open-checksum type="sha256">df1350cf09aaf3e420a6c2ae477794bd3f1d25d129ee5bc2746584863c8556ec</open-checksum>
  <location href="repodata/9689e8e61c3e280793f7b82772c3d24f674445e493b08a8ebe389b447e046c8a-filelists.xml.gz"/>
  <timestamp>1438992854</timestamp>
  <size>1190341</size>
  <open-size>18150416</open-size>
</data>
<data type="other">
  <checksum type="sha256">a05067df29ea0ab11405dca141c723f47fc4e73690f0d341801c364012153ea4</checksum>
  <open-checksum type="sha256">ea9f3136249367f29013bbb877c48ccb0b3b3b9383909784e32cd36e00364fc3</open-checksum>
  <location href="repodata/a05067df29ea0ab11405dca141c723f47fc4e73690f0d341801c364012153ea4-other.xml.gz"/>
  <timestamp>1438992854</timestamp>
  <size>137371</size>
  <open-size>766354</open-size>
</data>
<data type="primary">
  <checksum type="sha256">f3f4f48e9e6cf41f8e4e4cd778b98435b0b7da6bd1c2e23187f064ae749cd960</checksum>
  <open-checksum type="sha256">1983a1129feed7d9f955f58202779439e76b17903131a6a1f3930ca7f94e252a</open-checksum>
  <location href="repodata/f3f4f48e9e6cf41f8e4e4cd778b98435b0b7da6bd1c2e23187f064ae749cd960-primary.xml.gz"/>
  <timestamp>1438992854</timestamp>
  <size>380552</size>
  <open-size>3499085</open-size>
</data>
</repomd>
* Connection #1 to host download.opensuse.org left intact

[tkelman]

https://gist.github.com/5e107713a51a9d6c6870

[staticfloat]

Looks like your command-line curl downloads just fine. How exactly does WinRPM attempt to download this file? Using curl? Using a different curl than you're running there?

[tkelman]

HTTPClient.jl, on unix

[staticfloat]

This works for me on an Ubuntu 14.04 machine:

julia> using HTTPClient.HTTPC

julia> z = HTTPC.get("https://cache.e.ip.saba.us/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1/repodata/repomd.xml")
HTTP Code   :200
RequestTime :1.352728
Headers     :
    Connection : keep-alive
    Date : Mon, 10 Aug 2015 04:04:25 GMT
    Date : Mon, 10 Aug 2015 04:04:25 GMT
    Accept-Ranges : bytes
    Cache-Control : must-revalidate
    Server : nginx
    Server : Apache/2.2.12 (Linux/SUSE)
    Content-Length : 401
    Content-Length : 1620
    ETag : "654-51cc1a1e5a180"
    Expires : Mon, 10 Aug 2015 04:04:25 GMT
    Last-Modified : Sat, 08 Aug 2015 00:14:14 GMT
    X-Content-Type-Options : nosniff
    Strict-Transport-Security : max-age=63072000; includeSubdomains; preload
    Content-Type : text/html; charset=utf-8
    Content-Type : text/xml
    Location : http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1/repodata/repomd.xml
    X-Frame-Options : DENY
Length of body : 1620

julia> versioninfo()
Julia Version 0.3.10
Commit c8ceeef (2015-06-24 13:54 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7 CPU         960  @ 3.20GHz
  WORD_SIZE: 64
  BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Nehalem)
  LAPACK: liblapack.so.3
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

[tkelman]

Aha. How bizarre. Well, not a WinRPM issue I suppose.

julia> using HTTPClient.HTTPC

julia> z = HTTPC.get("https://cache.e.ip.saba.us/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1/repodata/repomd.xml")
ERROR: "Error executing request : Peer certificate cannot be authenticated with given CA certificates"
 in exec_as_multi at /home/tkelman/.julia/v0.3/HTTPClient/src/HTTPC.jl:702
 in get at /home/tkelman/.julia/v0.3/HTTPClient/src/HTTPC.jl:371
 in get at /home/tkelman/.julia/v0.3/HTTPClient/src/HTTPC.jl:519

julia> versioninfo()
Julia Version 0.3.11
Commit 483dbf5 (2015-07-27 06:18 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM)2 Duo CPU     E8400  @ 3.00GHz
  WORD_SIZE: 64
  BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Penryn)
  LAPACK: liblapack.so.3
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

[staticfloat]

You don't, by any chance, have some environment variables getting set that could be mucking with the certificates like CURL_CA_BUNDLE do you?

[tkelman]

Not as far as I can tell from env, unless opam or miniconda went and put a different libcurl earlier in my path. Doesn't look like they've done so though.

[staticfloat]

What version of HTTPClient.jl do you have installed? Your backtrace isn't quite matching up with the latest release as far as I can tell...

[tkelman]

v0.1.4. Wonder what's holding it back. Doesn't look dirty, not sure what the best way is to find out whether something else I have installed is imposing an upper bound...

[tkelman]

Oh, strange, JuliaLang/METADATA.jl#2704 is causing AWS and HTTPClient to fight it out over who gets to be on the latest tag. AWS doesn't have any newer versions that are compatible with 0.3.

[staticfloat]

Well, if you somehow force HTTPClient to v0.1.5, does it solve the problem?

[tkelman]

No, bummer, false alarm.

julia> z = HTTPC.get("https://cache.e.ip.saba.us/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1/repodata/repomd.xml")
ERROR: "Error executing request : Peer certificate cannot be authenticated with given CA certificates"
 in exec_as_multi at /home/tkelman/.julia/v0.3/HTTPClient/src/HTTPC.jl:717
 in get at /home/tkelman/.julia/v0.3/HTTPClient/src/HTTPC.jl:386
 in get at /home/tkelman/.julia/v0.3/HTTPClient/src/HTTPC.jl:534

julia> Pkg.installed("HTTPClient")
v"0.1.5"

[staticfloat]

Have you done an apt-get update && apt-get upgrade recently? Perhaps there's an update to your local CA list? Is your system time significantly wrong?

[tkelman]

I update almost daily, no updates that apt-get wants to give me, and system time looks fine. Maybe overdue for a reboot, will try that didn't help. Thanks for the troubleshooting ideas.