Installation failed even when new version listed on list-qt (just released 6.4.0 today)

[DioEgizio]

Describe the bug
can't download 6.4.0 even if it's out and can be downloaded with the official installer

To Reproduce
Steps to reproduce the behavior:

1. run aqt with this command: python3 -m aqt install-qt mac desktop 6.4.0
2. See error

ERROR : Failed to locate XML data for Qt version '6.4.0'.
==============================Suggested follow-up:==============================

• Please use 'aqt list-qt mac desktop' to show versions available.

Expected behavior
it working

Desktop (please complete the following information):

• OS: macOS
• aqt version v2.2.3

Additional context
it made me update the mantainance tool trying to update with the official installer, perhaps smth changed there?

[DioEgizio]

also
python3 -m aqt list-qt mac desktop

shows

5.9.0 5.9.1 5.9.2 5.9.3 5.9.4 5.9.5 5.9.6 5.9.7 5.9.8 5.9.9
5.10.0 5.10.1
5.11.0 5.11.1 5.11.2 5.11.3
5.12.0 5.12.1 5.12.2 5.12.3 5.12.4 5.12.5 5.12.6 5.12.7 5.12.8 5.12.9 5.12.10 5.12.11 5.12.12
5.13.0 5.13.1 5.13.2
5.14.0 5.14.1 5.14.2
5.15.0 5.15.1 5.15.2
6.0.0 6.0.1 6.0.2 6.0.3 6.0.4
6.1.0 6.1.1 6.1.2 6.1.3
6.2.0 6.2.1 6.2.2 6.2.3 6.2.4
6.3.0 6.3.1 6.3.2
6.4.0

so this is quite weird

[ddalcino]

I will have a look at this when I get home from work, but you are right that this is a problem. aqt list-qt should not be allowed to list items that you cannot successfully install.

I have noticed that there have been problems recently with installing Qt or tools less than a day after they have been released, because there’s a delay between the time the server uploads files and when the sha256 hashes for those files are available. Right now, aqt install-* won’t install anything without the hash files for security reasons. aqt list-* does not check for hash files at all, but it probably should. See #577, #521

The most recent update for Qt 6.4.0 is dated about 2 hours before this issue was created. By the time I get off work and take a look at this, the hash files might be up on the server and you’ll be able to install this properly again. I don’t know how long the delays are, but I know they are less than 24 hours. They could be less than 12.

[miurahr]

Qt download site may create hashes with hashing command, not on-the-fly.

MirrorBrain, which Qt project uses, manual explain how to create hash, advises to use cron job.
https://mirrorbrain-docs.readthedocs.io/en/latest/installation/initial_config.html#creating-hashes

Add the hashing command to /etc/crontab to be run every few hours. Alternatively, run it after changes in the file tree happen, coupled to some trigger etc.

A job calculating hash requires reading all the binaries of versions, it will take some duration.

[miurahr]

I push a pin the issue because many other users may want to use aqtinstall just after new Qt releases.
It will become FAQ.

[ddalcino]

Sorry it took me so long to get to this, but I have confirmed that aqt v2.2.3 can now install Qt 6.4.0 properly, after the hashes were uploaded.

$ aqt install-qt mac desktop 6.4.0

$ aqt install-qt mac desktop 6.4.0
INFO    : aqtinstall(aqt) v2.2.3 on Python 3.10.6 [CPython Clang 13.1.6 (clang-1316.0.21.2.5)]
INFO    : Downloading qtdeclarative...
INFO    : Downloading qttools...
INFO    : Downloading qtbase...
INFO    : Downloading qtsvg...
INFO    : Redirected: mirrors.ocf.berkeley.edu
INFO    : Redirected: mirrors.ocf.berkeley.edu
INFO    : Redirected: mirrors.ocf.berkeley.edu
INFO    : Redirected: mirrors.ocf.berkeley.edu
INFO    : Finished installation of qtsvg-MacOS-MacOS_12-Clang-MacOS-MacOS_12-X86_64-ARM64.7z in 1.72755112
INFO    : Downloading qttranslations...
INFO    : Redirected: mirrors.ocf.berkeley.edu
INFO    : Finished installation of qtbase-MacOS-MacOS_12-Clang-MacOS-MacOS_12-X86_64-ARM64.7z in 6.48118783
INFO    : Finished installation of qttranslations-MacOS-MacOS_12-Clang-MacOS-MacOS_12-X86_64-ARM64.7z in 5.11139325
INFO    : Finished installation of qttools-MacOS-MacOS_12-Clang-MacOS-MacOS_12-X86_64-ARM64.7z in 10.17033025
INFO    : Finished installation of qtdeclarative-MacOS-MacOS_12-Clang-MacOS-MacOS_12-X86_64-ARM64.7z in 12.73643021
INFO    : Patching /Users/dave/PyCharmProjects/aqtinstall/6.4.0/macos/bin/qmake
INFO    : Finished installation
INFO    : Time elapsed: 15.17287929 second

Some notes:

• According to download.qt.io, the new files for Qt 6.4.0 were last modified at Thu, 29 Sep 2022 03:14:45 GMT (Unix time: 1664421285).
• I checked, but could not locate the hash files at 17:00:00 GMT.
• The hash files were present at 23:00:00 GMT.

This upload of Qt 6.4.0 replaced an earlier upload a week prior. I was able to install Qt 6.4.0 in CI without any problems last week. The Qt developers uploaded a new copy of these files yesterday, which caused the sha256 hashes to be removed, making it impossible to download Qt 6.4.0 for <24 hours. We should expect intermittent outages whenever these updates occur.

[ddalcino]

Here's one potential way to handle this problem:

When we fail to retrieve a sha256 checksum file, we could check the upload date of the file we're trying to download. If the file was uploaded very recently, we could print an error message explaining that the SHA256 hash is not present on the server and there's no secure way to download the file yet. For macOS/desktop/Qt6.4.0, the 'Last modified' date is present at this url: https://download.qt.io/online/qtsdkrepository/mac_x64/desktop/qt6_640/Updates.xml.mirrorlist

I have noticed that at the Updates.xml.mirrorlist url, in addition to showing the 'Last modified' date, it will also show the SHA256 hash. It will not show this hash if the Updates.xml.sha256 file is not available yet.

I like having good explanations in error messages, but I don't really like this solution. There will always be a few hours every month where whatever tool or version of Qt you're trying to download is unavailable. We just don't have a good way to make it available all the time. This could cause CI build failures for a lot of people, and that's probably very annoying.

[miurahr]

@ddalcino Thank you for the mirrorlist link.

This tell us Qt project uses MirrorBrain v2.16.0 and the mirrorlink is produced by apache mod_mirrorbrain , and mtime would be always returned, and hashes may sometimes omit when calculation has not been done.

https://github.com/poeml/mirrorbrain/blob/master/mod_mirrorbrain/mod_mirrorbrain.c#L2959

[miurahr]

I have noticed that at the Updates.xml.mirrorlist url, in addition to showing the 'Last modified' date, it will also show the SHA256 hash. It will not show this hash if the Updates.xml.sha256 file is not available yet.

Unfortunately No. As you can see the code of mirrorbrain,

L2512

        hashbag = hashbag_fill(r, dbd, filename);
        if (hashbag == NULL) {
            debugLog(r, cfg, "no hashes found in database");
        } 

L2914

  case MIRRORLIST:

        setenv_give(r, "mirrorlist");
        debugLog(r, cfg, "Sending mirrorlist");

        ap_set_content_type(r, "text/html; charset=UTF-8");

L2963

        ap_rprintf(r, "  <li><span class=\"mirrorbrain-label\">Filename:</span> %s</li>\n", basename);
	ap_rprintf(r, "  <li><span class=\"mirrorbrain-label\">Path:</span> %s</li>\n", r->uri);
        ap_rprintf(r, "  <li><span class=\"mirrorbrain-label\">Size:</span> %s (%s bytes)</li>\n",   apr_strfsize(r->finfo.size, buf), apr_off_t_toa(r->pool, r->finfo.size));
        time_str = apr_palloc(r->pool, APR_RFC822_DATE_LEN);
        apr_rfc822_date(time_str, r->finfo.mtime);
        ap_rprintf(r, "  <li><span class=\"mirrorbrain-label\">Last modified:</span> " 
      "%s (Unix time: %" APR_INT64_T_FMT ")</li>\n",    time_str, apr_time_sec(r->finfo.mtime));

        if (hashbag != NULL) {
            if (hashbag->sha256hex)
                ap_rprintf(r, "  <li><span class=\"mirrorbrain-label\">"

The last line tell us mirrorlist will omit sha256 when hash database (hashbag) is not available, but always return filename, path, file size and mtime.

[LanderN]

The same thing is happening with the release of Qt 6.4.1 today, I'd be happy with an --insecure or --skip-hash-verify option to temporarily bypass the verification.

[Leon0402]

Same problem with Qt 6.5.0, which was released a few hours ago. By the way Qt 6.5.0 is not marked any special in aqtinstall despite the fact that it's a beta release. Is this itentional? I think it's a good idea to allow beta releases, but they probably be marked as such.

[DioEgizio]

Qt 6.5 isn't out yet though?

[Leon0402]

Qt 6.5.0 is released as a beta version: https://download.qt.io/online/qtsdkrepository/mac_x64/desktop/qt6_650/Updates.xml.mirrorlist and https://download.qt.io/development_releases/qt/6.5/6.5.0-beta1/

Also see: https://wiki.qt.io/Qt_6.5_Release

[ddalcino]

By the way Qt 6.5.0 is not marked any special in aqtinstall despite the fact that it's a beta release.

You are correct that 6.5.0 is a beta version. Unfortunately, I don’t see any way to work that out programmatically using the information available in the Updates.xml file or the Updates.xml.mirrorlist files in the https://download.qt.io/online/qtsdkrepository/mac_x64/desktop/qt6_650/ directory. I don’t like the idea of scraping blog posts at wiki.Qt.io to figure out which release is which.

Is this intentional?

No. It’s just an artifact of the directory structure of https://download.qt.io/online/qtsdkrepository/<host>/<target>/.

I think it's a good idea to allow beta releases, but they probably be marked as such.

If you know of any reliable way to determine which releases are beta and which are not, I’d love to hear it, or to see a PR.

[Leon0402]

If you know of any reliable way to determine which releases are beta and which are not, I’d love to hear it, or to see a PR.

Would it be an option to checkout https://download.qt.io/official_releases/qt/ and https://download.qt.io/development_releases/qt/ perhaps?
One could compare both of them.

But other than that the only way is to combine hardcoding with a little bit of logic. Like comparing when the Updates.xml was published with some hardcoded date of the actual release. This way it should work as well if the release schedule changes slightly afterwards, so the maintance burden is not to high. Obviously still requires manual changes every 6 month, but maybe that would be acceptable?

[adenis78]

I have the same.

aqt list-qt linux desktop
5.9.0 5.9.1 5.9.2 5.9.3 5.9.4 5.9.5 5.9.6 5.9.7 5.9.8 5.9.9
5.10.0 5.10.1
5.11.0 5.11.1 5.11.2 5.11.3
5.12.0 5.12.1 5.12.2 5.12.3 5.12.4 5.12.5 5.12.6 5.12.7 5.12.8 5.12.9 5.12.10 5.12.11 5.12.12
5.13.0 5.13.1 5.13.2
5.14.0 5.14.1 5.14.2
5.15.0 5.15.1 5.15.2
6.0.0 6.0.1 6.0.2 6.0.3 6.0.4
6.1.0 6.1.1 6.1.2 6.1.3
6.2.0 6.2.1 6.2.2 6.2.3 6.2.4
6.3.0 6.3.1 6.3.2
6.4.0 6.4.1 6.4.2 6.4.3
6.5.0 6.5.1

aqt install-qt --outputdir /opt/qt linux desktop 6.2.0 gcc_64
INFO : aqtinstall(aqt) v3.1.6 on Python 3.10.4 [CPython GCC 12.0.1 20220308 (Red Hat 12.0.1-0)]
ERROR : Failed to locate XML data for Qt version '6.2.0'.
==============================Suggested follow-up:==============================

• Please use 'aqt list-qt linux desktop' to show versions available.
  [psd@localhost ~]$ aqt install-qt --outputdir /opt/qt linux desktop 6.2 gcc_64
  INFO : aqtinstall(aqt) v3.1.6 on Python 3.10.4 [CPython GCC 12.0.1 20220308 (Red Hat 12.0.1-0)]
  INFO : Resolved spec '6.2' to 6.2.4
  ERROR : Failed to locate XML data for Qt version '6.2.4'.
  ==============================Suggested follow-up:==============================
• Please use 'aqt list-qt linux desktop' to show versions available.
  [psd@localhost ~]$ aqt install-qt --outputdir /opt/qt linux desktop 6.5.0 gcc_64INFO : aqtinstall(aqt) v3.1.6 on Python 3.10.4 [CPython GCC 12.0.1 20220308 (Red Hat 12.0.1-0)]
  ERROR : Failed to locate XML data for Qt version '6.5.0'.
  ==============================Suggested follow-up:==============================
• Please use 'aqt list-qt linux desktop' to show versions available.
  [psd@localhost ~]$ aqt install-qt --outputdir /opt/qt linux desktop 5.15.2 gcc_64
  INFO : aqtinstall(aqt) v3.1.6 on Python 3.10.4 [CPython GCC 12.0.1 20220308 (Red Hat 12.0.1-0)]
  ERROR : Failed to locate XML data for Qt version '5.15.2'.
  ==============================Suggested follow-up:==============================
• Please use 'aqt list-qt linux desktop' to show versions available.

This program is now unusable, isn't it? Or what am I doing wrong?

[ddalcino]

@adenis78 this is an extremely unusual result; see CI runs for proof that the program works. Please show us the contents of your settings.ini file; it's easy to break the program by changing it.

[nrbnlulu]

@ddalcino The issue exists on the workflows as well.
https://github.com/nrbnlulu/aqtinstall/actions/runs/5209030601/jobs/9398343913#step:8:96

[kelteseth]

I having the same issue with 6.5.1:

PS C:\Users\Eli> python -m aqt install-qt -O  C:\Code\cpp\ScreenPlay\aqt windows desktop 6.5.1 win64_msvc2019_64 -m qt3d qtquick3d qtconnectivity qt5compat qtimageformats qtmultimedia qtshadertools qtwebchannel qtwebengine qtwebsockets qtwebview qtpositioning
INFO    : aqtinstall(aqt) v3.1.6 on Python 3.11.0 [CPython MSC v.1933 64 bit (AMD64)]
WARNING : Specified Qt version "6.5.1" did not exist when this version of aqtinstall was released. This may not install properly, but we will try our best.
ERROR   : Failed to locate XML data for Qt version '6.5.1'.
==============================Suggested follow-up:==============================
* Please use 'aqt list-qt windows desktop' to show versions available.

It does get listed here:

PS C:\Users\Eli> python -m aqt list-qt windows desktop
5.9.0 5.9.1 5.9.2 5.9.3 5.9.4 5.9.5 5.9.6 5.9.7 5.9.8 5.9.9
5.10.0 5.10.1
5.11.0 5.11.1 5.11.2 5.11.3
5.12.0 5.12.1 5.12.2 5.12.3 5.12.4 5.12.5 5.12.6 5.12.7 5.12.8 5.12.9 5.12.10 5.12.11 5.12.12
5.13.0 5.13.1 5.13.2
5.14.0 5.14.1 5.14.2
5.15.0 5.15.1 5.15.2
6.0.0 6.0.1 6.0.2 6.0.3 6.0.4
6.1.0 6.1.1 6.1.2 6.1.3
6.2.0 6.2.1 6.2.2 6.2.3 6.2.4
6.3.0 6.3.1 6.3.2
6.4.0 6.4.1 6.4.2 6.4.3
6.5.0 6.5.1

This command always worked for me so far...

Edit: Looks like a hiccup it works now...

[nrbnlulu]

Seems like an issue with Qt's download center... As for now it works.

[miurahr]

Qt download site operated by The Qt Company Ltd, uses MirrorBrain 2.17.0 or later for its web server. SHA256 hashes are calculated by the MirrorBrain server application.

It has a problem that it return error during calculate hash value when server contents are changed as I previously commented.
#578 (comment)
#578 (comment)

If anyone can help improving MirrorBrain software, qt downlaod site will be improved.

[nrbnlulu]

I think providing a hint on aqt would help as well.

[miurahr]

MirrorBrain team seems to stop its maintenance. OpenSuSE member continues development at https://github.com/openSUSE/mirrorbrain