cannot download specific Qt Creator version

[christian-rauch]

There seems to be no option to download specific Qt Creator versions (4.15, 5.0, ...).

aqt list-qt linux desktop lists the Qt versions whereas aqt list-tool linux desktop tools_qtcreator only lists the modules (qtcreator, qtcreatordev, ...). It would be useful to also specify a specific tool version.

[ddalcino]

If you can find them on https://download.qt.io/online/qtsdkrepository/, that's what we support.

aqt list-tool linux desktop tools_qtcreatorshould list everything in the https://download.qt.io/online/qtsdkrepository/linux_x64/desktop/tools_qtcreator/ directory; as far as I know, the Qt repo maintainers only keep one version there at a time.

[christian-rauch]

You are right, https://download.qt.io/online/qtsdkrepository/linux_x64/desktop/tools_qtcreator/qt.tools.qtcreatordev/ only lists a single (the most recent) version. The Qt Maintainance tool also provides no way to install an older version.

This might be problematic if you want to run a CI job for an older Qt Creator plugin as aqtinstall will then download a newer incompatible version.

I previously used https://download.qt.io/official_releases/qtcreator which provides many older Qt Creator releases. https://download.qt.io/development_releases/qtcreator/ then also provides the development releases. However, the same repo does only provide source releases of Qt and not the precompiled distribution. I guess supporting two different repo schemes is too much to ask here.

[ddalcino]

I guess supporting two different repo schemes is too much to ask here.

A lot of our code is tightly coupled to the design of the https://download.qt.io/online/qtsdkrepository/ repo; I think it would be nice to decouple it. This repo is extremely inconsistent, and each new release of Qt adds new wrinkles and idiosyncracies. It's hard enough to support just one repo. I don't want to say we can't support another repo, but we would need a lot of help to do so.

[ddalcino]

I am thinking about this, and I think there might be another way to install archives from https://download.qt.io/official_releases/qtcreator. I think that all we're looking for is a multi-platform replacement for curl "some_exe" in the case of an exe, or curl "some_zip_or_tar" | tar -xf. The standard python tarfile and zipfile libraries make this really easy to do.

Here's the interface I'm thinking about:

$ # Download QtCreator 5.0.2
$ aqt install-extra "official_releases/qtcreator/5.0/5.0.2/qt-creator-opensource-windows-x86_64-5.0.2.exe"
$ # Download & extract zip file
$ aqt install-extra "official_releases/jom/jom_1_1_3.zip" --extract
$ # Download & extract compressed tar
$ aqt install-extra "official_releases/gdb/windows-7/android-ndk-r10-windows-gdb.tar.xz" --extract

The Qt repos in the "official_releases" section often, but not always, include an "md5sums.txt" or "sha256sums.txt" file. I think it would be useful to read this file, when available, and compare its contents with the checksum of the downloaded file. This would at least give us some added value over curl | tar.

This interface is a lot less useful than something like aqt install-qtcreator 5.0.2, but it's a lot less work to implement and maintain, and it would work for a lot more tools.

Is this worth implementing? Would anyone use this? Is the interface OK?

[christian-rauch]

I implemented a custom python script to download given Qt and Qt Creator versions from the release or development channel: https://github.com/ros-industrial/ros_qtc_plugin/blob/devel/setup.py. I think you can also parse the Updates.xml to get the most recent version number.

[ddalcino]

I think you can also parse the Updates.xml to get the most recent version number.

Yes, we are doing this already with aqt install-tool and aqt list-tool, but the Updates.xml file only exists in the "online/qtsdkrepository/" that we already support.

url_repo_qtc_fmt = "https://download.qt.io/{release_type}_releases/qtcreator/{qtcv_maj}/{qtcv_full}/installer_source/{os}_{arch}/"
url_repo_qt_fmt = "https://download.qt.io/online/qtsdkrepository/{os}_{arch}/desktop/qt{ver_maj}_{ver_concat}/"

Interesting ... maybe if we use a data file to define format strings for all the directories, and define constants that describe how the variables must be filled in ... that could work. Let me think about this.

[ddalcino]

Ok, I have a preliminary description of the "official_releases" repo encoded in JSON: official_releases.txt. I think that this structure can be easily extended (probably not outright reused) for the other "*_releases" repos. The structure can be used to turn a list of arguments into a url, or a list of urls if you pass "all" as an argument. The structure defines what arguments are required, and in many cases, what values are acceptable for an argument.

Right now, there are a few urls in the repo that are unreachable using this structure, but I think the structure can be extended to include them.

It would be fairly straightforward to connect this structure to a CLI. I think it would be pretty reasonable to construct an interface like this:

$ aqt install-official jom 1.1.3              # download jom version 1.1.3 and verify checksum
$ aqt install-official jom 1.1.3 --extract    # download jom version 1.1.3, verify checksum, and extract zip
$ aqt install-official gdb windows 64         # download android-ndk-r10-windows-x86_64-gdb.tar.xz

Qt Creator is a more difficult case. As far as I can tell, there are 3-4 different kinds of binaries/source archives that you can download for Qt Creator. To allow downloads for all of them, I defined 4 separate schemas for Qt Creator, which you would have to choose using a CLI flag. Each schema would require a separate set of arguments:

$ aqt install-official 5.0.2 windows      # qtcreator/5.0/5.0.2/qt-creator-opensource-windows-x86_64-5.0.2.exe
$ aqt install-official 5.0.2 windows qtcreator_dev 64 --schema installer_source   # qtcreator/5.0/5.0.2/installer_source/windows_x64/qtcreator_dev.7z
$ aqt install-official 5.0.2 zip --schema opensource   # qtcreator/5.0/5.0.2/qt-creator-opensource-src-5.0.2.zip
$ aqt install-official 5.0.2 --schema installer_mac    # qtcreator/5.0/5.0.2/qt-creator-opensource-mac-x86_64-5.0.2_installer.dmg

This particular setup would assume that if you don't specify the schema, you want the binary. I'm not sure that's the right default behavior.

For your particular case, it looks like you're downloading specific 7z archives in the "installer_source" folder. Using this proposed interface, you would be able to download all of them using:

$ aqt install-official <version> <host> all <bits> --schema installer_source

Unresolved issues

There are some problems with this approach that I haven't figured out yet:

1. How are users supposed to figure out how to use this tool, or what's available with it? I think the tool might be almost unusable if I don't add a corresponding list-official command.
2. What should the subcommand be called? My choice, install-official, looks strange to me. install-extra looks better. However, if we extend the tool to allow downloads from the other folders in https://download.qt.io/ (snapshots, ministro, linguist_releases, etc.), we will need to pass it another argument for the folder name.
3. Writing a fast tool that lists available versions could be impossible. For example, QtCreator has 12 folders 4.5-5.0, and each of these folders contains several folders for each patch release (the folder 4.5 contains folders 4.5.0, 4.5.1, 4.5.2...). I can use bs4 to scrape the download.qt.io/official_releases/index.html file to discover every minor release, and then scrape the 12 folders 4.5-5.0 to find out what versions are available; that's 13 HTTP GET requests, and each one takes roughly 1 second on my PC. We can do some of these in parallel, but all that ends if the Qt server decides that 12 simultaneous GET requests from the same IP is malicious. A normal human being does not request that many web pages at once. Maybe I'm overthinking this, but I do not want to abuse the Qt servers.
4. I suspect that my json description will go out of date quickly, and could need revisions every time the Qt devs add or remove packages. I have no idea how I could update this file programmatically, like we did with combinations.json in CI.
5. I have not attempted to add pyside or QtForPython support; there's so much stuff in those folders it makes me want to cry. 😢

Would this proposed tool be useful to anyone? I would probably use it to install jom

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[christian-rauch]

While I think it is very useful to only download a subset of archives of a package, I agree that the command-line interface for querying different packages, archives, etc becomes quite complicated. Right now, the CLI is already complex because aqtinstall allows querying different operations systems, architectures and versions. Overall, I see why you are reluctant to add this functionality for users.

In the end, I did not use aqtinstall because I usually only want a givens version of Qt and Qt Creator for the operating system and architecture the script is running on. The implementation of this custom tool is quite small and it achieves what I need.

@ddalcino The short answer to your question

Would this proposed tool be useful to anyone?

is: Yes, I believe such a tool would be useful because I would have used it if the functionality would have been available. But I am not going to use it anymore, now that I implemented a custom script that does exactly what I need with less amount of code :-)

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days