Add versions of curl between 7.73 and 7.30

[SirLynix]

Hi,

I was trying to add libcurl to my project and I ran into some issues as 7.64.1 has a test that doesn't compile on Windows because of an empty file. (btw, maybe we should disable test building in the cmake)

I tried to upgrade to another version but it seems versions have to be registered in the repository first, so I wrote a little script to fetch all versions from GitHub, download them and compute the SHA256 of the file.

I didn't go below 7.29 because it doesn't have a .tar.bz2 on the official website, and instead of fixing this I though there was enough curl versions for now.

Script for reference:

-- Import parameter option module
import("net.http")

-- Entrance function
function main(...)
	local githubPageFile = "github.html"
	local curlFile = "curl.tar.bz2"
	local resultFile = "result.lua"

	local result = io.open(resultFile, "w+")

	local page = 1

	local nextPage = "https://github.com/curl/curl/releases"
	repeat
		print("Downloading page " .. page .. " ...")
		http.download(nextPage, githubPageFile)
		local file = io.open(githubPageFile, "r")
		local content = file:read("*a")
		file:close()

		local pattern = [[<a href="/curl/curl/releases/tag/curl%-(%d+)_(%d+)_(%d+)">]]
		for major, minor, patch in content:gmatch(pattern) do
			print("Found version " .. major .. "." .. minor .. "." .. patch .. ", downloading...")
			local url = "https://curl.haxx.se/download/curl-" .. major .. "." .. minor .. "." .. patch .. ".tar.bz2"
			http.download(url, curlFile)
			local hash = hash.sha256(curlFile)
			print("SHA256: " .. hash)
			local lua = string.format([[add_versions("%d.%d.%d", "%s")]], major, minor, patch, hash)
			print(lua)

			result:write(lua .. "\n")
			result:flush()
		end

		pattern = [[href="(https://github%.com/curl/curl/releases%?after=curl%-%d+_%d+_%d+)">Next]]
		nextPage = content:match(pattern)

		page = page + 1
	until nextPage == nil
end```

[waruqi]

see xmake-io/xmake#1009

[SirLynix]

Even though I understand maintenance can become difficult, it still bother me a bit that only a few versions should be listed.

What if I reduce the list to the last 10 major versions of libcurl?

[waruqi]

You can reduce the list to the last 10 major versions or you can use add_requires("curl 7.67.0", {verify = false}) to use any version.

[SirLynix]

Yes thank you, this new feature fixed my problem, however I'm thinking about other users of xmake here. I don't understand how the version feature is supposed to work if we limit the number of known versions.

[waruqi]

Although I also want to support a complete version list, in order to consider file integrity, we must add the sha256 sum for each version, which will cause us to be unable to maintain a too long version list.

In order to consider maintainability, we must limit the number of version lists. I think that including the last 10 major versions can already meet most user needs.

[SirLynix]

I understand, although using a script makes it fairly easy to generate such a version list.

I will reopen a PR with the last ten major versions of libcurl.

[waruqi]

However, each package maintains a long list, which will also cause xmake-repo to become very large, affect the pull time, and also cause the package/xmake.lua content to be very bloated and poorly readable.

In addition, even with script automation, it will take a long time to download each package if the network is not good.

[SirLynix]

I understand your considerations.

However isn't having at least the good versions number (without hashs) better than not having them? I don't think it's possible nor a good thing to decide which versions xmake keeps and which xmake drop.

Wouldn't it be a good solution to split xmake.lua add_versions calls to a separate version.lua (and just includes it in the xmake.lua) if there's too many calls? It would keep the xmake.lua as readable as today.

Having shitty Internet I also understand your considerations about the downloading time however:

• This would be done only once per package version (a better script would not download already registered versions and hashs)
• It would be better to have versions listed (even without hash), I suppose xmake could show a warning to the user in case the hash is missing when installing, but still take it into account.
• Not all libraries have as many versions as curl has.
• You can ask xmake community to use the script themselves and give you pull requests (like this one :) ) to fill the blanks.

Please note that I understand and agrees with your considerations, I just think having only a few versions of each package instead of as many as humanly possible is a worse solution.

Thank you for this amazing software!

[waruqi]

I think sha256 verification is more important, at least it can ensure that the package will not be tampered with, and it can also ensure the integrity of the package. This is more safe and reliable for users.

And compared with homebrew/vcpkg only provides one version, xmake have the last 10 major versions available.

In addition, users can also choose to add verify = false parameter to choose an unsafe way to install any version package.

I think this is a better solution for now.

Not all libraries have as many versions as curl has.

But every package is constantly being updated and released, and there will be more and more.

[SirLynix]

Fair enough, I didn't know vcpkg worked the same way.

But doesn't that defeat semver range on the long run? Let's suppose I write something like add_requires("libcurl <=7.64") and 7.64 gets dropped in this repository someday (because we only keep the last 10 major versions available), it means I'll have to upgrade or force the libcurl version with a verifiy = false the day it happens or my project will automatically break (maybe not on my computer due to cache, but for new users).

[waruqi]

Then we can relax the restriction standards appropriately, keep all major versions, and keep ten minor versions for each major version.

for example:

8.0.0, 8.1.0, 8.2.0, ...
7.0.0, 7.1.0, 7.2.0, 7.3.0, 7.4.0, 7.5.0, 7.6.0, 7.7.0, 7.8.0, 7.9.0
6.0.0, 6.1.0, 6.2.0, ...

[SirLynix]

Seems good, but I suppose it depends a lot on the library release cycle (libcurl isn't really the standard about this).

How about splitting the versions in a separate versions.lua file when there's too many of them?

[waruqi]

How about splitting the versions in a separate versions.lua file when there's too many of them?

includes("versions.lua")

[SirLynix]

I meant how do you feel about it, as a solution to make the code readable. But I'll take it as a "no problem" 😃

[waruqi]

If you keep the major version and 10 minor versions, the version list is still very long. We can consider placing the version list in a separate versions.lua file.

[SirLynix]

About #104, how many versions should I keep? As many as there is in this PR (but in a separate file)?

[waruqi]

Considering that most versions of curl are 7.x, we can use 7.x0.00 as the major version and 7.xy.00 as the minor version.

xmake.lua

includes("versions.lua")
package("curl")
    add_urls()
    add_versions_list()

versions.lua

function add_versions_list()
    add_versions("7.73.0", "cf34fe0b07b800f1c01a499a6e8b2af548f6d0e044dca4a29d88a4bee146d131")
    add_versions("7.72.0", "ad91970864102a59765e20ce16216efc9d6ad381471f7accceceab7d905703ef")
    add_versions("7.71.0", "600f00ac2481a89548a4141ddf983fd9386165e1960bac91d0a1c81dca5dd341")
    add_versions("7.70.0", "a50bfe62ad67a24f8b12dd7fd655ac43a0f0299f86ec45b11354f25fbb5829d0")
    add_versions("7.69.0", "668d451108a7316cff040b23c79bc766e7ed84122074e44f662b8982f2e76739")
    add_versions("7.68.0", "207f54917dd6a2dc733065ccf18d61bb5bebeaceb5df49cd9445483e8623eeb9")
    add_versions("7.67.0", "dd5f6956821a548bf4b44f067a530ce9445cc8094fd3e7e3fc7854815858586c")
    add_versions("7.66.0", "6618234e0235c420a21f4cb4c2dd0badde76e6139668739085a70c4e2fe7a141")
    add_versions("7.65.0", "ea47c08f630e88e413c85793476e7e5665647330b6db35f5c19d72b3e339df5c")
    add_versions("7.64.0", "d573ba1c2d1cf9d8533fadcce480d778417964e8d04ccddcc76e591d544cf2eb")
    add_versions("7.63.0", "9bab7ed4ecff77020a312d84cc5fb7eb02d58419d218f267477a724a17fd8dd8")
    add_versions("7.62.0", "7802c54076500be500b171fde786258579d60547a3a35b8c5a23d8c88e8f9620")
    add_versions("7.61.0", "5f6f336921cf5b84de56afbd08dfb70adeef2303751ffb3e570c936c6d656c9c")
    add_versions("7.60.0", "897dfb2204bd99be328279f88f55b7c61592216b0542fcbe995c60aa92871e9b")
    add_versions("7.59.0", "b5920ffd6a8c95585fb95070e0ced38322790cb335c39d0dab852d12e157b5a0")
    add_versions("7.58.0", "1cb081f97807c01e3ed747b6e1c9fee7a01cb10048f1cd0b5f56cfe0209de731")
    add_versions("7.57.0", "c92fe31a348eae079121b73884065e600c533493eb50f1f6cee9c48a3f454826")
    add_versions("7.56.0", "de60a4725a3d461c70aa571d7d69c788f1816d9d1a8a2ef05f864ce8f01279df")
    add_versions("7.55.0", "af1d69ec6f15fe70a2cabaa98309732bf035ef2a735e4e1a3e08754d2780e5b1")
    add_versions("7.54.0", "f50ebaf43c507fa7cc32be4b8108fa8bbd0f5022e90794388f3c7694a302ff06")
    add_versions("7.53.0", "b2345a8bef87b4c229dedf637cb203b5e21db05e20277c8e1094f0d4da180801")
    add_versions("7.52.0", "b9a2e18b4785eb75ad84598720e1559e1c53550ea011c0e00becdb94e2df5cc6")
    add_versions("7.51.0", "7f8240048907e5030f67be0a6129bc4b333783b9cca1391026d700835a788dde")
    add_versions("7.50.0", "608dfe2db77f48db792c387e7791aca55a25f0b42385707ad927164199ecfa9a")
    add_versions("7.49.0", "14f44ed7b5207fea769ddb2c31bd9e720d37312e1c02315def67923a4a636078")
    add_versions("7.48.0", "864e7819210b586d42c674a1fdd577ce75a78b3dda64c63565abe5aefd72c753")
    add_versions("7.47.0", "2b096f9387fb9b2be08d17e518c62b6537b1f4d4bb59111d5b4fa0272f383f66")
    add_versions("7.46.0", "b7d726cdd8ed4b6db0fa1b474a3c59ebbbe4dcd4c61ac5e7ade0e0270d3195ad")
    add_versions("7.45.0", "65154e66b9f8a442b57c436904639507b4ac37ec13d6f8a48248f1b4012b98ea")
    add_versions("7.44.0", "1e2541bae6582bb697c0fbae49e1d3e6fad5d05d5aa80dbd6f072e0a44341814")
    add_versions("7.43.0", "baa654a1122530483ccc1c58cc112fec3724a82c11c6a389f1e6a37dc8858df9")
    add_versions("7.42.0", "32557d68542f5c6cc8437b5b8a945857b4c5c6b6276da909e35b783d1d66d08f")
    add_versions("7.41.0", "9f8b546bdc5c57d959151acae7ce6610fe929d82b8d0fc5b25a3a2296e5f8bea")
    add_versions("7.40.0", "899109eb3900fa6b8a2f995df7f449964292776a04763e94fae640700f883fba")
    add_versions("7.39.0", "b222566e7087cd9701b301dd6634b360ae118cc1cbc7697e534dc451102ea4e0")
    add_versions("7.38.0", "035bd41e99aa1a4e64713f4cea5ccdf366ca8199e9be1b53d5a043d5165f9eba")
    add_versions("7.37.0", "24502492de3168b0556d8e1a06f14f7589e57b204917d602a572e14239b3e09e")
    add_versions("7.36.0", "1fbe82b89bcd6b7ccda8cb0ff076edc60e911595030e27689f4abd5ef7f3cfcd")
    add_versions("7.35.0", "d774d1701454f1b7d331c2075fc4f6dd972bddc2d171f43645ef3647c7fc0d83")
    add_versions("7.34.0", "10beade56b48311499e444783df3413405b22f20a147ed4a1d8a8125f1cc829b")
    add_versions("7.33.0", "0afde4cd949e2658eddc3cda675b19b165eea1af48ac5f3e1ec160792255d1b3")
    add_versions("7.32.0", "8e3db42548e01407cb2f1407660c0f528b89ec7afda6264442fc2b229b95223b")
    add_versions("7.31.0", "a73b118eececff5de25111f35d1d0aafe1e71afdbb83082a8e44d847267e3e08")
    add_versions("7.30.0", "6b1c410387bea82601baec85d6aa61955794672e36766407e99ade8d55aaaf11")
end

[waruqi]

You can continue to modify the version list in #104 .