Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package "lock file" support to freeze dependencies #1567

Closed
jorgenpt opened this issue Aug 9, 2021 · 12 comments
Closed

Package "lock file" support to freeze dependencies #1567

jorgenpt opened this issue Aug 9, 2021 · 12 comments

Comments

@jorgenpt
Copy link

jorgenpt commented Aug 9, 2021

Is your feature request related to a problem? Please describe.

When I check in requirement for my project, I want to make sure that everyone else (including the build server) use the exact same version until we intentionally upgrade. Using add_requirement("foo ~1.2") is helpful to constrain the version, but I don't want xmake to upgrade the package unless I explicitly e.g. run xmake upgrade-requirements. If xrepo adds support for transitive dependencies (i.e. my package can depend on a different package), I'd like to be able to stick with a specific version of those dependencies as well.

I also want to make sure that the dependencies haven't been tampered with, so I'd like some guarantee that the checksums in the package lua file haven't been modified without bumping the version.

Describe the solution you'd like

Many package managers (npm w/ package.lock, cargo w/ cargo.lock, pip freeze w/ requirements.txt, etc) use a .lock file that specifies more specific information about the dependencies I've installed.

For xrepo, this could be a file that contains URLs to all my packages' xmake.lua files, the version of the packages, and the specific commit hash of the xmake.lua file that should be used. By default, installing requirements via e.g. xmake should default to using the specific package listed in the lock file if it is present, otherwise it should use the current rules to determine the version and record it in a .lock file. Then there should be an explicit command to update my packages that would re-evaluate any requirements listed in the xmake.lua and update xmake.lock to the latest version of the package that meets the requirements and the latest commit of the package's xmake.lua file.

@waruqi waruqi transferred this issue from xmake-io/xrepo Aug 9, 2021
@waruqi waruqi added this to the v2.5.7 milestone Aug 10, 2021
@waruqi
Copy link
Member

waruqi commented Aug 19, 2021

I have supported it in #1587

We need only add set_policy("package.requires_lock", true) to enable xmake-requires.lock file

add_requires("zlib")
set_policy("package.requires_lock", true)

target("test1")
    set_kind("binary")
    add_files("src/*.c")
    add_packages("zlib")

We can use xmake-requires.lock to lock packages for each platform/archs, it will also lock their repositories with the special commit/branch and their install scripts.

$ cat ./xmake-requires.lock
{
    __meta__ = {
        version = "1.0"
    },
    ["zlib#ddbdd8b8f76d455a8c0ef45fffe37160"] = {
        arch = "x86_64",
        buildhash = "b76a297309d14c09b42cfe3927260a51",
        name = "zlib",
        plat = "macosx",
        repo = {
            branch = "master",
            commit = "df6cad5cf5701de3705c7afce54d732e1f5b14b9",
            url = "https://gitee.com/tboox/xmake-repo.git"
        },
        version = "1.2.11"
    },
    ["zlib~debug#d72a8a937b694e449bce540c8eb65a8a"] = {
        arch = "x86_64",
        buildhash = "e6a6fa6a86ad4fc7ae3c2cf7c60cc084",
        name = "zlib",
        plat = "macosx",
        repo = {
            branch = "master",
            commit = "df6cad5cf5701de3705c7afce54d732e1f5b14b9",
            url = "https://gitee.com/tboox/xmake-repo.git"
        },
        version = "1.2.11"
    },
    ["zlib~shared#b286744a1898416faad2e5cab0104b2f"] = {
        arch = "x86_64",
        buildhash = "8e5b898b0f344715bac89cc6a1577506",
        name = "zlib",
        plat = "macosx",
        repo = {
            branch = "master",
            commit = "df6cad5cf5701de3705c7afce54d732e1f5b14b9",
            url = "https://gitee.com/tboox/xmake-repo.git"
        },
        version = "1.2.11"
    }
}

@waruqi
Copy link
Member

waruqi commented Aug 19, 2021

and we can run xmake require --upgrade to upgrade packages and update xmake-requires.lock

@waruqi waruqi closed this as completed Aug 20, 2021
@SirLynix
Copy link
Member

I noticed xmake only writes the lock file when installing dependencies, but what if the dependencies are already installed (from another project per example)?

@SirLynix
Copy link
Member

Also there may be a problem with local repositories:

-- from my project xmake.lua, containing nazaraengine package
add_repositories("burgwar-repo xmake-repo")

it gives this in xmake lock file:

    ["nazaraengine 2021.06.10#7636c10f9dfd4b49b85e757089e235b3"] = {
        arch = "x64",
        buildhash = "e14b1a7c2240427a8515d0036dfb4267",
        name = "nazaraengine",
        plat = "windows",
        repo = {
            commit = "58c748891dac51f86fb2715aafa69da2d66adb71",
            url = "xmake-repo"
        },
        version = "2021.06.10"
    },

then resetting xmake cache:

lynix@SirLynixVanDesktop:/mnt/c/Projets/Burgwar/BurgWar$ xmake.exe f -c
checking for platform ... windows
checking for architecture ... x64
checking for Microsoft Visual Studio (x64) version ... 2019
error: Cloning into 'C:\Projets\Burgwar\BurgWar\.xmake\windows\x64\repositories\57fd85b63a814e0cb82668796999d0bd.lock'..
.
fatal: 'C:\Projets\Burgwar\BurgWar\xmake-repo' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

@waruqi
Copy link
Member

waruqi commented Aug 21, 2021

I have improved it.

@SirLynix
Copy link
Member

It works better, however I still have this issue:

1>update vsxmake project ..
1>checking for releasedbg.x64 ...
1>checking for Microsoft Visual Studio (x64) version ... 2019
1>EXEC : error : Cloning into 'C:\Projets\Burgwar\BurgWar\.xmake\windows\x64\repositories\57fd85b63a814e0cb82668796999d0bd.lock'...
1>fatal: 'C:\Projets\Burgwar\BurgWar\xmake-repo' does not appear to be a git repository
1>fatal: Could not read from remote repository.
1>
1>Please make sure you have the correct access rights
1>and the repository exists.
1>
1>EXEC : warning : package(cxxopts): buildhash is not matched in xmake-requires.lock
1>EXEC : warning : add -v for getting more warnings ..
1>C:\Program Files\xmake\plugins\project\vsxmake\vsproj\Xmake.targets(103,5): error MSB3073: La commande "..." s'est arrêtée avec le code -1.

With this in the lock file:

        ["nazaraengine~server 2021.06.10#e1c6474d"] = {
            buildhash = "2169054d0b36425faff95e9c7e162d05",
            repo = {
                url = "xmake-repo"
            },
            version = "2021.06.10"
        },

@waruqi
Copy link
Member

waruqi commented Aug 22, 2021

try it again.

@SirLynix
Copy link
Member

SirLynix commented Aug 22, 2021

Seems better, I no longer have the error. However I have a lot of "build hash does not match" error (which doesn't make sense since I generated the lock right before using xmake).

Here's the full verbose log:

lynix@SirLynixVanDesktop:/mnt/c/Projets/Burgwar/BurgWar$ xmake.exe project -vk vsxmake
checking for platform ... windows
checking for architecture ... x64
checking for vswhere.exe ... C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe
checking for cl.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Host
X64\x64\cl.exe
checking for Microsoft Visual Studio (x64) version ... 2019
checking for dmd ... no
checking for ldc2 ... no
checking for gdc ... no
checking for zig ... no
checking for zig ... no
checking for unzip ... no
checking for 7z ... C:\Program Files\xmake\winenv\bin\7z
checking for git ... ok
checking for gzip ... no
git checkout eda7adee81bac151f87c507030cc0dd8ab299462
HEAD is now at eda7ade update automake (#583)
git rev-parse HEAD
git checkout eda7adee81bac151f87c507030cc0dd8ab299462
HEAD is now at eda7ade update automake (#583)
git checkout eda7adee81bac151f87c507030cc0dd8ab299462
HEAD is now at eda7ade update automake (#583)
git checkout d0e1f98585a4ec31b056fe3bda65a4fe37e0b375
Warning: you are leaving 3 commits behind, not connected to
any of your branches:

  eda7ade update automake (#583)
  4498f11 update imgui (#582)
  a3a520f Chipmunk fixes (#581)

If you want to keep them by creating a new branch, this may be a good time
to do so with:

 git branch <new-branch-name> eda7ade

HEAD is now at d0e1f98 fix style
git checkout eda7adee81bac151f87c507030cc0dd8ab299462
Previous HEAD position was d0e1f98 fix style
HEAD is now at eda7ade update automake (#583)
git checkout eda7adee81bac151f87c507030cc0dd8ab299462
HEAD is now at eda7ade update automake (#583)
git checkout eda7adee81bac151f87c507030cc0dd8ab299462
HEAD is now at eda7ade update automake (#583)
git checkout eda7adee81bac151f87c507030cc0dd8ab299462
HEAD is now at eda7ade update automake (#583)
git checkout eda7adee81bac151f87c507030cc0dd8ab299462
HEAD is now at eda7ade update automake (#583)
git checkout eda7adee81bac151f87c507030cc0dd8ab299462
HEAD is now at eda7ade update automake (#583)
git checkout eda7adee81bac151f87c507030cc0dd8ab299462
HEAD is now at eda7ade update automake (#583)
checking for xmake::cxxopts ... cxxopts v2.2.0
checking for xmake::concurrentqueue ... concurrentqueue master
checking for xmake::hopscotch-map ... hopscotch-map v2.3.0
checking for xmake::nlohmann_json ... nlohmann_json v3.10.0
checking for xmake::tl_expected ... tl_expected v1.0.0
checking for xmake::tl_function_ref ... tl_function_ref v1.0.0
checking for cmake ... C:\Program Files\CMake\bin\cmake
checking for cmake ... C:\Program Files\CMake\bin\cmake
checking for xmake::fmt ... fmt 8.0.1
checking for xmake::libcurl ... libcurl 7.73.0
checking for xmake::nazaraengine ... nazaraengine 2021.06.10
checking for xmake::nazaraengine ... nazaraengine 2021.06.10
checking for xmake::sol2 ... sol2 v3.2.1
checking for xmake::stackwalker ... stackwalker master
checking for qmake ... C:\Projets\Libs\Qt\5.15.0\msvc2019_64\bin\qmake
checking for Qt SDK directory ... C:/Projets/Libs/Qt/5.15.0/msvc2019_64
checking for Qt SDK version ... 5.15.0
checking for link.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Ho
stX64\x64\link.exe
checking for the shared library linker (sh) ... link.exe
checking for link.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Ho
stX64\x64\link.exe
checking for the linker (ld) ... link.exe
checking for link.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Ho
stX64\x64\link.exe
checking for the static library archiver (ar) ... link.exe
configure
{
    menu = true
    host = windows
    arch = x64
    corelib_static = false
    clientlib_static = false
    buildir = build
    pkg_searchdirs = Azure-Kinect-Sensor-SDK
    mode = debug
    plat = windows
    network = public
    qt = C:/Projets/Libs/Qt/5.15.0/msvc2019_64
    build_mapeditor = true
    ndk_stdcxx = true
    kind = static
    mingw = C:/Projets/Softs/mingw64-10.3
    theme = default
    ccache = true
    vs = 2019
    proxy_pac = pac.lua
    qt_sdkver = 5.15.0
}
using project kind vs2019
checking for releasedbg.x64 ...
checking for vswhere.exe ... C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe
checking for cl.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Host
X64\x64\cl.exe
checking for Microsoft Visual Studio (x64) version ... 2019
checking for dmd ... no
checking for ldc2 ... no
checking for gdc ... no
checking for zig ... no
checking for zig ... no
checking for unzip ... no
checking for 7z ... C:\Program Files\xmake\winenv\bin\7z
checking for git ... ok
checking for gzip ... no
git checkout eda7adee81bac151f87c507030cc0dd8ab299462
HEAD is now at eda7ade update automake (#583)
checking for xmake::cxxopts ... cxxopts v2.2.0
checking for xmake::concurrentqueue ... concurrentqueue master
checking for xmake::hopscotch-map ... hopscotch-map v2.3.0
checking for xmake::nlohmann_json ... nlohmann_json v3.10.0
checking for xmake::tl_expected ... tl_expected v1.0.0
checking for xmake::tl_function_ref ... tl_function_ref v1.0.0
checking for cmake ... C:\Program Files\CMake\bin\cmake
checking for cmake ... C:\Program Files\CMake\bin\cmake
checking for xmake::fmt ... fmt 8.0.1
checking for xmake::libcurl ... libcurl 7.73.0
checking for xmake::nazaraengine ... nazaraengine 2021.06.10
checking for xmake::nazaraengine ... nazaraengine 2021.06.10
checking for xmake::sol2 ... sol2 v3.2.1
checking for xmake::stackwalker ... stackwalker master
checking for qmake ... C:\Projets\Libs\Qt\5.15.0\msvc2019_64\bin\qmake
checking for Qt SDK directory ... C:/Projets/Libs/Qt/5.15.0/msvc2019_64
checking for Qt SDK version ... 5.15.0
checking for link.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Ho
stX64\x64\link.exe
checking for the shared library linker (sh) ... link.exe
checking for link.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Ho
stX64\x64\link.exe
checking for the linker (ld) ... link.exe
checking for link.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Ho
stX64\x64\link.exe
checking for the static library archiver (ar) ... link.exe
checking for asan.x64 ...
checking for vswhere.exe ... C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe
checking for cl.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Host
X64\x64\cl.exe
checking for Microsoft Visual Studio (x64) version ... 2019
checking for dmd ... no
checking for ldc2 ... no
checking for gdc ... no
checking for zig ... no
checking for zig ... no
checking for unzip ... no
checking for 7z ... C:\Program Files\xmake\winenv\bin\7z
checking for git ... ok
checking for gzip ... no
checking for xmake::cxxopts ... cxxopts v2.2.0
checking for xmake::concurrentqueue ... concurrentqueue master
checking for xmake::hopscotch-map ... hopscotch-map v2.3.0
checking for xmake::nlohmann_json ... nlohmann_json v3.10.0
checking for xmake::tl_expected ... tl_expected v1.0.0
checking for xmake::tl_function_ref ... tl_function_ref v1.0.0
checking for cmake ... C:\Program Files\CMake\bin\cmake
checking for cmake ... C:\Program Files\CMake\bin\cmake
checking for xmake::fmt ... fmt 8.0.1
checking for xmake::libcurl ... libcurl 7.73.0
checking for xmake::nazaraengine ... nazaraengine 2021.06.10
checking for xmake::nazaraengine ... nazaraengine 2021.06.10
checking for xmake::sol2 ... sol2 v3.2.1
checking for xmake::stackwalker ... stackwalker master
checking for qmake ... C:\Projets\Libs\Qt\5.15.0\msvc2019_64\bin\qmake
checking for Qt SDK directory ... C:/Projets/Libs/Qt/5.15.0/msvc2019_64
checking for Qt SDK version ... 5.15.0
checking for link.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Ho
stX64\x64\link.exe
checking for the shared library linker (sh) ... link.exe
checking for link.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Ho
stX64\x64\link.exe
checking for the linker (ld) ... link.exe
checking for link.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Ho
stX64\x64\link.exe
checking for the static library archiver (ar) ... link.exe
checking for debug.x64 ...
checking for vswhere.exe ... C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe
checking for cl.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Host
X64\x64\cl.exe
checking for Microsoft Visual Studio (x64) version ... 2019
checking for dmd ... no
checking for ldc2 ... no
checking for gdc ... no
checking for zig ... no
checking for zig ... no
checking for unzip ... no
checking for 7z ... C:\Program Files\xmake\winenv\bin\7z
checking for git ... ok
checking for gzip ... no
checking for xmake::cxxopts ... cxxopts v2.2.0
checking for xmake::concurrentqueue ... concurrentqueue master
checking for xmake::hopscotch-map ... hopscotch-map v2.3.0
checking for xmake::nlohmann_json ... nlohmann_json v3.10.0
checking for xmake::tl_expected ... tl_expected v1.0.0
checking for xmake::tl_function_ref ... tl_function_ref v1.0.0
checking for cmake ... C:\Program Files\CMake\bin\cmake
checking for cmake ... C:\Program Files\CMake\bin\cmake
checking for xmake::fmt ... fmt 8.0.1
checking for xmake::libcurl ... libcurl 7.73.0
checking for xmake::nazaraengine ... nazaraengine 2021.06.10
checking for xmake::nazaraengine ... nazaraengine 2021.06.10
checking for xmake::sol2 ... sol2 v3.2.1
checking for xmake::stackwalker ... stackwalker master
checking for qmake ... C:\Projets\Libs\Qt\5.15.0\msvc2019_64\bin\qmake
checking for Qt SDK directory ... C:/Projets/Libs/Qt/5.15.0/msvc2019_64
checking for Qt SDK version ... 5.15.0
checking for link.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Ho
stX64\x64\link.exe
checking for the shared library linker (sh) ... link.exe
checking for link.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Ho
stX64\x64\link.exe
checking for the linker (ld) ... link.exe
checking for link.exe ... C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\Ho
stX64\x64\link.exe
checking for the static library archiver (ar) ... link.exe
create ok!
warning: package(cxxopts): buildhash is not matched in xmake-requires.lock
warning: package(concurrentqueue): buildhash is not matched in xmake-requires.lock
warning: package(nlohmann_json): buildhash is not matched in xmake-requires.lock
warning: package(fmt): buildhash is not matched in xmake-requires.lock
warning: package(libcurl): buildhash is not matched in xmake-requires.lock
warning: package(nazaraengine): buildhash is not matched in xmake-requires.lock
warning: package(nazaraengine~server): buildhash is not matched in xmake-requires.lock
warning: package(stackwalker): buildhash is not matched in xmake-requires.lock

Also it looks like it does unnecessary checkout the repo multiple times to the same hash

@waruqi
Copy link
Member

waruqi commented Aug 23, 2021

I have improved it on dev.

@SirLynix
Copy link
Member

It works without any problem now.
Is there a reason why the buildhash was removed/not working in the first place?

I tried it by forcing fmt version to 7.1.3 and then running xmake require --upgrade which set it to the last version (8.0.1), I just feel it's lacking a feedback.

lynix@SirLynixVanDesktop:/mnt/c/Projets/Burgwar/BurgWar$ xmake.exe require --upgrade
checking for platform ... windows
checking for architecture ... x64
checking for Microsoft Visual Studio (x64) version ... 2019
checking for Qt SDK directory ... C:/Projets/Libs/Qt/5.15.0/msvc2019_64
checking for Qt SDK version ... 5.15.0
updating repositories .. ok

I feel it would be better with a message "upgraded fmt (7.1.3 => 8.0.1)", if that's not too much work 😄

@waruqi
Copy link
Member

waruqi commented Aug 23, 2021

Is there a reason why the buildhash was removed/not working in the first place?

At present, we can only calculate the key of the locked package based on the earliest requires info. At this time, the package configs have not been finalized, so we cannot correspond to the exact package configs and buildhash based on this key.

But this does not affect the lock of the package version and warehouse.

@waruqi
Copy link
Member

waruqi commented Aug 23, 2021

I feel it would be better with a message "upgraded fmt (7.1.3 => 8.0.1)", if that's not too much work 😄

I added it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants