[bug] Wrong context applied to build_requires when using lockfiles

[stefansli]

The bison package has a build_requires dependency on flex. When trying to use lockfiles to build the required packages in the correct order I get mismatch between package-id and locked package-id (see log below).

As you can see from the lockfile, the flex package is marked to be in the build context (x86_64). But when trying to build the dependency using this lockfile, for some reason the host context is applied when calculating the package-id (see the conan info log below).

Environment Details (include every applicable attribute)

• Operating System+version: Windows 10
• Conan version: 1.47.0, 1.48.0 from chocolatey

Steps to reproduce (Include if Applicable)

Use the following lockfile:

{
 "graph_lock": {
  "nodes": {
   "1": {
    "ref": "bison/3.7.6#448fa7d0feeabb38324eda4681892f81",
    "options": "fPIC=True",
    "package_id": "ae4c50636a6d49e6d7b04a5fbfeacfe182f704d4",
    "requires": [
     "2"
    ],
    "build_requires": [
     "4"
    ],
    "context": "host"
   },
   "2": {
    "ref": "m4/1.4.18#062a12c8324b342d46e17b5a5b258cc9",
    "options": "",
    "package_id": "6173abd988ee2e3159da6cce836868055fdb1cb4",
    "context": "build"
   },
   "4": {
    "ref": "flex/2.6.4#4ee653bf6e3f8dee58fe7e8b74368f6c",
    "options": "fPIC=True\nshared=False",
    "package_id": "d4e2c4fda75c443daaf5f26224d0044231fb93f3",
    "requires": [
     "2"
    ],
    "context": "build"
   }
  },
  "revisions_enabled": true
 },
 "version": "0.4",
 "profile_host": "[settings]\narch=x86\nbuild_type=Release\ncompiler=gcc\ncompiler.libcxx=libstdc++\ncompiler.version=5\nos=Linux\n[options]\n[build_requires]\n[env]\nCC=gcc-5\nCXX=g++-5",
 "profile_build": "[settings]\narch=x86_64\nbuild_type=Release\ncompiler=gcc\ncompiler.libcxx=libstdc++\ncompiler.version=5\nos=Linux\n[options]\n[build_requires]\n[env]\nCC=gcc-5\nCXX=g++-5"
}

Logs (Executed commands with output) (Include/Attach if Applicable)

$ conan install flex/2.6.4@#4ee653bf6e3f8dee58fe7e8b74368f6c -b flex --lockfile=lockfile.lock 
Using lockfile: 'lockfile.lock'
Configuration (profile_host):
[settings]
arch=x86
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=5
os=Linux
[options]
[build_requires]
[env]
CC=gcc-5
CXX=g++-5
Configuration (profile_build):
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=5
os=Linux
[options]
[build_requires]
[env]
CC=gcc-5
CXX=g++-5
flex/2.6.4: Forced build from source
ERROR: 'flex/2.6.4#4ee653bf6e3f8dee58fe7e8b74368f6c' package-id 'ae4c50636a6d49e6d7b04a5fbfeacfe182f704d4' doesn't match the locked one 'd4e2c4fda75c443daaf5f26224d0044231fb93f3'

Output of conan info:

$ conan info flex/2.6.4@#4ee653bf6e3f8dee58fe7e8b74368f6c -pr:h linux-x86-gcc-5-libstdc++-release
flex/2.6.4
    ID: ae4c50636a6d49e6d7b04a5fbfeacfe182f704d4

Expected package-id:

$ conan info flex/2.6.4@#4ee653bf6e3f8dee58fe7e8b74368f6c -pr:h linux-x86_64-gcc-5-libstdc++-release 
flex/2.6.4
    ID: d4e2c4fda75c443daaf5f26224d0044231fb93f3

[memsharded]

Hi @vectorsli

please try to add the --build-require to the conan install command, that is necessary to define that such package is in the build context while installing, it could also be in the host context.

Note, however, that it might fail with error: ERROR: Cannot build 'flex/2.6.4#e4696e6333f2d486e7073de6a9559712' because it is already locked in the input lockfile```, as the binary is locked in the lockfile. You should do --build=flex`` too while capturing the lockfile if you want to be able to build it later.

I would also recommend, if you haven't yet, to have a look to the 2.0 proposal for lockfiles, which are already available in the released 2.0-alpha.6: conan-io/tribe#34

[stefansli]

@memsharded
Thanks for the quick answer. I'm building a simple build orchestration and I was trying to keep it as simple as possible, following the guide for lockfiles and lockfile bundles (we try to use bundles for the build-order).

If I understand this correctly, after determining the lockfile I'll use to build the package I have to also check the context and then apply the correct flag? If the flag is required since host is the default context, then I'd expect a warning/error if the specified reference is not found in the lockfile.

Just for reference, I created the lockfile using this command:

conan lock create --ref=bison/3.7.6@#448fa7d0feeabb38324eda4681892f81 -pr:h linux-x86-gcc-5-libstdc++-release -pr:b linux-x86_64-gcc-5-libstdc++-release --lockfile=base.lock --lockfile-out=lockfile.lock --build=missing

I tried your suggestion with --build-require but this didn't work either:

$ conan install flex/2.6.4@#4ee653bf6e3f8dee58fe7e8b74368f6c -b flex --lockfile=lockfile.lock --build-require
ERROR: 'm4/1.4.18#062a12c8324b342d46e17b5a5b258cc9' package-id '24647d9fe8ec489125dfbae4b3ebefaf7581674c' doesn't match the locked one '6173abd988ee2e3159da6cce836868055fdb1cb4'

So this is again a very special case since both flex and bison have a dependency on m4. For this specific graph I'd expect two instances of m4 to be generated into the lockfile, one in the build context and one for the host context. For some reason this is not the case in my example.

I'll have a look at the lockfiles 2.0

[memsharded]

If I understand this correctly, after determining the lockfile I'll use to build the package I have to also check the context and then apply the correct flag? If the flag is required since host is the default context, then I'd expect a warning/error if the specified reference is not found in the lockfile.

But that information is returned by the build-order, isn't it? It is not something that you need to guess.

So this is again a very special case since both flex and bison have a dependency on m4. For this specific graph I'd expect two instances of m4 to be generated into the lockfile, one in the build context and one for the host context. For some reason this is not the case in my example.

This is weird. I didn't report it, but on my side, when I am reproducing, I am actually getting 2 instances of m4 in the graph:

{
 "graph_lock": {
  "nodes": {
   "1": {
    "ref": "bison/3.7.6#de3449489624dbc45cfb8a868818def8",
    "options": "fPIC=True",
    "package_id": "ae4c50636a6d49e6d7b04a5fbfeacfe182f704d4",
    "requires": [
     "2"
    ],
    "build_requires": [
     "3"
    ],
    "context": "host"
   },
   "2": {
    "ref": "m4/1.4.19#d9741f0aa0ac45e6b54a59f79e32ac81",
    "options": "",
    "package_id": "6173abd988ee2e3159da6cce836868055fdb1cb4",
    "context": "host"
   },
   "3": {
    "ref": "flex/2.6.4#e4696e6333f2d486e7073de6a9559712",
    "options": "fPIC=True\nshared=False",
    "package_id": "d4e2c4fda75c443daaf5f26224d0044231fb93f3",
    "prev": "4d19984729c3933e2ad93991a7bf6b6a",
    "requires": [
     "4"
    ],
    "context": "build"
   },
   "4": {
    "ref": "m4/1.4.19#d9741f0aa0ac45e6b54a59f79e32ac81",
    "options": "",
    "package_id": "24647d9fe8ec489125dfbae4b3ebefaf7581674c",
    "prev": "9d688c9ef4c12a69e6f0824ebc6848bb",
    "context": "build"
   }
  },
  "revisions_enabled": true
 },
 "version": "0.4",
 "profile_host": 
}

I am trying to think what could be the difference in your case, but I am struggling to find a reason...

[memsharded]

This referred to old Conan 1.X lockfiles. Lockfiles have been fully revamped in Conan 2.0, I am closing this ticket as outdated, please create new tickets referring to the new lockfiles for any further question or issue. Thanks!