Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiple parallel configure with Colcon and CPM: not supported #293

Closed
huand opened this issue Oct 7, 2021 · 5 comments
Closed

Multiple parallel configure with Colcon and CPM: not supported #293

huand opened this issue Oct 7, 2021 · 5 comments
Labels
enhancement New feature or request

Comments

@huand
Copy link

huand commented Oct 7, 2021

Hello,
Im not too sure this is the cause of the problem, but when using colcon (a build tool for ros2 packages), I think colcon starts all leaf dependencies config and build in "parallel", if there are multiple duplicate dependencies among those parallel build, CPM fails. Example, where MyProject1 is used by several leaf projects and downloaded by cpm:

[ 11%] Creating directories for 'MyProject1'
[ 22%] Performing download step (git clone) for 'MyProject1'
Cloning into 'fc4e6fc51c1494235cb68e60c5b7fb54c701a612'...
fatal: shallow file has changed since we read it
fatal: destination path 'fc4e6fc51c1494235cb68e60c5b7fb54c701a612' already exists and is not an empty directory.
fatal: destination path 'fc4e6fc51c1494235cb68e60c5b7fb54c701a612' already exists and is not an empty directory.
-- Had to git clone more than once:
          3 times.

however if i start the process again, then it works.
would it be possible for CPM to check if another CPM is already active for some deps and if another cpm process wants the same deps to just wait for the other to finish instead of failing? I understand this is probably a niche case for build system like Colcon.
@TheLartians
Copy link
Member

Hey, thanks for the issue! I don't know about how the colcon build tool works, but I think if we want to support parallel builds we would need to introduce a sort of lock for the CPM source cache directory. There will probably be some complexity involved if we want to avoid creating hangups and edge-cases.

Does deactivating the source cache, e.g. by calling unset CPM_SOURCE_CACHE before building, resolve the issue?

@egorodet
Copy link
Contributor

egorodet commented Jul 11, 2022 •
edited
Loading

Hi @TheLartians, I'm experiencing a similar issue in CLion IDE, which allows to work with multiple CMake configurations and runs cmake configure for active configurations in parallel. In such case it became impossible to use CPM_SOURCE_CACHE because of update collisions, so I had to fallback to separate source locations, which seems to be slowly working. I wish CPM would have a directory locking protection to allow parallel use cases.

@egorodet egorodet mentioned this issue Jul 18, 2022
Merged
egorodet added a commit to MethanePowered/MethaneKit that referenced this issue Jul 19, 2022
@egorodet
Merge pull request #98 from egorodet/develop
d95ef69 
Version 0.6.4: Hot-Fixes

- **Samples** apps:
  - Attempted replacement of `PerlinNoise` with `FastNoise2` library, but reverted it back because of no ARM support for M1 Macs (see [FastNoise2 issue](Auburn/FastNoise2#93))
- **Graphics** libraries:
  - Fixed runtime errors in GPU profiling builds with `METHANE_GPU_PROFILING_ENABLED=ON`:
    - DirectX GPU timestamps re-calibration issue leading to GPU ranges shifting was fixed.
    - Fixed command list execution waiting threads synchronisation in Profiling builds in Typography tutorial.
  - Fixed resources retaining in command lists. Retained resources were incorrectly cleared on `CommandList::Reset()`, while they should be cleared on `CommandList::Commit()`.
  - Fixed DirectX descriptor heaps allocations after `Context::Reset()` by always using deferred heap allocation in all cases. Deferred heap initialisation flag was removed, since it became unconditionally deferred.
  - Fixed sporadic hang in `CommandQueueTrackingBase::WaitForExecution()`
  - Fixed sporadic crash on destruction of `CommandQueueTrackingBase` with proper shutdown procedure called from destructor of derived class.
  - Fixed Vulkan build and some initialisation errors on MacOS.
- **Tests**:
  - All unit tests were updated to support breaking changes in Catch v3.
  - `Point<T,size>` wrapper class was extended with workarounds of MacOS & ARM specific bugs in HLSL++ integer vector comparison and division operators (see [1](redorav/hlslpp#60) and [2](redorav/hlslpp#61)).
  - Logging of `Point`, `Rect` and `RectSize` values was added in `DataTypes` unit tests.
- **External** libraries:
  - External dependencies management via Git submodules was replaced with [CPM.cmake](https://github.com/cpm-cmake/CPM.cmake) package manager. No submodules anymore - it greatly simplifies external library updates!
    - All externally dependent repositories are downloaded to CPM cache directory during CMake configuration stage, to `Build/Output/ExternalsCache` by default (it can be changed with CMake option `CPM_SOURCE_CACHE`).
    - When CMake project is configured under CLion, external repositories are downloaded to individual build directories of each configuration to workaround [parallel cache update collision issue of the CPM.cmake](cpm-cmake/CPM.cmake#293).
  - New `README.md` description of the external dependencies was added in `Externals` directory.
  - Almost all libraries were updated to latest version, except SPIRV-Cross and DirectX Shader Compiler pre-built binary tools.
- **Build** infrastructure:
  - Root `CMakeLists.txt` was simplified by moving all compiler configuration options to `CMake/MethaneBuildOptions.cmake`.
  - Use caching of Externals CPM package sources in Azure Pipelines and GitHub Workflows to speedup builds.
  - MacOS builders were switched from v10.15 to v11.
@TheLartians TheLartians added the enhancement New feature or request label Aug 3, 2022
@PercentBoat4164 PercentBoat4164 mentioned this issue Nov 27, 2022
Merged
@TheLartians
Copy link
Member

Is this fixed in v0.37.0 with #427?

@egorodet
Copy link
Contributor

Is this fixed in v0.37.0 with #427?

@TheLartians I've just tested v0.37 with multiple CMake configurations initializing dependencies in parallel in CLion and it worked just fine without any errors! Thanks a lot for fixing this painful issue!

@TheLartians TheLartians mentioned this issue Jan 28, 2023
Closed
@TheLartians
Copy link
Member

Great, thanks for testing! I close the issue for now as I assume this directly relates to the Colcon error as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@TheLartians @huand @egorodet