Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The whole system depends solely on looo #158

Open
adrianinsaval opened this issue Mar 22, 2023 · 41 comments
Open

The whole system depends solely on looo #158

adrianinsaval opened this issue Mar 22, 2023 · 41 comments

Comments

@adrianinsaval
Copy link
Member

For some weeks now the conda builds have been stuck, I've been asking around trying to find the problem and solutions but looo seems to not have much availability lately (can't blame him for that) so things move slow, it might be the case that we manage to fix this without his intervention or that later he has time and can get it working again, but this is very fragile and completely depends on this one person.

The actual compilation of freecad for conda even happens on a personal repo from him: https://github.com/looooo/freecad-feedstock
so if something needs fixing there we can't even push changes without his intervention.

I also don't know who else has access to the anaconda freecad account, is it also just him?
https://anaconda.org/freecad

and how does the uploading of those packages work? how to set up another repo to upload stuff there, how is the authentication done and who can approve it?

What can we do to improve this? Can we move the feedstocks repos for the packages in that account into the github org account?

@adrianinsaval
Copy link
Member Author

ping @looooo @yorikvanhavre

@luzpaz
Copy link
Collaborator

luzpaz commented Mar 22, 2023

looooo is doing a tremendous job AND we do need more people involved that understand the whole conda workflow.

@yorikvanhavre
Copy link
Member

I'd be all for moving stuff to the FreeCAD org repo, if @looooo finds it a good idea. But it won't magically make things go faster.
Usually "one-person" tasks are because someone volunteers for something, and nobody else wants to do it... We have a worse situation with the Ubuntu PPA package, that basically nobody wants to maintain :/ So if anyone else volunteers to help, that can only do good... Would you be interested in helping, @adrianinsaval ?

@yorikvanhavre
Copy link
Member

I wonder if the whole packaging area woudn't be something where the FPA could help...

@JohnOCFII
Copy link

Between this thread https://forum.freecad.org/viewtopic.php?t=76777&start=20 and this thread https://forum.freecad.org/viewtopic.php?t=76795&start=10 user "oursland" may be able to help. I've been able to build FreeCAD on macOS ARM using his conda recipe.

@luzpaz
Copy link
Collaborator

luzpaz commented Mar 22, 2023

We do need a packaging team that has a way to communicate. For obvious reasons but specifically to keep all builds and their dependencies in unison (a quick example would be keeping all OCC libs ideally be at the same version across all permutations)

@adrianinsaval
Copy link
Member Author

adrianinsaval commented Mar 22, 2023

Well yes I am trying to help where I can with conda (but don't have enough knowledge to be a proper maintainer)
I know moving repos to the org account won't immediately make things faster, but it would mean someone can take over when looo is absent, as opposed to the current situation where we're left with nothing.

I'm open to accepting a role as a fallback guy for merging PRs or other mundane activities.

An additional benefit is that it makes those repos discoverable, I can't even remember how I found that compilation is happening from his repo but it sure as hell wasn't straightforward to find! A discoverable repo might attract more contributors.

@luzpaz
Copy link
Collaborator

luzpaz commented Mar 22, 2023

User ppd was documenting how to maintain the Snap packaging. But it still incomplete. Nevertheless, it has been somewhat helpful, for example at release time. I think it would be important to document the workflow of all package building/managing (PPD, Conda (AppImage, Windows Installer, macOS), Snap, Flatpak, Docker (GUI, CLI)).

@adrianinsaval
Copy link
Member Author

case in point: https://forum.freecad.org/viewtopic.php?p=670177#p670177

Found the problem, an fmt package is being installed (likely as dependency of something else) when building the linux and macOS conda packages but it is not installed when installing the freecad package in conda, so it is then missing at runtime causing errors, which makes the bundling process fail.

On windows fmt is not installed during the build process or at least not detected by cmake, so it downloads and builds fmt from source so it's packaged with freecad so there's no error at runtime so the bundling works.

Made a PR to the feedstock repo with what I think is how to add fmt as dependency:
https://github.com/looooo/freecad-feedstock/pull/13

but I don't know if looo will be available to merge it, so I also made a PR to the bundle repo as workaround for macOS and linux:
#160

we need to merge a change in looo's repo to fix current issues the proper way, if he's not available there's no telling when/if that is ever getting merged. A hacky solution can be made on the bundle side (this repo) but that is not ideal and might not always be possible.

Who else has merge access to this repo that can merge my PR as workaround in the meantime?

@chennes
Copy link
Member

chennes commented Mar 23, 2023

I have merge access here, so I can do the local merge, but of course I can't do the feedstock. Next time @looooo has some availability we should discuss moving the feedstock to under the FreeCAD git account so that other maintainers can help out.

@yorikvanhavre
Copy link
Member

Note that the actual build config is not on @looooo 's repo (he has only fork) but on conda itself: https://github.com/conda-forge/freecad-feedstock . But AFAICS it only configures how to build FreeCAD on the conda forge, similarly to other CI platforms. I can't find anything there that says how the different release packages are built

@luzpaz
Copy link
Collaborator

luzpaz commented Apr 27, 2023

It's build on my fork looooo/freecad-feedstock (master branch) but I want to move all the necessary stuff to conda-forge and update to py3.11 but this is currently blocked by a windows issue (boost1.68 migration).

regarding macOS, loooo still uses his own fork per #171 (comment)

@adrianinsaval
Copy link
Member Author

Note that the actual build config is not on @looooo 's repo (he has only fork) but on conda itself: https://github.com/conda-forge/freecad-feedstock . But AFAICS it only configures how to build FreeCAD on the conda forge, similarly to other CI platforms. I can't find anything there that says how the different release packages are built

that's not entirely correct, yes some builds are made on the conda-forge repo, those get uploaded here: https://anaconda.org/conda-forge/freecad/files when I opened this issue only stable releases where being uploaded there, but it seems it has now enabled dev builds there too: conda-forge/freecad-feedstock@b0af988#diff-f4fdc7679c30abcace9506f9d2cb7c79b22b5bfb24787e0ffb18f37349a0a178 could you comment on that @looooo ?

on the other hand there are scheduled builds running at: https://github.com/looooo/freecad-feedstock which get uploaded to: https://anaconda.org/freecad/freecad/files

and these are the ones we are using in our bundles AFAIK, these are the ones I'm worried about because it's using what looks like an official account using our name but it seems nobody besides lorenz has the credentials to access that and the builds are configured on his personal repo. Maybe @sgrogan knows somethings? Although I haven't seen him around for a long time either.

@adrianinsaval
Copy link
Member Author

adrianinsaval commented May 4, 2023

Trying to understand how the system works, its seems you can generate and get CI running on a repo using https://github.com/conda-forge/conda-smithy
looo is using azure pipelines, which apparently give you unlimited minutes for 10 parallel jobs for open source projects: https://azure.microsoft.com/en-us/products/devops/pipelines/#pricing
we likely qualify for using that service to run the builds so I think there is a way for us to start running builds from our own repo, but we remain with the problem of not having credentials to upload to https://anaconda.org/freecad/freecad/files which is not necessarily critical to keep the system running but it would be better if we can.

Perhaps the FPA can contact anaconda and try to get access to it, it's listed as an organization account.

@looooo
Copy link
Contributor

looooo commented May 5, 2023

that's not entirely correct, yes some builds are made on the conda-forge repo, those get uploaded here: https://anaconda.org/conda-forge/freecad/files when I opened this issue only stable releases where being uploaded there, but it seems it has now enabled dev builds there too: conda-forge/freecad-feedstock@b0af988#diff-f4fdc7679c30abcace9506f9d2cb7c79b22b5bfb24787e0ffb18f37349a0a178 could you comment on that @looooo ?

Sorry for my late response. Actually I tried to address this issue by moving the builds to conda-forge. So there is now the conda-forge/freecad-feedstock development branch which is a replacement for my fork. I also moved the script which builds freecad twice a week to this repo.
What is missing is the windows build which I am not able to fix. There is still an issue with the updated boost library. It works with PCH enabled but this runs into azure limitation. So to get the windows build working again there needs to be investigation into why the fix from @wwmayer is not working for pch-disabled and windows.
Also I am not sure how to handle the osx-arm and Linux-aarch64 builds. In the ideal case these builds should be done with azure and cross-compilation. But this also needs some efforts to solve the remaining build-errors.

So to keep the conda-infrastructure for freecad running some work is needed. I tried to do this but I failed several times with the mentioned tasks.

Also I don't see why this all depends only on me. All the scripts are openly available and everyone can experiment with the builds. But maybe new forces are needed to keep this running.

@adrianinsaval
Copy link
Member Author

Thank you for all your work looo, sorry to be bothering you

Sorry for my late response. Actually I tried to address this issue by moving the builds to conda-forge. So there is now the conda-forge/freecad-feedstock development branch which is a replacement for my fork. I also moved the script which builds freecad twice a week to this repo.

are there not changes from your fork that need to be pushed to the conda forge repo? I can see about making a PR for this next week.

Will this account/repo be disabled then: https://anaconda.org/freecad/ ? I ask because I think only you have credentials for that one, and it looks kinda official. If we'll keep that online I think it's important that the FPA and/or other maintainers also have access to the credentials for that account.

What is missing is the windows build which I am not able to fix. There is still an issue with the updated boost library. It works with PCH enabled but this runs into azure limitation. So to get the windows build working again there needs to be investigation into why the fix from @wwmayer is not working for pch-disabled and windows.

well the one in your repo seems to be working on windows, that's why I say there's probably stuff that needs to be pushed from your repo to the conda-forge one. I tried to work today on fixing the build error and it got really confusing trying to track which repo I should be trying to fix, and why they are different.

Also I am not sure how to handle the osx-arm and Linux-aarch64 builds. In the ideal case these builds should be done with azure and cross-compilation. But this also needs some efforts to solve the remaining build-errors.

doesn't azure have arm machines? why do we need cross-compilation? For example occt-feedstock appears to be configured to build in arm too. We can tackle this problem later though, I don't think we're in a rush on that front.

So to keep the conda-infrastructure for freecad running some work is needed. I tried to do this but I failed several times with the mentioned tasks.

I will try to help fix the compilation problems.

Also I don't see why this all depends only on me. All the scripts are openly available and everyone can experiment with the builds. But maybe new forces are needed to keep this running.

yes, here on the bundle repo we have other people who can merge stuff, but on the feedstock side where the actual compilation happens only you had write access, I want to contribute but there's nothing I can do if there's nobody around to accept my contributions, that's why I think we need more people with write access on the relevant repos and perhaps to the anaconda and azure accounts associated. I mean we could start from scratch if it really comes down to it but I think it would be better to have more people on the current setup.

Now that you are moving the compilation to conda-forge, who else has write access there that we can contact when you are not around? If there's nobody else do you have the authority to give someone else in our team access there as fallback?

@looooo
Copy link
Contributor

looooo commented May 6, 2023

Will this account/repo be disabled then: https://anaconda.org/freecad/ ? I ask because I think only you have credentials for that one, and it looks kinda official. If we'll keep that online I think it's important that the FPA and/or other maintainers also have access to the credentials for that account.

Yeah if we move everything to conda-forge we should be able to disable this account. But I am not sure how to upload local builds to conda-forge. So for osx-arm and Linux-aarch64 the freecad account is still needed.

well the one in your repo seems to be working on windows, that's why I say there's probably stuff that needs to be pushed from your repo to the conda-forge one. I tried to work today on fixing the build error and it got really confusing trying to track which repo I should be trying to fix, and why they are different.

The one in my repo is not updated to boost 1.78.

doesn't azure have arm machines? why do we need cross-compilation? For example occt-feedstock appears to be configured to build in arm too. We can tackle this problem later though, I don't think we're in a rush on that front.

Afaik azure has no arm machines. The occt-feedstock is also cross-compiled. There are 2 options: building with qemu (which is too slow for freecad) and cross-compiling.

Now that you are moving the compilation to conda-forge, who else has write access there that we can contact when you are not around? If there's nobody else do you have the authority to give someone else in our team access there as fallback?

Maintainers can add them self by:
https://conda-forge.org/docs/maintainer/infrastructure.html#conda-forge-admin-please-add-user-username

I will try to help fix the compilation problems.

Thanks, solving this issue would be really great.

Additionally there is also a crash with osx-arm64. I reported in the forum. There it would be nice to check if the brew-build also has this issue. The crash can be reduced by drawing a line in sketcher and toggle the constraint type to blue and "constraint" the length of the line. To me it seems like a coin callback issue, but I am not sure. This crash was introduced by some osx-updates...

@yorikvanhavre
Copy link
Member

Could you describe briefly how the appimages are produced from the conda builds @looooo ? I couldn't find anywhere something that shows how it is done...

@looooo
Copy link
Contributor

looooo commented May 8, 2023

@yorikvanhavre The bundling is done via a script which gathers all the necessary dependencies via the conda package manager. Then it copies the files and compress them to a appimage/DMG/zip file.

@yorikvanhavre
Copy link
Member

Okay found it: https://github.com/FreeCAD/FreeCAD-Bundle/blob/master/conda/linux/create_bundle.sh Thanks!

I'd like to make a script that one can run locally, and that creates an appimage build, without conda. I'll have a look at how that one works

@adrianinsaval
Copy link
Member Author

Running locally is certainly possible but not using conda would mean we have to get all the other dependencies from some other source. We would have to use some form of package manager either way (unless you want to compile everything, we're partially doing that for flatpak), using a distro's repo would be more stable but also less flexible, when there are problems in a 3rd party package it'll be harder to solve, and if we use Debian or Ubuntu as a base updating things like opencascade would be kind of a pain.

One not frequently considered advantage of running conda builds is that we can detect early problems with new versions of our deps, so we solve them before they become a problem on other distros.

@yorikvanhavre
Copy link
Member

yorikvanhavre commented May 11, 2023

If using a standard distro like debian, most of the packages are there already, and there are scripts like https://github.com/AppImageCommunity/pkg2appimage that can use it. The rest can be obtained via pip.

My idea here is not to substitute the conda workflow for our releases. Rather have an alternative in case, for some reason, the conda workflow is not available. Basically something that can be documented and used by someone else to either produce a release package or, who knows, adapt to something else... It's mostly to build knowledge ;)

@adrianinsaval
Copy link
Member Author

Yeah if we move everything to conda-forge we should be able to disable this account. But I am not sure how to upload local builds to conda-forge. So for osx-arm and Linux-aarch64 the freecad account is still needed.

@looooo could you then pass along credentials for this account to @yorikvanhavre so that the FPA has access to it and can then give access to others as required? I will not do it right now but eventually I'd like to setup a feedstock repo in our org account where I would pin all or most of the dependencies so that we can easily make stable releases without having to chase after new library releases every time. My idea is to make the process smoother so that we can increase our release frequency without adding absurd amounts of work to volunteer packagers. It seems to me that for this type of usage that conda repository is the most appropriate.

@looooo
Copy link
Contributor

looooo commented Jun 3, 2023

i transferred my fork of the freecad-feedstock to the freecad organisation. Still not sure if this makes a lot of sense as osx and linux are now build via conda-forge/freecad-feedstock (development-branch).

@adrianinsaval
Copy link
Member Author

adrianinsaval commented Jun 5, 2023

Thank you, that still leaves the question of credentials to the anaconda account though, I've seen that circle CI has mac m1 machines available so I was thinking of moving the mac builds to that as I recall that you told me in azure we are doing cross compilation (which seems to not be working) and bundling for arm from x86 had some issues. (but we probably can't move everything to circle CI as there's a limit for free use)

About using conda-forge, that is indeed good and I think it's a good strategy for the weekly builds, however from what I remember it's been problematic to keep freecad up to date with the constantly updated dependencies on conda specially for point releases. I hope that if we pin most of the dependencies that would not be a problem and we would be able to more easily make releases, which would help if we aim to increase our release frequency. So my idea would be to get things working on the conda-forge repo, use that for our weekly builds and once we make a major release pin all dependencies to what is working at the moment then use that for all subsequent point releases in that series.

@adrianinsaval
Copy link
Member Author

Maintainers can add them self by:
https://conda-forge.org/docs/maintainer/infrastructure.html#conda-forge-admin-please-add-user-username

@looooo I've created issues for some of the meaningful repos, I don't know if you are able to approve these:
conda-forge/freecad-feedstock#94
conda-forge/occt-feedstock#99
conda-forge/smesh-feedstock#70

are there other significant feedstocks where we could benefit from having more people from our team?

@adrianinsaval
Copy link
Member Author

I hate to keep nagging you @looooo but now that the feedstock repo has been moved to the org account it seems the automated builds are not running on it anymore, I suspect it has to be registered again with azure and anaconda, could you please provide the account and password to the anaconda account? (perhaps via forum PM)

and if you can please also accept the issues I created to add myself to the conda-forge smesh and freecad feedstocks, and thank you for approving the one for occt.

@M4xTheMan
Copy link

Okay found it: https://github.com/FreeCAD/FreeCAD-Bundle/blob/master/conda/linux/create_bundle.sh Thanks!

I'd like to make a script that one can run locally, and that creates an appimage build, without conda. I'll have a look at how that one works

@yorikvanhavre did you have the time to make any progress regarding building AppImages locally?

@yorikvanhavre
Copy link
Member

I shamefully have to say no 😅 will try ASAP

@M4xTheMan
Copy link

Great to hear that but no pressure!

@adrianinsaval
Copy link
Member Author

given that the people of conda-forge don't want weekly builds done on their infrastructure: conda-forge/freecad-feedstock#99 I think we have to put our freecad-feedstock repo back in action @looooo, I essentially have it working already on my personal repo. And we don't need to use azure since conda-smithy also supports github actions, plus I figured out how to use cirrus-ci for native arm builds (although this might not actually be needed: https://github.com/adrianinsaval/freecad-feedstock/actions/runs/5502596625, I'll have to make a bundle out of the cross compiled packages and have some people test them to see if no functionality is lost, particularly I'm uncertain if the spacemouse driver will work on the cross compiled version). I'm just missing access to the anaconda.org/freecad channel.

@yorikvanhavre
Copy link
Member

How does it work to put our stable releases to conda-forge?

@looooo
Copy link
Contributor

looooo commented Jul 22, 2023

I have a checklist here:
https://github.com/FreeCAD/FreeCAD-Bundle/blob/master/notes.md

@yorikvanhavre
is there already a stable release? I didn't see one here:
https://github.com/FreeCAD/FreeCAD/releases

Btw. Thanks for continuing the work on the feedstocks @adrianinsaval

@adrianinsaval
Copy link
Member Author

Not yet, but soon we'll have a release candidate

@luzpaz
Copy link
Collaborator

luzpaz commented Aug 17, 2024

@adrianinsaval is this ticket still relevant ?

@luzpaz
Copy link
Collaborator

luzpaz commented Oct 23, 2024

@adrianinsaval 👆

@looooo
Copy link
Contributor

looooo commented Oct 23, 2024

I think you can close this issue. @oursland and @adrianinsaval have done a great job continuing the effort to maintain freecad via conda.

@adrianinsaval
Copy link
Member Author

There's just one thing I would like done here, the anaconda repo has only me and looooo as owners but I would like to have an FPA member to have ownership of it too so they can easily pass it on if/when someone else needs to take over. @yorikvanhavre can you create an account in anaconda.org so that I can add you as another owner of our cones repo there?

@oursland
Copy link
Collaborator

I request to be added as an owner.

@adrianinsaval
Copy link
Member Author

I request to be added as an owner.

Are you oursland in anaconda.org too?

@oursland
Copy link
Collaborator

Yes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants