Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for Windows ARM64 support #3107

Open
dennisameling opened this issue Mar 12, 2021 · 218 comments
Open

Tracking issue for Windows ARM64 support #3107

dennisameling opened this issue Mar 12, 2021 · 218 comments

Comments

@dennisameling
Copy link

dennisameling commented Mar 12, 2021

There's been a discussion about Windows ARM64 support here: #3021

I made a discussion comment there to keep track of the outstanding tasks to get basic ARM64 support to Git for Windows. However, @Alovchin91 informed me via Twitter that it was hard to keep track of the progress that way, requesting a dedicated issue. So here it is 😊

Open tasks for basic ARM64 support:

Thanks to the changes that we did in the last few days, there's now a fully working build of Git for Windows ARM64.

Test build for ARM64 users, built on March 12, 2021: https://github.com/dennisameling/git/releases/tag/v2.31.0-rc2.windows.2

I think we're pretty much ready to publish a beta version of Git for Windows ARM64! @dscho what do you think? Please let me know if I missed something in the list above.

@dscho
Copy link
Member

dscho commented Mar 12, 2021

I'm a bit focused on making sure that the v2.31.0 release goes smoothly, and will be happy to pay more attention to the ARM64 side of things after that release is out (which will probably happen Tuesday or Wednesday this coming week).

@Alovchin91
Copy link

Hi @dennisameling ,

Thanks for your wonderful work! ❤️ Have been using MinGit lately (which is a 💎 by itself) and it works great on my SPX!

One thing I just found:

C:\Projects\rustup>git log --help
fatal: 'C:/MinGit/arm64/share/doc/git-doc': not a documentation directory.

While MinGit doesn't seem to contain git-doc anyway, what caught my attention is that git.exe tries to search for documentation in the arm64 directory, while docs seem to still be in mingw32.

Could you please check on your side, on a complete Git setup? Thanks a lot! 😃

@dscho
Copy link
Member

dscho commented Mar 14, 2021

The documentation is not built as part of the CMake-based build. It will take quite some work to implement that. @Alovchin91 that's your chance to get involved! 😉

@Alovchin91
Copy link

@dscho I would be happy to help, but right now my capacity is extremely limited unfortunately and I have never had any proper experience with Cmake.

But wouldn't it be enough to point Git.exe from arm64 folder to the documentation in mingw32? 🤔

@dscho
Copy link
Member

dscho commented Mar 14, 2021

But wouldn't it be enough to point Git.exe from arm64 folder to the documentation in mingw32? 🤔

Not really, because the build configuration may differ in more than just the CPU architecture, even in ways affecting how/what parts of the documentation are compiled.

@Alovchin91
Copy link

Alovchin91 commented Mar 14, 2021

Hmm, I think right now my concern is mostly about where does arm64 version look for docs in a full installer-based version of Git 🤔

I saw that it's also absent from the "normal" mingw32 MinGit build, so I suppose missing docs in those builds is not arm64 specific issue.

@dscho
Copy link
Member

dscho commented Mar 14, 2021

Right, documentation is excluded from MinGit builds.

But looking for the documentation in the wrong location, just because there is no documentation in the correct location, isn't a solution for the non-MinGit flavors of Git for Windows, either ;-)

@Alovchin91
Copy link

Ah, right, now I get your point 😁 For some reason I thought that the most part of the tools are redirected to mingw32 folder and arm64 only contains wrappers or helpers 😅

@dscho
Copy link
Member

dscho commented Mar 14, 2021

Thanks to @dennisameling tireless work, the Git executables are built for Windows/ARM64. The i686 version of the MINGW tools we're still relying on are things like curl.exe, but we do include all of the i686 Git executables at the moment, too.

@dennisameling
Copy link
Author

Can confirm the help docs are only present in mingw32\share\doc\git-doc on a full installation and not in arm64\share\doc\git-doc.

The help function looks at GIT_HTML_PATH to find the help docs:

git/builtin/help.c

Lines 464 to 482 in 959b2f0

static void get_html_page_path(struct strbuf *page_path, const char *page)
{
struct stat st;
char *to_free = NULL;
if (!html_path)
html_path = to_free = system_path(GIT_HTML_PATH);
/* Check that we have a git documentation directory. */
if (!strstr(html_path, "://")) {
if (stat(mkpath("%s/git.html", html_path), &st)
|| !S_ISREG(st.st_mode))
die("'%s': not a documentation directory.", html_path);
}
strbuf_init(page_path, 0);
strbuf_addf(page_path, "%s/%s.html", html_path, page);
free(to_free);
}

This is defined in the CMake file:

add_compile_definitions(PAGER_ENV="LESS=FRX LV=-c"
GIT_EXEC_PATH="libexec/git-core"
GIT_LOCALE_PATH="share/locale"
GIT_MAN_PATH="share/man"
GIT_INFO_PATH="share/info"
GIT_HTML_PATH="share/doc/git-doc"
DEFAULT_HELP_FORMAT="html"
DEFAULT_GIT_TEMPLATE_DIR="share/git-core/templates"
GIT_VERSION="${PROJECT_VERSION}.GIT"
GIT_USER_AGENT="git/${PROJECT_VERSION}.GIT"
BINDIR="bin"
GIT_BUILT_FROM_COMMIT="")

@dscho would you be okay with a clause in the CMake file for Win ARM64 that sets GIT_HTML_PATH to ../mingw32/share/doc/git-doc for the time being? We can leverage the already available CMAKE_GENERATOR_PLATFORM for that logic.

@dscho
Copy link
Member

dscho commented Mar 19, 2021

would you be okay with a clause in the CMake file for Win ARM64 that sets GIT_HTML_PATH to ../mingw32/share/doc/git-doc for the time being?

I wonder how much work it would be to optionally use asciidoctor or asciidoc, if configured. These lines describe how asciidoc is called: https://github.com/git/git/blob/v2.31.0/Documentation/Makefile#L125-L133 And these describe how asciidoctor would be called instead: https://github.com/git/git/blob/v2.31.0/Documentation/Makefile#L179-L190

@dennisameling
Copy link
Author

I wonder how much work it would be to optionally use asciidoctor or asciidoc, if configured.

Will have a look in the coming days.

Just created a PR for 64-bit ARM64 installer support: git-for-windows/build-extra#333

@dennisameling
Copy link
Author

@dscho now that the 64-bit ARM64 installer PR has been merged, do you think we're ready to release a beta build of Git for Windows ARM64? I'm happy to look into the docs thing with asciidoctor and asciidoc mentioned above, but will only have time for that after May 7 due to work responsibilities and a break 😊 Thanks in advance!

@rimrul
Copy link
Member

rimrul commented Apr 22, 2021

Check if the daily "Git for Windows update check" works correctly on ARM64

It'll probably run daily and check for updates, but when it detects an update, it'll try to download and install the x86 installer.

@dscho
Copy link
Member

dscho commented Apr 23, 2021

do you think we're ready to release a beta build of Git for Windows ARM64?

Yes, I think we're getting to the point where we want to invite testers (I, unfortunately, am not eligible because I lack the hardware).

I'm happy to look into the docs thing with asciidoctor and asciidoc mentioned above

Fantastic.

I do think that it should be possible to add that to the CMakeLists.txt definition (a quick search brought up gdamore/nanomsg@5592124 which might be a good example to follow, although I would try not to have a separate Boolean to enable documentation; Rather, I would make it conditional on a check that tries to find asciidoc or asciidoctor, allowing the user to specify a hard-coded path to either to side-step the discovery).

It'll probably run daily and check for updates, but when it detects an update, it'll try to download and install the x86 installer.

Indeed.

For microsoft/git, I already did something slightly ugly to redirect the git update-git-for-windows command: microsoft#321 (comment). Essentially, it is a hack that edits the git-update-git-for-windows script to target a different update site, via a few sed commands.

I could imagine that we would probably want to improve on that e.g. by enhancing git-update-git-for-windows itself. One option would be to search for a file, say, git-update-git-for-windows.ini next to the script (when called via the PATH, as git update-git-for-windows will do internally, "$0" contains the absolute path to the script, therefore "$0.ini"would refer to that.inifile's path). If found, it would override a couple of things. Something along these lines (applies tobuild-extra`, but reader beware: this is totally untested):

diff --git a/git-extra/git-update-git-for-windows b/git-extra/git-update-git-for-windows
index 7121dd7be..50ff5524a 100755
--- a/git-extra/git-update-git-for-windows
+++ b/git-extra/git-update-git-for-windows
@@ -174,7 +174,22 @@ update_git_for_windows () {
 		;;
 	esac
 
-	latest_tag_url=https://gitforwindows.org/latest-tag.txt
+	if test -f "$0.ini"
+	then
+		fork="$(git config -f "$0.ini" update.fromFork)"
+		test -n "$releases_url" || {
+			echo "Could not find update.fromFork in $0.ini" >&2
+			return 1
+		}
+		releases_url=https://api.github.com/repos/$fork/releases
+		latest_tag_url=$releases_url/latest
+		latest_eval='latest=${latest_tag#*\"tag_name\": \"}; latest=${latest%%\"*}'
+	else
+		releases_url=https://api.github.com/repos/git-for-windows/git/releases
+		latest_tag_url=https://gitforwindows.org/latest-tag.txt
+		latest_eval='latest=${latest_tag#v}'
+	fi
+
 	latest_tag=$(http_get $latest_tag_url) ||
 	case $?,"$proxy" in
 	7,)
@@ -190,7 +205,7 @@ update_git_for_windows () {
 		;;
 	esac
 
-	latest=${latest_tag#v}
+	eval "$latest_eval"
 	# Did we ask about this version already?
 	recently_seen="$(get_recently_seen)"
 	test -n "$quiet" && test "x$recently_seen" = "x$latest" && return
@@ -215,7 +230,6 @@ update_git_for_windows () {
 	esac
 
 	echo "Update $latest is available" >&2
-	releases_url=https://api.github.com/repos/git-for-windows/git/releases
 	releases=$(http_get $releases_url/latest) || return
 	download=$(echo "$releases" |
 		grep '"browser_download_url": "' |

@dscho
Copy link
Member

dscho commented Apr 23, 2021

❓ Check if the daily "Git for Windows update check" works correctly on ARM64

Right, completely forgot to address that part. We do trust uname -m in the git-update-git-for-windows script, but on ARM, we cannot (because our MSYS2 will be an i686 or an x86_64 one).

@rimrul
Copy link
Member

rimrul commented Apr 23, 2021

I could imagine that we would probably want to improve on that e.g. by enhancing git-update-git-for-windows itself. One option would be to search for a file, say, git-update-git-for-windows.ini next to the script (when called via the PATH, as git update-git-for-windows will do internally, "$0" contains the absolute path to the script, therefore "$0.ini"would refer to that.ini` file's path). If found, it would override a couple of things.

I was thinking a lot simpler: We could check $MSYSTEM or $MINGW_MOUNT_POINT to detect we're on an ARM install. Admittedly, I did not think about microsoft/git when I had that idea.

@dscho
Copy link
Member

dscho commented Apr 26, 2021

I don't really trust MSYSTEM and MINGW_MOUNT_POINT at this stage ;-) If git.exe sets MSYSTEM, for example, it will not have the expected value.

@dscho
Copy link
Member

dscho commented Apr 28, 2021

I opened git-for-windows/build-extra#338 to help with this.

@dennisameling
Copy link
Author

Incredibly excited that git-for-windows/build-extra#340 has been merged. I almost don't dare to ask, but do you think we're now ready to release a beta version of Git for Windows ARM64, @dscho (apart from the request for the docs, which I hope to look into in the coming two weeks)? 🤞🏼

Yes, I think we're getting to the point where we want to invite testers (I, unfortunately, am not eligible because I lack the hardware).

I ordered and received a Raspberry Pi 4 last week, then installed Windows on ARM 19043 (21H1) on it. The performance is surprisingly good, but every once in a while there's a little driver-related hiccup that the Windows on Raspberry team is working hard on. If you want, I can send you the TeamViewer ID + password so you can test a bit on your end as well 👍🏼

@Alovchin91
Copy link

I have the Surface Pro X so I can also help with testing 🙂 RDP is also an option.

@dscho
Copy link
Member

dscho commented May 18, 2021

Right now, I would like to focus on Git for Windows v2.32.0, for which -rc0 came out yesterday, and the final version is expected around June 7th/8th.

After that, I'm all game for ARM64 and will take y'all up on the kind offers to let me connect to your hardware.

@Alovchin91
Copy link

Well... I wonder if BUILD next week brings anything interesting to arm64 world anyway 🤷 Maybe an arm64 GitHub Actions pipeline? One could dream...

@rimrul
Copy link
Member

rimrul commented May 19, 2021

@dennisameling Could you add the issue with git --version --build-options that I pointed out in #3083 to the list?

While it isn't a show stopper for the ARM beta it would be nice to have that fixed before we get more ARM bug reports.

@Alovchin91
Copy link

Okay, I hope it's finally time to prepare the first ARM64 beta! 🤗 Please let me know if I can help with anything. I could provide an ARM64 virtual machine on my Surface Pro X for a build pipeline, for example 🙂

@dennisameling
Copy link
Author

dennisameling commented Jul 10, 2021

Just updated the list above with open issues. I recently came across the fact that Visual C++ is needed since we use MSVC for the ARM64 builds. We can either include the Visual C++ Redistributable package in the installer, or statically build using MSVC (if possible). Do you have any preference here @dscho? I guess the latter option would be best as software like GitHub Desktop is depending on MinGit, so a static build would make their lives easier as well.

@rimrul
Copy link
Member

rimrul commented Jul 10, 2021

I don't think UCRT can be statically linked and redistributing vcredist would probably be a GPL violation, so linking the vcredist download page is probably our only option.

@dscho
Copy link
Member

dscho commented Jun 3, 2024

Would you be so kind to run the “Build git-artifacts (aarch64)” pipeline for 2.45.2?

Hopefully this run will do the job.

@Alovchin91
Copy link

Alovchin91 commented Jun 4, 2024

Do I understand correctly that git-for-windows-automation could be forked and run on a hosted runner? Is there any kind of instruction on how to set it up? I would be happy to help by opening a Pull Request if there's something that needs fixing in the job.

@dscho
Copy link
Member

dscho commented Jun 4, 2024

Do I understand correctly that git-for-windows-automation could be forked and run on a hosted runner?

@Alovchin91 no, it must be run on a self-hosted runner because it requires a native Windows/ARM64 to run.

Is there any kind of instruction on how to set it up?

You could use the create-azure-self-hosted-runners workflow after setting up a couple of repository secrets.

I would be happy to help by opening a Pull Request if there's something that needs fixing in the job.

The main problem right now is that the MINGW-packages commit was already pushed. I have a couple of ideas how to address that, the most promising approach is to polish and integrate the changes I made to build the embargoed Git for Windows v2.45.1.

@Alovchin91
Copy link

Alovchin91 commented Jun 4, 2024

no, it must be run on a self-hosted runner because it requires a native Windows/ARM64 to run.

Yes, sorry, that's what I meant 😅

You could use the create-azure-self-hosted-runners workflow after setting up a couple of repository secrets.

I don't really have an Azure subscription, is there a free/trial tier for open-source work? I'm interested in this work as an individual user mostly.

Yet again, I would be happy to contribute if there's anything I could help with, so please let me know.

@dscho
Copy link
Member

dscho commented Jun 4, 2024

I don't really have an Azure subscription

Do you have an AWS account? It should be possible to do something equivalent there.

@Alovchin91
Copy link

Do you have an AWS account?

Nope, and the last time I checked they didn't have any Windows on Arm offers. But I do have a Surface Pro X and a MacBook with Parallels, so I can set up a private self-hosted runner for testing purposes.

@dscho
Copy link
Member

dscho commented Jun 4, 2024

Do you have an AWS account?

Nope, and the last time I checked they didn't have any Windows on Arm offers. But I do have a Surface Pro X and a MacBook with Parallels, so I can set up a private self-hosted runner for testing purposes.

That should be enough, you only need to imitate the labels and register with your fork of git-for-windows-automation, then you can run the git-artifacts workflow.

@MatthewSteeples
Copy link

https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/ Not yet available for Opensource projects, but coming later this year (unless anyone knows someone...)

@dscho
Copy link
Member

dscho commented Jun 4, 2024

https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/ Not yet available for Opensource projects, but coming later this year (unless anyone knows someone...)

@MatthewSteeples I already reported this 15 hours ago.

@MatthewSteeples
Copy link

MatthewSteeples commented Jun 4, 2024

So you did! Apologies, don't know how I missed that, especially as I remember reading the follow up comment about the schedule having slipped. Having one of those weeks...!

@rimrul
Copy link
Member

rimrul commented Jun 4, 2024

I don't really have an Azure subscription, is there a free/trial tier for open-source work?

https://opensource.microsoft.com/azure-credits

There is this offer of free azure credits for open source, but the aproval/rejection timeframe is 3-4 weeks and it doesn't read like it's targeted at individual contributors.

@dscho
Copy link
Member

dscho commented Jun 4, 2024

So you did! Apologies, don't know how I missed that

@MatthewSteeples No worries. This ticket is one of those really, really horrible ones where I wish for a more sophisticated UI than a single-threaded one:
image

@Alovchin91
Copy link

Alovchin91 commented Jun 17, 2024

That should be enough, you only need to imitate the labels and register with your fork of git-for-windows-automation, then you can run the git-artifacts workflow.

I must be missing something. I've essentially followed whatever is done by post-deployment-script.ps1 but the workflow fails with

Error: Need an appId
    at initCheckRunState (C:\actions-runner\_work\git-for-windows-automation\git-for-windows-automation\check-runs.js:129:23)

Turns out I need to create a GitHub App. How do I configure it?

And what did you mean by "register with your fork of git-for-windows-automation"?

@Alovchin91
Copy link

...And now it fails with some other cryptic message:

FAILED GitHub REST API call!
{
  port: 443,
  hostname: 'api.github.com',
  method: 'GET',
  path: '/repos/git-for-windows/git/installation',
  headers: {
    'User-Agent': 'GitForWindowsHelperApp/0.0',
    Accept: 'application/json',
    Authorization: '***'
  }
}
{
  statusCode: 404,
  statusMessage: 'Not Found',
  body: '{"message":"Not Found","documentation_url":"https://docs.github.com/rest/apps/apps#get-a-repository-installation-for-the-authenticated-app","status":"404"}',
  json: {
    message: 'Not Found',
    documentation_url: 'https://docs.github.com/rest/apps/apps#get-a-repository-installation-for-the-authenticated-app',
    status: '404'
  }
}

I swear that repo was not made to be forked...

@dscho
Copy link
Member

dscho commented Jun 17, 2024

Turns out I need to create a GitHub App. How do I configure it?

Yes. There is some documentation in the still-unpolished embargoed-builds branch of the gfw-helper-github-app repository, see in particular git-for-windows/gfw-helper-github-app@52a5c8a (you'll just want to drop the default_events section and you're good).

And what did you mean by "register with your fork of git-for-windows-automation"?

The runner should be specific to your fork, if you use the create-azure-self-hosted-runners workflow, it will do that by default. My comment was meant more for the case that you set up a VM manually, and then register that manually as a self-hosted runner.

@dscho
Copy link
Member

dscho commented Jun 17, 2024

...And now it fails with some other cryptic message:

That makes me believe that the GitHub App is maybe not installed on your fork?

I swear that repo was not made to be forked...

Indeed it wasn't. It was made to support Git for Windows' own processes.

@Alovchin91
Copy link

Alovchin91 commented Jun 17, 2024

That makes me believe that the GitHub App is maybe not installed on your fork?

It fails when trying to do "Mirror Check Run" on git-for-windows/git repo:

https://github.com/git-for-windows/git-for-windows-automation/blob/292b89bd88329ba348c08e906438a0967a2cc1a1/.github/workflows/git-artifacts.yml#L34-L35

Does that mean I also need to fork the git repo? 🤔

Or maybe I can skip this step altogether? All I want to do is essentially build pacman packages from an existing tag.

Okay, scratch all that. I have removed this step altogether and now I can continue.

@Alovchin91
Copy link

I believe there's now an issue with git-sdk-arm64, check out this run: https://github.com/Alovchin91/git-for-windows-automation/actions/runs/9555876276/job/26340005470

error: target not found: mingw-w64-clang-aarch64-asciidoctor

And quite a number of the following lines (though not sure if they are errors):

/usr/bin/makepkg: line 935: /usr/bin/gettext: No such file or directory

@Alovchin91
Copy link

Alovchin91 commented Jun 17, 2024

Using the full setup-git-for-windows-sdk flavor allowed to move on, but now there are thousands of errors like:

module error : failed to open /usr/lib/libxslt-plugins/nwalsh_com_xsl_documentation_1_0.dll

See https://github.com/Alovchin91/git-for-windows-automation/actions/runs/9556010330/job/26340435149, Build mingw-w64-clang-aarch64-git step.

The packages have been successfully built though.

@dennisameling
Copy link
Author

I believe there's now an issue with git-sdk-arm64, check out this run: https://github.com/Alovchin91/git-for-windows-automation/actions/runs/9555876276/job/26340005470

error: target not found: mingw-w64-clang-aarch64-asciidoctor

That should be fixed with this PR: git-for-windows/git-sdk-arm64#19

And quite a number of the following lines (though not sure if they are errors):

/usr/bin/makepkg: line 935: /usr/bin/gettext: No such file or directory

I noticed those too, but they're also in the x86 and x64 flavors, e.g. here: https://github.com/git-for-windows/git-for-windows-automation/actions/runs/9346419477/job/25721230283#step:19:112

Not sure if that's a known issue at all.

@dscho
Copy link
Member

dscho commented Jun 18, 2024

And quite a number of the following lines (though not sure if they are errors):

/usr/bin/makepkg: line 935: /usr/bin/gettext: No such file or directory

I noticed those too, but they're also in the x86 and x64 flavors, e.g. here: https://github.com/git-for-windows/git-for-windows-automation/actions/runs/9346419477/job/25721230283#step:19:112

Not sure if that's a known issue at all.

#4952

@dscho
Copy link
Member

dscho commented Jun 19, 2024

And quite a number of the following lines (though not sure if they are errors):

/usr/bin/makepkg: line 935: /usr/bin/gettext: No such file or directory

I noticed those too, but they're also in the x86 and x64 flavors, e.g. here: https://github.com/git-for-windows/git-for-windows-automation/actions/runs/9346419477/job/25721230283#step:19:112
Not sure if that's a known issue at all.

#4952

I believe that this will be addressed via git-for-windows/build-extra#560.

@tuanle07
Copy link

Any updates on this? Been waiting for git on Arm64 for years. :(

@dscho
Copy link
Member

dscho commented Aug 20, 2024

Any updates on this? Been waiting for git on Arm64 for years. :(

A better use of your time might be to help the effort.

As to updates: there is e.g. git-for-windows/build-extra#574, git-for-windows/build-extra#572, git-for-windows/git-sdk-arm64#21, git-for-windows/build-extra#574 and of course https://inbox.sourceware.org/cygwin/a3ec694e-e9d5-04db-fa21-b93b28f7a0d4@jdrake.com/#t.

Git for Windows/ARM64 would certainly benefit from competent help by motivated contributors.

@dennisameling
Copy link
Author

dennisameling commented Sep 9, 2024

Now that the asciidoctor issue is out of the way (thanks @dscho!), I was able to build the arm64 artifacts again. And I have some good news:

@dscho given that the pipelines seem to be a lot more stable now, how do you feel about releasing a beta version of GfW arm64? I think one final blocker would be the discussion on the artifact names. WDYT?

@Alovchin91
Copy link

Alovchin91 commented Sep 9, 2024

Just FYI, I have been using the artefacts from Dennis's runs for more than half a year now (I'm installing it inside MSYS2 proper from the packages archive) and I have never had a single issue with it. The only thing that I had to workaround is Pacman but that's because Pacman itself kept hanging randomly after the 6.0.2-12 patch (I've posted it somewhere in one of the discussions; never tried to upgrade it past 6.0.2-11 afterwards). I work in a humongous mono-repo (although without LFS but I believe Git LFS is a separate thing) so there it is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests