Dependency download fails on Linux: (403) Forbidden

[Zaratusa]

Hi, I've generated a Dockerfile to build a custom version 4.27.2 on Linux in Gitlab CI with the following settings:

# This file was generated by ue4-docker version 0.0.106 with the following options:
# 
# - combine: true
# - credential_mode: "secrets"
# - excluded_components: {"ddc": false, "debug": true, "templates": true}
# - gitdependencies_args: "--exclude=Android --exclude=Mac --exclude=Win32 --exclude=Win64"
# - source_mode: "copy"
# 
# This Dockerfile combines the steps for the following images:
#
# - ue4-build-prerequisites
# - ue4-source
# - ue4-minimal

Running the Setup.sh causes the following error, even when running it without any excludes:

Failed to download 'http://cdn.unrealengine.com/dependencies/UnrealEngine-16546836/01aeef9d6f354d0b789fe0f90786078b0395495f': The remote server returned an error: (403) Forbidden. (WebException)

I've never had any problems downloading the dependencies on Windows or MacOS. Is this a known issue on Linux?

[TBBle]

I also get an Access Denied error for that URL from my Windows-based Chome from Japan. So possibly an issue with Epic's CDN.

[Zaratusa]

Read more about the fix here: https://forums.unrealengine.com/t/upcoming-disruption-of-service-impacting-unreal-engine-users-on-github/1155880

[slonopotamus]

Niiiice. This effectively means they broke all git tags.

[slonopotamus]

We're in trouble. I did some Python (#320) so we repair existing UE tags. The problem is that it doesn't work. Because we need to authenticate when we're downloading Commit.gitdeps.xml attached to a release.

[Zaratusa]

As mentioned before, the thread describes the fix: Manually update the Engine/Build/Commit.gitdeps.xml in your custom Engine. There is currently no automatic fix for this issue.

[tongwenbo]

Hello, thanks for the information. I have replaced this file on my local windows UE repo and it worked. Do you know where should I replace this file to in the Ubuntu machine where I run 'ue4-docker build'? Should I first clone the repo of Unreal Engine mannually first and replace this file and then run 'ue4-docker build'? Thanks a lot!

[jcastillo-lazr]

Hello, thanks for the information. I have replaced this file on my local windows UE repo and it worked. Do you know where should I replace this file to in the Ubuntu machine where I run 'ue4-docker build'? Should I first clone the repo of Unreal Engine manually first and replace this file and then run 'ue4-docker build'? Thanks a lot!

I have the same question. Right now I have local copy of UE4 but I would like to do modify that file in the container. The image is not created correctly therefore I cannot get inside the container and modify that file manually. I'm not sure if I should try to hack the python package and the Dockerfile.

[jonititan]

If I understand correctly this is an issue for anyone using ue4-docker and won't be fixed. Perhaps it needs including in the documentation?
I've got this issue as well but as a beginner to this area its a little more tricky to fix since its only documented in an issue and not mentioned in the docs.

[muety]

Any chances to get this fixed?

[TBBle]

This should only be a problem for users using custom (local) UE4 forks. In that case, you can apply Epic's fix to your local source, and things will work again. Per that announcement, UE4 4.25 and later custom forks should be able to merge (or cherry-pick) the latest tag from Epic and get the fixed file, but I haven't tried that myself.

Trying to automate this in ue4-docker is a hassle because we either need access to the user's GitHub credentials to extract the relevant file (which is what blocked #320), which usually isn't available when building from local forks; or we'd need to have available every updated Commit.gitdeps.xml for each release version, and I don't think the license allows us to distribute those.

Potentially we could add a flag that lets the ue4-docker user override Commit.gitdeps.xml, but that's pretty specific and requires the user to go download the correct file, in which case they may as well just commit it to their local fork and the problem is fixed.

If anyone is trying to build an old commit ID directly, rather than using local source, and hitting this issue, then #320's approach would be feasible, as you're already providing GitHub credentials for the download stage. If people are in this situation, then an approach like #320's could work, and if it's gated on providing GitHub credentials, then it'd work for people doing local source builds too if they additionally provide GitHub credentials.

To be clear, my understanding is that this only affects people (re)building the ue4-source image (and maybe ue4-minimal?), it didn't retroactively break existing images.

[jonititan]

Yes following the instructions to build airsim from source.
https://microsoft.github.io/AirSim/docker_ubuntu/

[muety]

I also don't think I'm using a custom fork. I'm really not versed about UE4 and its build process, was just following CARLA's docs to build a Docker image. I simply did pip3 install ue4-docker and ran into the above issue. Possibly the pip package is outdated or something?

[slonopotamus]

Yes following the instructions to build airsim from source.

I believe you should open an issue in airsim to ask them to fix their build instructions.

[TBBle]

Their instructions are using 4.19.2, which is ancient, and didn't have the in-repo fix applied by Epic. In this case, they're not using a custom local fork, so they could benefit from the approach from #320, since their build-from-source step requires GitHub credentials already. This is the "old commit ID directly" situation I describe above, I forgot to mention that it also applies to 4.26 and older too.

4.19 has fallen out of support in ue4-docker as well, so honestly it'd be simpler for everyone if they could upgrade to a newer UE4 version. Their non-container builds use 4.27 (and specify that is the current minimum supported version) so I would guess that it's just their instructions that need to be updated. The 4.27 branch or tag (I'm unclear which, honestly...) has been fixed upstream so it should not suffer this issue when used the way the AirSim docs describe.

Potentially, when following their instructions, replace 4.19.2 with 4.27.2 and it'll "just work"™. microsoft/AirSim#4846 suggests that maybe it won't actually work though... microsoft/AirSim#4561 shows issues with either 4.27.2 or Ubuntu 22.10. Anyway, that project is currently dead as of July 2022, and so I don't expect they'll take any fixes upstream... Even their "successor project" link appears dead. It is MIT licensed and has over four thousand forks, so maybe someone else is continuing the work and fixing stuff like this?

Edit: Oh dear, the successor project was cancelled in October and the entire team was to be laid off as of last week. So I expect that repo will not see another update.

Also, a quick check suggests that the fixed Commit.gitdeps.xml was in the release branches, but the release tags were not updated, so just building 4.27.2 will not work, you'd need to build the tip of the branch (which I don't recall the syntax for exactly, it'll be the "custom branch" syntax I guess) so the local fix for building releases before 5.2 appears not as simple as I hoped, but at least for building "latest patch release" it should not be too problematic.

[chenhengwei1999]

Is there any solutions that can build the ue4 (or ue5) automatically by the ue4-docker python packgae? From the answers above, I find it is difficult to build the ue4 from scratch with a clean docker container, I just want to use the ue4-docker package to build a successfuly container, that can be used to do something in carla.

In short, how can I solve this issue? @slonopotamus @Zaratusa

Last but not least, thank you for the discussion with the above attempted solutions.

[TBBle]

You should be able to build containers for UE 4.25 or later by doing a "custom" build against the relevant Epic release branch, e.g., https://github.com/EpicGames/UnrealEngine/tree/4.25, rather than using the tag. Any releases from 5.2 onwards should work by tag as usual.

For anything older versions or releases, you'll need to create a custom branch of your own, apply Epic's fix and then build from that custom branch.

If Carla depends on a specific version of UE, it might make sense for them to maintain a fork of UE 4 with that version plus the patched CommitDeps, and point their users at that fork when using ue4-docker. That would also make it easier for them to maintain any local patches that turn out to be needed. Edit: Turns out they do this already: https://github.com/CarlaUnreal/UnrealEngine, but for some reason aren't using this in their ue4-docker instructions. I'd suggest raising this with them directly, since they're already maintaining a custom UE4 fork. In fact, looking at their instructions, I'm not sure why they're using ue4-docker, since they seem to ignore its output and directly build their own containers with ue4 inside? So yeah, for Carla-specific situation, I'd suggest opening a ticket with Carla to clarify.

[adamrehn]

On the note of the Carla fork actually, if anyone has recently linked their Epic Games account to their GitHub account and can't access that or any other forks of the Unreal Engine, it's due to Epic's ongoing GitHub organisation maintenance. Fortunately it looks like Epic are aware of the issue and are working to address it.

@slonopotamus @TBBle just a heads-up, in case we see any issues filed against ue4-docker from users who are trying to build images for a fork they don't have access to.