Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Android 2022.2.X+ Build Issues #205

Merged
merged 6 commits into from
Apr 9, 2023
Merged

Fix Android 2022.2.X+ Build Issues #205

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3cf009d
Update Unity hub to latest beta which fixes 2022.2.X android build is…
AndrewKahr Apr 5, 2023
9369090
Improve error catching during installation, refactor android variable…
AndrewKahr Apr 6, 2023
80236fe
Fix source not found
AndrewKahr Apr 6, 2023
b7fcb1d
Fix source
AndrewKahr Apr 6, 2023
481012f
Export CMDLINE tools path for Unity 2021+
AndrewKahr Apr 6, 2023
e147c5c
Test timeout
AndrewKahr Apr 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,7 @@ jobs:
# Test #
############
- name: Test project
timeout-minutes: 10
Copy link
Member

@GabLeRoux GabLeRoux Apr 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For other reviewers:
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes

Default is 360, reducing it to 10 will most likely make it fail faster if necessary 👍

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only a valid setting if you know the maximum amount a project would take to build. Useful for this repo. Perhaps also worth documenting in our docs for GHA?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general you don't really need this setting. I think what is happening is these big matrices of Unity test runs exceed some kind of rate limit for activation endpoint silently which causes it to hang. I just set it to 10 minutes because I saw every test run with our empty test project takes about 1 min so by 10 min it's probably hung and guaranteed to fail. This time is heavily driven by the actual tests and whether the library folder already exists. The activation errors are also generally rare in standard CI runs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are in fact a few cases where implementers of GameCI using GHA are waiting for their run and they report it times out after 5 hours not knowing the error.

Probably the fastest way to setup GameCI (in case anything goes wrong) is to set it up using a test project, and having that setting set to 5 minutes :) Just a suggestion though.

uses: game-ci/unity-test-runner@main
with:
unityVersion: ${{ matrix.version }}
Expand Down
67 changes: 57 additions & 10 deletions images/ubuntu/editor/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@ FROM $hubImage AS builder
# Install editor
ARG version
ARG changeSet
RUN unity-hub install --version "$version" --changeset "$changeSet" | tee /var/log/install-editor.log && grep 'Error' /var/log/install-editor.log | exit $(wc -l)
RUN unity-hub install --version "$version" --changeset "$changeSet" | tee /var/log/install-editor.log && grep 'Failed to install\|Error while installing an editor\|Completed with errors' /var/log/install-editor.log | exit $(wc -l)

# Install modules for that editor
ARG module="non-existent-module"
RUN for mod in $module; do \
if [ "$mod" = "base" ] ; then \
echo "running default modules for this baseOs"; \
else \
unity-hub install-modules --version "$version" --module "$mod" --childModules | tee /var/log/install-module-${mod}.log && grep 'Missing module' /var/log/install-module-${mod}.log | exit $(wc -l); \
unity-hub install-modules --version "$version" --module "$mod" --childModules | tee /var/log/install-module-${mod}.log && grep 'Missing module\|Completed with errors' /var/log/install-module-${mod}.log | exit $(wc -l); \
fi \
done \
# Set execute permissions for modules
Expand Down Expand Up @@ -95,7 +95,7 @@ RUN echo "$version-$module" | grep -q -v '^\(2020.1\|2020.2.0f\|2020.2.1f\).*-we
#=======================================================================================
# [2018.x-android] Install 'Android SDK 26.1.1' and 'Android NDK 16.1.4479499'
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^\(2018.3\|2018.4\).*android' \
RUN echo "$version-$module" | grep -q -v '^2018\.[34].*android' \
&& exit 0 \
|| : \
# Versions
Expand Down Expand Up @@ -138,28 +138,25 @@ RUN echo "$version-$module" | grep -q -v '^\(2018.3\|2018.4\).*android' \
&& echo ". /usr/bin/unity-editor.d/android-2018.3-4.sh" >> ~/.bashrc

#=======================================================================================
# [2019.x/2020.x/2021.x-android] Support usage of installed Android SDKs outside of editor
# [2019.x/2020.x/2021.x/2022.x-android] Setup Android SDK and NDK Variables
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^20(?!18).*android' \
&& exit 0 \
|| : \
# Environment Variables
&& export RAW_ANDROID_SDK_ROOT=$(jq -cr '(.[] | select(.id == "android-sdk-platform-tools")).destination' $UNITY_PATH/modules.json) \
&& export RAW_ANDROID_SDK_ROOT=$(jq -cr '(.[] | select(.id | contains("android-sdk-platform-tools"))).destination' $UNITY_PATH/modules.json) \
# We need to replace some characters common to paths that will break the sed expression when expanded
&& export ESCAPED_UNITY_PATH=$(printf '%s' "$UNITY_PATH" | sed 's/[#\/]/\\\0/g') \
&& export ANDROID_SDK_ROOT=$(echo $RAW_ANDROID_SDK_ROOT | sed -e "s/{UNITY_PATH}/$ESCAPED_UNITY_PATH/g") \
&& export ANDROID_HOME=${ANDROID_SDK_ROOT} \
&& export RAW_ANDROID_NDK_ROOT=$(jq -cr '(.[] | select(.id == "android-ndk")).destination' $UNITY_PATH/modules.json) \
&& export RAW_ANDROID_NDK_ROOT=$(jq -cr '(.[] | select(.id | contains("android-ndk"))).destination' $UNITY_PATH/modules.json) \
&& export ANDROID_NDK_HOME=$(echo $RAW_ANDROID_NDK_ROOT | sed -e "s/{UNITY_PATH}/$ESCAPED_UNITY_PATH/g") \
&& export RAW_JAVA_HOME=$(jq -cr '(.[] | select(.id == "android-open-jdk")).destination' $UNITY_PATH/modules.json) \
&& export RAW_JAVA_HOME=$(jq -cr '(.[] | select(.id | contains("android-open-jdk"))).destination' $UNITY_PATH/modules.json) \
&& export ESCAPED_JAVA_HOME=$(echo $RAW_JAVA_HOME | sed -e "s/{UNITY_PATH}/$ESCAPED_UNITY_PATH/g") \
# Unity 2019.x doesn't have the jdk in the modules, so put in a fallback. sdkmanager will fail if invaild
&& export JAVA_HOME=${ESCAPED_JAVA_HOME:-$UNITY_PATH/Editor/Data/PlaybackEngines/AndroidPlayer/Tools/OpenJDK/Linux} \
&& export PATH=$JAVA_HOME/bin:${ANDROID_SDK_ROOT}/tools:${ANDROID_SDK_ROOT}/tools/bin:${ANDROID_SDK_ROOT}/platform-tools:${PATH} \
\
# Accept licenses
&& yes | "${ANDROID_HOME}/tools/bin/sdkmanager" --licenses \
\
# Update alias 'unity-editor'
&& { \
echo "export ANDROID_SDK_ROOT=${ANDROID_SDK_ROOT}"; \
Expand All @@ -171,6 +168,56 @@ RUN echo "$version-$module" | grep -q -vP '^20(?!18).*android' \
# Update '~/.bashrc' to enable using variables when logging in
&& echo ". /usr/bin/unity-editor.d/android-2019+.sh" >> ~/.bashrc

#=======================================================================================
# [2021.x/2022.x-android] Set CMDLINE Tools Path
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^(202[1-9]|20[3-9]).*android' \
&& exit 0 \
|| : \
&& . ~/.bashrc \
&& export RAW_CMDLINE_TOOLS_PATH=$(jq -cr '(.[] | select(.id | contains("android-sdk-command-line-tools"))).renameTo' $UNITY_PATH/modules.json) \
&& export ESCAPED_UNITY_PATH=$(printf '%s' "$UNITY_PATH" | sed 's/[#\/]/\\\0/g') \
&& export ANDROID_CMDLINE_TOOLS_PATH=$(echo $RAW_CMDLINE_TOOLS_PATH | sed -e "s/{UNITY_PATH}/$ESCAPED_UNITY_PATH/g") \
# Prefer cmdline tools over legacy tools
&& export PATH=${ANDROID_CMDLINE_TOOLS_PATH}/bin:${PATH} \
&& { \
echo "export ANDROID_SDK_ROOT=${ANDROID_SDK_ROOT}"; \
echo "export ANDROID_HOME=${ANDROID_HOME}"; \
echo "export ANDROID_NDK_HOME=${ANDROID_NDK_HOME}"; \
echo "export JAVA_HOME=${JAVA_HOME}"; \
echo "export ANDROID_CMDLINE_TOOLS_PATH=${ANDROID_CMDLINE_TOOLS_PATH}"; \
echo "export PATH=${PATH}"; \
} > /usr/bin/unity-editor.d/android-2019+.sh

#=======================================================================================
# [2019.x/2020.x-android] Accept Android SDK licenses via old sdkmanager
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^(20(19|20)).*android' \
&& exit 0 \
|| : \
&& . ~/.bashrc \
&& yes | "${ANDROID_HOME}/tools/bin/sdkmanager" --licenses

#=======================================================================================
# [2021.x/2022.x-android] Accept Android SDK licenses via new cmdline-tools sdkmanager
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^(202[1-9]|20[3-9]).*android' \
&& exit 0 \
|| : \
&& . ~/.bashrc \
&& yes | "${ANDROID_CMDLINE_TOOLS_PATH}/bin/sdkmanager" --licenses

#=======================================================================================
# [2022.x-android] Fix for symlink issue on Android 2022.2+
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^(2022.[2-9]|202[3-9]|20[3-9]).*android' \
&& exit 0 \
|| : \
&& . ~/.bashrc \
&& cd "${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin" \
# Symlink any file less than 64 bytes to the file name within the file. We assume there are no real files that small
&& for f in $(find . -type f -size -64c); do target=$(cat $f) && echo "Making symlink $f -> $target" && rm $f && ln -s $target $f ; done
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we using file size here for the loop? Is it to identify files that are symlinks?
If so, we can do so by using the -L test to determine whether it is a symbolic link.

Example:

RUN echo "$version-$module" | grep -q -vP '^(2022.[2-9]|202[3-9]|20[3-9]).*android' \
  && exit 0 \
  || : \
  && . ~/.bashrc \
  && cd "${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin" \
  && for f in *; do \
       if [ -L "$f" ]; then \
         target=$(readlink "$f") && echo "Making symlink $f -> $target" && rm "$f" && ln -s "$target" "$f"; \
       fi \
     done

I haven't tried this

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the files aren't actually symbolic links, they're a mix of ascii and binary files but inside they have the file name of the executable they should point to. We suspect hub isn't using the proper flags when unzipping so the symbolic links get lost hence the need for us to make the links ourselves.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly more manageable to separate that part into a different PR? Not sure though. And not preference from my side.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the main point of the PR to fix the android build issues of 2022.2+ so I'm inclined to keep this as a single PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, makes sense!


#=======================================================================================
# [webgl] Support audio using ffmpeg (~99MB)
#=======================================================================================
Expand Down
3 changes: 2 additions & 1 deletion images/ubuntu/hub/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
ARG baseImage="unityci/base"
FROM $baseImage
ARG hubVersion="3.0.0"
ARG hubVersion="3.4.2-beta.1"

# Hub dependencies
RUN apt-get -q update \
Expand All @@ -10,6 +10,7 @@ RUN apt-get -q update \
# Install Unity Hub
# https://docs.unity3d.com/hub/manual/InstallHub.html#install-hub-linux
RUN sh -c 'echo "deb https://hub.unity3d.com/linux/repos/deb stable main" > /etc/apt/sources.list.d/unityhub.list' \
&& sh -c 'echo "deb https://hub.unity3d.com/linux/repos/deb unstable main" > /etc/apt/sources.list.d/unityhub-beta.list' \
&& wget -qO - https://hub.unity3d.com/linux/keys/public | apt-key add - \
&& apt-get -q update \
&& apt-get -q install -y "unityhub=$hubVersion" \
Expand Down
3 changes: 1 addition & 2 deletions images/windows/hub/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
ARG baseImage="unityci/base:windows-latest"

FROM $baseImage
ARG hubVersion="3.0.0"

# Install unity hub
RUN choco install unity-hub --version=%hubVersion --no-progress -y
RUN choco install unity-hub --no-progress -y
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove the version flag? Is the argument no longer valid?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It actually wasn't relevant from the start. Reading more into the readme for the chocolatey package, Unity doesn't store an archive of the older Unity hubs so it will always pull the latest version. The chocolatey script used a hard coded hash for each version to verify the download so putting an older version can cause errors during installs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They must have changed that. I wouldn't have added this param without it being documented.
Nice catch!