Intermittent crashes when running unity builds on docker. #243

diyang100 · 2024-04-05T15:11:08Z

Bug description

We are previously running our Unity CI builds on Windows, and wanted to run them in an isolated docker container environment for security reasons. The unity version is 2021.3.21f1 and we tried to set up under Ubuntu 22.04. The GameCI image tested is ubuntu-2021.3.21f1-webgl-3.0.1 and our own Dockerfile based on the Dockerfile by GameCI. We had only edited the file to support multiple modules. Unfortunately, we get two errors below occasionally, about an error every 3 or 4 runs.

Start importing Assets/TestPackagesShared/Art/Epic_BlueSunset/Epic_BlueSunset_Cam_2_Left+X.png using Guid(604c8e250817f485c97f85e8237f7524) Importer(-1,00000000000000000000000000000000) ./External/TextureCompressors/Crunch/crnlib/crn_threading_pthreads.cpp(194): Failure: ""semaphore: sem_wait() or sem_timedwait() failed""
Fatal error: Interrupted system call
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x2f
Obtained 43 stack frames.
#0  0x007f5a18e35520 in __sigaction
#1  0x007f5a18e899fc in pthread_kill
#2  0x007f5a18e35476 in raise
#3  0x007f5a18e1b7f3 in abort
#4  0x007f58d01685d2 in FailWithErrno()
#5  0x007f58d01685f1 in Mutex::~Mutex()
#6  0x007f5a18e38495 in secure_getenv
#7  0x007f5a18e38610 in exit
#8  0x00562e304653cc in crnlib_fail(char const*, char const*, unsigned int)
#9  0x00562e30463723 in crnlib::task_pool::join()
#10 0x00562e3048a4c8 in crnlib::tree_clusterizer<crnlib::vec<6u, float> >::split_node(std::priority_queue<crnlib::tree_clusterizer<crnlib::vec<6u, float> >::NodeInfo, std::vector<crnlib::tree_clusterizer<crnlib::vec<6u, float> >::NodeInfo, std::allocator<crnlib::tree_clusterizer<crnlib::vec<6u, float> >::NodeInfo> >, std::less<crnlib::tree_clusterizer<crnlib::vec<6u, float> >::NodeInfo> >&, unsigned int&, crnlib::task_pool*)
#11 0x00562e304790d6 in crnlib::tree_clusterizer<crnlib::vec<6u, float> >::generate_codebook(crnlib::vec<6u, float>*, unsigned int*, unsigned int, unsigned int, bool, crnlib::task_pool*)
#12 0x00562e3047238a in crnlib::dxt_hc::determine_color_endpoints()
#13 0x00562e3046fa45 in crnlib::dxt_hc::compress(crnlib::color_quad<unsigned char, int> (*) [16], crnlib::vector<crnlib::dxt_hc::endpoint_indices_details>&, crnlib::vector<crnlib::dxt_hc::selector_indices_details>&, crnlib::vector<unsigned int>&, crnlib::vector<unsigned int>&, crnlib::vector<unsigned int>&, crnlib::vector<unsigned long long>&, crnlib::dxt_hc::params const&)
#14 0x00562e304a3b82 in crnlib::crn_comp::quantize_images()
#15 0x00562e304a7ba1 in crnlib::crn_comp::compress_internal()
#16 0x00562e304a8263 in crnlib::crn_comp::compress_pass(crn_comp_params const&, float*)
#17 0x00562e304af1da in crnlib::create_compressed_texture(crn_comp_params const&, crnlib::vector<unsigned char>&, unsigned int*, float*)
#18 0x00562e304affeb in crnlib::create_compressed_texture(crn_comp_params const&, crn_mipmap_params const&, crnlib::vector<unsigned char>&, unsigned int*, float*)
#19 0x00562e3046411b in crn_compress(crn_comp_params const&, crn_mipmap_params const&, unsigned int&, unsigned int*, float*)
#20 0x00562e2de312e8 in CompressTextureWithMultipleImagesCrunch(Texture2D&, TextureFormat, int, bool, void (*)(float, void*), void*)
#21 0x00562e2f422f54 in CompressTexture(Texture2D&, TextureFormat, int, bool, bool)
#22 0x00562e2f400cbf in TextureGenerator::GenerateTexture(TextureGlobalSettingsProvider&, TextureGenerationSettings const&, TextureGenerationOutput&, AssetImportContext*)
#23 0x00562e3012c06f in TextureImporter::GenerateAssetData(AssetImportContext&)
#24 0x00562e2f9051a5 in ImportToObjects(core::basic_string<char, core::StringStorageDefault<char> > const&, core::hash_map<AssetDatabase::ImporterID, ImporterInfo, core::hash<AssetDatabase::ImporterID>, std::equal_to<AssetDatabase::ImporterID> > const&, dynamic_array<PostprocessorInfo, 0ul> const&, AssetResolveTracker const&, ImportAssetInputs const&, ImportAssetOutputs&, IImportProgress*)
#25 0x00562e2f90412e in ImportAsset(core::hash_map<AssetDatabase::ImporterID, ImporterInfo, core::hash<AssetDatabase::ImporterID>, std::equal_to<AssetDatabase::ImporterID> > const&, dynamic_array<PostprocessorInfo, 0ul> const&, AssetResolveTracker const&, ImportAssetInputs const&, ImportAssetOutputs&, IImportProgress*)
#26 0x00562e2f92eb0e in AssetImportWorker::Import(AssetResolver&, ImportAssetInputs const&, WorkerImportResponse&, IImportProgress*, bool*)
#27 0x00562e2f97417c in AssetImportManager::ImportInProcess(AssetResolver&, ArtifactKey const&, AssetImportParameters const&, AssetDatabase::ArtifactID&, IImportProgress*, core::hash_set<core::basic_string<char, core::StringStorageDefault<char> >, core::hash<core::basic_string<char, core::StringStorageDefault<char> > >, std::equal_to<core::basic_string<char, core::StringStorageDefault<char> > > >&, CrashedAssetImportsDatabase&)
#28 0x00562e2f95fc58 in AssetImportManager::Import(core::hash_map<AssetDatabase::ImporterID, ImporterInfo, core::hash<AssetDatabase::ImporterID>, std::equal_to<AssetDatabase::ImporterID> > const&, dynamic_array<PostprocessorInfo, 0ul> const&, AssetResolver&, CategorizedAssets&, IImportProgress*, core::hash_set<UnityGUID, core::hash<UnityGUID>, std::equal_to<UnityGUID> > const&, LoadedImportedAssetsSnapshot*, LoadedSourceAssetsSnapshot*, core::hash_set<core::basic_string<char, core::StringStorageDefault<char> >, core::hash<core::basic_string<char, core::StringStorageDefault<char> > >, std::equal_to<core::basic_string<char, core::StringStorageDefault<char> > > >&, CrashedAssetImportsDatabase&, OnDemandScheduler&)
#29 0x00562e2f987b81 in RefreshInternalV2(AssetDatabase::UpdateAssetOptions, ScanFilter const&, InternalRefreshFlagsV2)
#30 0x00562e2f96222b in StopAssetImportingV2Internal(AssetDatabase::UpdateAssetOptions, InternalRefreshFlagsV2, ScanFilter const*, char const*)
#31 0x00562e2f964969 in InitialRefreshV2(bool)
#32 0x00562e2f5179e7 in Application::InitializeProject()
#33 0x00562e2f5c176a in InitializeUnity(void*)
#34 0x007f5a1c51c2c8 in g_main_context_check
#35 0x007f5a1c51bc44 in g_main_context_dispatch
#36 0x007f5a1c571258 in g_io_channel_new_file
#37 0x007f5a1c51b2b3 in g_main_loop_run
#38 0x007f5a1961fcfd in gtk_main
#39 0x00562e2f5c0efc in main
#40 0x007f5a18e1cd90 in __libc_init_first
#41 0x007f5a18e1ce40 in __libc_start_main
#42 0x00562e2d775029 in _start
Aborted (core dumped)

The second (separate) error we occasionally get is:

An error occurred while resolving packages:
  One or more packages could not be added to the local file system:
    com.unity.burst: aborted

A re-import of the project may be required to fix the issue or a manual modification of ./Packages/manifest.json file.
/home/bokken/build/output/unity/unity/External/baselib/builds/Include/C/Internal/Baselib_Semaphore_FutexBased.inl.h(151): Assertion failed (count >= 0) - Destruction is not allowed when there are still threads waiting on the semaphore.

To try to fix the error, we set vm.max_map_count and kernel.sem on both the build machine and docker container. But it still meets the errors above. Script to set sysctl:

memorymaps=$(cat /proc/sys/vm/max_map_count)
echo "max_map_count:" $memorymaps
if [ $memorymaps -le 100000 ]; then
    echo "memory maps less than 100000. updating config."
    sysctl -w vm.max_map_count=262144
fi
semaphoreconfig=$(cat /proc/sys/kernel/sem)
semaphoreconfig=( $semaphoreconfig ) # expand into array
echo "sem:" ${semaphoreconfig[0]} ${semaphoreconfig[1]} ${semaphoreconfig[2]} ${semaphoreconfig[3]}
if [ ${semaphoreconfig[0]} -le 50000 ]; then
    echo "semaphore config less than 50000. updating config."
    sysctl -w kernel.sem="50100 128256000 50100 2560"
fi

How to reproduce

Dockerfile:

# Licensed from Game-CI, see full license in LICENSE.txt
# Modified by Spatial

ARG hubImage="unityci/hub"
ARG baseImage="unityci/base"

###########################
#         Builder         #
###########################

FROM $hubImage AS builder

# Install editor

# Note that the arguments `version`, `changeSet`, and `module` are placeholders and are overriden
# in `Pipeline/tasks/unity-docker.yml` when the Dockerfile is run. This provides auto-updates for
# when we decide to upgrade unity.
ARG version="2021.3.36f1"
ARG changeSet="7a0645017be0"
RUN unity-hub install --version "$version" --changeset "$changeSet" | tee /var/log/install-editor.log && grep 'Failed to install\|Error while installing an editor\|Completed with errors' /var/log/install-editor.log | exit $(wc -l)

# Install modules for that editor
ARG module="android ios webgl windows-mono"
RUN for mod in $module; do \
      if [ "$mod" = "base" ] ; then \
        echo "running default modules for this baseOs"; \
      else \
        unity-hub install-modules --version "$version" --module "$mod" --childModules | tee /var/log/install-module-${mod}.log && grep 'Missing module\|Completed with errors' /var/log/install-module-${mod}.log | exit $(wc -l); \
      fi \
    done \
	# Set execute permissions for modules
	&& chmod -R 755 /opt/unity/editors/$version/Editor/Data/PlaybackEngines

RUN echo "$version-$module" | grep -q -vP '^(2021.2.(?![0-4](?![0-9]))|2021.[3-9]|202[2-9]|20[3-9]).*linux' \
  && exit 0 \
  || unity-hub install-modules --version "$version" --module "linux-server" --childModules | tee /var/log/install-module-linux-server.log && grep 'Missing module' /var/log/install-module-linux-server.log | exit $(wc -l);

RUN echo "$version-$module" | grep -q -vP '^(2021.2.(?![0-4](?![0-9]))|2021.[3-9]|202[2-9]|20[3-9]).*windows' \
  && exit 0 \
  || unity-hub install-modules --version "$version" --module "windows-server" --childModules | tee /var/log/install-module-windows-server.log && grep 'Missing module' /var/log/install-module-windows-server.log | exit $(wc -l);

###########################
#          Editor         #
###########################

FROM $baseImage

# Always put "Editor" and "modules.json" directly in $UNITY_PATH
ARG version="2021.3.21f1"
ARG module="android ios webgl windows-mono"
ENV UNITY_PATH="/opt/unity/editors/$version"
COPY --from=builder /opt/unity/editors/$version/ "$UNITY_PATH/"
COPY --from=builder /opt/unityhub /opt/unityhub

# Add a file containing the version for this build
RUN echo $version > "$UNITY_PATH/version"


###########################
#  Alias to unity-editor  #
###########################

RUN /bin/echo -e '#!/bin/bash\n\
\n\
if [ -d /usr/bin/unity-editor.d ] ; then\n\
  for i in /usr/bin/unity-editor.d/*.sh; do\n\
    if [ -r $i ]; then\n\
      . $i\n\
    fi\n\
  done\n\
fi\n\
\n\
xvfb-run -ae /dev/stdout "$UNITY_PATH/Editor/Unity" -batchmode "$@"' > /usr/bin/unity-editor \
  && chmod 755 /usr/bin/unity-editor \
  && mkdir /usr/bin/unity-editor.d \
  && echo > ~/.bashrc # start from empty to keep "Validate Android Utils" CI step happy.

#=======================================================================================
# [2019.3.[0-5]-linux-il2cpp] https://github.com/game-ci/docker/issues/76
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^2019.3.[0-5]f.*linux-il2cpp' \
  && exit 0 \
  || echo 'export IL2CPP_ADDITIONAL_ARGS=--tool-chain-path=/' >> /usr/bin/unity-editor.d/linux-il2cpp-2019.3.5.and.older.sh

#=======================================================================================
# [2019.3.6+/2019.4.0-linux-il2cpp] https://forum.unity.com/threads/unity-2019-3-linux-il2cpp-player-can-only-be-built-with-linux-error.822210/#post-5633977
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^\(2019.3.[6-9]f\|2019.3.1[0-9]f\|2019.4.0\).*linux-il2cpp' \
  && exit 0 \
  || echo 'export IL2CPP_ADDITIONAL_ARGS="--sysroot-path=/ --tool-chain-path=/"' >> /usr/bin/unity-editor.d/linux-il2cpp-2019.3-4.sh

#=======================================================================================
# [2020.x/2020.2.0/2020.2.1-webgl] Support GZip compression: https://github.com/game-ci/docker/issues/75
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^\(2020.1\|2020.2.0f\|2020.2.1f\).*-webgl' \
  && exit 0 \
  || : \
  && wget https://old-releases.ubuntu.com/ubuntu/pool/main/g/gzip/gzip_1.6-5ubuntu2_amd64.deb \
  && dpkg -i gzip_1.6-5ubuntu2_amd64.deb \
  && rm gzip_1.6-5ubuntu2_amd64.deb \
  && echo 'export GZIP=-f' >> /usr/bin/unity-editor.d/webgl-2020.1-2.sh


###########################
#       Extra steps       #
###########################

#=======================================================================================
# [2018.x-android] Install 'Android SDK 26.1.1' and 'Android NDK 16.1.4479499'
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^2018\.[34].*android' \
  && exit 0 \
  || : \
  # Versions
  && export ANDROID_BUILD_TOOLS_VERSION=28.0.3 \
  && export ANDROID_NDK_VERSION=16.1.4479499 \
  \
  # Environment Variables
  && export ANDROID_INSTALL_LOCATION=${UNITY_PATH}/Editor/Data/PlaybackEngines/AndroidPlayer \
  && export ANDROID_SDK_ROOT=${ANDROID_INSTALL_LOCATION}/SDK \
  && export ANDROID_HOME=${ANDROID_SDK_ROOT} \
  && export ANDROID_NDK_HOME=${ANDROID_SDK_ROOT}/ndk/${ANDROID_NDK_VERSION} \
  && export JAVA_HOME=${UNITY_PATH}/Editor/Data/PlaybackEngines/AndroidPlayer/Tools/OpenJDK/Linux \
  && export PATH=$JAVA_HOME/bin:${ANDROID_SDK_ROOT}/tools:${ANDROID_SDK_ROOT}/tools/bin:${ANDROID_SDK_ROOT}/platform-tools:${PATH} \
  \
  # Download Android SDK (commandline tools) 26.1.1
  && mkdir -p ${ANDROID_SDK_ROOT} \
  && chmod -R 777 ${ANDROID_INSTALL_LOCATION} \
  && wget -q https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip -O /tmp/android-sdk.zip \
  && unzip -q /tmp/android-sdk.zip -d ${ANDROID_SDK_ROOT} \
  \
  # Install platform-tools, NDK 16.1.4479499 and build-tools 28.0.3
  && yes | sdkmanager \
    "platform-tools" \
    "ndk;${ANDROID_NDK_VERSION}" \
    "build-tools;${ANDROID_BUILD_TOOLS_VERSION}" \
    > /dev/null \
  \
  # Accept licenses
  && yes | "${ANDROID_HOME}/tools/bin/sdkmanager" --licenses \
  \
  # Update alias 'unity-editor'
  && { \
    echo "export ANDROID_SDK_ROOT=${ANDROID_SDK_ROOT}"; \
    echo "export ANDROID_HOME=${ANDROID_HOME}"; \
    echo "export ANDROID_NDK_HOME=${ANDROID_NDK_HOME}"; \
    echo "export JAVA_HOME=${JAVA_HOME}"; \
    echo "export PATH=${PATH}"; \# Licensed from Game-CI, see full license in LICENSE.txt
# Modified by Spatial

ARG hubImage="unityci/hub"
ARG baseImage="unityci/base"

###########################
#         Builder         #
###########################

FROM $hubImage AS builder

# Install editor

# Note that the arguments `version`, `changeSet`, and `module` are placeholders and are overriden
# in `Pipeline/tasks/unity-docker.yml` when the Dockerfile is run. This provides auto-updates for
# when we decide to upgrade unity.
ARG version="2021.3.36f1"
ARG changeSet="7a0645017be0"
RUN unity-hub install --version "$version" --changeset "$changeSet" | tee /var/log/install-editor.log && grep 'Failed to install\|Error while installing an editor\|Completed with errors' /var/log/install-editor.log | exit $(wc -l)

# Install modules for that editor
ARG module="android ios webgl windows-mono"
RUN for mod in $module; do \
      if [ "$mod" = "base" ] ; then \
        echo "running default modules for this baseOs"; \
      else \
        unity-hub install-modules --version "$version" --module "$mod" --childModules | tee /var/log/install-module-${mod}.log && grep 'Missing module\|Completed with errors' /var/log/install-module-${mod}.log | exit $(wc -l); \
      fi \
    done \
	# Set execute permissions for modules
	&& chmod -R 755 /opt/unity/editors/$version/Editor/Data/PlaybackEngines

RUN echo "$version-$module" | grep -q -vP '^(2021.2.(?![0-4](?![0-9]))|2021.[3-9]|202[2-9]|20[3-9]).*linux' \
  && exit 0 \
  || unity-hub install-modules --version "$version" --module "linux-server" --childModules | tee /var/log/install-module-linux-server.log && grep 'Missing module' /var/log/install-module-linux-server.log | exit $(wc -l);

RUN echo "$version-$module" | grep -q -vP '^(2021.2.(?![0-4](?![0-9]))|2021.[3-9]|202[2-9]|20[3-9]).*windows' \
  && exit 0 \
  || unity-hub install-modules --version "$version" --module "windows-server" --childModules | tee /var/log/install-module-windows-server.log && grep 'Missing module' /var/log/install-module-windows-server.log | exit $(wc -l);

###########################
#          Editor         #
###########################

FROM $baseImage

# Always put "Editor" and "modules.json" directly in $UNITY_PATH
ARG version="2021.3.36f1"
ARG module="android ios webgl windows-mono"
ENV UNITY_PATH="/opt/unity/editors/$version"
COPY --from=builder /opt/unity/editors/$version/ "$UNITY_PATH/"
COPY --from=builder /opt/unityhub /opt/unityhub

# Add a file containing the version for this build
RUN echo $version > "$UNITY_PATH/version"


###########################
#  Alias to unity-editor  #
###########################

RUN /bin/echo -e '#!/bin/bash\n\
\n\
if [ -d /usr/bin/unity-editor.d ] ; then\n\
  for i in /usr/bin/unity-editor.d/*.sh; do\n\
    if [ -r $i ]; then\n\
      . $i\n\
    fi\n\
  done\n\
fi\n\
\n\
xvfb-run -ae /dev/stdout "$UNITY_PATH/Editor/Unity" -batchmode "$@"' > /usr/bin/unity-editor \
  && chmod 755 /usr/bin/unity-editor \
  && mkdir /usr/bin/unity-editor.d \
  && echo > ~/.bashrc # start from empty to keep "Validate Android Utils" CI step happy.

#=======================================================================================
# [2019.3.[0-5]-linux-il2cpp] https://github.com/game-ci/docker/issues/76
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^2019.3.[0-5]f.*linux-il2cpp' \
  && exit 0 \
  || echo 'export IL2CPP_ADDITIONAL_ARGS=--tool-chain-path=/' >> /usr/bin/unity-editor.d/linux-il2cpp-2019.3.5.and.older.sh

#=======================================================================================
# [2019.3.6+/2019.4.0-linux-il2cpp] https://forum.unity.com/threads/unity-2019-3-linux-il2cpp-player-can-only-be-built-with-linux-error.822210/#post-5633977
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^\(2019.3.[6-9]f\|2019.3.1[0-9]f\|2019.4.0\).*linux-il2cpp' \
  && exit 0 \
  || echo 'export IL2CPP_ADDITIONAL_ARGS="--sysroot-path=/ --tool-chain-path=/"' >> /usr/bin/unity-editor.d/linux-il2cpp-2019.3-4.sh

#=======================================================================================
# [2020.x/2020.2.0/2020.2.1-webgl] Support GZip compression: https://github.com/game-ci/docker/issues/75
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^\(2020.1\|2020.2.0f\|2020.2.1f\).*-webgl' \
  && exit 0 \
  || : \
  && wget https://old-releases.ubuntu.com/ubuntu/pool/main/g/gzip/gzip_1.6-5ubuntu2_amd64.deb \
  && dpkg -i gzip_1.6-5ubuntu2_amd64.deb \
  && rm gzip_1.6-5ubuntu2_amd64.deb \
  && echo 'export GZIP=-f' >> /usr/bin/unity-editor.d/webgl-2020.1-2.sh


###########################
#       Extra steps       #
###########################

#=======================================================================================
# [2018.x-android] Install 'Android SDK 26.1.1' and 'Android NDK 16.1.4479499'
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^2018\.[34].*android' \
  && exit 0 \
  || : \
  # Versions
  && export ANDROID_BUILD_TOOLS_VERSION=28.0.3 \
  && export ANDROID_NDK_VERSION=16.1.4479499 \
  \
  # Environment Variables
  && export ANDROID_INSTALL_LOCATION=${UNITY_PATH}/Editor/Data/PlaybackEngines/AndroidPlayer \
  && export ANDROID_SDK_ROOT=${ANDROID_INSTALL_LOCATION}/SDK \
  && export ANDROID_HOME=${ANDROID_SDK_ROOT} \
  && export ANDROID_NDK_HOME=${ANDROID_SDK_ROOT}/ndk/${ANDROID_NDK_VERSION} \
  && export JAVA_HOME=${UNITY_PATH}/Editor/Data/PlaybackEngines/AndroidPlayer/Tools/OpenJDK/Linux \
  && export PATH=$JAVA_HOME/bin:${ANDROID_SDK_ROOT}/tools:${ANDROID_SDK_ROOT}/tools/bin:${ANDROID_SDK_ROOT}/platform-tools:${PATH} \
  \
  # Download Android SDK (commandline tools) 26.1.1
  && mkdir -p ${ANDROID_SDK_ROOT} \
  && chmod -R 777 ${ANDROID_INSTALL_LOCATION} \
  && wget -q https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip -O /tmp/android-sdk.zip \
  && unzip -q /tmp/android-sdk.zip -d ${ANDROID_SDK_ROOT} \
  \
  # Install platform-tools, NDK 16.1.4479499 and build-tools 28.0.3
  && yes | sdkmanager \
    "platform-tools" \
    "ndk;${ANDROID_NDK_VERSION}" \
    "build-tools;${ANDROID_BUILD_TOOLS_VERSION}" \
    > /dev/null \
  \
  # Accept licenses
  && yes | "${ANDROID_HOME}/tools/bin/sdkmanager" --licenses \
  \
  # Update alias 'unity-editor'
  && { \
    echo "export ANDROID_SDK_ROOT=${ANDROID_SDK_ROOT}"; \
    echo "export ANDROID_HOME=${ANDROID_HOME}"; \
    echo "export ANDROID_NDK_HOME=${ANDROID_NDK_HOME}"; \
    echo "export JAVA_HOME=${JAVA_HOME}"; \
    echo "export PATH=${PATH}"; \
  } > /usr/bin/unity-editor.d/android-2018.3-4.sh \
  # Update '~/.bashrc' to enable using variables when logging in
  && echo ". /usr/bin/unity-editor.d/android-2018.3-4.sh" >> ~/.bashrc

#=======================================================================================
# [2019.x/2020.x/2021.x/2022.x-android] Setup Android SDK and NDK Variables
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^20(?!18).*android' \
  && exit 0 \
  || : \
  # Environment Variables
  && export RAW_ANDROID_SDK_ROOT=$(jq -cr '(.[] | select(.id | contains("android-sdk-platform-tools"))).destination' $UNITY_PATH/modules.json) \
  # We need to replace some characters common to paths that will break the sed expression when expanded
  && export ESCAPED_UNITY_PATH=$(printf '%s' "$UNITY_PATH" | sed 's/[#\/]/\\\0/g') \
  && export ANDROID_SDK_ROOT=$(echo $RAW_ANDROID_SDK_ROOT | sed -e "s/{UNITY_PATH}/$ESCAPED_UNITY_PATH/g") \
  && export ANDROID_HOME=${ANDROID_SDK_ROOT} \
  && export RAW_ANDROID_NDK_ROOT=$(jq -cr '(.[] | select(.id | contains("android-ndk"))).destination' $UNITY_PATH/modules.json) \
  && export ANDROID_NDK_HOME=$(echo $RAW_ANDROID_NDK_ROOT | sed -e "s/{UNITY_PATH}/$ESCAPED_UNITY_PATH/g") \
  && export RAW_JAVA_HOME=$(jq -cr '(.[] | select(.id | contains("android-open-jdk"))).destination' $UNITY_PATH/modules.json) \
  && export ESCAPED_JAVA_HOME=$(echo $RAW_JAVA_HOME | sed -e "s/{UNITY_PATH}/$ESCAPED_UNITY_PATH/g") \
  # Unity 2019.x doesn't have the jdk in the modules, so put in a fallback. sdkmanager will fail if invaild
  && export JAVA_HOME=${ESCAPED_JAVA_HOME:-$UNITY_PATH/Editor/Data/PlaybackEngines/AndroidPlayer/Tools/OpenJDK/Linux} \
  && export PATH=$JAVA_HOME/bin:${ANDROID_SDK_ROOT}/tools:${ANDROID_SDK_ROOT}/tools/bin:${ANDROID_SDK_ROOT}/platform-tools:${PATH} \
  \
  # Update alias 'unity-editor'
  && { \
    echo "export ANDROID_SDK_ROOT=${ANDROID_SDK_ROOT}"; \
    echo "export ANDROID_HOME=${ANDROID_HOME}"; \
    echo "export ANDROID_NDK_HOME=${ANDROID_NDK_HOME}"; \
    echo "export JAVA_HOME=${JAVA_HOME}"; \
    echo "export PATH=${PATH}"; \
  } > /usr/bin/unity-editor.d/android-2019+.sh \
  # Update '~/.bashrc' to enable using variables when logging in
  && echo ". /usr/bin/unity-editor.d/android-2019+.sh" >> ~/.bashrc

#=======================================================================================
# [2021.x/2022.x-android] Set CMDLINE Tools Path
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^(202[1-9]|20[3-9]).*android' \
  && exit 0 \
  || : \
  && . ~/.bashrc \
  && export RAW_CMDLINE_TOOLS_PATH=$(jq -cr '(.[] | select(.id | contains("android-sdk-command-line-tools"))).renameTo' $UNITY_PATH/modules.json) \
  && export ESCAPED_UNITY_PATH=$(printf '%s' "$UNITY_PATH" | sed 's/[#\/]/\\\0/g') \
  && export ANDROID_CMDLINE_TOOLS_PATH=$(echo $RAW_CMDLINE_TOOLS_PATH | sed -e "s/{UNITY_PATH}/$ESCAPED_UNITY_PATH/g") \
  # Prefer cmdline tools over legacy tools
  && export PATH=${ANDROID_CMDLINE_TOOLS_PATH}/bin:${PATH} \
  && { \
    echo "export ANDROID_SDK_ROOT=${ANDROID_SDK_ROOT}"; \
    echo "export ANDROID_HOME=${ANDROID_HOME}"; \
    echo "export ANDROID_NDK_HOME=${ANDROID_NDK_HOME}"; \
    echo "export JAVA_HOME=${JAVA_HOME}"; \
    echo "export ANDROID_CMDLINE_TOOLS_PATH=${ANDROID_CMDLINE_TOOLS_PATH}"; \
    echo "export PATH=${PATH}"; \
  } > /usr/bin/unity-editor.d/android-2019+.sh

#=======================================================================================
# [2019.x/2020.x-android] Accept Android SDK licenses via old sdkmanager
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^(20(19|20)).*android' \
  && exit 0 \
  || : \
  && . ~/.bashrc \
  && yes | "${ANDROID_HOME}/tools/bin/sdkmanager" --licenses

#=======================================================================================
# [2021.x/2022.x-android] Accept Android SDK licenses via new cmdline-tools sdkmanager
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^(202[1-9]|20[3-9]).*android' \
  && exit 0 \
  || : \
  && . ~/.bashrc \
  && yes | "${ANDROID_CMDLINE_TOOLS_PATH}/bin/sdkmanager" --licenses

#=======================================================================================
# [2022.x-android] Fix for symlink issue on Android 2022.2+
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^(2022.[2-9]|202[3-9]|20[3-9]).*android' \
  && exit 0 \
  || : \
  && . ~/.bashrc \
  && cd "${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin" \
  # Symlink any file less than 64 bytes to the file name within the file. We assume there are no real files that small
  && for f in $(find . -type f -size -64c); do target=$(cat $f) && echo "Making symlink $f -> $target" && rm $f && ln -s $target $f ; done

#=======================================================================================
# [webgl] Support audio using ffmpeg (~99MB)
#=======================================================================================
RUN echo "$module" | grep -q -v 'webgl' \
  && exit 0 \
  || : \
  && apt-get update \
  && apt-get -q install -y --no-install-recommends --allow-downgrades \
    ffmpeg \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

#=======================================================================================
# [webgl, il2cpp] build-essential clang
#=======================================================================================
RUN echo "$module" | grep -q -v '\(webgl\|linux-il2cpp\)' \
  && exit 0 \
  || : \
  && apt-get -q update \
  && apt-get -q install -y --no-install-recommends --allow-downgrades \
    build-essential \
    clang \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

#=======================================================================================
# [2019.x] libnotify4 libunwind-dev libssl1.0.0
#=======================================================================================
RUN echo "$version" | grep -q -v '^2019.' \
  && exit 0 \
  || : \
  && apt-get -q update \
  && apt-get -q install -y --no-install-recommends --allow-downgrades \
    libnotify4 \
    libunwind-dev \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/* \
  # Install libssl1.0.0
  && wget http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.0.0_1.0.2g-1ubuntu4.20_amd64.deb \
  && dpkg -i libssl1.0.0_1.0.2g-1ubuntu4.20_amd64.deb \
  && rm libssl1.0.0_1.0.2g-1ubuntu4.20_amd64.deb

#=======================================================================================
# [2018.x/2019.x/2020.x/2021.1.x-webgl] python2
#=======================================================================================

RUN echo "$version-$module" | grep -q -v '^\(2018\|2019\|2020\|2021.1\).*webgl' \
  && exit 0 \
  || : \
  && apt-get -q update \
  && apt-get -q install -y --no-install-recommends --allow-downgrades \
    python-setuptools \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/* \
  && ln -s /usr/bin/python2 /usr/bin/python

#=======================================================================================
# [2018.x/2019.x/2020.1.x-webgl] support brotli compression for linux
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^\(2018\|2019\|2020.1\).*webgl' \
  && exit 0 \
  || : \
  && cp \
    $UNITY_PATH/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/Brotli/dist/Brotli-0.4.0-py2.7-linux-x86_64.egg \
    $UNITY_PATH/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/Brotli/dist/Brotli-0.4.0-py2.7-macosx-10.10-x86_64.egg

#=======================================================================================
# [2021.x/2022.x-mac-mono] x64arm64/x64ARM64 case issue https://github.com/game-ci/unity-builder/issues/320
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^202[12].*mac-mono' \
  && exit 0 \
  || : \
  && ln -s \
    $UNITY_PATH/Editor/Data/PlaybackEngines/MacStandaloneSupport/Variations/macos_x64arm64_player_nondevelopment_mono \
    $UNITY_PATH/Editor/Data/PlaybackEngines/MacStandaloneSupport/Variations/macos_x64ARM64_player_nondevelopment_mono \
  && ln -s \
    $UNITY_PATH/Editor/Data/PlaybackEngines/MacStandaloneSupport/Variations/macos_x64arm64_player_development_mono \
    $UNITY_PATH/Editor/Data/PlaybackEngines/MacStandaloneSupport/Variations/macos_x64ARM64_player_development_mono

#=======================================================================================
# [2021.x-linux-il2cpp] lld
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^2021.*linux-il2cpp' \
  && exit 0 \
  || : \
  && apt-get -q update \
  && apt-get -q install -y --no-install-recommends --allow-downgrades \
    lld \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

#=======================================================================================
# Install blender
#=======================================================================================

RUN mkdir /blender && \
  curl -o \
    /tmp/blender.tar.xz -L \
    https://mirrors.iu13.net/blender/release/Blender3.6/blender-3.6.4-linux-x64.tar.xz && \
  tar xf \
    /tmp/blender.tar.xz -C \
    /blender/ --strip-components=1 && \
  ln -s \
    /blender/blender \
    /usr/bin/blender
  } > /usr/bin/unity-editor.d/android-2018.3-4.sh \
  # Update '~/.bashrc' to enable using variables when logging in
  && echo ". /usr/bin/unity-editor.d/android-2018.3-4.sh" >> ~/.bashrc

#=======================================================================================
# [2019.x/2020.x/2021.x/2022.x-android] Setup Android SDK and NDK Variables
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^20(?!18).*android' \
  && exit 0 \
  || : \
  # Environment Variables
  && export RAW_ANDROID_SDK_ROOT=$(jq -cr '(.[] | select(.id | contains("android-sdk-platform-tools"))).destination' $UNITY_PATH/modules.json) \
  # We need to replace some characters common to paths that will break the sed expression when expanded
  && export ESCAPED_UNITY_PATH=$(printf '%s' "$UNITY_PATH" | sed 's/[#\/]/\\\0/g') \
  && export ANDROID_SDK_ROOT=$(echo $RAW_ANDROID_SDK_ROOT | sed -e "s/{UNITY_PATH}/$ESCAPED_UNITY_PATH/g") \
  && export ANDROID_HOME=${ANDROID_SDK_ROOT} \
  && export RAW_ANDROID_NDK_ROOT=$(jq -cr '(.[] | select(.id | contains("android-ndk"))).destination' $UNITY_PATH/modules.json) \
  && export ANDROID_NDK_HOME=$(echo $RAW_ANDROID_NDK_ROOT | sed -e "s/{UNITY_PATH}/$ESCAPED_UNITY_PATH/g") \
  && export RAW_JAVA_HOME=$(jq -cr '(.[] | select(.id | contains("android-open-jdk"))).destination' $UNITY_PATH/modules.json) \
  && export ESCAPED_JAVA_HOME=$(echo $RAW_JAVA_HOME | sed -e "s/{UNITY_PATH}/$ESCAPED_UNITY_PATH/g") \
  # Unity 2019.x doesn't have the jdk in the modules, so put in a fallback. sdkmanager will fail if invaild
  && export JAVA_HOME=${ESCAPED_JAVA_HOME:-$UNITY_PATH/Editor/Data/PlaybackEngines/AndroidPlayer/Tools/OpenJDK/Linux} \
  && export PATH=$JAVA_HOME/bin:${ANDROID_SDK_ROOT}/tools:${ANDROID_SDK_ROOT}/tools/bin:${ANDROID_SDK_ROOT}/platform-tools:${PATH} \
  \
  # Update alias 'unity-editor'
  && { \
    echo "export ANDROID_SDK_ROOT=${ANDROID_SDK_ROOT}"; \
    echo "export ANDROID_HOME=${ANDROID_HOME}"; \
    echo "export ANDROID_NDK_HOME=${ANDROID_NDK_HOME}"; \
    echo "export JAVA_HOME=${JAVA_HOME}"; \
    echo "export PATH=${PATH}"; \
  } > /usr/bin/unity-editor.d/android-2019+.sh \
  # Update '~/.bashrc' to enable using variables when logging in
  && echo ". /usr/bin/unity-editor.d/android-2019+.sh" >> ~/.bashrc

#=======================================================================================
# [2021.x/2022.x-android] Set CMDLINE Tools Path
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^(202[1-9]|20[3-9]).*android' \
  && exit 0 \
  || : \
  && . ~/.bashrc \
  && export RAW_CMDLINE_TOOLS_PATH=$(jq -cr '(.[] | select(.id | contains("android-sdk-command-line-tools"))).renameTo' $UNITY_PATH/modules.json) \
  && export ESCAPED_UNITY_PATH=$(printf '%s' "$UNITY_PATH" | sed 's/[#\/]/\\\0/g') \
  && export ANDROID_CMDLINE_TOOLS_PATH=$(echo $RAW_CMDLINE_TOOLS_PATH | sed -e "s/{UNITY_PATH}/$ESCAPED_UNITY_PATH/g") \
  # Prefer cmdline tools over legacy tools
  && export PATH=${ANDROID_CMDLINE_TOOLS_PATH}/bin:${PATH} \
  && { \
    echo "export ANDROID_SDK_ROOT=${ANDROID_SDK_ROOT}"; \
    echo "export ANDROID_HOME=${ANDROID_HOME}"; \
    echo "export ANDROID_NDK_HOME=${ANDROID_NDK_HOME}"; \
    echo "export JAVA_HOME=${JAVA_HOME}"; \
    echo "export ANDROID_CMDLINE_TOOLS_PATH=${ANDROID_CMDLINE_TOOLS_PATH}"; \
    echo "export PATH=${PATH}"; \
  } > /usr/bin/unity-editor.d/android-2019+.sh

#=======================================================================================
# [2019.x/2020.x-android] Accept Android SDK licenses via old sdkmanager
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^(20(19|20)).*android' \
  && exit 0 \
  || : \
  && . ~/.bashrc \
  && yes | "${ANDROID_HOME}/tools/bin/sdkmanager" --licenses

#=======================================================================================
# [2021.x/2022.x-android] Accept Android SDK licenses via new cmdline-tools sdkmanager
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^(202[1-9]|20[3-9]).*android' \
  && exit 0 \
  || : \
  && . ~/.bashrc \
  && yes | "${ANDROID_CMDLINE_TOOLS_PATH}/bin/sdkmanager" --licenses

#=======================================================================================
# [2022.x-android] Fix for symlink issue on Android 2022.2+
#=======================================================================================
RUN echo "$version-$module" | grep -q -vP '^(2022.[2-9]|202[3-9]|20[3-9]).*android' \
  && exit 0 \
  || : \
  && . ~/.bashrc \
  && cd "${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin" \
  # Symlink any file less than 64 bytes to the file name within the file. We assume there are no real files that small
  && for f in $(find . -type f -size -64c); do target=$(cat $f) && echo "Making symlink $f -> $target" && rm $f && ln -s $target $f ; done

#=======================================================================================
# [webgl] Support audio using ffmpeg (~99MB)
#=======================================================================================
RUN echo "$module" | grep -q -v 'webgl' \
  && exit 0 \
  || : \
  && apt-get update \
  && apt-get -q install -y --no-install-recommends --allow-downgrades \
    ffmpeg \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

#=======================================================================================
# [webgl, il2cpp] build-essential clang
#=======================================================================================
RUN echo "$module" | grep -q -v '\(webgl\|linux-il2cpp\)' \
  && exit 0 \
  || : \
  && apt-get -q update \
  && apt-get -q install -y --no-install-recommends --allow-downgrades \
    build-essential \
    clang \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

#=======================================================================================
# [2019.x] libnotify4 libunwind-dev libssl1.0.0
#=======================================================================================
RUN echo "$version" | grep -q -v '^2019.' \
  && exit 0 \
  || : \
  && apt-get -q update \
  && apt-get -q install -y --no-install-recommends --allow-downgrades \
    libnotify4 \
    libunwind-dev \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/* \
  # Install libssl1.0.0
  && wget http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.0.0_1.0.2g-1ubuntu4.20_amd64.deb \
  && dpkg -i libssl1.0.0_1.0.2g-1ubuntu4.20_amd64.deb \
  && rm libssl1.0.0_1.0.2g-1ubuntu4.20_amd64.deb

#=======================================================================================
# [2018.x/2019.x/2020.x/2021.1.x-webgl] python2
#=======================================================================================

RUN echo "$version-$module" | grep -q -v '^\(2018\|2019\|2020\|2021.1\).*webgl' \
  && exit 0 \
  || : \
  && apt-get -q update \
  && apt-get -q install -y --no-install-recommends --allow-downgrades \
    python-setuptools \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/* \
  && ln -s /usr/bin/python2 /usr/bin/python

#=======================================================================================
# [2018.x/2019.x/2020.1.x-webgl] support brotli compression for linux
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^\(2018\|2019\|2020.1\).*webgl' \
  && exit 0 \
  || : \
  && cp \
    $UNITY_PATH/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/Brotli/dist/Brotli-0.4.0-py2.7-linux-x86_64.egg \
    $UNITY_PATH/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/Brotli/dist/Brotli-0.4.0-py2.7-macosx-10.10-x86_64.egg

#=======================================================================================
# [2021.x/2022.x-mac-mono] x64arm64/x64ARM64 case issue https://github.com/game-ci/unity-builder/issues/320
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^202[12].*mac-mono' \
  && exit 0 \
  || : \
  && ln -s \
    $UNITY_PATH/Editor/Data/PlaybackEngines/MacStandaloneSupport/Variations/macos_x64arm64_player_nondevelopment_mono \
    $UNITY_PATH/Editor/Data/PlaybackEngines/MacStandaloneSupport/Variations/macos_x64ARM64_player_nondevelopment_mono \
  && ln -s \
    $UNITY_PATH/Editor/Data/PlaybackEngines/MacStandaloneSupport/Variations/macos_x64arm64_player_development_mono \
    $UNITY_PATH/Editor/Data/PlaybackEngines/MacStandaloneSupport/Variations/macos_x64ARM64_player_development_mono

#=======================================================================================
# [2021.x-linux-il2cpp] lld
#=======================================================================================
RUN echo "$version-$module" | grep -q -v '^2021.*linux-il2cpp' \
  && exit 0 \
  || : \
  && apt-get -q update \
  && apt-get -q install -y --no-install-recommends --allow-downgrades \
    lld \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

#=======================================================================================
# Install blender
#=======================================================================================

RUN mkdir /blender && \
  curl -o \
    /tmp/blender.tar.xz -L \
    https://mirrors.iu13.net/blender/release/Blender3.6/blender-3.6.4-linux-x64.tar.xz && \
  tar xf \
    /tmp/blender.tar.xz -C \
    /blender/ --strip-components=1 && \
  ln -s \
    /blender/blender \
    /usr/bin/blender

create bash script build.sh based on the following:

unity-editor \
            -logfile /usr/unitylibs/spatial-unity-environments/package-builder/$packageSKU-$packageVersion.log \
            -batchmode \
            -projectPath $projectPath \
            -buildTarget $unityBuildTarget \
            -executeMethod $scriptExecuteMethod \
            -platforms $platforms \
            -sku $packageSKU \
            -packageVersion $packageVersion \
            -brotliCompressionLevel $brotliCompressionLevel

run docker script:

 docker run --name ${{ parameters.packageSKU }}-${{ parameters.packageVersion }}\
            --mount type=bind,source=/root/unityconfig,target=/usr/share/unity3d/config/ \
            --mount type=bind,source="$(pwd)"/${{ parameters.packageBuilderProject }},target=/usr/unitylibs/spatial-unity-environments/package-builder \
            docker_image_name
            /bin/bash ./build.sh \
            ${{ parameters.unityVersion }} \
            /usr/unitylibs/spatial-unity-environments/package-builder \
            ${{ parameters.unityBuildTarget }} \
            ${{ parameters.scriptExecuteMethod }} \
            ${{ parameters.platforms }} \
            ${{ parameters.packageSKU }} \
            ${{ parameters.packageVersion }} \
            ${{ parameters.brotliCompressionLevel }}

Expected behavior

The build should be succeeding 100% of the time, but errors out 30% of the time.

Additional details

The text was updated successfully, but these errors were encountered:

portwatcher · 2024-08-02T09:06:36Z

I also encounter this probelm:

Begin MonoManager ReloadAssembly
TYPE: 3
V: wbarrier_conc
* Assertion at ../../mono/arch/amd64/../x86/x86-codegen.h:410, condition `offset == (gint32)offset' not met

Caught fatal signal - signo:6 code:-6 errno:0 addr:0x
...

The x86 ubuntu-2021.3.27f1-android-3.1.0 container is running by jenkins on a arm64 mac mini. I think the arch compatibility is very doubtful. But I have to stick to arm64 because my assigned job is to dockerize the ci with jenkins to eliminate the gap of archs of different machines.

portwatcher · 2024-08-02T09:14:12Z

very likely to be a bug in Mono
mono/mono#21423

CheeryLee · 2024-11-08T13:08:12Z

It was reported to Unity a few months ago:
https://issuetracker.unity3d.com/issues/linux-unity-editor-fails-to-import-the-project-in-batch-mode-when-building-on-the-linux-vm-environment

I noticed there that it's not really resolved and still persists in modern releases. Currently I'm looking forward to find some workarounds.

@portwatcher the problem posted by topic creator is not related to bug in Mono - crnlib is a DXT-cruncher that is a 3rd party stuff included in Unity installation.

diyang100 added the bug Something isn't working label Apr 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intermittent crashes when running unity builds on docker. #243

Intermittent crashes when running unity builds on docker. #243

diyang100 commented Apr 5, 2024 •

edited

Loading

portwatcher commented Aug 2, 2024 •

edited

Loading

portwatcher commented Aug 2, 2024

CheeryLee commented Nov 8, 2024 •

edited

Loading

Intermittent crashes when running unity builds on docker. #243

Intermittent crashes when running unity builds on docker. #243

Comments

diyang100 commented Apr 5, 2024 • edited Loading

portwatcher commented Aug 2, 2024 • edited Loading

portwatcher commented Aug 2, 2024

CheeryLee commented Nov 8, 2024 • edited Loading

diyang100 commented Apr 5, 2024 •

edited

Loading

portwatcher commented Aug 2, 2024 •

edited

Loading

CheeryLee commented Nov 8, 2024 •

edited

Loading