Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install assertpy on test VMs #2886

Merged
merged 4 commits into from
Jul 27, 2023
Merged

Install assertpy on test VMs #2886

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f657199
Install assertpy on test VMs
Jul 26, 2023
395682d
set versions
Jul 26, 2023
dcdd9f3
Merge branch 'develop' into assertpy
narrieta Jul 27, 2023
856a48c
Merge branch 'develop' into assertpy
narrieta Jul 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 18 additions & 5 deletions tests_e2e/orchestrator/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,27 +68,40 @@ RUN \
export PATH="$HOME/.local/bin:$PATH" && \
\
# \
# Install LISA \
# Install LISA. \
# \
# (note that we use a specific commit, which is the version of LISA that has been verified to work with our \
# tests; when taking a new LISA version, make sure to verify that the tests work OK before pushing the \
# Docker image to our registry) \
# \
cd $HOME && \
git clone https://github.com/microsoft/lisa.git && \
cd lisa && \
git checkout a030c5e6a0695db77dbf5bd52a45d07cbbf00087 && \
\
python3 -m pip install --upgrade pip && \
python3 -m pip install --editable .[azure,libvirt] --config-settings editable_mode=compat && \
\
# \
# Install additional test dependencies \
# \
# (note that we update azure-mgmt-compute to 29.1.0 - LISA installs 26.1; this is needed in order to access \
# osProfile.linuxConfiguration.enableVMAgentPlatformUpdates in the VM model - that property is used by some \
# tests, such as Agent versioning) \
# \
python3 -m pip install distro msrestazure pytz && \
python3 -m pip install azure-mgmt-compute --upgrade && \
python3 -m pip install azure-mgmt-compute==29.1.0 --upgrade && \
\
# \
# Download Pypy to a known location, from which it will be installed to the test VMs. \
# \
mkdir $HOME/bin && \
wget https://downloads.python.org/pypy/pypy3.7-v7.3.5-linux64.tar.bz2 -O /tmp/pypy3.7-x64.tar.bz2 && \
wget https://downloads.python.org/pypy/pypy3.7-v7.3.5-aarch64.tar.bz2 -O /tmp/pypy3.7-arm64.tar.bz2 && \
wget https://dcrdata.blob.core.windows.net/python/pypy3.7-x64.tar.bz2 -O /tmp/pypy3.7-x64.tar.bz2 && \
wget https://dcrdata.blob.core.windows.net/python/pypy3.7-arm64.tar.bz2 -O /tmp/pypy3.7-arm64.tar.bz2 && \
\
# \
# Install pudb, which can be useful to debug issues in the image \
# \
python3 -m pip install pudb && \
\
# \
# The setup for the tests depends on a few paths; add those to the profile \
Expand Down
4 changes: 2 additions & 2 deletions tests_e2e/orchestrator/lib/agent_test_suite.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,10 +294,10 @@ def _setup_node(self, install_test_agent: bool) -> None:
#
if self.context.ssh_client.get_architecture() == "aarch64":
pypy_path = Path("/tmp/pypy3.7-arm64.tar.bz2")
pypy_download = "https://downloads.python.org/pypy/pypy3.7-v7.3.5-aarch64.tar.bz2"
pypy_download = "https://dcrdata.blob.core.windows.net/python/pypy3.7-arm64.tar.bz2"
else:
pypy_path = Path("/tmp/pypy3.7-x64.tar.bz2")
pypy_download = "https://downloads.python.org/pypy/pypy3.7-v7.3.5-linux64.tar.bz2"
pypy_download = "https://dcrdata.blob.core.windows.net/python/pypy3.7-x64.tar.bz2"
if pypy_path.exists():
log.info("Found Pypy at %s", pypy_path)
else:
Expand Down
56 changes: 56 additions & 0 deletions tests_e2e/orchestrator/scripts/prepare-pypy
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/usr/bin/env bash
nagworld9 marked this conversation as resolved.
Show resolved Hide resolved

# Microsoft Azure Linux Agent
#
# Copyright 2018 Microsoft Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#
# This script is used to prepare a tarball containing Pypy with the assert-py module pre-installed.
# It needs to be run on x64 and arm64 VMs and the resulting tarballs need to be uploaded to storage,
# from where they are downloaded and installed to the test VMs (see wiki for detail).
#

set -euo pipefail

cd /tmp
rm -rf pypy3.7-*

arch=$(uname -m)
printf "Preparing Pypy for architecture %s...\n" $arch

printf "\n*** Downloading Pypy...\n"
if [[ $arch == "aarch64" ]]; then
tarball="pypy3.7-arm64.tar.bz2"
wget https://downloads.python.org/pypy/pypy3.7-v7.3.5-aarch64.tar.bz2 -O $tarball
else
tarball="pypy3.7-x64.tar.bz2"
wget https://downloads.python.org/pypy/pypy3.7-v7.3.5-linux64.tar.bz2 -O $tarball
fi

printf "\n*** Installing assertpy...\n"
tar xf $tarball
./pypy3.7-v7.3.5-*/bin/pypy -m ensurepip
./pypy3.7-v7.3.5-*/bin/pypy -mpip install assertpy

printf "\n*** Creating new tarball for Pypy...\n"
# remove the cache files created when Pypy, and set the owner to 0/0, in order to match the original tarball
find pypy3.7-v7.3.5-* -name '*.pyc' -exec rm {} \;
mv -v $tarball "$tarball.original"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is .original means? does it add extension name to the zip name? or its just way of defining for something?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the original tarball that we downloaded on line 37/40. I keep the original in case it is needed for debugging

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that's the case, to keep original this needs to move up before we install other modules? is itn't?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never mind, we are not modifying the tar at this point, its effecting the extraction folder I guess. I think we are good

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, the modules are in the new tarball; original remains unchanged

tar cf $tarball --bzip2 --owner 0:0 --group 0:0 pypy3.7-v7.3.5-*
rm -rf pypy3.7-v7.3.5-*

printf "\nPypy is ready at %s\n" "$(pwd)/$tarball"