Skip to content

Speed up CodeBuild CI #1426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 8, 2024
Merged

Speed up CodeBuild CI #1426

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d5bcee2
Speed up CodeBuild CI:
andrewhop Jan 30, 2024
cc2e7fe
PR feedback: go back to using parallel=auto for mtr
andrewhop Jan 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions tests/ci/cdk/cdk/codebuild/github_ci_integration_omnibus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ batch:
variables:
AWS_LC_CI_TARGET: "tests/ci/integration/run_crt_integration.sh"

- identifier: openssh_integration_x86_64
- identifier: openssh_integration_master_x86_64
buildspec: tests/ci/codebuild/common/run_simple_target.yml
env:
type: LINUX_CONTAINER
Expand All @@ -45,8 +45,31 @@ batch:
image: 620771051181.dkr.ecr.us-west-2.amazonaws.com/aws-lc-docker-images-linux-x86:amazonlinux-2023_clang-15x_sanitizer_latest
variables:
AWS_LC_CI_TARGET: "tests/ci/integration/run_openssh_integration.sh"
OPENSSH_BRANCH: "master"

- identifier: openssh_integration_aarch
- identifier: openssh_integration_8_9_x86_64
buildspec: tests/ci/codebuild/common/run_simple_target.yml
env:
type: LINUX_CONTAINER
privileged-mode: false
compute-type: BUILD_GENERAL1_SMALL
image: 620771051181.dkr.ecr.us-west-2.amazonaws.com/aws-lc-docker-images-linux-x86:amazonlinux-2023_clang-15x_sanitizer_latest
variables:
AWS_LC_CI_TARGET: "tests/ci/integration/run_openssh_integration.sh"
OPENSSH_BRANCH: "V_8_9"

- identifier: openssh_integration_master_aarch
buildspec: tests/ci/codebuild/common/run_simple_target.yml
env:
type: ARM_CONTAINER
privileged-mode: false
compute-type: BUILD_GENERAL1_LARGE
Copy link
Contributor

@samuel40791765 samuel40791765 Feb 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need BUILD_GENERAL1_LARGE here considering we use BUILD_GENERAL1_SMALL for x86?

image: 620771051181.dkr.ecr.us-west-2.amazonaws.com/aws-lc-docker-images-linux-aarch:amazonlinux-2023_clang-15x_sanitizer_latest
variables:
AWS_LC_CI_TARGET: "tests/ci/integration/run_openssh_integration.sh"
OPENSSH_BRANCH: "master"

- identifier: openssh_integration_8_9_aarch
buildspec: tests/ci/codebuild/common/run_simple_target.yml
env:
type: ARM_CONTAINER
Expand All @@ -55,6 +78,7 @@ batch:
image: 620771051181.dkr.ecr.us-west-2.amazonaws.com/aws-lc-docker-images-linux-aarch:amazonlinux-2023_clang-15x_sanitizer_latest
variables:
AWS_LC_CI_TARGET: "tests/ci/integration/run_openssh_integration.sh"
OPENSSH_BRANCH: "V_8_9"

- identifier: postgres_integration_x86_64
buildspec: tests/ci/codebuild/common/run_nonroot_target.yml
Expand Down
4 changes: 2 additions & 2 deletions tests/ci/integration/run_mysql_integration.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ function mysql_patch_reminder() {
}

function mysql_build() {
cmake ${MYSQL_SRC_FOLDER} -GNinja -DWITH_BOOST=${BOOST_INSTALL_FOLDER} -DWITH_SSL=${AWS_LC_INSTALL_FOLDER} "-B${MYSQL_BUILD_FOLDER}"
ninja -C ${MYSQL_BUILD_FOLDER}
cmake ${MYSQL_SRC_FOLDER} -GNinja -DWITH_BOOST=${BOOST_INSTALL_FOLDER} -DWITH_SSL=${AWS_LC_INSTALL_FOLDER} "-B${MYSQL_BUILD_FOLDER}" -DCMAKE_BUILD_TYPE=RelWithDebInfo
time ninja -C ${MYSQL_BUILD_FOLDER}
ls -R ${MYSQL_BUILD_FOLDER}
}

Expand Down
24 changes: 12 additions & 12 deletions tests/ci/integration/run_openssh_integration.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@ function openssh_build() {

function checkout_openssh_branch() {
pushd "${OPENSSH_WORKSPACE_FOLDER}"
make clean
git clean -f -d
git checkout --track origin/"$1"
popd
Expand All @@ -80,24 +79,25 @@ function openssh_run_tests() {
mkdir -p "${AWS_LC_BUILD_FOLDER}" "${AWS_LC_INSTALL_FOLDER}" "${OPENSSH_INSTALL_FOLDER}"

# Get latest OpenSSH version.
git clone https://github.com/openssh/openssh-portable.git
git clone https://github.com/openssh/openssh-portable.git "${OPENSSH_WORKSPACE_FOLDER}"
ls

# Buld AWS-LC as a shared library
# Build AWS-LC as a shared library
aws_lc_build "$SRC_ROOT" "$AWS_LC_BUILD_FOLDER" "$AWS_LC_INSTALL_FOLDER" -DBUILD_TESTING=OFF -DBUILD_TOOL=OFF -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_SHARED_LIBS=1
install_aws_lc

CODEBUILD_SKIPPED_TESTS="agent-subprocess forwarding multiplex channel-timeout forward-control agent-restrict connection-timeout"
if [ "$OPENSSH_BRANCH" != "master" ]; then
checkout_openssh_branch "$OPENSSH_BRANCH"
fi
Comment on lines +89 to +91
Copy link
Contributor

@WillChilds-Klein WillChilds-Klein Jan 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you do this checkout unconditionally? if OpenSSH ever changes the name of their default branch, this might break.

Copy link
Contributor Author

@andrewhop andrewhop Jan 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Git was complaining about a 'master' branch already existing, I couldn't find a nice option to checkout if it doesn't already exist so i went with this.

Copy link
Contributor

@WillChilds-Klein WillChilds-Klein Jan 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, maybe you could do something like

if [ "$OPENSSH_BRANCH != $(git branch --show-current) ]; then
    checkout_openssh_branch "$OPENSSH_BRANCH
fi


# Using default branch. Build openssh and run tests.
openssh_build
openssh_run_tests "${CODEBUILD_SKIPPED_TESTS}"

# Using branch V_8_9
checkout_openssh_branch V_8_9
openssh_build
# In v8.9, the "percent" test requires the 'openssl' cli command
openssh_run_tests "percent ${CODEBUILD_SKIPPED_TESTS}"
CODEBUILD_SKIPPED_TESTS="agent-subprocess forwarding multiplex channel-timeout forward-control agent-restrict connection-timeout"
if [ "$OPENSSH_BRANCH" == "V_8_9" ]; then
# In v8.9, the "percent" test requires the 'openssl' cli command
openssh_run_tests "percent ${CODEBUILD_SKIPPED_TESTS}"
else
openssh_run_tests "${CODEBUILD_SKIPPED_TESTS}"
fi

popd

4 changes: 2 additions & 2 deletions tests/ci/run_fuzz_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ set -u
# 18 minutes for cleanup and merging files
if [[ $PLATFORM == "aarch64" ]]; then
# Arm sanitizers are very slow which causes the clean up time to take longer per
# fuzz test, only run for 16 minutes
TOTAL_FUZZ_TEST_TIME=1000
# fuzz test, only run for 8 minutes
TOTAL_FUZZ_TEST_TIME=500
else
TOTAL_FUZZ_TEST_TIME=1500
fi
Expand Down