Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copying gcsfuse logs and fio output to gcs bucket via Kokoro #1140

Merged
merged 2 commits into from
May 24, 2023
Merged

Copying gcsfuse logs and fio output to gcs bucket via Kokoro #1140

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c52feb1
adding logic to push logs to kokoro
vadlakondaswetha May 24, 2023
5b5c4fe
collecting fio output also via kokoro
vadlakondaswetha May 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions perfmetrics/scripts/continuous_test/gcp_ubuntu/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ sudo dpkg -i $HOME/release/packages/gcsfuse_${GCSFUSE_VERSION}_amd64.deb
cd "./perfmetrics/scripts/"
echo "Mounting gcs bucket"
mkdir -p gcs
LOG_FILE=log-$(date '+%Y-%m-%d').txt
LOG_FILE=${KOKORO_ARTIFACTS_DIR}/gcsfuse-logs.txt
GCSFUSE_FLAGS="--implicit-dirs --max-conns-per-host 100 --enable-storage-client-library --debug_fuse --debug_gcs --log-file $LOG_FILE --log-format \"text\" --stackdriver-export-interval=30s"
BUCKET_NAME=periodic-perf-tests
MOUNT_POINT=gcs
Expand All @@ -55,11 +55,6 @@ gcsfuse $GCSFUSE_FLAGS $BUCKET_NAME $MOUNT_POINT
chmod +x run_load_test_and_fetch_metrics.sh
./run_load_test_and_fetch_metrics.sh

# Copying gcsfuse logs to bucket
gsutil -m cp $LOG_FILE gs://periodic-perf-tests/fio-gcsfuse-logs/

# Deleting logs older than 10 days
python3 utils/metrics_util.py gcs/fio-gcsfuse-logs/ 10
sudo umount $MOUNT_POINT

# ls_metrics test. This test does gcsfuse mount first and then do the testing.
Expand Down
8 changes: 8 additions & 0 deletions perfmetrics/scripts/continuous_test/gcp_ubuntu/continuous.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,10 @@
action {
define_artifacts {
regex: "gcsfuse-logs.txt"
regex: "github/gcsfuse/perfmetrics/scripts/fio-output.json"
strip_prefix: "github/gcsfuse/perfmetrics/scripts"
}
}

build_file: "gcsfuse/perfmetrics/scripts/continuous_test/gcp_ubuntu/build.sh"

10 changes: 4 additions & 6 deletions perfmetrics/scripts/run_load_test_and_fetch_metrics.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,16 @@ set -e
echo Print the time when FIO tests start
date
echo Running fio test..
fio job_files/seq_rand_read_write.fio --lat_percentiles 1 --output-format=json --output='output.json'
echo Logging fio results
cp output.json gcs/fio-logs/output-$(date '+%Y-%m-%d').json
python3 utils/metrics_util.py gcs/fio-logs/ 10
fio job_files/seq_rand_read_write.fio --lat_percentiles 1 --output-format=json --output='fio-output.json'

echo Installing requirements..
pip install --require-hashes -r requirements.txt --user
gsutil cp gs://periodic-perf-tests/creds.json gsheet
echo Fetching results..
# Upload data to the gsheet only when it runs through kokoro.
if [ "${KOKORO_JOB_TYPE}" != "RELEASE" ] && [ "${KOKORO_JOB_TYPE}" != "CONTINUOUS_INTEGRATION" ] && [ "${KOKORO_JOB_TYPE}" != "PRESUBMIT_GITHUB" ];
then
python3 fetch_metrics.py output.json
python3 fetch_metrics.py fio-output.json
else
python3 fetch_metrics.py output.json --upload
python3 fetch_metrics.py fio-output.json --upload
fi