Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-20162 [nightly] depending on pipeline execution we sometimes re… #74

Closed
wants to merge 1 commit into from
Closed

HBASE-20162 [nightly] depending on pipeline execution we sometimes re… #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions dev-support/Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,6 @@ pipeline {
skipDefaultCheckout()
}
environment {
TOOLS = "${env.WORKSPACE}/tools"
// where we check out to across stages
BASEDIR = "${env.WORKSPACE}/component"
YETUS_RELEASE = '0.7.0'
// where we'll write everything from different steps. Need a copy here so the final step can check for success/failure.
OUTPUT_DIR_RELATIVE_GENERAL = 'output-general'
Expand All @@ -42,14 +39,14 @@ pipeline {

PROJECT = 'hbase'
PROJECT_PERSONALITY = 'https://raw.githubusercontent.com/apache/hbase/master/dev-support/hbase-personality.sh'
PERSONALITY_FILE = 'tools/personality.sh'
// This section of the docs tells folks not to use the javadoc tag. older branches have our old version of the check for said tag.
AUTHOR_IGNORE_LIST = 'src/main/asciidoc/_chapters/developer.adoc,dev-support/test-patch.sh'
WHITESPACE_IGNORE_LIST = '.*/generated/.*'
// output from surefire; sadly the archive function in yetus only works on file names.
ARCHIVE_PATTERN_LIST = 'TEST-*.xml,org.apache.h*.txt,*.dumpstream,*.dump'
// These tests currently have known failures. Once they burn down to 0, remove from here so that new problems will cause a failure.
TESTS_FILTER = 'cc,checkstyle,javac,javadoc,pylint,shellcheck,whitespace,perlcritic,ruby-lint,rubocop,mvnsite'
BRANCH_SPECIFIC_DOCKERFILE = "${env.BASEDIR}/dev-support/docker/Dockerfile"
// Flaky urls for different branches. Replace '-' and '.' in branch name by '_' because those
// characters are not allowed in bash variable name.
// Not excluding flakies from the nightly build for now.
Expand Down Expand Up @@ -101,7 +98,8 @@ if [ ! -d "${YETUS_DIR}" ]; then
gunzip -c yetus.tar.gz | tar xpf - -C "${YETUS_DIR}" --strip-components 1
fi
'''
dir ("${env.TOOLS}") {
// Set up the file we need at PERSONALITY_FILE location
dir ("tools") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will always be relative to root? To $BASEDIR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's relative to the workspace root.

Copy link
Contributor Author

@busbey busbey Mar 9, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at the end of this stage we stash the tools directory and in future stages we unstash it relative to those stages' workspace root.

sh """#!/usr/bin/env bash
echo "Downloading Project personality."
curl -L -o personality.sh "${env.PROJECT_PERSONALITY}"
Expand Down Expand Up @@ -130,6 +128,7 @@ curl -L -o personality.sh "${env.PROJECT_PERSONALITY}"
}
}
environment {
BASEDIR = "${env.WORKSPACE}/component"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I... don't understand how these are different from what we had previously?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the captured context is different. jenkins captures things like WORKSPACE per executor. how it turns the given stages and initial environment into executors is magic AFAICT.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so down here, because we had a node on the stage we get a new executor and a new WORKSPACE gets made.

// TODO does hadoopcheck need to be jdk specific?
// Should be things that work with multijdk
TESTS = 'all,-unit,-findbugs'
Expand Down Expand Up @@ -191,6 +190,7 @@ curl -L -o personality.sh "${env.PROJECT_PERSONALITY}"
branch 'branch-1*'
}
environment {
BASEDIR = "${env.WORKSPACE}/component"
TESTS = 'mvninstall,compile,javac,unit,htmlout'
OUTPUT_DIR_RELATIVE = "${env.OUTPUT_DIR_RELATIVE_JDK7}"
OUTPUT_DIR = "${env.WORKSPACE}/${env.OUTPUT_DIR_RELATIVE_JDK7}"
Expand Down Expand Up @@ -259,6 +259,7 @@ curl -L -o personality.sh "${env.PROJECT_PERSONALITY}"
}
}
environment {
BASEDIR = "${env.WORKSPACE}/component"
TESTS = 'mvninstall,compile,javac,unit,findbugs,htmlout'
OUTPUT_DIR_RELATIVE = "${env.OUTPUT_DIR_RELATIVE_HADOOP2}"
OUTPUT_DIR = "${env.WORKSPACE}/${env.OUTPUT_DIR_RELATIVE_HADOOP2}"
Expand Down Expand Up @@ -334,6 +335,7 @@ curl -L -o personality.sh "${env.PROJECT_PERSONALITY}"
}
}
environment {
BASEDIR = "${env.WORKSPACE}/component"
TESTS = 'mvninstall,compile,javac,unit,htmlout'
OUTPUT_DIR_RELATIVE = "${env.OUTPUT_DIR_RELATIVE_HADOOP3}"
OUTPUT_DIR = "${env.WORKSPACE}/${env.OUTPUT_DIR_RELATIVE_HADOOP3}"
Expand Down Expand Up @@ -408,6 +410,9 @@ curl -L -o personality.sh "${env.PROJECT_PERSONALITY}"
// this needs to be set to the jdk that ought to be used to build releases on the branch the Jenkinsfile is stored in.
jdk "JDK 1.8 (latest)"
}
environment {
BASEDIR = "${env.WORKSPACE}/component"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the difference between ${BASEDIR} and ${env.BASEDIR} below

Do they both read this value, or does only env.BASEDIR read this and the other one comes from ~magic

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

env.BASEDIR gets replaced with the literal value by jenkins. BASEDIR is a bash variable that will be set to a value of the same thing that Jenkins would rewrite.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this one doesn't have a node? so where does it exec?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stages that don't have their own node label exec on the same node as the task that launches the entire pipeline (so the one that runs the "install yetus" and "init health results" steps)

}
steps {
sh '''#!/bin/bash -e
echo "Setting up directories"
Expand Down
8 changes: 4 additions & 4 deletions dev-support/hbase_nightly_yetus.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@

declare -i missing_env=0
# Validate params
for required_env in "TESTS" "TOOLS" "BASEDIR" "ARCHIVE_PATTERN_LIST" "OUTPUT_DIR_RELATIVE" \
"BRANCH_SPECIFIC_DOCKERFILE" "OUTPUT_DIR" "PROJECT" "AUTHOR_IGNORE_LIST" \
for required_env in "TESTS" "PERSONALITY_FILE" "BASEDIR" "ARCHIVE_PATTERN_LIST" "OUTPUT_DIR_RELATIVE" \
"OUTPUT_DIR" "PROJECT" "AUTHOR_IGNORE_LIST" \
"WHITESPACE_IGNORE_LIST" "BRANCH_NAME" "TESTS_FILTER" "DEBUG" \
"USE_YETUS_PRERELEASE" "WORKSPACE" "YETUS_RELEASE"; do
if [ -z "${!required_env}" ]; then
Expand All @@ -44,15 +44,15 @@ if [[ -n "${SET_JAVA_HOME}" ]]; then
YETUS_ARGS=("--java-home=${SET_JAVA_HOME}" "${YETUS_ARGS[@]}")
fi
YETUS_ARGS=("--plugins=${TESTS}" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--personality=${TOOLS}/personality.sh" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--personality=${PERSONALITY_FILE}" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--basedir=${BASEDIR}" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--archive-list=${ARCHIVE_PATTERN_LIST}" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--console-urls" "${YETUS_ARGS[@]}")
# YETUS-532, repeat this twice in case the fix is to update args rather than docs
YETUS_ARGS=("--build-url-patchdir=artifact/${OUTPUT_DIR_RELATIVE}" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--build-url-artifacts=artifact/${OUTPUT_DIR_RELATIVE}" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--docker" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--dockerfile=${BRANCH_SPECIFIC_DOCKERFILE}" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--dockerfile=${BASEDIR}/dev-support/docker/Dockerfile" "${YETUS_ARGS[@]}")
# Yetus sets BUILDMODE env variable to "full" if this arg is passed.
YETUS_ARGS=("--empty-patch" "${YETUS_ARGS[@]}")
YETUS_ARGS=("--html-report-file=${OUTPUT_DIR}/console-report.html" "${YETUS_ARGS[@]}")
Expand Down