Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mirror introspector log to a rotating file in 'log home' (if configured) #1827

Merged
merged 6 commits into from
Jul 22, 2020
Merged

Mirror introspector log to a rotating file in 'log home' (if configured) #1827

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d5f53b3
Mirror introspector log to a rotating file in 'log home' (if configured)
Jul 20, 2020
4318ef0
minor fix
Jul 20, 2020
bafd971
remove comment
Jul 20, 2020
0ed4ce5
doc that logHome includes introspector out
Jul 21, 2020
8a096fb
minor doc update
Jul 21, 2020
a7fc7cb
minor doc update
Jul 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
199 changes: 133 additions & 66 deletions operator/src/main/resources/scripts/introspectDomain.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,23 @@
# - encrypted admin user password passed in via a plain-text secret (for use in sit config)
# - md5 checksum of the DOMAIN_HOME/security/SerializedSystemIni.dat domain secret file
# - situational config files for overriding the configuration within the DOMAIN_HOME
# - Model in Image domain home zips and md5s (when the domain source type is MII)
#
# It works as part of the following flow:
#
# (1) When an operator discovers a new domain, it launches this script via an
# introspector k8s job.
# (2) This script then:
# (2A) Configures and starts a NM via startNodeManager.sh (in NODEMGR_HOME)
# (2B) Calls introspectDomain.py, which depends on the NM
# (2C) Exits 0 on success, non-zero otherwise.
# (2A) Generates MII domain home zips/md5 files if the domain home source type is MII
# (2B) Configures and starts a NM via startNodeManager.sh (in NODEMGR_HOME)
# (2C) Calls introspectDomain.py, which depends on the NM and puts files in stdout
# (2D) Exits 0 on success, non-zero otherwise.
# (5) Operator parses the output of introspectDomain.py into files and:
# (5A) Uses one of the files to get the domain's name, cluster name, ports, etc.
# (5B) Deploys a config map for the domain containing the files.
# (6) Operator starts pods for domain's WebLogic servers.
# (7) Pod 'startServer.sh' script loads files from the config map,
# generates a domain home from the files for the MII case,
# copies/uses encrypted files, and applies sit config files. It
# also checks that domain secret md5 cksum matches the cksum
# obtained by this script.
Expand All @@ -43,16 +46,20 @@

SCRIPTPATH="$( cd "$(dirname "$0")" > /dev/null 2>&1 ; pwd -P )"

#
# setup tracing
#

source ${SCRIPTPATH}/utils.sh
[ $? -ne 0 ] && echo "[SEVERE] Missing file ${SCRIPTPATH}/utils.sh" && exit 1

traceTiming "INTROSPECTOR '${DOMAIN_UID}' MAIN START"


#
# Local createFolder method which does an 'exit 1' instead of exitOrLoop for
# immediate failure during introspection
#

function createFolder {
mkdir -m 750 -p $1
if [ ! -d $1 ]; then
Expand All @@ -61,21 +68,68 @@ function createFolder {
fi
}

trace "Introspecting the domain"
#
# setup MII functions in case this is a MII domain
#

source ${SCRIPTPATH}/modelInImage.sh
[ $? -ne 0 ] && trace SEVERE "Error sourcing ${SCRIPTPATH}/modelInImage.sh" && exit 1

#
# setup introspector log file
#

traceDirs before LOG_HOME

if [ ! -z "${LOG_HOME}" ] && [ ! -d "${LOG_HOME}" ]; then
trace "Creating log home directory: '${LOG_HOME}'"
createFolder ${LOG_HOME}
fi

ilog_dir="${LOG_HOME:-/tmp}"
ilog_timestamp="$(date --utc '+%Y-%m-%d_%H.%M.%S')"
ilog_prefix="${ilog_dir}/introspector_script"
ilog_file="${ilog_prefix}_${ilog_timestamp}_.out"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it the intention for the introspector logs to end with '_' before .out?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - I found it made the generated file name more readable.

ilog_max=${MAX_INTROSPECTOR_LOG_FILES:-10}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation note: MAX_INTROSPECTOR_LOG_FILES is internal for now - there are no plans to document this env var. Default is 10.

if [ ! -d "${ilog_dir}" ]; then
trace "Creating introspector log directory: '${ilog_dir}'"
createFolder "${ilog_dir}"
fi

echo "" >> ${ilog_file}

#
# main introspection function
#

function doIntrospect() {

# list potentially interesting env-vars and dirs before they're updated by export.*Homes
trace "Introspecting domain '${DOMAIN_UID}', log location: '$ilog_file'"

traceEnv before
traceDirs before
traceDirs after LOG_HOME

# set defaults
# set ORACLE_HOME/WL_HOME/MW_HOME to defaults if needed
# keep only 10 total log files by default (delete oldest first)
for ilog_cur in \
$(ls -1r ${ilog_prefix}* 2>/dev/null | tail -n +$((ilog_max + 1)))
do
trace "Removing old introspector log file '${ilog_cur}'"
rm ${ilog_cur}
done

exportInstallHomes
# list potentially interesting env-vars and dirs before they're updated by export.*Homes

# check if prereq env-vars, files, and directories exist
traceEnv before
traceDirs before DOMAIN_HOME DATA_HOME

checkEnv -q \
# set defaults
# set ORACLE_HOME/WL_HOME/MW_HOME to defaults if needed

exportInstallHomes

# check if prereq env-vars, files, and directories exist

checkEnv -q \
DOMAIN_UID \
NAMESPACE \
ORACLE_HOME \
Expand All @@ -86,37 +140,30 @@ checkEnv -q \
OPERATOR_ENVVAR_NAMES \
|| exit 1

for script_file in "${SCRIPTPATH}/wlst.sh" \
"${SCRIPTPATH}/startNodeManager.sh" \
"${SCRIPTPATH}/modelInImage.sh" \
"${SCRIPTPATH}/introspectDomain.py"; do
[ ! -f "$script_file" ] && trace SEVERE "Missing file '${script_file}'." && exit 1
done

for dir_var in JAVA_HOME WL_HOME MW_HOME ORACLE_HOME; do
[ ! -d "${!dir_var}" ] && trace SEVERE "Missing ${dir_var} directory '${!dir_var}'." && exit 1
done

#
# DATA_HOME env variable exists implies override directory specified. Attempt to create directory
#
if [ ! -z "${DATA_HOME}" ] && [ ! -d "${DATA_HOME}" ]; then
trace "Creating data home directory: '${DATA_HOME}'"
createFolder ${DATA_HOME}
fi


traceTiming "INTROSPECTOR '${DOMAIN_UID}' MII CREATE DOMAIN START"
for script_file in "${SCRIPTPATH}/wlst.sh" \
"${SCRIPTPATH}/startNodeManager.sh" \
"${SCRIPTPATH}/introspectDomain.py"; do
[ ! -f "$script_file" ] && trace SEVERE "Missing file '${script_file}'." && exit 1
done

for dir_var in JAVA_HOME WL_HOME MW_HOME ORACLE_HOME; do
[ ! -d "${!dir_var}" ] && trace SEVERE "Missing ${dir_var} directory '${!dir_var}'." && exit 1
done

#
# DATA_HOME env variable exists implies override directory specified. Attempt to create directory
#
if [ ! -z "${DATA_HOME}" ] && [ ! -d "${DATA_HOME}" ]; then
trace "Creating data home directory: '${DATA_HOME}'"
createFolder ${DATA_HOME}
fi

source ${SCRIPTPATH}/modelInImage.sh
if [ $? -ne 0 ]; then
trace SEVERE "Error sourcing modelInImage.sh" && exit 1
fi
traceTiming "INTROSPECTOR '${DOMAIN_UID}' MII CREATE DOMAIN START"

# Add another env/attribute in domain yaml for model in image
# log error if dir exists and attribute set
DOMAIN_CREATED=0
if [ ${DOMAIN_SOURCE_TYPE} == "FromModel" ]; then
# Add another env/attribute in domain yaml for model in image
# log error if dir exists and attribute set
DOMAIN_CREATED=0
if [ ${DOMAIN_SOURCE_TYPE} == "FromModel" ]; then
trace "Beginning Model In Image"
command -v gzip
if [ $? -ne 0 ] ; then
Expand All @@ -142,58 +189,78 @@ if [ ${DOMAIN_SOURCE_TYPE} == "FromModel" ]; then
createWLDomain || exit 1
created_domain=$DOMAIN_CREATED
trace "Create domain return code = " ${created_domain}
else
else
created_domain=1
fi

traceTiming "INTROSPECTOR '${DOMAIN_UID}' MII CREATE DOMAIN END"

fi

# check DOMAIN_HOME for a config/config.xml, reset DOMAIN_HOME if needed
traceTiming "INTROSPECTOR '${DOMAIN_UID}' MII CREATE DOMAIN END"

exportEffectiveDomainHome || exit 1
# check DOMAIN_HOME for a config/config.xml, reset DOMAIN_HOME if needed

# list potentially interesting env-vars and dirs after they're updated by export.*Homes
exportEffectiveDomainHome || exit 1

traceEnv after
traceDirs after
# list potentially interesting env-vars and dirs after they're updated by export.*Homes

# check if we're using a supported WebLogic version
# (the check will log a message if it fails)
traceEnv after
traceDirs after DOMAIN_HOME DATA_HOME

checkWebLogicVersion || exit 1
# check if we're using a supported WebLogic version
# (the check will log a message if it fails)

# start node manager
# run instrospector wlst script
if [ ${created_domain} -ne 0 ]; then
checkWebLogicVersion || exit 1

# start node manager
# run instrospector wlst script
if [ ${created_domain} -ne 0 ]; then
traceTiming "INTROSPECTOR '${DOMAIN_UID}' MII NM START"

# start node manager -why ??
# introspectDomain.py uses an NM to setup credentials for the server NMs
# (see 'nmConnect' in introspectDomain.py)
trace "Starting node manager"
${SCRIPTPATH}/startNodeManager.sh || exit 1

traceTiming "INTROSPECTOR '${DOMAIN_UID}' MII NM END"

traceTiming "INTROSPECTOR '${DOMAIN_UID}' MII MD5 START"

traceTiming "INTROSPECTOR '${DOMAIN_UID}' MII NM END"

traceTiming "INTROSPECTOR '${DOMAIN_UID}' MII MD5 START"

# put domain secret's md5 cksum in file '/tmp/DomainSecret.md5'
# the introspector wlst script and WL server pods will use this value
generateDomainSecretMD5File '/tmp/DomainSecret.md5' || exit 1

traceTiming "INTROSPECTOR '${DOMAIN_UID}' MII MD5 END"

traceTiming "INTROSPECTOR '${DOMAIN_UID}' INTROSPECT START"

trace "Running introspector WLST script ${SCRIPTPATH}/introspectDomain.py"
${SCRIPTPATH}/wlst.sh ${SCRIPTPATH}/introspectDomain.py || exit 1

traceTiming "INTROSPECTOR '${DOMAIN_UID}' INTROSPECT END"
fi
trace "Domain introspection complete"
fi
trace "Domain introspection complete"
}

exit 0
# we have different log file modes in case we need to revert 'tee' mode

case "${INTROSPECTOR_LOG_FILE_MODE:-tee}" in
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation note: There are no plans to document INTROSPECTOR_LOG_FILE_MODE - default is 'tee'.

tee)
set -o pipefail
doIntrospect |& tee $ilog_file
exit $?
;;
bg_and_tail)
${SCRIPTPATH}/tailLog.sh $ilog_file /tmp/dontcare &
tail_log_pid=$!
doIntrospect >> $ilog_file 2>&1 &
wait $!
exitCode=$?
# sleep 1 second in case background 'tail' needs time to catch up
sleep 1
kill -9 $tail_log_pid
exit $exitCode
;;
*)
# no log file - everything simply goes to stdout/stderr (old behavior)
doIntrospect
exit $?
;;
esac
35 changes: 13 additions & 22 deletions operator/src/main/resources/scripts/modelInImage.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
#
# This script contains the all the function of model in image
# It is used by introspectDomain.sh job and starServer.sh
# It is used by introspectDomain.sh job and startServer.sh

source ${SCRIPTPATH}/utils.sh

Expand Down Expand Up @@ -532,8 +532,8 @@ function diff_model() {
org.python.util.jython \
${SCRIPTPATH}/model_diff.py $1 $2 > ${WDT_OUTPUT} 2>&1
if [ $? -ne 0 ] ; then
trace SEVERE "Failed to compare models. Check logs for error."
trace SEVERE "$(cat ${WDT_OUTPUT})"
trace SEVERE "Failed to compare models. Check logs for error. Comparison output:"
cat ${WDT_OUTPUT}
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation note: When the introspector java parses out a SEVERE it includes everything after the SEVERE up to the next trace statement. (So the 'cat' above is included in the SEVERE just before it.)

exitOrLoop
fi
trace "Exiting diff_model"
Expand Down Expand Up @@ -648,11 +648,8 @@ function generateMergedModel() {
${archive_list} ${variable_list} -domain_type ${WDT_DOMAIN_TYPE} > ${WDT_OUTPUT}
ret=$?
if [ $ret -ne 0 ]; then
trace SEVERE "WDT Failed: Validate Model Failed "
if [ -d ${LOG_HOME} ] && [ ! -z ${LOG_HOME} ] ; then
cp ${WDT_OUTPUT} ${LOG_HOME}/introspectJob_validateDomain.log
fi
trace SEVERE "$(cat ${WDT_OUTPUT})"
trace SEVERE "WDT Failed: Validate Model Failed:"
cat ${WDT_OUTPUT}
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation note: we no longer need to check if LOG_HOME is set and/or do the 'cp' because the calling script is now logging the entire output of this script.

exitOrLoop
fi

Expand All @@ -679,11 +676,8 @@ function wdtCreatePrimordialDomain() {
> ${WDT_OUTPUT}
ret=$?
if [ $ret -ne 0 ]; then
trace SEVERE "WDT Create Domain Failed ${ret}"
if [ -d ${LOG_HOME} ] && [ ! -z ${LOG_HOME} ] ; then
cp ${WDT_OUTPUT} ${LOG_HOME}/introspectJob_createDomain.log
fi
trace SEVERE "$(cat ${WDT_OUTPUT})"
trace SEVERE "WDT Create Domain Failed, ret=${ret}:"
cat ${WDT_OUTPUT}
exitOrLoop
fi

Expand Down Expand Up @@ -712,11 +706,8 @@ function wdtUpdateModelDomain() {
ret=$?

if [ $ret -ne 0 ]; then
trace SEVERE "WDT Update Domain Failed "
if [ -d ${LOG_HOME} ] && [ ! -z ${LOG_HOME} ] ; then
cp ${WDT_OUTPUT} ${LOG_HOME}/introspectJob_updateDomain.log
fi
trace SEVERE "$(cat ${WDT_OUTPUT})"
trace SEVERE "WDT Update Domain Failed:"
cat ${WDT_OUTPUT}
exitOrLoop
fi

Expand Down Expand Up @@ -788,8 +779,8 @@ function encrypt_decrypt_model() {
trace SEVERE "Fatal Error: Failed to $1 domain model. This error is irrecoverable. Check to see if the secret " \
"described in the configuration.model.runtimeEncryptionSecret domain resource field has been changed since the " \
"creation of the domain. You can either reset the password to the original one and try again or delete "\
"and recreate the domain."
trace SEVERE "$(cat ${WDT_OUTPUT})"
"and recreate the domain. Failure output:"
cat ${WDT_OUTPUT}
exitOrLoop
fi

Expand Down Expand Up @@ -829,8 +820,8 @@ function encrypt_decrypt_domain_secret() {
trace SEVERE "Fatal Error: Failed to $1 domain secret. This error is irrecoverable. Check to see if the secret " \
"described in the configuration.model.runtimeEncryptionSecret domain resource field has been changed since the " \
"creation of the domain. You can either reset the password to the original one and try again or delete "\
"and recreate the domain."
trace SEVERE "$(cat ${WDT_OUTPUT})"
"and recreate the domain. Failure output:"
cat ${WDT_OUTPUT}
exitOrLoop
fi

Expand Down
4 changes: 2 additions & 2 deletions operator/src/main/resources/scripts/startServer.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,7 @@ function prepareMIIServer() {
# trace env vars and dirs before export.*Home calls

traceEnv before
traceDirs before
traceDirs before DOMAIN_HOME LOG_HOME DATA_HOME

traceTiming "POD '${SERVICE_NAME}' MII UNZIP START"

Expand Down Expand Up @@ -316,7 +316,7 @@ exportEffectiveDomainHome || exitOrLoop
# trace env vars and dirs after export.*Home calls

traceEnv after
traceDirs after
traceDirs after DOMAIN_HOME LOG_HOME DATA_HOME

#
# Check if introspector actually ran. This should never fail since
Expand Down
Loading