Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix up change-yaml.sh to generate md5 sum for spark #281

Merged
merged 1 commit into from
May 23, 2019

Conversation

tmckayus
Copy link
Collaborator

This is transitional while we wait for sha512 support in cekit.

This is transitional while we wait for sha512 support in cekit.
@todo
Copy link

todo bot commented May 23, 2019

remove this download when sha512 support lands in upstream cekit (tmckay)

# TODO remove this download when sha512 support lands in upstream cekit (tmckay)
# Since this is big let's see if it's already there
if [ -f "/tmp/spark-${SPARK}-bin-hadoop${HVER}.tgz" ]; then
echo
echo Using existing "/tmp/spark-${SPARK}-bin-hadoop${HVER}.tgz", if this is not what you want delete it and run again
echo


This comment was generated by todo based on a TODO comment in 9deb637 in #281. cc @tmckayus.

@todo
Copy link

todo bot commented May 23, 2019

remove this checksum calculation when sha512 support lands in upstream cekit (tmckay)

# TODO remove this checksum calculation when sha512 support lands in upstream cekit (tmckay)
calcsum=$(sha512sum /tmp/spark-${SPARK}-bin-hadoop${HVER}.tgz | cut -d" " -f1)
sum=$(cat /tmp/spark-${SPARK}-bin-hadoop${HVER}.tgz.sha512 | tr -d [:space:] | cut -d: -f2 | tr [:upper:] [:lower:])
if [ "$calcsum" != "$sum" ]; then
echo "Failed to confirm authenticity of Spark archive, checksum mismatch"
echo "sha512sum : ${calcsum}"


This comment was generated by todo based on a TODO comment in 9deb637 in #281. cc @tmckayus.

@todo
Copy link

todo bot commented May 23, 2019

replace this with sha512 when it lands in upstream cekit (tmckay)

oshinko-s2i/change-yaml.sh

Lines 113 to 118 in 9deb637

# TODO replace this with sha512 when it lands in upstream cekit (tmckay)
calcsum=$(md5sum /tmp/spark-${SPARK}-bin-hadoop${HVER}.tgz | cut -d" " -f1)
$SED -i '\@url: https://archive.apache.org/dist/spark/@!b;n;s/md5.*/md5: '$calcsum'/' image.*.yaml
#change the spark version in the env var
$SED -i '\@name: SPARK_VERSION@!b;n;s/value:.*/value: '$SPARK'/' image.java.yaml


This comment was generated by todo based on a TODO comment in 9deb637 in #281. cc @tmckayus.

@tmckayus
Copy link
Collaborator Author

@elmiko ptal
This is part of the in-flight spark 2.4 update

Copy link
Contributor

@elmiko elmiko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks good, i think my change has gone into upstream cekit for the sha512 stuff. maybe we should confirm it in our build pipeline?

@tmckayus
Copy link
Collaborator Author

is that part of cekit 3? if so we need to modify the image.yamls, we can do that all at once

@elmiko
Copy link
Contributor

elmiko commented May 23, 2019

looks like my changes are in cekit 3.1 for sure, so maybe we should just proceed as is and leave the todos for later.

@tmckayus
Copy link
Collaborator Author

Merging before tests complete, this is just a utility script

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants