Update Storage.py and initialiser image

[adriangonz]

Fixes #1341

Changelog

• Import latest version of Storage.py from KFServing, including tests.

TODO

• Update default kfserving/storage-initializer image.

Notes

To make SSL optional in S3 URLs, both Seldon and KFServing introduced different environment variables with the same finality: USE_SSL and S3_USE_HTTPS. You can find the respective PRs here:

• configurable secure argument for minio client #827
• make connection to minio configurable kserve/kserve#362

In order to make Storage.py compatible with the kfserving/storage-initializer image and to avoid any breaking changes on users who were already using Seldon's USE_SSL, I've made both compatible (with priority for USE_SSL).

[seldondev]

Fri Jan 24 17:46:58 UTC 2020
The logs for [pr-build] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1368/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1368 --build=1

[seldondev]

Fri Jan 24 17:47:06 UTC 2020
The logs for [lint] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1368/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1368 --build=2

[adriangonz]

I've rebased from master which had the latest version of the Helm charts (and already uses the latest kfserving/storage-initializer image).

[seldondev]

Tue Jan 28 13:17:50 UTC 2020
The logs for [pr-build] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1368/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1368 --build=3

[adriangonz]

/cc @axsaucedo

[seldondev]

Tue Jan 28 13:18:02 UTC 2020
The logs for [lint] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1368/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1368 --build=4

[axsaucedo]

@adriangonz this looks solid. I think it would be best to add this PR into the KFServing repo, and update the image to use the latest KFServing version of the image downloader - at least that's how we're doing it atm

[axsaucedo]

/hold

[adriangonz]

@axsaucedo this PR just copies over the latest version of KFServing's storage.py so that it's aligned with the kfserving/storage-initializer image. It doesn't really add any extra functionality on top of that.

The only change wrt their repo is adding support for different names of the environment variable which disables SSL in S3 links. But that's mainly so that old versions of Seldon Core don't break with this change.

[axsaucedo]

/hold cancel

[axsaucedo]

/approve
/ltgm

[axsaucedo]

/approve
/lgtm

[seldondev]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: axsaucedo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [axsaucedo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[seldondev]

Tue Feb 4 21:53:43 UTC 2020
The logs for [pr-build] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1368/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1368 --build=5

[seldondev]

Tue Feb 4 21:53:51 UTC 2020
The logs for [lint] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1368/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1368 --build=6

[seldondev]

failed to trigger Pull Request pipeline

• failed to create agent
• failed to calculate in repo config
• failed to load trigger config for repository SeldonIO/seldon-core for ref 5205434
• failed to find any lighthouse configuration files in repo SeldonIO/seldon-core at sha 5205434
• failed to process repo SeldonIO/seldon-core refref 5205434
• failed to list files in directory /var/tmp/gitrepo660600678/.lighthouse
• open /var/tmp/gitrepo660600678/.lighthouse
• no such file or directory