Support retries to push to a registry

[rmannibucau]

Description of the issue:

On laggy registries - which means you often get a 504 timeout issue - it is not possible to push images with jib whereas it works with docker with a retry (it retries not yet uploaded layers).

Expected behavior:

A retry would happen on layers (default to 3 maybe?).

[chanseokoh]

#691 is related, although that isn't about dealing with 504.

[briandealwis]

We should also investigate using chunked uploads since some layers may be rather huge. The only worrisome aspect is that it's unclear what should be our minimum chunk size:

There is no enforcement on layer chunk splits other than that the server must receive them in order. The server may enforce a minimum chunk size. If the server cannot accept the chunk, a 416 Requested Range Not Satisfiable response will be returned and will include a Range header indicating the current status:

It appears that chunks must be provided in sequence. The documentation describing the 416 result says:

Out of order chunk: the range of the next chunk must start immediately after the “last valid range” from the previous response

[rmannibucau]

up? on openshift we easily get SSL issues (indeed without any modification a relaunch works) when pushing concurrently layers so a proper retry would help on client side.

edit: checked that just looping inside com.google.cloud.tools.jib.registry.RegistryEndpointCaller#call(java.net.URL) (potentially with a jitter) to retry solves the mentionned issue. A default of 3 retries sounds very acceptable or alternatively a max duration per push (if the image push didn't suceed in 10mn, don't retry anymore anything for ex).

[chanseokoh]

One form retry is implemented in #3351. There can be other ways of retrying, but #3351 should already be very helpful. Closing the issue.