Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: add pouch load functionality #1391

Merged
merged 1 commit into from
May 30, 2018
Merged

feature: add pouch load functionality #1391

merged 1 commit into from
May 30, 2018

Conversation

fuweid
Copy link
Contributor

@fuweid fuweid commented May 23, 2018

Signed-off-by: Wei Fu fhfuwei@163.com

Ⅰ. Describe what this PR did

Allow user to load oci.v1 format image by tar stream.

Ⅱ. Does this pull request fix one issue?

NONE

Ⅲ. Describe how you did it

Use containerd image oci importer to load the image.

Ⅳ. Describe how to verify it

First one is help:

➜  /tmp pouch load -h
load a set of images by tar stream

Usage:
  pouch load [OPTIONS] [IMAGE_NAME]

Examples:
$ pouch load -i busybox.tar busybox

Flags:
  -h, --help           help for load
  -i, --input string   Read from tar archive file, instead of STDIN

Since we don't have dump functionality for pouch, we need to verify it by ctr.

➜  /tmp ctr images export test.tar registry.hub.docker.com/library/busybox:1.28
➜  /tmp pouch load test < test.tar
➜  /tmp pouch images
IMAGE ID       IMAGE NAME   SIZE
8ac48589692a   test:1.28    710.83 KB
➜  /tmp pouch run 8ac48589692a ls
bin
dev
etc
home
proc
root
run
sys
tmp
usr
var

➜  /tmp # or use input
➜  /tmp pouch load -i test.tar test1
➜  /tmp pouch images
IMAGE ID       IMAGE NAME   SIZE
8ac48589692a   test1:1.28   710.83 KB
8ac48589692a   test:1.28    710.83 KB

➜  /tmp # or without name
➜  /tmp pouch load -i test.tar
➜  /tmp pouch images
IMAGE ID       IMAGE NAME             SIZE
8ac48589692a   unknown/unknown:1.28   710.83 KB
8ac48589692a   test:1.28              710.83 KB
8ac48589692a   test1:1.28             710.83 KB

NOTE: The oci.v1 spec says that the org.opencontainers.image.ref.name is used as tag in common.
Based on this, we don't allow user to apply any name which contains digest or tag information.

➜  /tmp pouch load -i test.tar test:1
Error: {"message":"the image name should not contains any digest or tag information"}

➜  /tmp pouch load -i test.tar test@sha256:123
Error: {"message":"the image name should not contains any digest or tag information"}

Ⅴ. Special notes for reviews

The output of docker save doesn't meet the oci.v1 spec and moby has the enhanced issue for this.

For now, this commit have not supported docker image layout tar yet.
We need to implement docker to oci related tool to handle the gap between docker and pouchd.

@codecov-io
Copy link

codecov-io commented May 24, 2018

Codecov Report

Merging #1391 into master will increase coverage by 21.06%.
The diff coverage is 31.35%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #1391       +/-   ##
==========================================
+ Coverage   16.24%   37.3%   +21.06%     
==========================================
  Files         200     254       +54     
  Lines       13572   17340     +3768     
==========================================
+ Hits         2205    6469     +4264     
+ Misses      11212   10042     -1170     
- Partials      155     829      +674
Impacted Files Coverage Δ
ctrd/image.go 79.42% <0%> (+79.42%) ⬆️
apis/server/image_bridge.go 64.93% <0%> (+51.89%) ⬆️
cli/load.go 0% <0%> (ø)
daemon/mgr/image.go 69.09% <0%> (+69.09%) ⬆️
daemon/mgr/image_load.go 0% <0%> (ø)
cli/main.go 0% <0%> (ø) ⬆️
pkg/multierror/def.go 100% <100%> (ø)
apis/server/router.go 91.24% <100%> (+91.24%) ⬆️
client/image_load.go 100% <100%> (ø)
client/request.go 55.68% <60%> (-1.47%) ⬇️
... and 131 more

apis/swagger.yml Outdated
schema:
$ref: "#/responses/500ErrorResponse"
parameters:
- name: "name"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we add input as a body parameter? the image load api consumes application/x-tar

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Roger that!

return bytes.NewReader(b), nil
}

return nil, nil
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the obj is nil, shall we return an error?
If we return nil, nil, the is.Reader is nil, then we need to judge if it is nil in the caller. WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the obj is nil, it means that we don't need to send the data to the server. HTTP Client will handle the nil case for us. It's ok I think. How do you think?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I am OK on this, just a remind to ensure this.

// may fails to restart.
for _, img := range imgs {
if err := mgr.storeImageReference(ctx, img); err != nil {
return err
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we try to store every image?
And aggregate the errors of each store error, and then return the aggregated error?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Will update

@fuweid
Copy link
Contributor Author

fuweid commented May 29, 2018

Hi @allencloud and @shaloulcy , I have updated the code and please take a look. Thanks

@allencloud
Copy link
Collaborator

LGTM, and I would like to invite @shaloulcy to take a look as well. In addition, the missing integration test is what we need to finish ASAP. @fuweid

@pouchrobot pouchrobot added the LGTM one maintainer or community participant agrees to merge the pull reuqest. label May 29, 2018
@fuweid
Copy link
Contributor Author

fuweid commented May 29, 2018

@allencloud yes. will do

// NOTE: in the image ocispec.v1, the org.opencontainers.image.ref.name
// annotation represents a "tag" for image. For example, an image may
// have a tag for different versions or builds of the software.
// And containerd.importer will append ":" and annotation so that we
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

":" and annotation

%s/and/to

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch!

@@ -16,6 +16,38 @@ import (
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)

// Multierrors contains a slice of errors.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think Multierrors not attached to image, so we should put this code to pkg dir, maybe pkg/error, WDYT ?😆

Copy link
Contributor Author

@fuweid fuweid May 29, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can move it out to pkg if necessary. The pkg/error only contains the Multierrors. It is weird... 😈 make senses?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's ok, some cli command also need the Multierrors, so it's proper to move it to pkg, WDTY?? 😆

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you mind to give me a example?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can move it to the pkg/utils in case that some one wants to use.

Signed-off-by: Wei Fu <fhfuwei@163.com>
@fuweid
Copy link
Contributor Author

fuweid commented May 30, 2018

@HusterWan I have moved the Multierror into pkg/multierror. Please take a look. Thanks! 😄

@HusterWan
Copy link
Contributor

LGTM

@HusterWan HusterWan merged commit 0ffabc9 into AliyunContainerService:master May 30, 2018
@fuweid fuweid deleted the feature_add_image_load branch May 30, 2018 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature LGTM one maintainer or community participant agrees to merge the pull reuqest. size/XXL
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants