image-rs: Support to reuse meta_store

[ChengyuZhu6]

Support to write meta_store to meta_store.json.
Set the absolute path for meta store with meta_store.json

[ChengyuZhu6]

Needs to add related unit tests.

[ChengyuZhu6]

I’ve added code and unit tests to enable the reuse of the meta store and layers in the image client.

[ChengyuZhu6]

cc @arronwy @Xynnn007 @bpradipt

[ChengyuZhu6]

The template of meta_store.json is as follows:

{
    "image_db": {
        "sha256:fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e": {
            "id": "sha256:fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e",
            "digest": "sha256:92c7f9c92844bbbb5d0a101b22f7c2a7949e40f8ea90c8b3bc396879d95e899a",
            "reference": "mcr.microsoft.com/hello-world",
            "image_config": {
                "created": "2019-01-01T01:29:27.650294696Z",
                "architecture": "amd64",
                "os": "linux",
                "config": {
                    "User": "",
                    "Env": [
                        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
                    ],
                    "Cmd": [
                        "/hello"
                    ],
                    "WorkingDir": ""
                },
                "rootfs": {
                    "type": "layers",
                    "diff_ids": [
                        "sha256:af0b15c8625bb1938f1d7b17081031f649fd14e6b233688eea3c5483994a66a3"
                    ]
                },
                "history": [
                    {
                        "created": "2019-01-01T01:29:27.416803627Z",
                        "created_by": "/bin/sh -c #(nop) COPY file:f77490f70ce51da25bd21bfc30cb5e1a24b2b65eb37d4af0c327ddc24f0986a6 in / "
                    },
                    {
                        "created": "2019-01-01T01:29:27.650294696Z",
                        "created_by": "/bin/sh -c #(nop)  CMD [\"/hello\"]",
                        "empty_layer": true
                    }
                ]
            },
            "signed": false,
            "layer_metas": [
                {
                    "decoder": "Gzip",
                    "encrypted": false,
                    "compressed_digest": "sha256:1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced",
                    "uncompressed_digest": "sha256:af0b15c8625bb1938f1d7b17081031f649fd14e6b233688eea3c5483994a66a3",
                    "store_path": "/tmp/.tmpYioDI2/layers/sha256_1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced"
                }
            ]
        }
    },
    "layer_db": {
        "sha256:1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced": {
            "decoder": "Gzip",
            "encrypted": false,
            "compressed_digest": "sha256:1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced",
            "uncompressed_digest": "sha256:af0b15c8625bb1938f1d7b17081031f649fd14e6b233688eea3c5483994a66a3",
            "store_path": "/tmp/.tmpYioDI2/layers/sha256_1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced"
        }
    },
    "snapshot_db": {}
}

[bpradipt]

The template of meta_store.json is as follows:

{
    "image_db": {
        "sha256:fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e": {
            "id": "sha256:fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e",
            "digest": "sha256:92c7f9c92844bbbb5d0a101b22f7c2a7949e40f8ea90c8b3bc396879d95e899a",
            "reference": "mcr.microsoft.com/hello-world",
            "image_config": {
                "created": "2019-01-01T01:29:27.650294696Z",
                "architecture": "amd64",
                "os": "linux",
                "config": {
                    "User": "",
                    "Env": [
                        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
                    ],
                    "Cmd": [
                        "/hello"
                    ],
                    "WorkingDir": ""
                },
                "rootfs": {
                    "type": "layers",
                    "diff_ids": [
                        "sha256:af0b15c8625bb1938f1d7b17081031f649fd14e6b233688eea3c5483994a66a3"
                    ]
                },
                "history": [
                    {
                        "created": "2019-01-01T01:29:27.416803627Z",
                        "created_by": "/bin/sh -c #(nop) COPY file:f77490f70ce51da25bd21bfc30cb5e1a24b2b65eb37d4af0c327ddc24f0986a6 in / "
                    },
                    {
                        "created": "2019-01-01T01:29:27.650294696Z",
                        "created_by": "/bin/sh -c #(nop)  CMD [\"/hello\"]",
                        "empty_layer": true
                    }
                ]
            },
            "signed": false,
            "layer_metas": [
                {
                    "decoder": "Gzip",
                    "encrypted": false,
                    "compressed_digest": "sha256:1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced",
                    "uncompressed_digest": "sha256:af0b15c8625bb1938f1d7b17081031f649fd14e6b233688eea3c5483994a66a3",
                    "store_path": "/tmp/.tmpYioDI2/layers/sha256_1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced"
                }
            ]
        }
    },
    "layer_db": {
        "sha256:1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced": {
            "decoder": "Gzip",
            "encrypted": false,
            "compressed_digest": "sha256:1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced",
            "uncompressed_digest": "sha256:af0b15c8625bb1938f1d7b17081031f649fd14e6b233688eea3c5483994a66a3",
            "store_path": "/tmp/.tmpYioDI2/layers/sha256_1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced"
        }
    },
    "snapshot_db": {}
}

Thanks @ChengyuZhu6 for fixing this. Few questions
As you know one of the aspects I'm experimenting with is to figure out a way to embed image layers in rootfs to avoid downloading the image layers again during pod creation. So for this scenario as I understand from this PR I need to do the following

1. Populate image layers in a specific path in rootfs
2. Create a meta_store.json file to indicate the layer details and the path inside the rootfs
3. Ensure this meta_store.json is reference by kata-agent/image-rs to reuse the embedded layers

Is my understanding correct?
Would you know of a way (tool/script etc) to create meta_store.json ?

[ChengyuZhu6]

As you know one of the aspects I'm experimenting with is to figure out a way to embed image layers in rootfs to avoid downloading the image layers again during pod creation.

image-rs will download the image only once, even if multiple containers in a pod use same image.

[ChengyuZhu6]

Populate image layers in a specific path in rootfs
Create a meta_store.json file to indicate the layer details and the path inside the rootfs
Ensure this meta_store.json is reference by kata-agent/image-rs to reuse the embedded layers

Just a reminder: we don't need to populate the image layers to rootfs. Instead, we should store the layers in the image-rs workdir/layers directory. For example, in kata, the layers (such as sha256:1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced) should be stored in /run/kata-containers/image/layers/sha256_1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced. Image-rs will then overlay the layers onto rootfs.

[ChengyuZhu6]

Would you know of a way (tool/script etc) to create meta_store.json ?

I'm not sure if there is an existing tool or script for this, but it sounds like an interesting idea.

[Xynnn007]

nice code, thanks @ChengyuZhu6 ! Let's wait for an approval from @bpradipt

[bpradipt]

/lgtm
Thanks @ChengyuZhu6

[fitzthum]

LGTM

We have had some issues with layer caching on the host before (i.e. encrypted and unencrypted images can share layers), but I don't think that will be a problem here.