Oras commands like attach, discover, and pull support an option to format output as JSON #638

plooploops · 2022-10-10T18:32:01Z

What is the version of your ORAS CLI

Version: 0.15.0 Go version: go1.19 Git commit: 94abaec Git tree state: clean

What would you like to be added?

An option to format output for commands like attach, discover, and pull as JSON.

Updated version per comments:

For oras discover

For example, suppose you make the call:

oras discover <myacr>.azurecr.io/my/repo:tag -o json

And then you would get a JSON output like the following:

{
  "referrers": [
    {
      "digest": "sha256:someHash",
      "mediaType": "someType",
      "artifactType": "someArtifactType",
      "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
      "size": sizeInBytes
    },
  ]
}

This would allow you to pull in the artifact (e.g. oras pull <reference value>.

For oras pull

For oras pull, it would be helpful to get the downloaded artifactType, reference, and files along with the other fields described in the below sample output.

Suppose you make the following call:

oras pull <reference value> -o json

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
  "files" : [
    "path/to/my/downloaded/artifact.json",
    "path/to/another/downloaded/artifact.json"
  ],
  "size": sizeInBytes
}

For oras attach

For oras attach, it would also be helpful to get the uploaded artifactDigest, reference, and files along with the other fields described in the below sample output.

Suppose you make the following call:

oras attach "<myacr>.azurecr.io/my/repo:tag" --artifact-type "someArtifactType" <path/to/some/artifact>:<type> -o json

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
  "files" :  [
    "path/to/my/uploaded/artifact.json",
    "path/to/my/other/uploaded/artifact.json"
  ],
  "size": sizeInBytes
}

Why is this needed for ORAS?

This would be helpful for programmatically checking artifact status vs. parsing text lines (which could change depending on environment).

Are you willing to submit PRs to contribute to this feature?

Yes, I am willing to implement it.

The text was updated successfully, but these errors were encountered:

sajayantony · 2022-10-10T19:16:58Z

I would also hope that the json output could contain more scriptable values like fully qualified references so that clients don’t have to join strings.

qweeah · 2022-10-11T07:10:58Z

Thanks @plooploops for this feature request. Can you elaborate on the scenarios of programmatically checking artifact status for oras attach, discover and pull repectively?

Below are some minimal information needed in the JSON output in my mind:
oras discover: digest, media type, size, artifact type (already supportted with -o json flag) and URL
oras pull: digest, media type and size of the pulled artifact manifest
oras attach: digest, media type, size and URL of the newly created artifact

plooploops · 2022-10-12T19:24:04Z

For oras discover

Thanks @qweeah, you're right that oras discover supports the -o json flag!

It would be helpful to include the URL as you had mentioned, I'm assuming in this case it would be for the artifact itself?

For example, suppose you make the call:

oras discover <myacr>.azurecr.io/my/repo:tag -o json

And then you would get a JSON output like the following:

{
  "referrers": [
    {
      "digest": "sha256:someHash",
      "mediaType": "someType",
      "artifactType": "someArtifactType",
      "artifactUrl" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
      "size": sizeInBytes
    },
  ]
}

This would allow you to pull in the artifact (e.g. oras pull <artifactUrl value>.

For oras pull

The fields you had mentioned make sense, and it would also be helpful to get the downloaded artifactType, artifactUrl, and artifactFilepath too.

Suppose you make the following call:

oras pull <artifactUrl value>

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "artifactUrl" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
  "artifactFilepath" : "path/to/my/downloaded/artifact.json",
  "size": sizeInBytes
}

For oras attach

The fields you had mentioned make sense, and it would also be helpful to get the uploaded artifactDigest, artifactUrl, and artifactFilePath.

Suppose you make the following call:

oras attach "<myacr>.azurecr.io/my/repo:tag" --artifact-type "someArtifactType" <path/to/some/artifact>:<type>

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "artifactDigest" : "sha256:someOtherHash", 
  "artifactUrl" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
  "artifactFilepath" : "path/to/my/uploaded/artifact.json",
  "size": sizeInBytes
}

If you think these make sense, I can pull the notes to the top post.

qweeah · 2022-10-13T02:21:09Z

@plooploops Thanks for the clarification. Here are some comments:

artifactUrl: I suggest to name it to reference since both OCI image and OCI artifact could have this field. And as you have described, this field specifies a URI from which this object MAY be downloaded, conforming to RFC 3986 with a scheme as defined in RFC 7230. Also user won't need this field in the result of oras pull since it's in the CLI argument.
artifactFilepath: I suggest to use files with an array type since more than one file might be processed. oras pull need this field. However oras attach doesn't because the blob file paths are specified by end users via the CLI arguments.
artifactDigest: This field is duplicated to digest field and only having one field digest would be enough.

/cc @FeynmanZhou for UX clarification and @shizhMSFT for planning.

sajayantony · 2022-10-13T03:05:54Z

Please use reference since thats has a well know definition in OCI. I’m curious why do we need to digest and artifactDigest?

qweeah · 2022-10-13T03:14:34Z

Please use reference since thats has a well know definition in OCI.

Good point. Updated inline.

I’m curious why do we need to digest and artifactDigest?

We don't need both. digest only would be enough.

plooploops · 2022-10-14T19:08:45Z

Updated version per comments:

For oras discover

For example, suppose you make the call:

oras discover <myacr>.azurecr.io/my/repo:tag -o json

And then you would get a JSON output like the following:

{
  "referrers": [
    {
      "digest": "sha256:someHash",
      "mediaType": "someType",
      "artifactType": "someArtifactType",
      "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
      "size": sizeInBytes
    },
  ]
}

This would allow you to pull in the artifact (e.g. oras pull <reference value>.

For oras pull

For oras pull, it would be helpful to get the downloaded artifactType, reference, and files along with the other fields described in the below sample output.

Suppose you make the following call:

oras pull <reference value>

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
  "files" : [
    "path/to/my/downloaded/artifact.json",
    "path/to/another/downloaded/artifact.json"
  ],
  "size": sizeInBytes
}

For oras attach

For oras attaach, it would also be helpful to get the uploaded artifactDigest, reference, and files along with the other fields described in the below sample output.

Suppose you make the following call:

oras attach "<myacr>.azurecr.io/my/repo:tag" --artifact-type "someArtifactType" <path/to/some/artifact>:<type>

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
  "files" :  [
    "path/to/my/uploaded/artifact.json",
    "path/to/my/other/uploaded/artifact.json"
  ],
  "size": sizeInBytes
}

Again, open for comments (and thank you for the input already @sajayantony and @qweeah!)

If we think this covers a good first version, then I'll pull it up to the top of the post.

qweeah · 2022-10-16T02:48:34Z

@plooploops This version looks good to me. One small thing to add: reference should contain scheme, e.g. http[s]://<myacr>.azurecr.io/my/repo@sha256:someHash

plooploops · 2022-10-24T17:31:06Z

@qweeah Sounds good. Updated the version here (and will copy it to the top of the post).

Updated version per comments:

For oras discover

For example, suppose you make the call:

oras discover <myacr>.azurecr.io/my/repo:tag -o json

And then you would get a JSON output like the following:

{
  "referrers": [
    {
      "digest": "sha256:someHash",
      "mediaType": "someType",
      "artifactType": "someArtifactType",
      "reference" : "http[s]://<myacr>.azurecr.io/my/repo@sha256:someHash",
      "size": sizeInBytes
    },
  ]
}

This would allow you to pull in the artifact (e.g. oras pull <reference value>.

For oras pull

For oras pull, it would be helpful to get the downloaded artifactType, reference, and files along with the other fields described in the below sample output.

Suppose you make the following call:

oras pull <reference value>

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "reference" : "http[s]://<myacr>.azurecr.io/my/repo@sha256:someHash",
  "files" : [
    "path/to/my/downloaded/artifact.json",
    "path/to/another/downloaded/artifact.json"
  ],
  "size": sizeInBytes
}

For oras attach

For oras attaach, it would also be helpful to get the uploaded artifactDigest, reference, and files along with the other fields described in the below sample output.

Suppose you make the following call:

oras attach "<myacr>.azurecr.io/my/repo:tag" --artifact-type "someArtifactType" <path/to/some/artifact>:<type>

And then you would get a JSON output like the following:

{
  "digest": "sha256:someHash",
  "mediaType": "someType",
  "artifactType": "someArtifactType",
  "reference" : "<http[s]://<myacr>.azurecr.io/my/repo@sha256:someHash",
  "files" :  [
    "path/to/my/uploaded/artifact.json",
    "path/to/my/other/uploaded/artifact.json"
  ],
  "size": sizeInBytes
}

sajayantony · 2022-10-24T18:06:34Z

The reference doesn’t contain a scheme as per the spec. Also you can’t use that format in most tools and so if you pipe that to say oras tag then that wouldn’t work.
My vote is to follow the spec and use
fqdn/repo:tag

plooploops · 2022-10-26T22:37:24Z

I'm interested in simplifying @sajayantony. Which spec are you referring to in this case?

Using "<myacr>.azurecr.io/my/repo@sha256:someHash" instead would make subsequent automation easier, for example, when looking to pull in your manifest with oras manifest fetch.

sajayantony · 2022-10-26T23:29:47Z

I was referring to about only removing the http(s) prefix. All other fields seem reasonable. /cc @shizhMSFT for any further input.

plooploops · 2022-10-26T23:38:11Z

Sounds good @sajayantony.

@qweeah are you open to removing the http(s) part for the reference field?

For example, when you call oras attach "<myacr>.azurecr.io/my/repo:tag" --artifact-type "someArtifactType" <path/to/some/artifact>:<type> you would get something like this:

{
 "digest": "sha256:someHash",
 "mediaType": "someType",
 "artifactType": "someArtifactType",
 "reference" : "<myacr>.azurecr.io/my/repo@sha256:someHash",
 "files" :  [
   "path/to/my/uploaded/artifact.json",
   "path/to/my/other/uploaded/artifact.json"
 ],
 "size": sizeInBytes
}

sajayantony · 2022-10-27T01:00:33Z

SGTM -

Does this sound like what you had in mind @plooploops

oras attach myreg.azurecr.io/my/repo:tag \
      --artifact-type "example/foo" logo.jpg:image/jpg \
      -o json  \
| jq .reference | xargs -n1 oras manifest fetch

Ensure you ask for -o json since simple text is what it defaults to
Use fields to pipe to jq/oras and avoid having to concat things like repo?

qweeah · 2022-10-27T02:29:13Z

@qweeah are you open to removing the http(s) part for the reference field?

@plooploops Sounds good if it helps automation.

oras attach myreg.azurecr.io/my/repo:tag \
      --artifact-type "example/foo" logo.jpg:image/jpg \
      -o json  \
| jq .reference | xargs -n1 oras manifest fetch --output <export-file-path>

@sajayantony Worth to mention that be above use case is already supported via --export-manifest (and would be safer if user is trying sign the uploaded manifest)

oras attach myreg.azurecr.io/my/repo:tag \
      --artifact-type "example/foo" logo.jpg:image/jpg \
      --export-manifest <export-file-path>

sajayantony · 2022-10-27T06:32:49Z

Ok my example may be bad. But the fact that I can use the fully qualified reference to the next command is still valuable. I don’t have to concat stuff.

shizhMSFT · 2022-10-27T11:45:20Z

It is bit weird to have --output for oras pull as we already have that option.

  -o, --output string                               output directory (default ".")

Generally speaking, this proposal and #497 are proposing the same thing: It is better to let the oras CLI to output some metadata so that it can be further processed.

How about --metadata-file? For example,

oras attach myreg.azurecr.io/my/repo:tag \
      --artifact-type "example/foo" logo.jpg:image/jpg \
      --metadata-file <metadata-file-path>

The metadata file can be further processed later. Even if there is an error in the middle of oras attach, you still can get some logs although oras is not able to produce the final metadata file.

plooploops · 2022-10-27T17:07:44Z

@sajayantony @qweeah - I've updated the top comment to reflect removing the scheme <http[s]> from the reference field. I also added the -o json flag as part of the example calls for oras discover, oras pull, and oras attach.

@shizhMSFT if there's an output formatting option for -o json or --output json that would let you format your output for stdout for oras discover, oras pull, and for oras attach would be helpful.

As you mentioned, oras pull does have -o json but this will put an output file somewhere, and you would need to parse the message to get the path, and then parse the output file.

oras pull <myacr>.azurecr.io/my/repo/image@sha256:somehash -o json --allow-path-traversal

Downloading differentHash C:\path\to\output\payload.json
Downloaded  differentHash C:\path\to\output\payload.json
Pulled <myacr>.azurecr.io/my/repo/image@sha256:somehash
Digest: sha256:somehash

If you can format the output that you get from stdout to json for oras discover, oras pull, and oras attach, then you can process the response without needing to parse text. For example, Azure CLI commands support multiple output formats, and having different oras commands implementing an output formatting option would be helpful from an automation standpoint.

After looking through the metadata file example from #497, it's possible to parse the metadata file to find the relevant fields like digest or reference, but is the schema of the metadata file stable? In this case, we'd still be parsing a file.

plooploops · 2022-10-28T00:38:44Z

@sajayantony @qweeah @shizhMSFT

Here's a hackmd to cover what's been included so far: https://hackmd.io/@plooploops/SkJn5qONs

FeynmanZhou · 2022-10-28T15:41:59Z

@sajayantony @qweeah @shizhMSFT

Here's a hackmd to cover what's been included so far: https://hackmd.io/@plooploops/SkJn5qONs

The proposal looks good and now I understand the scenario that you are requesting. For the tag name, there is already a flag output -o in some commands to export the output to a file. Will it be easy for users to memorize if we use just one --output flag and provide multiple options (file, JSON, other formats)?

shizhMSFT · 2022-10-28T16:30:33Z

The --output option means output file path or output directory in many commands like oras pull, oras blob fetch, oras manifest fetch, etc..

This proposal is to change the CLI display to the stdout. How about naming this kind of feature as --display. It takes options like text, json, and even tty, csv, yaml in the future. Let me put them in a single example,

$ oras pull example.contoso.com/myapp:v1 --output Downloads/myapp --display json
{
  "digest": "sha256:01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b",
  "mediaType": "application/vnd.oci.artifact.manifest.v1+json",
  "artifactType": "application/vnd.myapp",
  "reference" : "example.contoso.com/myapp@sha256:01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b",
  "files" : [
    "Downloads/myapp/artifact.json",
    "Downloads/myapp/app.tar"
  ],
  "size": 532
}
$ ls Downloads/myapp
artifact.json
app.tar

plooploops · 2022-10-28T19:13:54Z

@FeynmanZhou we could use the -o as you had mentioned for different output types like file, JSON.

However, @shizhMSFT brought up an interesting alternative in keeping the -o and --output reserved for the file path for output, and then using --display json to display the results in JSON to stdout.

In the hackmd I was using the --format json option instead, but I do like the option you had mentioned @shizhMSFT.

I updated the hackmd to reflect the proposal (and pull in your example @shizhMSFT) so we can have options for getting files, and for formatting responses to stdout as JSON.

sajayantony · 2023-07-19T01:15:21Z

Reviving this back up for 1.2.0.
Improving the scriptability of oras is important. I think having individual issues for each of these commands would be good to discuss the respective outputs while implementing it. @FeynmanZhou @shizhMSFT

plooploops added the enhancement New feature or request label Oct 10, 2022

shizhMSFT added this to the future milestone Oct 28, 2022

shizhMSFT added this to ORAS-Planning Oct 28, 2022

jasminetMSFT mentioned this issue Nov 3, 2022

feat: export manifest descriptor by push #524

Closed

shizhMSFT mentioned this issue Nov 25, 2022

spec: update cli verify spec for UX improvement notaryproject/notation#440

Merged

qweeah mentioned this issue Feb 11, 2023

Consistent logging through the project #786

Closed

1 task

qweeah mentioned this issue Mar 6, 2023

feat: add digest hint in oras tag logs #853

Merged

4 tasks

sajayantony added the logging label Jul 18, 2023

FeynmanZhou modified the milestones: future, v1.2.0 Jul 19, 2023

qweeah mentioned this issue Sep 12, 2023

support --output - to pull to stdout #346

Open

FeynmanZhou mentioned this issue Dec 4, 2023

doc(spec): formatted ORAS CLI output #1199

Merged

qweeah mentioned this issue Jan 9, 2024

feat: support formatted output in oras attach and push #1237

Merged

4 tasks

shizhMSFT assigned qweeah Mar 12, 2024

This was referenced Mar 14, 2024

feat: Use cobra.Command.OutOrStdout method for output #1288

Merged

feat: support --format for oras pull #1293

Merged

feat: support --format in manifest fetch command #1295

Merged

feat: support --format in oras discover #1299

Merged

qweeah closed this as completed in #1299 Apr 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Oras commands like attach, discover, and pull support an option to format output as JSON #638

Oras commands like attach, discover, and pull support an option to format output as JSON #638

plooploops commented Oct 10, 2022 •

edited

Loading

sajayantony commented Oct 10, 2022

qweeah commented Oct 11, 2022 •

edited

Loading

plooploops commented Oct 12, 2022

qweeah commented Oct 13, 2022 •

edited

Loading

sajayantony commented Oct 13, 2022

qweeah commented Oct 13, 2022

plooploops commented Oct 14, 2022 •

edited

Loading

qweeah commented Oct 16, 2022

plooploops commented Oct 24, 2022

sajayantony commented Oct 24, 2022

plooploops commented Oct 26, 2022

sajayantony commented Oct 26, 2022

plooploops commented Oct 26, 2022

sajayantony commented Oct 27, 2022

qweeah commented Oct 27, 2022 •

edited

Loading

sajayantony commented Oct 27, 2022

shizhMSFT commented Oct 27, 2022

plooploops commented Oct 27, 2022 •

edited

Loading

plooploops commented Oct 28, 2022

FeynmanZhou commented Oct 28, 2022

shizhMSFT commented Oct 28, 2022

plooploops commented Oct 28, 2022

sajayantony commented Jul 19, 2023

Oras commands like attach, discover, and pull support an option to format output as JSON #638

Oras commands like attach, discover, and pull support an option to format output as JSON #638

Comments

plooploops commented Oct 10, 2022 • edited Loading

What is the version of your ORAS CLI

What would you like to be added?

For oras discover

For oras pull

For oras attach

Why is this needed for ORAS?

Are you willing to submit PRs to contribute to this feature?

sajayantony commented Oct 10, 2022

qweeah commented Oct 11, 2022 • edited Loading

plooploops commented Oct 12, 2022

For oras discover

For oras pull

For oras attach

qweeah commented Oct 13, 2022 • edited Loading

sajayantony commented Oct 13, 2022

qweeah commented Oct 13, 2022

plooploops commented Oct 14, 2022 • edited Loading

For oras discover

For oras pull

For oras attach

qweeah commented Oct 16, 2022

plooploops commented Oct 24, 2022

For oras discover

For oras pull

For oras attach

sajayantony commented Oct 24, 2022

plooploops commented Oct 26, 2022

sajayantony commented Oct 26, 2022

plooploops commented Oct 26, 2022

sajayantony commented Oct 27, 2022

qweeah commented Oct 27, 2022 • edited Loading

sajayantony commented Oct 27, 2022

shizhMSFT commented Oct 27, 2022

plooploops commented Oct 27, 2022 • edited Loading

plooploops commented Oct 28, 2022

FeynmanZhou commented Oct 28, 2022

shizhMSFT commented Oct 28, 2022

plooploops commented Oct 28, 2022

sajayantony commented Jul 19, 2023

plooploops commented Oct 10, 2022 •

edited

Loading

qweeah commented Oct 11, 2022 •

edited

Loading

qweeah commented Oct 13, 2022 •

edited

Loading

plooploops commented Oct 14, 2022 •

edited

Loading

qweeah commented Oct 27, 2022 •

edited

Loading

plooploops commented Oct 27, 2022 •

edited

Loading