refactor: status, metadata and content handlers for `manifest index` commands #1509

wangxiaoxuan273 · 2024-09-24T08:55:43Z

What this PR does / why we need it:
Implements output handlers for manifest index commands.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1503

Please check the following list:

Does the affected code have corresponding tests, e.g. unit test, E2E test?
Does this change require a documentation update?
Does this introduce breaking changes that would require an announcement or bumping the major version?
Do all new files have an appropriate license header?

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

codecov · 2024-09-24T08:58:28Z

Codecov Report

Attention: Patch coverage is 64.32161% with 71 lines in your changes missing coverage. Please review.

Project coverage is 82.90%. Comparing base (97cb376) to head (f6c0790).

Files with missing lines	Patch %	Lines
cmd/oras/root/manifest/index/update.go	35.18%	23 Missing and 12 partials ⚠️
cmd/oras/root/manifest/index/create.go	30.00%	14 Missing and 7 partials ⚠️
cmd/oras/internal/display/status/discard.go	57.14%	6 Missing ⚠️
cmd/oras/internal/display/status/text.go	90.24%	3 Missing and 1 partial ⚠️
...md/oras/internal/display/content/manifest_index.go	85.00%	2 Missing and 1 partial ⚠️
cmd/oras/internal/display/metadata/discard.go	71.42%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1509      +/-   ##
==========================================
- Coverage   83.84%   82.90%   -0.95%     
==========================================
  Files         118      119       +1     
  Lines        5156     5281     +125     
==========================================
+ Hits         4323     4378      +55     
- Misses        592      641      +49     
- Partials      241      262      +21

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

cmd/oras/internal/display/content/manifest_index.go

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

cmd/oras/root/manifest/index/create.go

cmd/oras/internal/display/content/manifest_index.go

cmd/oras/internal/display/metadata/text/manifest_index_create.go

cmd/oras/internal/display/metadata/discard.go

cmd/oras/internal/display/metadata/interface.go

cmd/oras/internal/display/status/interface.go

qweeah · 2024-09-26T04:58:32Z

cmd/oras/internal/display/status/interface.go

+}
+
+// ManifestIndexUpdateHandler handles status output for manifest index update command.
+type ManifestIndexUpdateHandler interface {


Contains duplicated methods ManifestIndexCreateHandler as below

OnIndexFetching(indexRef string) error OnIndexFetched(indexRef string, digest digest.Digest) error

You can create a new interface and use it in both interfaces.

Created an interface ManifestReferenceFetchHandler and both index create and index update handlers contain it.

Based on this comment, this interface is now removed.

cmd/oras/internal/display/handler.go

qweeah · 2024-09-26T06:20:45Z

cmd/oras/root/manifest/index/update.go

+			// ignore --pretty when output to a file
+			if opts.outputPath != "" && opts.outputPath != "-" {
+				opts.Pretty.Pretty = false
+			}


Can be merged into the handler construction.

If this is a general option thing, maybe a method in otpions?

If this is a general option thing, maybe a method in otpions?

@TerryHowe We can move it to content handler to avoid code repeat.

TerryHowe · 2024-09-27T17:50:46Z

cmd/oras/internal/display/content/manifest_index.go

+// OnContentCreated is called after index content is created.
+func (h *manifestIndexCreate) OnContentCreated(manifest []byte) error {
+	out := h.stdout
+	if h.outputPath != "-" && h.outputPath != "" {


if you changed outputPath to "" for "-" this logic would look less odd.

Did you mean to put "" before "-"? Changed the if condition to if outputPath != "" && outputPath != "-"

TerryHowe · 2024-09-27T17:52:08Z

cmd/oras/internal/display/handler.go

+}
+
+// NewManifestIndexUpdateHandler returns status, metadata and content handlers for index update command.
+func NewManifestIndexUpdateHandler(outputPath string, printer *output.Printer, pretty bool) (


should there just be a NewManivestIndexHandler method for both for now and someone can add the other in the future if needed?

Currently we need both create and update handlers.

wangxiaoxuan273 · 2024-10-09T07:00:29Z

cmd/oras/internal/display/status/text.go

+	tmich := TextManifestIndexCreateHandler{
+		printer: printer,
+	}
+	return &tmich


The current naming style is aligned with the rest of the file.

wangxiaoxuan273 · 2024-10-09T07:00:36Z

cmd/oras/internal/display/status/text.go

+	miuh := TextManifestIndexUpdateHandler{
+		printer: printer,
+	}
+	return &miuh


The current naming style is aligned with the rest of the file.

cmd/oras/root/manifest/index/create.go

TerryHowe · 2024-09-27T18:11:21Z

cmd/oras/root/manifest/index/create.go

@@ -188,7 +197,8 @@ func getPlatform(ctx context.Context, target oras.ReadOnlyTarget, manifestBytes
 	return &platform, nil
 }

-func pushIndex(ctx context.Context, target oras.Target, desc ocispec.Descriptor, content []byte, ref string, extraRefs []string, path string, printer *output.Printer) error {
+func pushIndex(ctx context.Context, onIndexPushed func(path string) error, onTagged func(desc ocispec.Descriptor, tag string) error,


This is a private method here, seems like it would make more sense for it to take handlers than function pointers.

Another thing, it seems like an extremely long list of params like something is not right here.

Since both create and update use this method and index create handlers and index update handlers have different types, the function argument can't take handler argument.

Yeah this is an extremely long list of parameters, but I can't find another way :(

TerryHowe · 2024-09-27T18:13:44Z

cmd/oras/root/manifest/index/update.go

+			// ignore --pretty when output to a file
+			if opts.outputPath != "" && opts.outputPath != "-" {
+				opts.Pretty.Pretty = false
+			}


If this is a general option thing, maybe a method in otpions?

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

wangxiaoxuan273 · 2024-10-09T07:00:29Z

cmd/oras/internal/display/status/text.go

+	tmich := TextManifestIndexCreateHandler{
+		printer: printer,
+	}
+	return &tmich


The current naming style is aligned with the rest of the file.

wangxiaoxuan273 · 2024-10-09T07:00:36Z

cmd/oras/internal/display/status/text.go

+	miuh := TextManifestIndexUpdateHandler{
+		printer: printer,
+	}
+	return &miuh


The current naming style is aligned with the rest of the file.

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

cmd/oras/internal/display/metadata/text/manifest_index.go

cmd/oras/root/manifest/index/update.go

qweeah · 2024-10-11T06:12:59Z

cmd/oras/internal/display/metadata/text/manifest_index.go

+
+// ManifestIndexUpdateHandler handles text metadata output for index update events.
+type ManifestIndexUpdateHandler struct {
+	printer *output.Printer


Can you help separate ManifestIndexUpdateHandler and ManifestIndexCreateHandler into manifest_index_update.go and manifest_index_create.go?

qweeah · 2024-10-11T06:15:16Z

cmd/oras/internal/display/metadata/interface.go

+	OnManifestRemoved(digest digest.Digest) error
+	OnManifestAdded(manifestRef string, digest digest.Digest) error
+	OnIndexMerged(indexRef string, digest digest.Digest) error
 }


Change digest to descriptor since it will be used for formatted output in the future.

The parameter of OnManifestRemoved(digest digest.Digest) error must be a digest, since the descriptor is unknown. Changed the parameter of the other two functions as descriptors.

qweeah · 2024-10-11T06:19:53Z

cmd/oras/internal/display/handler.go

+	status.ManifestIndexCreateHandler,
+	metadata.ManifestIndexCreateHandler,
+	content.ManifestIndexCreateHandler,
+	error) {


When will an error be returned?

good catch. Removed the error return.

qweeah · 2024-10-11T06:20:56Z

cmd/oras/internal/display/status/utils.go

@@ -52,12 +52,6 @@ const (
 const (
 	IndexPromptFetching = "Fetching "


Extra space, same for fetched.

Suggested change

IndexPromptFetching = "Fetching "

IndexPromptFetching = "Fetching"

We are not doing status output indentation align anymore?

Yes, there are still aligning work but there is trailing space for both Fetching and Fetched. Should remove one space from both prompts.

qweeah · 2024-10-11T06:36:33Z

cmd/oras/root/manifest/index/create.go

+func pushIndex(ctx context.Context, onIndexPushed func(path string) error, onTagged func(desc ocispec.Descriptor, tag string) error,
+	target oras.Target, desc ocispec.Descriptor, content []byte, ref string, extraRefs []string, path string) error {


Try using handlers like:

Suggested change

func pushIndex(ctx context.Context, onIndexPushed func(path string) error, onTagged func(desc ocispec.Descriptor, tag string) error,

target oras.Target, desc ocispec.Descriptor, content []byte, ref string, extraRefs []string, path string) error {

func pushIndex(ctx context.Context, pmh metadata.ManifestPackHandler, th metadata.TaggedHandler,target oras.Target, desc ocispec.Descriptor, content []byte, ref string, extraRefs []string, path string) error {

This function signature is now func pushIndex(ctx context.Context, displayStatus status.ManifestIndexCreateHandler, taggedHandler metadata.TaggedHandler, target oras.Target, desc ocispec.Descriptor, content []byte, ref string, extraRefs []string, path string) after resolving this comment. Still long param list.

shizhMSFT · 2024-10-11T06:39:58Z

cmd/oras/internal/display/content/discard.go

 // NewDiscardHandler returns a new discard handler.
-func NewDiscardHandler() ManifestFetchHandler {
+func NewDiscardHandler() discardHandler {


Should we export discardHandler just like status.DiscardHandler?

As NewDiscardHandler is a public method, its returned type should be exported. Exported all discard handlers in content, metadata and status packages.

shizhMSFT · 2024-10-11T06:52:19Z

cmd/oras/internal/display/content/manifest_index.go

+		if err != nil {
+			return fmt.Errorf("failed to open %q: %w", h.outputPath, err)
+		}
+		defer f.Close()


We need to create an issue to track handling errors returned by Close() in all write paths.

Created #1516

shizhMSFT · 2024-10-11T06:54:35Z

cmd/oras/internal/display/content/manifest_index.go

+	// ignore --pretty when output to a file
+	if outputPath != "" && outputPath != "-" {
+		pretty = false
+	}


It is different from NewManifestFetchHandler. Is NewManifestFetchHandler wrong? /cc @qweeah

NewManifestFetchHandler is used only once by oras manifest fetch and such code snippet is in command code

oras/cmd/oras/root/manifest/fetch.go

Lines 80 to 83 in dcef719

// ignore --pretty when output to a file

case opts.outputPath != "" && opts.outputPath != "-":

opts.Pretty.Pretty = false

}

I think we need to remove it and change the flag reset to NewManifestFetchHandler too.

Added pretty value reset in manifest fetch handler too.

cmd/oras/internal/display/metadata/discard.go

shizhMSFT · 2024-10-11T07:04:43Z

cmd/oras/internal/display/metadata/interface.go

+type manifestPackHandler interface {
+	OnIndexPacked(desc ocispec.Descriptor) error
+	OnIndexPushed(path string) error
+	OnCompleted(desc ocispec.Descriptor) error
+}


Do you want to merge this to ManifestIndexCreateHandler?

Removed this interface and now moved these three methods to index create handler.

shizhMSFT · 2024-10-11T07:06:16Z

cmd/oras/internal/display/status/interface.go

+// referenceFetchHandler handles status output for reference fetch events.
+type referenceFetchHandler interface {
+	OnFetching(manifestRef string) error
+	OnFetched(manifestRef string, desc ocispec.Descriptor) error
+}


Do you want to merge it to ManifestIndexCreateHandler?

Removed this interface and now moved these three methods to index create handler.

shizhMSFT · 2024-10-11T07:07:51Z

cmd/oras/internal/display/status/utils.go

 	IndexPromptFetching = "Fetching "
 	IndexPromptFetched  = "Fetched  "


Why those 2 are left?

As per offline discussion, added back "added, removed, merged, pushed, packed"

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

wangxiaoxuan273 · 2024-10-21T03:05:32Z

As per offline discussion, added, merged, removed, pushed, packed should exist in both status and metadata output. And Status output is needed in the PR, metadata output can be added in a future pr when needed.

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

Xiaoxuan Wang added 5 commits September 24, 2024 10:39

create handler

182c8f0

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

compiles

495ab7e

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

align update output with tag

d81fd04

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

align update output with digest

f807af8

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

refinement

9134a8f

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

wangxiaoxuan273 changed the title ~~Output output~~ refactor: status, metadata and content handlers for manifest index commands Sep 24, 2024

Xiaoxuan Wang added 3 commits September 24, 2024 17:04

added error checks

5104cc9

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

content handlers

a9b1164

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

refinement

4d144be

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

wangxiaoxuan273 marked this pull request as ready for review September 25, 2024 05:57

wangxiaoxuan273 requested review from sajayantony, shizhMSFT, SteveLasker, qweeah and TerryHowe as code owners September 25, 2024 05:57

qweeah reviewed Sep 25, 2024

View reviewed changes

cmd/oras/internal/display/content/manifest_index.go Show resolved Hide resolved

fix pretty with file output

4d18845

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

qweeah requested changes Sep 26, 2024

View reviewed changes

qweeah reviewed Sep 26, 2024

View reviewed changes

qweeah requested changes Sep 26, 2024

View reviewed changes

TerryHowe reviewed Sep 27, 2024

View reviewed changes

discard parameter naming

8ea9b14

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

wangxiaoxuan273 commented Oct 9, 2024

View reviewed changes

Xiaoxuan Wang added 5 commits October 9, 2024 15:01

resolved comments

16cf5ab

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

manifest ref fetch handler

c56fdec

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

reset pretty in handler construction

49091a2

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

resolved comments

66249a8

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

Merge branch 'main' into output-output

6e3b09a

wangxiaoxuan273 commented Oct 11, 2024

View reviewed changes

cmd/oras/internal/display/metadata/text/manifest_index.go Outdated Show resolved Hide resolved

wangxiaoxuan273 commented Oct 11, 2024

View reviewed changes

cmd/oras/internal/display/metadata/text/manifest_index.go Outdated Show resolved Hide resolved

wangxiaoxuan273 commented Oct 11, 2024

View reviewed changes

cmd/oras/root/manifest/index/update.go Show resolved Hide resolved

qweeah requested changes Oct 11, 2024

View reviewed changes

shizhMSFT reviewed Oct 14, 2024

View reviewed changes

Xiaoxuan Wang added 2 commits October 14, 2024 14:55

resolved billy's comment

851701d

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

Merge branch 'main' into output-output

3a2fb99

wangxiaoxuan273 force-pushed the output-output branch from 4daf7d6 to 3a2fb99 Compare October 14, 2024 07:09

qweeah mentioned this pull request Oct 14, 2024

fix: update --add should add plain descriptor #1513

Merged

wangxiaoxuan273 mentioned this pull request Oct 21, 2024

oras should handle errors returned by Close() in all write paths #1516

Open

Xiaoxuan Wang added 2 commits October 21, 2024 15:24

resolved shiwei's comments

a7a88d8

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

Merge branch 'main' into output-output

f6c0790

Signed-off-by: Xiaoxuan Wang <xiaoxuanwang@microsoft.com>

		@@ -52,12 +52,6 @@ const (
		const (
		IndexPromptFetching = "Fetching "

	IndexPromptFetching = "Fetching "
	IndexPromptFetching = "Fetching"

		func pushIndex(ctx context.Context, onIndexPushed func(path string) error, onTagged func(desc ocispec.Descriptor, tag string) error,
		target oras.Target, desc ocispec.Descriptor, content []byte, ref string, extraRefs []string, path string) error {

	func pushIndex(ctx context.Context, onIndexPushed func(path string) error, onTagged func(desc ocispec.Descriptor, tag string) error,
	target oras.Target, desc ocispec.Descriptor, content []byte, ref string, extraRefs []string, path string) error {
	func pushIndex(ctx context.Context, pmh metadata.ManifestPackHandler, th metadata.TaggedHandler,target oras.Target, desc ocispec.Descriptor, content []byte, ref string, extraRefs []string, path string) error {

	// ignore --pretty when output to a file
	case opts.outputPath != "" && opts.outputPath != "-":
	opts.Pretty.Pretty = false
	}

		IndexPromptFetching = "Fetching "
		IndexPromptFetched = "Fetched "

refactor: status, metadata and content handlers for manifest index commands #1509

Are you sure you want to change the base?

refactor: status, metadata and content handlers for manifest index commands #1509

Conversation

wangxiaoxuan273 commented Sep 24, 2024

codecov bot commented Sep 24, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangxiaoxuan273 Oct 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangxiaoxuan273 Oct 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangxiaoxuan273 commented Oct 21, 2024

refactor: status, metadata and content handlers for `manifest index` commands #1509

refactor: status, metadata and content handlers for `manifest index` commands #1509

codecov bot commented Sep 24, 2024 •

edited

Loading

wangxiaoxuan273 Oct 21, 2024 •

edited

Loading

wangxiaoxuan273 Oct 21, 2024 •

edited

Loading