Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support progressed output in blob push and blob get #1113

Merged
merged 80 commits into from
Oct 13, 2023
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
b088f2b
feat: support progressed output in blob push and blob get
qweeah Sep 12, 2023
e773471
add error when --debug is used with terminal output
qweeah Sep 19, 2023
6044594
bug fix
qweeah Sep 19, 2023
b5d89dd
Merge branch 'main' into progress-bar-reader
qweeah Sep 20, 2023
6324b93
add test for console
qweeah Sep 20, 2023
f1077bd
fix lint & pass tty to manager
qweeah Sep 20, 2023
4c362f8
Merge branch 'main' into progress-bar-reader
qweeah Sep 20, 2023
941188f
rename flag to disable tty
qweeah Sep 20, 2023
c72ebe5
add test for blob push
qweeah Sep 21, 2023
5fc75a8
fix render racing
qweeah Sep 21, 2023
a2d3f8c
fix race
qweeah Sep 21, 2023
4a065fe
add test for `blob fetch`
qweeah Sep 21, 2023
00ca2fc
resolve comment
qweeah Sep 22, 2023
c85e809
resolve comment
qweeah Sep 22, 2023
997a63e
resolve comment
qweeah Sep 22, 2023
144e907
fix lint
qweeah Sep 22, 2023
1b1f23c
refactor spinner mark
qweeah Sep 22, 2023
008a61d
refactor: improve manager and status
qweeah Sep 25, 2023
95ad473
add error for manager functions
qweeah Sep 25, 2023
ed84a69
remove unnecessary lock
qweeah Sep 25, 2023
47ef494
update reader naming
qweeah Sep 25, 2023
7e249ed
remove reader.once
qweeah Sep 25, 2023
ca19602
Merge remote-tracking branch 'origin_src/main' into progress-bar-reader
qweeah Sep 25, 2023
a6476d4
ignore returned error of manager
qweeah Sep 25, 2023
8c7458b
code clean
qweeah Sep 25, 2023
1584fd2
resolve comment
qweeah Sep 28, 2023
8fef80b
Merge remote-tracking branch 'origin_src/main' into progress-bar-reader
qweeah Sep 28, 2023
b44e2e8
code clean
qweeah Sep 28, 2023
ea6476f
code clean
qweeah Sep 28, 2023
ebcc114
Merge remote-tracking branch 'origin_src/main' into progress-bar-reader
qweeah Oct 7, 2023
ad0608d
add test for spinner
qweeah Oct 7, 2023
1aab8bd
add cross-package coverage generation
qweeah Oct 7, 2023
78b0d42
Refactor tests
qweeah Oct 7, 2023
2b13d6a
cover common tests
qweeah Oct 7, 2023
6bf4f3d
add status coverage
qweeah Oct 7, 2023
a7f3f96
code clean
qweeah Oct 7, 2023
3ad8eaa
add doc for stdout
qweeah Oct 7, 2023
243dc37
add status tests
qweeah Oct 7, 2023
5f5a1d9
resolve comment
qweeah Oct 7, 2023
7554534
NIT
qweeah Oct 7, 2023
76de650
align timing and percentage
qweeah Oct 7, 2023
486a8ed
fix unit test
qweeah Oct 7, 2023
39b0247
Merge remote-tracking branch 'origin_src/main' into progress-bar-reader
qweeah Oct 7, 2023
4fe990e
increase coverage
qweeah Oct 7, 2023
3a7ed21
fix: won't show complete when aborted
qweeah Oct 8, 2023
ff2c230
code clean
qweeah Oct 8, 2023
9426fac
code clean
qweeah Oct 8, 2023
50c4181
code clean
qweeah Oct 8, 2023
aa5b847
code clean
qweeah Oct 8, 2023
f49bdcb
align units of processed data to total size
qweeah Oct 8, 2023
8bc5c37
add instant download speed
qweeah Oct 8, 2023
7327022
code clean
qweeah Oct 9, 2023
1abda95
change base for sizing
qweeah Oct 9, 2023
636f424
fix speed display
qweeah Oct 9, 2023
2ea395d
add test coverage
qweeah Oct 9, 2023
4f7ac01
bug fix
qweeah Oct 9, 2023
c47f967
reset render interval to 200ms
qweeah Oct 9, 2023
2489a61
turn humanize into package
qweeah Oct 10, 2023
9e7daec
bug fix
qweeah Oct 10, 2023
c4564fe
rename pts device
qweeah Oct 10, 2023
167adb1
Merge remote-tracking branch 'origin_src/main' into progress-bar-reader
qweeah Oct 10, 2023
6c324d1
reduce buffer size
qweeah Oct 10, 2023
d4298c1
fix typo
qweeah Oct 10, 2023
67574dd
change switch to if-else
qweeah Oct 10, 2023
25d8935
bug fix
qweeah Oct 10, 2023
c76c464
change switch to if-else
qweeah Oct 10, 2023
9403cca
revert rendering
qweeah Oct 10, 2023
0741396
reduce buffer size to 1
qweeah Oct 10, 2023
c5ab1ee
fix: enable pipe when output to stdout
qweeah Oct 11, 2023
6611186
fix display bug
qweeah Oct 11, 2023
e858d1a
revert change
qweeah Oct 11, 2023
e4d4439
fix bug
qweeah Oct 11, 2023
6f36708
resolve comment
qweeah Oct 12, 2023
45aad11
fix nit
qweeah Oct 13, 2023
9e7f876
apply comment
qweeah Oct 13, 2023
57c29f2
resolve comments
qweeah Oct 13, 2023
7c8b2d7
udpate last render time
qweeah Oct 13, 2023
932c5d5
resolve comments
qweeah Oct 13, 2023
223eede
Merge remote-tracking branch 'origin_src/main' into progress-bar-reader
qweeah Oct 13, 2023
edf7ab8
code clean
qweeah Oct 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions cmd/oras/internal/display/console/console.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package console

import (
"fmt"
"os"

"github.com/containerd/console"
"github.com/morikuni/aec"
)

// MinialWidth is the minimal width of supported console.
const MinialWidth = 80

// MinialHeight is the minimal height of supported console.
const MinialHeight = 10
qweeah marked this conversation as resolved.
Show resolved Hide resolved

// Console is a wrapper around containerd's console.Console and ANSI escape
// codes.
type Console struct {
console.Console
}

// Size returns the width and height of the console.
// If the console size cannot be determined, returns a default value of 80x10.
func (c *Console) Size() (width, height int) {
width = MinialWidth
height = MinialHeight
size, err := c.Console.Size()
if err == nil && size.Height > MinialHeight && size.Width > MinialWidth {
qweeah marked this conversation as resolved.
Show resolved Hide resolved
width = int(size.Width)
height = int(size.Height)
}
return
}

// GetConsole returns a Console from a file.
func GetConsole(f *os.File) (*Console, error) {
qweeah marked this conversation as resolved.
Show resolved Hide resolved
c, err := console.ConsoleFromFile(f)
if err != nil {
return nil, err
}
return &Console{c}, nil
}

// Save saves the current cursor position.
func (c *Console) Save() {
fmt.Fprint(c, aec.Hide)
// cannot use aec.Save since DEC has better compatilibity than SCO
fmt.Fprint(c, "\0337")
qweeah marked this conversation as resolved.
Show resolved Hide resolved
}

// NewRow allocates a horizontal space to the output area with scroll if needed.
func (c *Console) NewRow() {
// cannot use aec.Restore since DEC has better compatilibity than SCO
fmt.Fprint(c, "\0338")
fmt.Fprint(c, "\n")
fmt.Fprint(c, "\0337")
}

// OutputTo outputs a string to a specific line.
func (c *Console) OutputTo(upCnt uint, str string) {
fmt.Fprint(c, "\0338")
fmt.Fprint(c, aec.PreviousLine(upCnt))
fmt.Fprint(c, str+" ")
qweeah marked this conversation as resolved.
Show resolved Hide resolved
fmt.Fprint(c, aec.EraseLine(aec.EraseModes.Tail))
}

// Restore restores the saved cursor position.
func (c *Console) Restore() {
// cannot use aec.Restore since DEC has better compatilibity than SCO
fmt.Fprint(c, "\0338")
fmt.Fprint(c, aec.Column(0))
fmt.Fprint(c, aec.EraseLine(aec.EraseModes.All))
fmt.Fprint(c, aec.Show)
}
61 changes: 61 additions & 0 deletions cmd/oras/internal/display/console/console_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package console

import (
"testing"

"github.com/containerd/console"
)

func validateSize(t *testing.T, gotWidth, gotHeight, wantWidth, wantHeight int) {
t.Helper()
if gotWidth != wantWidth {
t.Errorf("Console.Size() gotWidth = %v, want %v", gotWidth, wantWidth)
}
if gotHeight != wantHeight {
t.Errorf("Console.Size() gotHeight = %v, want %v", gotHeight, wantHeight)
}
}

func TestConsole_Size(t *testing.T) {
pty, _, err := console.NewPty()
if err != nil {
t.Fatal(err)
}
c := &Console{
Console: pty,
}

// minimal width and height
gotWidth, gotHeight := c.Size()
validateSize(t, gotWidth, gotHeight, MinialWidth, MinialHeight)

// zero width
_ = pty.Resize(console.WinSize{Width: 0, Height: MinialHeight})
gotWidth, gotHeight = c.Size()
validateSize(t, gotWidth, gotHeight, MinialWidth, MinialHeight)

// zero height
_ = pty.Resize(console.WinSize{Width: MinialWidth, Height: 0})
gotWidth, gotHeight = c.Size()
validateSize(t, gotWidth, gotHeight, MinialWidth, MinialHeight)

// valid zero and height
_ = pty.Resize(console.WinSize{Width: 200, Height: 100})
gotWidth, gotHeight = c.Size()
validateSize(t, gotWidth, gotHeight, 200, 100)
}
149 changes: 149 additions & 0 deletions cmd/oras/internal/display/progress/manager.go
qweeah marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package progress

import (
"os"
"sync"
"time"

"oras.land/oras/cmd/oras/internal/display/console"
)

const BUFFER_SIZE = 20
qweeah marked this conversation as resolved.
Show resolved Hide resolved

// Status is print message channel
type Status chan<- *status

// Manager is progress view master
type Manager interface {
Add() Status
StopAndWait()
}

const (
bufFlushDuration = 100 * time.Millisecond
)

type manager struct {
statuses []*status
rwLock sync.RWMutex
renderTick *time.Ticker
c *console.Console
qweeah marked this conversation as resolved.
Show resolved Hide resolved
updating sync.WaitGroup
mu sync.Mutex
close sync.Once
// done used to stop render routine
// doneDone used to mark render routine stopped
done chan struct{}
doneDone chan struct{}
}

// NewManager initialized a new progress manager.
func NewManager(f *os.File) (Manager, error) {
var m manager
var err error

m.c, err = console.GetConsole(f)
if err != nil {
return nil, err
}
m.done = make(chan struct{})
m.doneDone = make(chan struct{})
m.renderTick = time.NewTicker(bufFlushDuration)
m.start()
return &m, nil
}

func (m *manager) start() {
m.renderTick.Reset(bufFlushDuration)
m.c.Save()
go func() {
for {
m.render()
select {
case <-m.done:
close(m.doneDone)
return
case <-m.renderTick.C:
}
}
}()
}

func (m *manager) render() {
m.mu.Lock()
defer m.mu.Unlock()
// todo: update size in another routine
width, height := m.c.Size()
len := len(m.statuses) * 2
offset := 0
if len > height {
// skip statuses that cannot be rendered
offset = len - height
}

for ; offset < len; offset += 2 {
m.rwLock.RLock()
status, progress := m.statuses[offset/2].String(width)
m.rwLock.RUnlock()
m.c.OutputTo(uint(len-offset), status)
m.c.OutputTo(uint(len-offset-1), progress)
}
}

// Add appends a new status with 2-line space for rendering.
func (m *manager) Add() Status {
m.mu.Lock()
defer m.mu.Unlock()
id := len(m.statuses)
m.statuses = append(m.statuses, nil)
defer m.c.NewRow()
defer m.c.NewRow()
return m.newStatus(id)
}

func (m *manager) newStatus(id int) Status {
ch := make(chan *status, BUFFER_SIZE)
m.updating.Add(1)
go m.update(ch, id)
return ch
}

func (m *manager) update(ch chan *status, id int) {
defer m.updating.Done()
for s := range ch {
m.rwLock.Lock()
m.statuses[id] = m.statuses[id].Update(s)
m.rwLock.Unlock()
}
}

// StopAndWait stops all status and waits for updating and rendering.
func (m *manager) StopAndWait() {
// 1. stop periodic render
m.renderTick.Stop()
close(m.done)
defer m.close.Do(func() {
// 4. restore cursor, mark done
m.c.Restore()
})
// 2. wait for all model update done
m.updating.Wait()
// 3. render last model
<-m.doneDone
m.render()
}
31 changes: 31 additions & 0 deletions cmd/oras/internal/display/progress/mark.go
qweeah marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package progress

var (
spinner = []rune("⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏")
spinnerLen = len(spinner)
spinnerPos = 0
)

// GetMark returns the rune of status mark.
func GetMark(s *status) rune {
if s.done {
return '√'
}
spinnerPos = (spinnerPos + 1) % spinnerLen
qweeah marked this conversation as resolved.
Show resolved Hide resolved
return spinner[spinnerPos/2]
}
Loading