Improve action log display with control chars

[wxiaoguang]

Close #23680

Some CLI programs use "\r" and control chars to print new content in current line.

So, the strings in one line are actually from \rReading...1%\rReading...5%\rReading...100%

This PR tries to make the output better.

[silverwind]

Would this have adverse effects of Windows-style newlines \r\n? Would suggest adding a test for that as well.

[wxiaoguang]

Would this have adverse effects of Windows-style newlines \r\n? Would suggest adding a test for that as well.

Nope, you can check the tests, one of them covers the ending "\r".

---

'\rabc\rx\r' there is an ending "\r", it won't affect the final result. And the lines have been splitted by "\n", so there wont' be "\n" in the string if I understand correctly, and that's why this function is called processConsoleLine, it only processes one line.

[wxiaoguang]

To make code more strict, I make the function remove the trailing "\r\n" or "\n" in 2c72257 , then it doesn't matter whether the line ends with new-line or not.

[codecov-commenter]

Codecov Report

Merging #23820 (2c72257) into main (f521e88) will decrease coverage by 0.16%.
The diff coverage is 28.98%.

❗ Current head 2c72257 differs from pull request most recent head c4249bd. Consider uploading reports for the commit c4249bd to get more accurate results

@@            Coverage Diff             @@
##             main   #23820      +/-   ##
==========================================
- Coverage   47.14%   46.98%   -0.16%     
==========================================
  Files        1149     1158       +9     
  Lines      151446   153175    +1729     
==========================================
+ Hits        71397    71971     +574     
- Misses      71611    72706    +1095     
- Partials     8438     8498      +60     

Impacted Files	Coverage Δ
cmd/dump.go	0.66% <0.00%> (-0.01%)	⬇️
cmd/mailer.go	0.00% <0.00%> (ø)
cmd/manager.go	0.00% <0.00%> (ø)
cmd/manager_logging.go	0.00% <0.00%> (ø)
cmd/migrate_storage.go	5.76% <0.00%> (-0.12%)	⬇️
cmd/restore_repo.go	0.00% <0.00%> (ø)
cmd/web.go	0.00% <0.00%> (ø)
models/actions/run.go	1.63% <0.00%> (-0.10%)	⬇️
models/actions/runner.go	1.44% <ø> (ø)
models/packages/package.go	45.45% <0.00%> (-1.13%)	⬇️
... and 65 more

... and 67 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[yp05327]

image: docker:23

tested in 2c72257

I prefer add a special process to handle \r or \n which can exchange one line log into several new lines, like:

1 Reading...10%
2 Reading...30%
3 Reading...50%
4 Reading...70%
5 Reading...Done

[wxiaoguang]

I see, I haven't spent enough time on this problem and it looks like not that simple. The comment TODO: control chars is also necessary for most cases.

So let's mark this PR as WIP.

To make a further improvement, I guess there could be 3 approaches:

1. Make backend split "\r\n" and "\n" and "\r", then the frontend can render:

   1 Reading... 1%
   2 Reading... 5%
   3 Reading... Done
   4 Next line
   

2. Make frontend split:

   1 Reading... 1%
     Reading... 5%
     Reading... Done
   2 Next line
   

3. Wait for more time to support all control chars and test for various cases.

Feel free to help or propose new PRs (while I am trying to implement approach 2)

[wxiaoguang]

@yp05327 what do you think about approach 2: "Make frontend split" ? I think it's clear and simple enough, and the output is still reasonable: because these strings are indeed rendered in one line if you run the command in terminal.

See 9d35890 and its tests.

---

If we want to implement approach 1, I am not sure how much work it needs for backend changes (if approach 2 is good enough, then we do not need to change backend's behavior).

If we want to implement approach 3, we need make AnsiToHTML package support Erase Control Chars , or choose other rendering packages.

[yp05327]

I agree approach 2 and 3. And using an existing package to handle it looks better, as we don’t need to maintain it by ourselves to support all possible chars and cases.
But I have no idea about it.

[wxiaoguang]

Thank you. I think 9d35890 could implement approach 2. If it looks good to you, let's move on ~~

I have done a quick test, the "\n" looks good to me 😁

[yp05327]

tested in 9d35890

I found the original log files, and it seems that the reason of this problem is ESC.

[yp05327]

There is a document maybe related?
https://man7.org/linux/man-pages/man4/console_codes.4.html

[yp05327]

Remove them or convert them into HTML code?

I found some solutions maybe related.

• Remove them
  https://stackoverflow.com/questions/14693701/how-can-i-remove-the-ansi-escape-sequences-from-a-string-in-python
• Convert them into HTML code (I prefer this solution)
  https://github.com/k0kubun/go-ansi
  https://pkg.go.dev/github.com/robert-nix/ansihtml#section-readme
  https://github.com/rburns/ansi-to-html

[wxiaoguang]

That's also the comment in code control chars like "\033[" , it should be handled by the AnsiToHTML packages (the color is also provided by these control chars). I will write some new tests to see why AnsiToHTML doesn't work for them in this case.

[wxiaoguang]

Hmm, I think it's a bug in AnsiToHTML package.

Without our code, AnsiToHTML itself doesn't work for \033[1A nor \033[1B, it only handles the \033[2K

  const ath = new AnsiToHTML({escapeXML: true});
  expect(ath.toHtml('\x1b[1A\x1b[2K\rtest\r\x1b[1B\x1b[1A\x1b[2K')).toEqual('A\rtest\rBA');
  expect(ath.toHtml('\x1b[1A\x1b[2Ktest\x1b[1B\x1b[1A\x1b[2K')).toEqual('AtestBA');

[wxiaoguang]

Although it's related to upstream bug/limitation (https://github.com/rburns/ansi-to-html) , upstream has many open issues/PRs, so I guess we could bypass the bug on our side first. And I guess some behaviors we like couldn't be accepted by upstream easily (eg: treat the [0K or [0J as new line)

So, let's try to use 2604c22 to patch.

[silverwind]

Maybe there better JS modules around for this. Many parts of GitHub's JS are open source, so I wonder if they may have their terminal rendering also available somewhere 😉.

[yp05327]

tested in 2604c22

[wxiaoguang]

I think it's much better than before.

Do you prefer to see full progress list or the only one final line?

• If you prefer to see the progress list, then current approach seems good enough.
• If you prefer to see the only final line, I think it's still feasible, we can only take the last message after [0K.
  Then the docker pull .... line 3 becomes one line : Digest: .....
  And the apk add ... line 16 becomes one line: Executing busybox-......