terminal input broken on OSX/IOS/Safari after recent upgrades.

[permezel]

Cockpit version: 255.1
OS: Fedora 34 server
Page: Terminal

After upgrading OSX to 12.0.1 (Monterey) and IOS to 15.1, input to Terminal has stopped working.
ENTER works.
Normal character input doesn't.

Steps to reproduce

1. Upgrade OSX or IOS to latest.
2. Might require installing additional keyboards: I have EN_us and JA_jp.
3. Login to cockpit via Safari
4. Select "Terminal"
5. Hit enter: see that line feed works.
6. Type 'ls -lrta' and see that the characters fail to register, but the ENTER 'works'.

If I switch to Japanese input, it does work, except that Japanese input is not useful for terminal interactions, typically.

If I switch back to ascii, and type 'ls -lrta' I get:

Works OK on chrome, though:

Here is my keyboard setting on OSX:

[martinpitt]

Does the keyboard work on other pages, like the "Filters" search box on the Logs page? I suppose it will, and MacOS' keyboard settings are most probably irrelevant.

Does keyboard input work for you on https://xtermjs.org/ ? Again, I suppose it does, as usually enabling screen reader mode (which that online demo doesn't do) is the more problematic part.

[permezel]

Does the keyboard work on other pages, like the "Filters" search box on the Logs page? I suppose it will, and MacOS' keyboard settings are most probably irrelevant.

Does keyboard input work for you on https://xtermjs.org/ ? Again, I suppose it does, as usually enabling screen reader mode (which that online demo doesn't do) is the more problematic part.

yes and yes.

[martinpitt]

Can you please confirm that you can type normally in https://jsfiddle.net/lowrey/cL4qntkw/ (standard XTerm demo). I then forked https://jsfiddle.net/8vb7qken/ to enable screen reader mode. How does that work for you? If it's not that, I'll create another one with different options; but that one is usually the most problematic.

[permezel]

On OSX/Safari, https://jsfiddle.net/lowrey/cL4qntkw/ fails. https://jsfiddle.net/8vb7qken/ also.  Even doing a cut/paste, when I hit ENTER:  "Terminal" still fails.  For some reason, today, after running those demos, the "Terminal" screen has the FG the same as the BG, for all settings of "Appearance". RESET does reset where the initial prompt appears (invisibly).  The font +/- moves where the cursor ends up, but it is all displayed in invisible ink. On IOS on my iPadPro, if I click the RESET button, Terminal starts to work. Does not work as a fix on OSX. I'm running Safari 15.1 on osx.  IOS should be up-to-date. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

…

On Wednesday, November 10th, 2021 at 16:51, Martin Pitt ***@***.***> wrote: Can you please confirm that you can type normally in https://jsfiddle.net/lowrey/cL4qntkw/ (standard XTerm demo). I then forked https://jsfiddle.net/8vb7qken/ to enable screen reader mode. How does that work for you? If it's not that, I'll create another one with different options; but that one is usually the most problematic. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.[https://github.com/notifications/beacon/AARU5FASHFDJGDWKG3JVTQLULIJATA5CNFSM5HPFQ37KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOHGBDPFI.gif]

[permezel]

Verifying that the xterm.js demo still works:

[permezel]

There are some errors loading the scripts you pointed me at. At least one is due to local PiHole filter rules.

Running thru PiHole:

sloth% host crbcdn.jsfiddle.net                                                                      
crbcdn.jsfiddle.net has address 0.0.0.0
crbcdn.jsfiddle.net has IPv6 address ::
crbcdn.jsfiddle.net has address 0.0.0.0

After enabling local VPN to bypass local PiHole:

sloth% host crbcdn.jsfiddle.net
crbcdn.jsfiddle.net has address 45.55.126.88

But this does not fix things. Here is the state of the world after a local cache flush and disabling VPN.
I need VPN off as my managed site is only accessible via local network. I can add a whitelist for the failing sites but in general, I do not like doing that, plus when I enabled VPN, the one site failing loaded, but then something else failed to load.

[permezel]

script.js still fails to load, even with VPN on.

no-VPN, yes-PiHole:

sloth% host cdn.usefathom.com
cdn.usefathom.com has address 0.0.0.0
cdn.usefathom.com has IPv6 address ::
cdn.usefathom.com has address 0.0.0.0

yes-VPN,no-PiHole:

sloth% host cdn.usefathom.com
cdn.usefathom.com is an alias for fathom-cdn.b-cdn.net.
fathom-cdn.b-cdn.net has address 103.77.233.210

[garrett]

The terminal works fine in GNOME Web, aka: Epiphany (WebKit browser on Linux)... even in Epiphany Canary (using the latest build of WebKitGTK 2.35.1 (r286326)). It seems that this is one of those rare ~1% of bugs that happens in Safari but not Epiphany.

Onward to macOS: I used quickemu to spin up a macOS VM and it doesn't work in Safari there, so this is reproducible. I specifically used Catalina and Safari 15.0 (15612.1.29.41.4, 15612). It's a little old, as I created this specific VM a while ago and I need to update, but the problem exists at least as far back as this version.

It does seem to take a very long time to start up the terminal too. I wonder if this is related. (The page loads fine but the terminal area just has a block cursor for a while and there's no prompt.) I did have a prompt the first time I tried (after several seconds) and saw the problem here. When I tried again, I didn't even get to the prompt; I just had the cursor.

I'm updating and will see what happens with the latest Safari version.

[garrett]

AHA!

It doesn't properly repaint until a resize event. So if you type ls /etc and then resize, you should see the results.

I think this is 100% related to the prompt not appearing at times. The issue is repainting. I wonder if our size handling is confusing Safari or if it's something else. I wonder if Safari is trying to optimize painting a little too much.

[garrett]

My first attempt was to see if any of the CSS changes we recently did might've triggered this behavior. I figured the easiest way to test this was basically removing the custom content we have in:

• pkg/lib/console.css
• pkg/systemd/terminal.scss

There are visual changes, so I can confirm these changes are in play, but there's no behavior change in the terminal. It still does not repaint until a resize forces it to repaint.

I think I'll have to hand this over to @marusak and @martinpitt. It doesn't seem to be CSS related and it looks like we'll have to do something in JavaScript to fix this.

[garrett]

Noteworthy: I also have the JSFiddle failing on Safari here in my VM as well. So I guess the bug is probably either in the terminal widget and/or Safari.

After resizing/reloading, I can manage to get the xterm demo text to show up, but it's not interactive. It's exhibiting the same frozen behavior.

[garrett]

There are a number of Safari related issues for xterm.js @ https://github.com/xtermjs/xterm.js/issues?q=is%3Aissue+is%3Aopen+safari ... perhaps one of them is related to the issue we're seeing here?

[garrett]

I took a crack at this again, and found xtermjs/xterm.js#2686 which suggested that canvas rendering is going away and will be moved to a plugin, with DOM rendering being set as default. When I tried switching to DOM rendering in the demo @ jsFiddle, it renders things and takes input just fine.

However, xtermjs/xterm.js#3271 states that the DOM renderer is "much slower" than canvas and suggests using WebGL instead.

However, both Canvas and WebGL read data from canvas, which is disabled in Tor, which has the same issues as Safari. xtermjs/xterm.js#2686 (comment) Is Safari possibly disabling this as well, as a security measure? That would explain why it doesn't even work with an old version of xterm.js, yet it used to work for us. I know Safari keeps tightening fingerprinting loopholes (as does Firefox), and it could explain the breakage.

So, our solution is probably to try out the DOM renderer, especially if it will be the default. I wonder what they mean by "much slower". That is: Is it slow enough to be noticed, or is this just statistically slower? If they're switching to it as the default (in a near-future release), it can't be so slow that it's unusable... right?

I tried adding rendererType: 'dom' as an option in https://github.com/cockpit-project/cockpit/blob/main/pkg/lib/cockpit-components-terminal.jsx#L118-L127 and it just breaks the terminal rendering everywhere... even after removing all of our custom CSS (for sizing). 🤷 Perhaps #16680 might help when it lands.

We may want to also consider WebGL rendering, but it looks like that might also have issues in Safari.

I think I really need assistance from @martinpitt and @marusak at this point. But we might have more clues to get this solved soon, I hope.

[garrett]

I've installed the last old version of macOS that doesn't support the latest Safari, 15.1. It's Mojave.

• Safari 14.1.2 (14611.3.10.1.7) on Mojave (10.14): Terminal works fine
• Catalina (10.15), Big Sur (10.16 / 11), Monterey (12) all have Safari 15.1 in their latest updates, where the Cockpit Terminal page does not work.

I'm not sure what Safari after 14.1.2 and before 15.1 does with the terminal, but all versions of macOS that support 15.x have a 15.1 update. (That is, the last 3 major versions of macOS support 15.1 and 14.x doesn't have an issue. So we should target 15.1 for the fix. Older versions would probably "just work" once we fix 15.x.)

[garrett]

We've updated xterm.js a few times since and this is still a problem...

[jelly]

We've updated xterm.js recently to 5.3.0. Hopefully this resolves the issue.

[garrett]

Still a problem:

It's Monterey 10.6.6 with Safari 16.5 (17615.2.9.116, 17615).

I'll try upgrading and see if more recent Safari builds might fix it. And I can try a newer version of macOS, but that's a hassle and will take a while.

(Tested using Quickemu, as usual.)

[garrett]

This was fixed in Safari 17!

Passes on:

macOS Ventura 13.4.1 (22F82)
Safari 17.0 (18616.1.27.111.22, 18616)

There are newer versions of macOS available, including Ventura. The important part is Safari 17.0.

https://developer.apple.com/documentation/safari-release-notes/safari-17-release-notes says:

Safari 17 is available for iOS 17, iPadOS 17, macOS Sonoma, macOS Ventura, and macOS Monterey.

Unfortunately, this means anyone stuck on an older version of macOS, iOS, and iPadOS that doesn't get Safari 17 is stuck with this bug.

I should note that WebKit itself has been fine, as has been the case on WebKitGTK, which includes both Cockpit Client and GNOME Web (aka: Epiphany). This has exclusively been a problem on Apple's Safari.

I don't think there's anything we can do about this. As it's fixed in the latest official Safari builds, I guess we should close this as fixed.

The Wikipedia page for Monterey (last version of macOS with Safari 17) lists this for support:

Receiving security updates. Drops support for most Macs released from mid 2013 to 2014, including all Macs with Nvidia GPUs.

So that means Macs after mid-2014 and up will have this fix. Macs older than that no longer get security fixes either, so hopefully people have since installed Linux (or at least Windows, as they were Intel) on those mid-2014 and earlier for some kind of support... They're likely in a drawer, landfill, or possibly even still in use (with security issues) at this point though. ☹️

[garrett]

According to https://en.wikipedia.org/wiki/MacOS_Big_Sur:

Unsupported as of September 26, 2023.

However, it seems stuck with Safari 16, according to https://www.macrumors.com/2022/09/12/apple-releases-safari-16/

So for all those people with almost a decade old or older Macs running Cockpit in Safari and not another browser: Sorry! I guess you might want to consider Firefox or Chrome. Or use ssh for your terminal experience.

(There is probably nobody using this specific old Mac with Safari + Cockpit now, as Safari 17 supports Macs for the last 9 years.)