webRTC screensharing support

[Elettrotecnica]

Dear developers,

at the university where I work, we have implemented proctoring of student exams via desktop and camera snaphots taken at random intervals using the browser's getDisplayMedia and getUserMedia APIs.

We plan to harden the security of our exams by introducing your great Safe Exam Browser. However, we are struggling with capturing the desktop, as it does not seem to be possible via getDisplayMedia. Capturing audio and camera instead works perfectly.

I got in contact with you already on Sourceforge (https://sourceforge.net/p/seb/feature-requests/71/). Apparently, the Chromium Embedded Framework you are using now as a base for your browser component does support screen-sharing. However, accroding to

https://bitbucket.org/chromiumembedded/cef/issues/2993/webrtc-based-screen-sharing-capturing-not
and
https://bitbucket.org/chromiumembedded/cef/issues/1065/add-support-for-webrtc-based-screen

there might be extra configuration needed on the integrator (your) side. I am also not sure how difficult it is to choose the correct screen. The basic issue is that CEF does not seem to support the screen selection UI, which cascades to a series of consequences (e.g. getDisplayMedia demands that the user chooses a source explicitly).

Please consider looking into the issue, as there is at least a chance that a relatively small effort could provide a very useful feature.

I am available for testing or to provide additional information in case.

All the best and thanks for a very useful tool.

[dbuechel]

Thanks for the issue creation. We'll have a look into it once we get the time, we're currently however pretty busy with the new proctoring feature supporting Jitsi Meet & Zoom.

[dbuechel]

Okay, so I added the two flags and basic screen capturing seems to work now. It however appears to be the case that only the currently active main display can be captured, at least according to the comments in the CEF issues mentioned above.

@Elettrotecnica Could you have a look at it and see whether it works for you as well? If yes, then we'd release this with version 3.3.0.

[dbuechel]

We will thus release 3.3.0 with the screen capturing capabilities mentioned above.

[Elettrotecnica]

Many thanks for your efforts! I think the behavior you describe is one of the aforementioned consequences of the embedded Chrome not supporting the screen selection UI.

Is there a preview version I can check? Otherwise I will report on the 3.3.0 release as soon as it is out.

Thanks again!

[Elettrotecnica]

I might have found the preview build, is the one one can download as an artifact here, right? https://sebdev-let.ethz.ch/project/appveyor/seb-win-refactoring

Will let you know soon

[dbuechel]

Yes exactly, those are the beta builds.

[Elettrotecnica]

I have tried the beta version and screen capture worked as expected. One cannot select the screen explicitly and the currently active one will be automatically taken, but this is not a real issue, as one can just limit the number of allowed displays to 1 in the seb conf.

Thanks again a lot!

[dbuechel]

Excellent, thanks for the feedback. Yes, unfortunately the browser component does not (yet) provide any functionality for screen selection. In case CEF / Chromium extend their API in the future we'd naturally revisit this issue and implement the necessary dialogs etc.

[dbuechel]

The feature is now released with version 3.3.0.

[lopezbertoni]

@Elettrotecnica @dbuechel
Sorry for the late message, but is this already working?
I try to share a screen using getDisplayMedia but I never get the screen popup to pick a tab or screen.
Please advise on how to proceed, greatly appreciated!

[dbuechel]

One cannot select the screen explicitly and the currently active one will be automatically taken, but this is not a real issue, as one can just limit the number of allowed displays to 1 in the seb conf.

#99 (comment)

[lopezbertoni]

@dbuechel Thanks for the response. I saw that comment but I was just unable to get it to work. What happened is that I had 2 monitors set up and that doesn't work.
After I unplugged one of my monitors, it still didn't work. I ended up rebooting my machine with only 1 monitor connected and then it started working as expected.
Thanks again for your help!

[dbuechel]

Aha, I see. Well, it might well be that it stopped working in the meantime, this has most unfortunately happened before when updating the browser engine. Are there any suspicious log entries in the client or browser log of SEB (to be found under %LocalAppData%\SafeExamBrowser\Logs)?

[lopezbertoni]

@dbuechel Some stuff in the logs.

E:\ProjectsGitHub\GreenLight\remote-proctoring\remote-proctoring\src\app\core\services\twilio.service.ts depends on 'twilio-video'. CommonJS or AMD dependencies can cause optimization bailouts.
For more info see: https://angular.io/guide/build#configuring-commonjs-dependencies", source: http://localhost:4300/polyfills.js (6230)
[0221/173330.310:INFO:CONSOLE(34497)] "2023-02-21T20:33:30.309Z warn [createLocalTracks #1] Call to getUserMedia failed: [object DOMException]", source: http://localhost:4300/vendor.js (34497)
[0221/173330.310:INFO:CONSOLE(62191)] "ERROR Error: Uncaught (in promise): NotReadableError: Could not start video source", source: http://localhost:4300/vendor.js (62191)
[0221/173330.557:WARNING:media_session.cc(949)] RED codec red is missing an associated payload type.
[0221/173331.508:INFO:CONSOLE(34497)] "2023-02-21T20:33:31.508Z warn [createLocalTracks #3] Call to getUserMedia failed: [object DOMException]", source: http://localhost:4300/vendor.js (34497)
[0221/173331.508:INFO:CONSOLE(62191)] "ERROR Error: Uncaught (in promise): NotReadableError: Could not start video source", source: http://localhost:4300/vendor.js (62191)
[0221/173332.304:WARNING:stun_port.cc(485)] Port[1109c00:0:1:0:local:Net[any:0:0:0❌x❌x:x/0:Wildcard:id=0]]: StunPort: stun host lookup received error 0
[0221/173332.304:WARNING:net_errors_win.cc(119)] Unknown error 1168 mapped to net::ERR_FAILED
[0221/173332.343:WARNING:basic_port_allocator.cc(1055)] Discarding candidate because port is already done gathering.
[0221/173332.484:WARNING:net_errors_win.cc(119)] Unknown error 1168 mapped to net::ERR_FAILED

I think this is non-issue at least for our use case. We're not going to support multiple monitors.
I also had the configuration set to 2 monitors max, maybe it was a combination of both that tripped this.
Regardless, thanks for your help!

Update

In case you're wondering about why this might fail on the MAC version of the SEB, you can get the following error if you try to run this automatically:

getdisplaymedia must be called from a user gesture

I worked around this by just showing a button that on click will share screen.

[dbuechel]

Excellent, that's good to know. I guess the issue is related to the following log entry:

[0221/173330.310:INFO:CONSOLE(34497)] "2023-02-21T20:33:30.309Z warn [createLocalTracks # 1] Call to getUserMedia failed: [object DOMException]", source: http://localhost:4300/vendor.js (34497)