Forward audio #3757
Forward audio #3757
Conversation
rom1v
force-pushed
the
audio
branch
4 times, most recently
from
6e948fe
to
2b2cf0a
Compare
|
Are the BUILD instructions outdated? |
|
Seems to be nowhere in the sources? I don't even see a src folder EDIT: Oh the folder is there just not that file in app/src/util |
Oops, it was not committed, sorry. Fixed. |
|
|
Aha java is too new (jdk 19). Trying an older version |
|
It's alive! Note 10 Plus on Android 12. Working well! |
test.mp4 |
|
Looking forward to this PR! |
Since scrcpy-server is not an Android application (it's a java executable), it has no Context. Some features will require a Context instance to get the package name and the UID. Add a FakeContext for this purpose. PR #3757 <#3757> Co-authored-by: Romain Vimont <rom@rom1v.com> Signed-off-by: Romain Vimont <rom@rom1v.com>
The audio demuxer thread is the one filling the audio buffer read by the SDL audio thread. It is time critical to avoid buffer underflow.
Recording is background task, writing the packets to a file is not urgent.
|
Let's merge this. I will continue some work on the |
Is the mod+r Android 11 bug fixed in this commit or will it be fixed in the next one? |
|
@rp1231 I reference previous commits since the github thread is unusable: #3757 (comment) #3757 (comment)
It is not fixed. Since the glitch is not recorded and is reduced with more buffering, here is my hypothesis. When buffer underflow occurs, the player inserts silence samples (it does not have any valid samples yet). Typically, buffer underflow occurs because some packets are late (due to jitter), so they are expected to arrive, but later. Therefore, inserting silence will delay the playback of future packets, and therefore I immediately shift the buffering level average by the number of silence samples inserted. If I didn't do that, then the overbuffering due to the inserted silence will be detected slowly (because the average buffering is smoothed), so it will first accelerate the playback (because underflow), then decelerate (because overbuffering will finally be detected). But there might be another cause of buffer underflow, which is currently not taken into account at all: when the device just misses to capture samples for a period of time. This is less likely than jitter, but I suppose that this is what happens when you rotate your device during fullscreen youtube playback: the device "freezes" and don't capture audio for a short period. In that case, considering that the missing samples will arrive (and count them for the buffering level) is a wrong prediction, which will be corrected "slowly" by the smoothed (observed) buffering level average. During this time, more underflow will occur. One possible fix could be to use the input packet PTS to detect that samples have not been captured by the device, but another issue makes it impossible: the default OPUS encoder writes its own timestamps in the encoded packets it produces, in a way that it matches the number of actual samples (it does not use the input timestamps provided by the audio capture), so missing packets could not be detected. The AAC encoder does not do that though (it forwards the input timestamp), so I build a specific version with more logs, to confirm my hypothesis. EDIT: do not take this binary, see the next comment.
Please run it with: diffdiff --git a/app/src/audio_player.c b/app/src/audio_player.c
index 41e7565f2..ed0934691 100644
--- a/app/src/audio_player.c
+++ b/app/src/audio_player.c
@@ -144,6 +144,17 @@ sc_audio_player_frame_sink_push(struct sc_frame_sink *sink,
const AVFrame *frame) {
struct sc_audio_player *ap = DOWNCAST(sink);
+ assert(frame->pts != AV_NOPTS_VALUE);
+ if (ap->expected_next_pts != AV_NOPTS_VALUE) {
+ int64_t diff = ap->expected_next_pts - frame->pts;
+ if (diff) {
+ LOGI("==== unexpected PTS diff = %" PRIi64, diff);
+ }
+ }
+ int64_t frame_duration_us = INT64_C(1000000) * frame->nb_samples
+ / ap->sample_rate;
+ ap->expected_next_pts = frame->pts + frame_duration_us;
+
SwrContext *swr_ctx = ap->swr_ctx;
int64_t swr_delay = swr_get_delay(swr_ctx, ap->sample_rate);
@@ -414,6 +425,7 @@ sc_audio_player_frame_sink_open(struct sc_frame_sink *sink,
ap->received = false;
ap->played = false;
ap->underflow = 0;
+ ap->expected_next_pts = AV_NOPTS_VALUE;
// The thread calling open() is the thread calling push(), which fills the
// audio buffer consumed by the SDL audio thread.
diff --git a/app/src/audio_player.h b/app/src/audio_player.h
index 3227f2fef..224abe62d 100644
--- a/app/src/audio_player.h
+++ b/app/src/audio_player.h
@@ -35,6 +35,9 @@ struct sc_audio_player {
// thread)
uint32_t previous_can_write;
+ // Useful to detect missing packets (e.g. due to a device freeze)
+ int64_t expected_next_pts;
+
// Resampler (only used from the receiver thread)
struct SwrContext *swr_ctx;
|
|
The version I built in the previous comment was on a branch with a major bug for estimation. Here is a new binary:
Also, please run with |
|
I'm getting this error: even on the previous build you posted, I got the error unknown option --Vdebug |
|
Also wanted to mention that this problem didn't occur in the very first build you had posted (scrcpy-win64-PR3757-1) |
|
Sorry, it is
Ok, but that could a side effect that in this first build buffering might be higher than expected. |
|
I could reproduce on a tablet: When the tablet is rotated, everything freezes, so captured packets are not sent immediately and the audio is not captured for a few milliseconds. On the client side, this causes buffer underflow (and glitches) as expected. The only way not to cause underflow is to increase the buffering (so that when the freeze happens, there are still enough samples in the buffer to absorb it). |
|
@rom1v |
mod+r requests the foreground app to switch orientation. It may refuse (typically if it does not support the other orientation, but not only). |
Yep but the thing is that this issue didn't occur before on the same android 11 phone in v1.25. |
If you retry now with v1.25, does it work? I think not. It's probably due to a youtube/os update. |
Ok I just tried it again on v2.0 and the shortcuts started working somehow..... |
|
A bit of a dumb question, but I wanted to ask: is it possible to patch the |
|
Processes started by root don't need any permission to capture audio. See #4127 |
Is running root directly with scrcpy a security risk, compared to running with the patched |
|
Recording does not support RAW audio codec... Why? Is it possible to add lossless audio recording? Any quick and dirty way |
|
Raw audio streams are not supported by mkv or mp4. It would require some lossless codec (but not implemented). In your stack trace, you don't use the official version, right? Because if I execute with your params, I get: |
|
Yeah I patched an exe just for fun |
|
Yes, it is possible to add flac support. On Windows, you need to build an FFmpeg with flac decoding support: https://github.com/rom1v/scrcpy-deps/blob/231e4e819127dc55f1bf394943ba9887c1cb6224/build_ffmpeg_windows.sh#L40-L41 And in scrcpy, you add flac everywhere in addition to opus and aac. Here is the commit for aac: 4601735 PR welcome 😉 |
Last month, @yume-chan submitted a working PoC to capture audio on Android. Big thanks for this discovery!
Since then I have worked to properly integrate audio support in scrcpy (capture, encode, transmit, record, decode, resample, play…), and my branch reached a working state (at least on my machine) 🎉
By default, audio is enabled (if supported):
If audio capture fails, then mirroring continues with video only (since audio is enabled by default, it is not acceptable to make scrcpy fail if audio is not available), unless
--require-audiois set.To disable audio:
If audio is enabled, it is also recorded when recording is enabled:
To play audio in real-time, the audio player implements drift compensation (it attempts to keep audio sample buffering to a certain level, quite low to get low-latency, but not too low to avoid underflow). It is experimental
(I just tinkered with this thing today)(I've made a lot of improvements since the initial MR, and it should work pretty fine), parameters and behavior are hardcoded, some configuration variables might need to be exposed in the future (or not).If you record, the timestamps are computed on the device, so they are not impacted by any jitter/underflow (this is the same as for the video), so the recording is always clean (if you use
--recordof course, not if you capture your audio output on the computer).Like the video, it works over TCP/IP (typically Wifi).
The following options have been renamed:
--codec->--video-codec--bit-rate->--video-bitrate--codec-options->--video-codec-options--encoder->--video-encoderAnd similar options for audio have been added:
--audio-codec--audio-bit-rate--audio-codec-options--audio-encoderThree audio formats are supported:
scrcpy --audio-codec=opusscrcpy --audio-codece=aacscrcpy --audio-codec=raw(more bandwidth, typically just for testing)It is possible to list the encoders available on the device (video and audio):
If you need a specific encoder, use
--audio-encoder:When recording, raw audio is not possible (it automatically switches to OPUS). In the future, I might add audio encoding on the client-side specifically for recording, but I don't plan to make it for this audio feature.
I also implemented
--audio-buffer=similar to existing--display-buffer=and--v4l2-buffer=.This branch is in working state.
Since I have refactored A LOT of code, I may have broken things (please tell me!).
Please test, review (good luck, that's a big PR 😉), and report any problem :)
Fixes #14
old
Here is a release build for the current MR for Windows (
audio.115):scrcpy-win64-PR3757-1.zipSHA-256: 8a46774889f0d6602e7f94c1e664b7184fdc0f0ccfe4d114045d2a8eefb408f6scrcpy-win32-PR3757-1.zipSHA-256: 7b07819377fa87f1a94f5a424b6757590492b32309d59457fb4eb1e543902bd4Here is a release build for the current MR for Windows (
audio.145):scrcpy-win64-PR3757-3.zipSHA-256: 6ef3c45292f5c3b713628e593bbf695be586506cd38a26f038d135cd88d08033scrcpy-win32-PR3757-3.zipSHA-256: 06fc50e289494554aa1d86a783960b5fd42268ac18395f224b0ea355aa377dd5