How to play audio with javacv?

[lonelyzerg]

I followed the samples on github, which use:

FloatBuffer channelSamplesFloatBuffer = (FloatBuffer) frame.samples[0];

When I run the code there is an exception says that cannot convert DirectShortBufferU to FloatBuffer.
I also tried frame.samples[0].array() and there is another exception. I googled and find that although this DirectShortBufferU extends ShortBuffer but there is no method array().
I tried some other ways too, but the audio quality is unsatisfying (actually very bad). Any ideas on how to fix it?

Thanks!

[saudet]

Which sample are you referring to?

[lonelyzerg]

JavaFxPlayVideoAndAudio.java

Here's part of my code:

// Setting audio format for recorder:
recorder.setVideoBitrate(2000000); 
recorder.setAudioOption("crf", "0");  
recorder.setAudioQuality(0);  
recorder.setAudioBitrate(192000);  
recorder.setSampleRate(44100);  
recorder.setAudioChannels(1);
recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);

// On the sender side, open the target data line:
audioFormat = getAudioFormat();
dataLineInfo = new DataLine.Info(TargetDataLine.class, audioFormat);
targetDataLine = (TargetDataLine) AudioSystem.getLine(dataLineInfo);
targetDataLine.open(audioFormat);
targetDataLine.start();
sampleRate = (int) audioFormat.getSampleRate();
numChannels = audioFormat.getChannels();
audioBufferSize = sampleRate * numChannels;
audioBytes = new byte[audioBufferSize];

public AudioFormat getAudioFormat() {
	float sampleRate = 44100.0F;
	// 8000,11025,16000,22050,44100
	int sampleSizeInBits = 16;
	// 8,16
	int channels = 1;
	// 1,2
	boolean signed = true;
	// true,false
	boolean bigEndian = false;
	// true,false
	return new AudioFormat(sampleRate, sampleSizeInBits, channels, signed, bigEndian);
}

// Recording:
int cnt = targetDataLine.read(audioBytes, 0, targetDataLine.available());
int sampleNum = cnt / 2;
short[] samples = new short[sampleNum];
ByteBuffer.wrap(audioBytes).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(samples);
ShortBuffer sBuff = ShortBuffer.wrap(samples, 0, sampleNum);
recorder.recordSamples(sampleRate, numChannels, sBuff);

// Decode and play:
CanvasFrame frame = new CanvasFrame("rcv");
fr = gb.grab();
if (fr.image != null)
	frame.showImage(gb.grabImage());
else if (fr.samples != null) {
	FloatBuffer temp =(FloatBuffer) fr.samples[0]); // Throws the exception.
}

[saudet]

I see. That sample needed to be updated since issue #18 has been fixed. I've done so in the latest commit above and now it works. Enjoy!

[lonelyzerg]

@saudet Thanks for the commit! The audio is working now, but I am countering another problem.
I am using ByteArrayOutputStream as the output of recorder that record both video and audio and I am using the following code to get the data from the stream:

// bos is the ByteArrayOutputStream, 
// and pub is a publisher that uses function pub(data) to send message to a subscriber.
if (bos.size() > 0) {
	pub.pub(bos.toByteArray());
	bos.reset();
	Thread.sleep(40);
}

Should I user two thread to capture video and audio and user the third thread to send the data, or send the data after recording a frame without caring about audio, or send the data every time the recorder encoded a video or audio?

[saudet]

We can do anything we want. What issue are you facing?

[lonelyzerg]

When I record video only or audio only it works well. But if I use one record to encode both video and audio and transmit it, the performance is not very good.
As the code I posted above, I am getting the data from the ByteArrayOutputStream and I need to decide when to dump all data in the stream. Cause there are 2 thread that encode video and audio, the data i get from the stream might contains broken image/audio frame.
For example, If I get the data when a image frame is encoded, there is about 5 seconds of delay on the receiver side. If I get the data when a image frame or an audio frame is encoded, the image/audio quality is very bad on the receiver side. If I get the data every 40ms (the frame rate is set to 25) using a third thread, the program throws exception after encoding several frames:

[libx264 @ 0x7fc7e87b6980] frame I:1     Avg QP:20.65  size: 23203
[libx264 @ 0x7fc7e87b6980] frame P:60    Avg QP:22.98  size:  1611
[libx264 @ 0x7fc7e87b6980] mb I  I16..4:  7.5% 90.4%  2.1%
[libx264 @ 0x7fc7e87b6980] mb P  I16..4:  1.0%  2.0%  0.0%  P16..4: 25.6%  2.1%  1.7%  0.0%  0.0%    skip:67.6%
[libx264 @ 0x7fc7e87b6980] 8x8 transform intra:75.5% inter:97.8%
[libx264 @ 0x7fc7e87b6980] coded y,uvDC,uvAC intra: 56.5% 90.6% 69.2% inter: 2.8% 23.7% 3.8%
[libx264 @ 0x7fc7e87b6980] i16 v,h,dc,p:  9% 25%  7% 59%
[libx264 @ 0x7fc7e87b6980] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 31% 35%  2%  2%  2%  4%  2%  4%
[libx264 @ 0x7fc7e87b6980] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 36% 27%  7%  5%  6%  2%  5%  2%  9%
[libx264 @ 0x7fc7e87b6980] i8c dc,h,v,p: 67% 19% 12%  2%
[libx264 @ 0x7fc7e87b6980] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fc7e87b6980] ref P L0: 54.4%  4.0% 26.4% 15.2%
[libx264 @ 0x7fc7e87b6980] kb/s:392.92
[aac @ 0x7fc7e87b4a60] Qavg: 120.000
org.bytedeco.javacv.FrameRecorder$Exception: No audio output stream (Is audioChannels > 0 and has start() been called?)

[saudet]

I'm not sure I understand, just encode everything as soon as you get it...?

[lonelyzerg]

I do encode everything. But when I am trying to get the data I am not sure that I get only one image/audio frame cause I am not sure that the recorder is not writing another audio/image data into the stream.

[saudet]

That's not an issue, we're supposed to encode both together, just like this: https://github.com/bytedeco/javacv/blob/master/samples/RecordActivity.java

[lonelyzerg]

I think I got this fixed...