Audio recording with FrameRecorder

[taskin224]

Hi,
Reading an mp4 file and an mp3 with FFmpegFrameGrabber and then recording these in a new mp4 file with FrameRecorder functions well.

However reading an mp4 video file and sound from microphone to record seems to have some problems. With the code below, although the sound seems to be "without noise", but it is extreme delayed and not recognizable. Soundbytes read from TargetDataLine are ok.

Not sure, if this is simply an error in Buffer handling or a bug. I appreciate it if you could have a look.

private Frame me_GetPlayAudioFrame(Frame audioFrame, TargetDataLine captureLine, SourceDataLine playLine, Vector storeBytes){
    int defBuffSize = captureLine.getBufferSize();
    //      System.out.println("defBuffSize="+defBuffSize);

// int buffSize = defBuffSize/5;
int buffSize = defBuffSize/10; // = 4410
byte[] soundData = new byte[buffSize];
int numBytesRead = captureLine.read(soundData, 0, buffSize);
playLine.write(soundData, 0, numBytesRead); // <<< OK, sound is played correctly...

    ByteBuffer buff = ByteBuffer.allocate(numBytesRead);
    buff.put(soundData); 
    buff.rewind();
    // buff.flip();

    ShortBuffer fBuff = ShortBuffer.allocate(numBytesRead/2);
    fBuff.put(buff.asShortBuffer());

    fBuff.flip();

    audioFrame.audioChannels = 2; 
    audioFrame.samples = new Buffer[audioFrame.audioChannels];
    audioFrame.samples[0] = fBuff;
    audioFrame.samples[1] = fBuff;
    storeBytes.add(soundData);
    return audioFrame;
}

...........................................

                recorder.record(frameVideo);
                recorder.record(frameAudio);

I can email the mp4 file if required.

Thanks,
Taskin

[taskin224]

Hi,
probably catureLine info is also of interest here...
TargetDataLine captureLine = me_GetStartTargetDataSource(SAMPLE_RATE_44100, SAMPLE_SIZE_16_BITS, AUDIO_CHANNELS_2, SIGNED_SAMPLE_VALUE, BIG_ENDIAN);

[taskin224]

Increasing the sampleRate, i.e. although received microphone sampling is with 44100, making it for Frame.java .as frame.sampleRate = 88200 has solved this issue. i.e. Audio is recorded correctly.
Issue is closed.

[saudet]

I'm not sure why increasing the sample rate to 88200 works (a value which I think is unsupported by FFmpeg, so that should throw an error AFAIK). But AFAIK, FFmpeg expects LITTLE_ENDIAN samples, so BIG_ENDIAN is probably not going to work according to libavutil/samplefmt.h:

* Audio sample formats
 *
 * - The data described by the sample format is always in native-endian order.

[taskin224]

Hello Samuel,

thank you for this response. Unfortunately it did not bring me any
further. In terms of output, it makes currently no difference, i.e.
still the same audio output with noise, if bigEndian or littleEndian is
used.

To be honest, I have had already searched for a way to convert a
floatButter to a shortBuffer. But I do not know how.
In fact I cannot even compare a float value with any other. A floating
value with "0" integer value, and a big floating point value. How to
convert it to a shortBuffer. It is clear floating point values can
also be represented in a byte stream, but I assume some of the bytes are
only for formatting, i.e. comma position, divider etc... So these make
noise.

Ok, here I am blocked. Why does FFmpegGrabber gets the correct
shortBuffer when a mp3 file is read but not with mp4?

Best regards,
Taskin

Am 19.07.2014 16:56, schrieb Samuel Audet:

I'm not sure why increasing the sample rate to 88200 works (a value
which I think is unsupported by FFmpeg, so that should throw an error
AFAIK). But AFAIK, FFmpeg expects LITTLE_ENDIAN samples, so BIG_ENDIAN
is probably not going to work according to |libavutil/samplefmt.h|:

|* Audio sample formats
*

• - The data described by the sample format is always in native-endian order.
  |

—
Reply to this email directly or view it on GitHub
#17 (comment).

[saudet]

The AAC decoder outputs as floats, while the MP3 decoder outputs at shorts, that's all. We can do the conversion with swresample, or we can do it manually. There's some info about how to do that here, for example:
http://www.jsresources.org/faq_audio.html#float_to_byte

But that does not seem to be your problem. I'm not sure I understand what is the problem from the discussion above. Please explain more clearly, thank you.

[taskin224]

Hello Samuel,

many thanks for this support. I definitively owe you.

my problem was simply to play an Mp4 file via FFmpegFrameGrabber. I
could manage it with the following code.
.....
playLine = me_GetStartSourceDataSource(SAMPLE_RATE_11025,
SAMPLE_SIZE_16_BITS, AUDIO_CHANNELS_2, SIGNED_SAMPLE_VALUE,
LITTLE_ENDIAN); <<<< SampleRate is unreasoanble, but this is the only
value that works with the code below...
.......
frame = grabber.grabFrame();
if(frame.image != null) {
da_CanvasFrame.showImage(frame.image);
}
else if(frame.samples != null) {
me_PlayFrameAudio(frame.samples.clone(), playLine);
}
.........
else if(buff instanceof FloatBuffer){
// the sample to process

         int len = buff.capacity();
         FloatBuffer fBuff = (FloatBuffer)buff;
         byte[] byteArr = new byte[len];
         int count = 0;
         for(int i=0; i<len; i++){
             float fSample = fBuff.get(i);
             if(fSample > -1 && fSample < 1){
                 int nSample = Math.round(fSample * 32767.0F);
                 byte high = (byte)((nSample >> 8) & 0xFF);
                 byte low = (byte) (nSample & 0xFF);
                 byteArr[count++] = high;
                 // byteArr[count++] = low; <<< according to the 

given reference, one needs to use the low byte too. By me, it does
cause noise.
}
}
playLine.write(byteArr, 0, byteArr.length);
fBuff.clear();

Any way, it works for me now. And I am very happy to have your
support. Hope to return back this contribution one day!
All the best wishes,
Taskin

Am 24.07.2014 14:42, schrieb Samuel Audet:

The AAC decoder outputs as floats, while the MP3 decoder outputs at
shorts, that's all. We can do the conversion with |swresample|, or we
can do it manually. There's some info about how to do that here, for
example:
http://www.jsresources.org/faq_audio.html#float_to_byte

But that does not seem to be your problem. I'm not sure I understand
what is the problem from the discussion above. Please explain more
clearly, thank you.

—
Reply to this email directly or view it on GitHub
#17 (comment).