About the Bpod Sound Module

The sound module contains a Respberry Pi, and Rpi Sound Card(eg. Our soundcard PCB). To provide low-latency sound service to the bpod.

It connect to the Bpod hardware (the ardunio) via serial interface, and connect to the Bpod System (MATLAB) via ZMQ Socket.

Fig 1. Architecture of the sound module

At the beginning of each training session, the MATLAB will sent a list of sound wave and settings to configure the sound service. At the beginning of each trial before running the State Meachine. The sound service will be turned on, then the Rpi will start listening on the serial port, the bpod can send 8-bits through info to turn on/off different sounds. After each trial, the sound service will be turned off.

Generally, the latency of the sound is about 7.5ms. With a low-latency configuration, we can get up to 1.3ms latency.

Fig 2. Super Low Latency Measured by an oscilloscope

By Jingjie Li from Erlich Lab. V1.0 software finished on 8th Feb, 2018. V2.0 PCB finished on 1st Aug.

How to use the Rpi Sound Module

For bpod protocol

For writing a training protocol, playing sound using Rpi Sound Module is easy. Just need several steps to initlize the sound service, add sounds and setup the latency.

Initilize sound service

For example, in method init, you need to initlize the sound service, and add sounds like that:

obj.Sounds = PiSoundServer('server',10.208.17.118,'port',3335)

We also provide a flexible support for psychtoolbox sound, for debugging on a personal computer without Rpi-Soundcard, also for lab training devices without Rpi-soundcard, so people can running the same code under different hardware configration. If you want to use Psychsound instead. You should call

obj.Sounds = PsychSoundServer();

In real session, you need to use obj.Sounds = SoundServerLauncher(), it will automatically choose to use psychsound or pisound based on settings in dbconf, and get ip address from db autometically.

If you are running bpod in emulator, obj.Sounds = SoundServerLauncher() can also lauch the emulator sound via PsyChsoundServer autometically.

In Erlich lab, every devices is using a dbconf file in home path to store the server info. We have a section in that file for storaging the bpod soundcard info. Our SoundServerLauncher() can automatically read our hardware configration(Rpi or Psych), ip address, port info.

Example .dbconf (Rig 696)

[sound]
server = Rpi
ip = 10.208.17.160
port = 3335

Noticed that the port number must be 3335, unless you changed it in the python file of the sound service in the Rpi.

set SF and latency for the sound service

After that, you should setup the sampling rate and latency. The sampling rate of the audio can be picked up from 48000, 96000 or 192000 Hz.

Setting the sampling rate: obj.Sounds.getSF(48000).

Setting the latency: obj.Sounds.setLatency('low') or obj.Sounds.setLatency('high'). In low latency mode, you can get a super short latency(up to 1.3ms), meanwhile, it cannot loop the specific audio. After starting a sound, it will stop after given repeation, you cannot control the exact stop time of the sound. We recommend you to use obj.Sounds.setLatency('high').

Make the sound list

Then you can start adding sounds using obj.Sounds.load(name,wave). You can customize several parameters of the sound using loop,volume,repeation and balance.

Such as using: obj.Sounds.load('HitSound',GenerateSineWave(SF,8,.5).*GenerateSineWave(SF,2000,.5),'loop',0,'volume',0.3,'repeation',5,'balance',1); To generate a hitsound, without autometic loop, volume 0,3, only using left sound channel, and repreat the sound for 5 times in each play.

After that, the sounds will be loaded into a local struct in the MATLAB, the next thing to do is to call obj.Sound.sync() to upload & sync sounds with the Rpi.

Sound configration code example

A complete sound setting-up code example is shown below:

SF = 48000;
% get connection to the sound service
obj.Sounds = SoundServerLauncher() 
% set sampling rate
obj.Sounds.setSF(SF);
% set latency
obj.Sounds.setLatency('high');
% add sounds
obj.Sounds.load('HitSound',GenerateSineWave(SF,8,.5).*GenerateSineWave(SF,2000,.5),'loop',0,'volume',0.3,'repeation',5,'balance',1);
obj.Sounds.load('MissSound',GenerateSineWave(SF,8,.5).*rand(size(GenerateSineWave(SF,8,.5)))-1,'loop',0,'volume',0.3,'balance',-1);
obj.Sounds.load('ViolationSound',rand(1,SF*.5)-1,'loop',1,'volume',0.8);
obj.Sounds.load('GoSound',GenerateSineWave(SF,2000,.1),'loop',0,'volume',0.5);
obj.Sounds.load('ShortViolSound',rand(1,SF*.1)*2-1,'loop',0,'volume',0.6);
obj.Sounds.load('CueSound',GenerateSineWave(SF,4000,.1),'loop',0,'volume',0.5);
% upload to the pi
obj.Sounds.sync()

For Erlich Lab members:

By now, the sound configration in the init methonds are done. When the protocol is running, the sound service will be autometically started when running dispatch.m.

For other people using traditional Bpod system:

before each trial, you should call OK = obj.Sounds.startServ(); to start the sound service.

After each trial, you need to call OK = obj.Sound.closeServ(); to stop sound service. which will allow you to modify sound list between each trial.

After each session, you need to call obj.Sounds.deleteall(); and obj.Sounds.closeConn()', which will detele all sounds and close socket connection. Also, we will turn off the amplifier's 12V power to prevent some overheating problem.

Trigger the sounds

You need to send 8 bits through serial port in Bpod Arduino To trigger the sounds, in 'low' latency mode, the 8 bits represent soundid in binary, in 'high' latency mode, the first bits represent the switch of the sound (1=ON/0=OFF), the other 7 bits represent sound id in binary. Figure shown below.

Fig 3. 8 bits triger info

You can specify sound name with obj.Sounds.trigger() to play specific sound in OutputActions while assembling state meachine, example shown below:

sma = AddState(sma, 'Name', 'SendSerial1', 'Timer', 2, ... 
       'StateChangeConditions', {'Tup', 'next'}, ... 
       'OutputActions', {obj.Sounds.trigger(),...
       obj.Sounds.GetSoundid('HitSound','play')});

If you want to stop specific sound, use obj.Sounds.GetSoundid(soundname,'stop') instead. In 'high' latency mode, you can also send 225 to stop all sounds.

Add/Delete/Modify Sounds

If you want to have some changes to the sound in a session. You can modify the soundlist in the PreparNextTrial method in your protocol.

Add Sounds example

[id,isSuss] = obj.Sounds.AddSound( 'GoSoundLong',GenerateSineWave(SF,2000,.1), 'repeation',5)

Delete Sounds example

isSuss = obj.Sounds.delete(soundid,isSync)

Soundid is the sound that to be deleted, you can also use the name of that sound instead. if isSync == 1, it will autometically delete the sound from the rpi, or it will only delete the sound from the local sound list, you will need to manually call obj.Sounds.sync() to sync the sounds.

Modify Sounds

isSuss = setParameter(obj, sndid, param_name, param_value, isSync)

param_name can be wav, loop, volume, rep, or bal, for example:

isSuss = setParameter(obj, 'GoSound', 'rep', 5, isSync)

Running test

If you are running test on the bpod emulator, you need to use psychsound server.

obj.Sounds = SoundServerLauncher() Using SoundServerLauncher to load sound server object, it will autometically load psychsound or pisound service based on your dbconf setting and autometically load psychsound when running on emulator mode.

Hardware connection

Using our soundcard PCB

We highly recommand our PCB solution. Our PCB is using TI's PCM5122 Hifi DAC Chip, with intergrated high-performance amplifier and sound detector chips. Which will be super easy and stable to use.

Fig 4. Bpod Soundcard PCB

Like other common used rpi based hardwares, you can directly plug the the card on the top of the RPi, it will pick up 5V and I2S sound signals from Rpi's GPIO pins.

The sound amplifier needs 12V power. So you need to have a 12V power source.

Connect the 12V DC power, 5V rpi power, Bpod-Rpi serial-'ethernet like' cable and the sound output wires(it can directly go to speakers, see the red and black wires in fig 3). Then it's done.

For the serial 1 or serial 2 port in the bpod shield board. normally, you need to Serial 2 (that is serial 1 in the bpod r0.5 arduino)

If you want to have a sound timing pluse feed back to the bpod, you can connect two BNC output ports to the bpod two BNC input ports.

Config the computer

Some of code in this function is calling something in Erlich Lab Matlab Utilities. For example, the ini file reading stuff, and the inputordefault functions. These functions are not included in this module, if you cannot edit & replace these, you will need to add Erlich Lab Matlab Utilities into you MATLAB search path.

After that, you can add this Module' folder (mainly three .m MATLAB file and a jar file) into your MATLAB searchpath, then it will work.

Auto Config the Rpi

If you are using our soundcard PCB. We have a simple bash script which can configuring the Rpi automatically.

Step 1. Git clone and download

in your Rpi command window, running git clone ...

Step 2. Close serial log-in

sudo raspi-config

close the serial log-in and enable serial port hardware

Step 3. Start to config

cd to the soundserv folder, change the access permissions of the autoconfig file, you may want to run:

sudo chmod +700 autoconfig.sh

then run the autoconfig script by

sudo ./autoconfig.sh

Start the soundserv and try

sudo python3 RespberryPiAudioService.py

Config auto-start service

This step is optional. If you want to make Rpi sound python program running as a inner service (auto start as a system service, and auto re-start after some error), you can turn this on by running a .sh script.

Before running this script, pls edit the .sh script named autostart-config.sh , you need to make sure the path of RespberryPiAudioService.py is right. Checking two lines marked by #.

then, run this command below to start that service

sudo chmod +700 autostart-config.sh
sudo ./autostart-config.sh

The Rpi will reboot automatically after this command.

Usage

Download the folder /SoundServfrom /bpod/Bpod/Bpod System File/Plugins/PiSoundServer from gitlab and put it into /home/erlichrig or /home/pi

For people in other lab, you can simply git clone this repo.

Then, cd into the SoundServ folder and start the main program by calling python3 RespberryPiAudioService.py if there's something wrong, try sudo python3 RespberryPiAudioService.py

And leave it running.

Manually Config

Serial Port Settings

Set Up

sudo raspi-config

close the serial log-in and enable serial port hardware

Enable UART Serial Code in config.txt sudo nano /boot/config.txt

enable UART by add or change: enable_uart=1

In cmdline.txt : sudo nano /boot/cmdline.txt

rewrite the file using: dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

Add user group: sudo usermod -aG dialout erlichrig (Username could be erlichrig or pi)

reboot the RPi: sudo reboot

Ckecking

Open the python: Python3

then, import the serial lib and try to start the port import serial ser=serial.Serial("/dev/ttyAMA0", baudrate=115200, timeout=5.0) If no error exist, the serial port setting is done.

Configuring the Sound Card Manually

Configure device tree overlay file sudo nano /boot/config.txt add this line: dtoverlay=iqaudio-dacplus uncomment the following line: dtparam=i2s=on

reboot the RPi sudo reboot

Install Jack Server

sudo apt-get install qjackctl (not so sure, maybe we don't need it)

Test Sound Card installation

aplay -l

If it can display the sound card as card 1 or device 1, it's done.

Numpy

sudo python3 -m pip install NumPy --user

CFFI

sudo python3 -m pip install cffi --user

PortAudio lib

sudo apt-get install libasound-dev

soundfile lib

sudo python3 -m pip install pysoundfile --user

sounddevice lib

sudo python3 -m pip install sounddevice --user

ZMQ installation

ZMQ Pre - Environment

sudo apt-get install libtool pkg-config build-essential autoconf automake

libsodium

wget https://github.com/jedisct1/libsodium/releases/download/1.0.3/
libsodium-1.0.3.tar.gz
tar -zxvf libsodium-1.0.3.tar.gz
cd libsodium-1.0.3/
./configure
make
sudo make install

Zeromq Package

wget http://download.zeromq.org/zeromq-4.1.3.tar.gz
tar -zxvf zeromq-4.1.3.tar.gz
cd zeromq-4.1.3/
./configure
make
sudo make install
sudo ldconfig

PyZMQ

sudo apt-get install python-dev

sudo python3 -m pip install pyzmq --user

Set .dbconf

If this rig is using rpi sound service, set like this

[sound]
server = Rpi
ip = 10.208.17.160
port = 3335

If not, set like this

[sound]
server = psych
ip = 10.208.17.160
port = 3335

Trouble Shooting

MATLAB Went error

1. Check the .dbconf by nano .dbconf. Check if there is a sound module and check the port settings.

2. That may also because the RPI audio service is not started. VNC or SSH to the RPi, and python3 RespberryPiAudioService.py or sudo python3 RespberryPiAudioService.py to start the audio service.

No error in MATLAB, but no sound

That usually because the serial line connection trouble. Check the enthernet-serial connection between bpod and rpi. Also try to switch between bpod serial 1 and serial 2 port and see.

If there are no serial recieveing display and you're pretty sure that the bpod has sent something, and the audio connection is correct. VNC or SSH to the RPi, check what did the program print.

Normally, it print like that:

Fig 5. Normal Print Out

If still no serial signal recieving, you may need to look into the python file named RpiSoundPlay_FAST.py, replace all /dev/ttyAMA0 with /dev/serial0.

If there is some error.

1. Ctrl+C to end the sound service and end the bpod protocol. Re start the soundservice and restart the protocol.

2. Might be the sound card issus. Using aplay -l to check is the sound card recognized and in the right device number (it should be device 1). If not, redo the sound card setting procedure and reboot the rpi.

3. Bugs in sound service program? Contact Jingjie via erlichlab mattermost or jingjie.li@nyu.edu