Two-way audio support

[felipecrs]

Describe what you are trying to accomplish and why in non-technical terms

A sizable number of cameras today, at a reasonable price, have a built-in speaker. It would be super if we could use Frigate to allow two-way audio with the camera over its interface.

This is the only reason I still have the app for my camera installed on my phone (Tuya).

Describe the solution you'd like

As an end-user perspective, a simple button to enable microphone in the Frigate UI which would reproduce the microphone in the camera and reproduce the camera's microphone on the Frigate UI (e.g. browser).

Describe alternatives you've considered

There are some projects or guides here and there on how to make something similar in Home Assistant, but they are either very device-specific, or not mature yet. Regardless, I think that Frigate aims, at some extent, to be the camera's home (like how it is adding PTZ control support) and having two-way audio capability is one of the features that the cameras provides which is not yet covered by Frigate, so we need to fallback to the original camera's app (which often means that cloud access to the camera needs to be enabled - something that Frigate allow us to prevent as well).

Additional context

I don't know, however, what would be the protocol or protocols that would be used to make the camera reproduce an audio. There is SIP, which some cameras, mainly doorbells, provides support, but I think SIP is a canon bullet for this problem -> Frigate's stream already contains audio FROM the camera, so sending audio TO the camera is the only thing pending.

[felipecrs]

Speaking about protocol, I think the way to go is to support the ONVIF backchannel specification (RTSP).

[felipecrs]

References:

https://developer.ridgerun.com/wiki/index.php?title=GStreamer_RTSP_negoiated_RTP_Transport_Streamer_Back_Channel_Communication

(will add more if I find)

[felipecrs]

In my Dahua VTO, there is a HTTP endpoint for posting audio (I saw in this manual here).

It works nicely:

video_2021-12-31_17-40-16.mp4

$ ffmpeg -i piano_original.wav -c:a pcm_alaw -ar 16000 -ac 1 -b:a 64k piano.wav

And looking at their manual, it seems to audio be possible to stream multiparts audio. I think it's should be possible to do something like:

1. Get microphone audio from browser
2. Supply this stream to Frigate backend, which would perform conversions using FFMPEG as needed by the camera
3. The Frigate backend to supply the output of the FFMPEG back to the camera HTTP API to reproduce the audio

[felipecrs]

Regardless, my last response is probably camera-specific (only Dahua, I guess).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pergolafabio]

i'm also interested in this

hi @felipecrs btw :-)

In my case for my hikvision camera, enabling two way audio is based on sending 2 ISAPI commands

[felipecrs]

@blakeblackshear this is one of those issues that deserves to keep open even though there might not be a foreseeable solution. Don't you think?

[zombielinux]

I'd be interested in this too. Especially considering the use case of doorbell cameras.

[pergolafabio]

Same here , for doorbell purposes...
I have hikvision, using two way audio on Synology surveillance system...

I have the API documentation, and also wireshark logs how SS is enabling the two way audio, it's a simple process for hikvision, it's just sending 2 http commands to open the audio stream

[leloch]

There is no universal standardised way to push audio to the camera (especially doorbells), supporting billion of vendors and implementations would be a nightmare - which is something I am sure author would like to avoid (there is enough confusion here with even simple receive only setups...). This is also THE reason why decent, mature solution does not exist in HA for this.

On the other hand, RTSP is pretty much standard for retrieving video/audio streams across cameras ecosystem - which makes Frigate possible and relatively malleable.

[zombielinux]

While I agree this would be a nightmare to maintain, I also suspect this would be something the community could help support, similar to how zigbee2mqtt is supported by the community adding devices.

In my mind there would be two parts.

1. Frontend system capturing audio to be sent (in a PTT buffer). This part could be standardized.
2. Plugin system to take that buffer, tweak with ffmpeg, and send it out via the hardware unique interface.

2 is the hard part, but, with the right underlying architecture, could be made significantly less painful. Further, separating the two would allow them to be maintained independently with 2 being shouldered by the community in a "this may work, this may not, ymmv" kind of way.

What works in our favor is the relatively small number of camera manufacturers, and the fact they are "lazy" and reuse code across their entire line. the audio out code for one Foscam, for example, may work on dozens of models.

[felipecrs]

ONVIF's RTSP backchannel for two way audio communication is yes a standard, albeit not all two-way audio capable cameras support it yet (because it's relatively new).

I do agree supporting camera specific protocols is not viable and sustainable.

I believe ONVIF RTSP backchannel is the way to go.

[distinctjuggle]

ONVIF Profile T requires that any camera which chooses to include two-way audio (backchannel) on the device and also wishes to be Profile T compliant must make the implementation of said two-way audio (backchannel) compliant with ONVIF's standard for such. There's a discussion confirming such here.

There are not an overwhelming amount of devices supporting the ONVIF backchannel standard yet, but there are definitely some. It would be great to see some progress/work made on this.

[NickM-27]

This will probably be supported via https://github.com/AlexxIT/go2rtc

[distinctjuggle]

Are there already plans for it's support, or should someone add an issue requesting it? I don't see it listed with a quick scan.

[NickM-27]

Are there already plans for it's support, or should someone add an issue requesting it? I don't see it listed with a quick scan.

It's already supported in that project

[distinctjuggle]

Ah, I see it now. I glanced right over that!

Thanks for informing me of that, I'm happy to see it!

[CV8R]

Hi,

Not sure if this is the best place or if I should create a new issue. Thought I would share my experience trying to get 2way audio working.

I have not been successful in getting 2 Way Audio to work via Frigate (docker) in the frigate-hass-card or via the Frigate web browser for Hikvision DS-KD8003-IME1(B) or DS-KV6113-WPE1(C).

Importantly, if I use the HA go2rtc addon 2 way audio works in the addon webpage.

I add the same config into the Frigate go2rtc config and the HA go2rtc addon.

My config is like this for the go2rtc addon and frigate per the go2rtc instructions for enabling Hikvision 2way audio.

streams:
  hikvision1:
    - rtsp://admin:password@192.168.1.123:554/Streaming/Channels/101
    - isapi://admin:password@192.168.1.123:80/

In the go2rtc addon I can use 2 way by going to the webpage selecting video+audio+microphone = two way audio from camera and clicking the webrtc.html local WebRTC viewer link.

I have tried using the frigate-hass-card in HA and enabled the microphone using:

cameras:
  - camera_entity: camera.front_door
    live_provider: go2rtc
    go2rtc:
      modes:
        - webrtc
menu:
  buttons:
    microphone:
      enabled: true

It looks like Frigate never makes the ISAPI connection - comparing the go2rtc stream info from frigate and go2rtc addon you can see the lack of "receivers" in the Frigate version.

Addon

  "producers": [
    {
      "type": "RTSP active producer",
      "url": "rtsp://10.0.6.22:554/Streaming/Channels/101/",
      "remote_addr": "10.0.6.22:554",
      "user_agent": "go2rtc/1.5.0",
      "medias": [
        "video, recvonly, H264",
        "audio, recvonly, PCMU/8000"
      ],
      "receivers": [
        "96 H264, bytes=286301, senders=1",
        "0 PCMU/8000, bytes=71680, senders=1"
      ],
      "recv": 364952
    },
    {
      "url": "isapi://uname:pass@10.0.6.22:80/"
    }
  ],

Frigate

  "producers": [
    {
      "type": "RTSP source",
      "url": "rtsp://10.0.6.22:554/Streaming/Channels/101/",
      "remote_addr": "10.0.6.22:554",
      "user_agent": "go2rtc/1.2.0",
      "medias": [
        "video, sendonly, 96 H264/90000",
        "audio, sendonly, 0 PCMU/8000"
      ],
      "tracks": [
        "96 H264/90000, sinks=3",
        "0 PCMU/8000, sinks=2"
      ],
      "recv": 2628824
    },
    {
      "url": "isapi://uname:pass@10.0.6.22:80/"
    }
  ],

In writing this I just realised that Frigate uses go2rtc from 1.2.0 and the addon in 1.5.0. A fix was introduced in 1.3.2 for ISAPI 2 way audio - I guess that is why its not working...

https://github.com/AlexxIT/go2rtc/pull/332

Hopefully the next Frigate update updates the go2rtc version. I will report back.

Cheers,
CV8R

[NickM-27]

Frigate supports using whatever version of go2rtc that you want https://docs.frigate.video/configuration/advanced#custom-go2rtc-version

[CV8R]

Nice! That worked. I now have two way to/from the frigate-hass-card. Happy days!

[pergolafabio]

thats nice indeed, maybe you can share it on the hikvision community thread :-)

[amkrafie]

@CV8R would you mind detailing how you got it to work ?

[CV8R]

@CV8R would you mind detailing how you got it to work ?

Configured exactly as my post above but copied the 1.5.0 version of go2rtc to frigate per instruction in subsequent post by NickM-27.

[hoanglongutc]

Hi,

Not sure if this is the best place or if I should create a new issue. Thought I would share my experience trying to get 2way audio working.

I have not been successful in getting 2 Way Audio to work via Frigate (docker) in the frigate-hass-card or via the Frigate web browser for Hikvision DS-KD8003-IME1(B) or DS-KV6113-WPE1(C).

Importantly, if I use the HA go2rtc addon 2 way audio works in the addon webpage.

I add the same config into the Frigate go2rtc config and the HA go2rtc addon.

My config is like this for the go2rtc addon and frigate per the go2rtc instructions for enabling Hikvision 2way audio.
...

@CV8R
Hi, I am using hikvision camera, my configurations are similar to yours.
can you tell more clearly step from camera on go2rtc to camera_entity: camera.front_door
I am using Integration Generic Camera on hass,

however when configuring on frigate-hass-card I get the following error:

Can you help me identify the error, thanks

[josurene]

Can someone share an example on how to set it up with dual audio?
Please.

[NickM-27]

implemented in #10408