Add ffmpeg-sidecar support

[Neptune650]

/claim #194
/closes #194

This PR adds ffmpeg-sidecar with auto install, it is partly inspired by the older PR but I've tested it and made significant important changes. Fork has been changed due to being more updated.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
screenpipe	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 9, 2024 6:29pm

[Neptune650]

@louis030195 Please check this #194 (comment) and review.

[Neptune650]

Hello, I've done the tests you requested

With h265:

With h264:

[Neptune650]

@louis030195
As you can see, when we use H265 for FFmpeg encoding - while resource usage only increases by around 1% (or sometimes even decreases), it brings huge benefits to storage usage, with a decrease of a 11%, this can matter a lot for extremely long duration files.

And another benefit is that we no longer need to use a different encoding for Windows and other platforms - we can just use libx265 for every platform.

[louis030195]

still think we can remove old ffmepg then

[Neptune650]

still think we can remove old ffmepg then

@louis030195 I think it's unnecessary - it just stays as a fallback and doesn't cause any issues at all.

[louis030195]

(env) (base) louisbeaumont@mac:~/Documents/screen-pipe$ cargo build --release --features metal
    Updating git repository `https://github.com/nathanbabcock/ffmpeg-sidecar`
    Updating crates.io index
error: failed to select a version for `tar`.
    ... required by package `ort-sys v2.0.0-rc.6`
    ... which satisfies dependency `ort-sys = "^2.0.0-rc.5"` (locked to 2.0.0-rc.6) of package `ort v2.0.0-rc.5`
    ... which satisfies dependency `ort = "^2.0.0-rc.5"` (locked to 2.0.0-rc.5) of package `screenpipe-audio v0.1.97 (/Users/louisbeaumont/Documents/screen-pipe/screenpipe-audio)`
    ... which satisfies path dependency `screenpipe-audio` (locked to 0.1.97) of package `screenpipe-server v0.1.97 (/Users/louisbeaumont/Documents/screen-pipe/screenpipe-server)`
versions that meet the requirements `^0.4` (locked to 0.4.41) are: 0.4.41

all possible versions conflict with previously selected packages.

  previously selected package `tar v0.4.42`
    ... which satisfies dependency `tar = "^0.4.42"` of package `ffmpeg-sidecar v2.0.0-rc (https://github.com/nathanbabcock/ffmpeg-sidecar?branch=main#3fd9caff)`
    ... which satisfies git dependency `ffmpeg-sidecar` of package `screenpipe-core v0.1.97 (/Users/louisbeaumont/Documents/screen-pipe/screenpipe-core)`
    ... which satisfies path dependency `screenpipe-core` (locked to 0.1.97) of package `screenpipe-server v0.1.97 (/Users/louisbeaumont/Documents/screen-pipe/screenpipe-server)`

is that only on mac?

can you add

tar = "=0.4.42" to macos dependencies in audio cargo? (fixed the build)

[louis030195]

(env) (base) louisbeaumont@mac:~/Documents/screen-pipe$ ./target/release/screenpipe setup
2024-10-30T00:55:25.278789Z  INFO screenpipe: logging initialized
2024-10-30T00:55:25.455469Z  INFO screenpipe: Audio permission requested. Please grant permission if prompted.
2024-10-30T00:55:55.535208Z  INFO screenpipe: Screen capture permission requested. Please grant permission if prompted.
2024-10-30T00:55:55.541221Z  INFO screenpipe_audio::whisper: device = Metal(MetalDevice(DeviceId(1)))    
2024-10-30T00:55:57.967964Z  INFO screenpipe_audio::vad_engine: Initializing SileroVad...
2024-10-30T00:55:57.968470Z  INFO screenpipe_audio::vad_engine: SileroVad Model downloaded to: "/Users/louisbeaumont/Library/Caches/screenpipe/vad/silero_vad.onnx"
2024-10-30T00:55:58.024703Z ERROR screenpipe: FFmpeg check failed: No such file or directory (os error 2)
2024-10-30T00:55:58.024999Z ERROR screenpipe: Please ensure FFmpeg is installed correctly and is in your PATH
Error: No such file or directory (os error 2)

[louis030195]

https://github.com/Neptune650/screenpipe/blob/704762f52279fd4fbc1e7a2d52835ded18e96f46/screenpipe-server/src/bin/screenpipe-server.rs#L167

did you run the code yourself?

[Neptune650]

https://github.com/Neptune650/screenpipe/blob/704762f52279fd4fbc1e7a2d52835ded18e96f46/screenpipe-server/src/bin/screenpipe-server.rs#L167

did you run the code yourself?

@louis030195 Oh - my bad, I hadn't checked with screenpipe setup.
Pushing a fix.

[Neptune650]

@louis030195 It doesn't cause any issues now.

[Neptune650]

tar = "=0.4.42"

I hadn't seen this error in Linux or Windows, adding it.

[louis030195]

🥹

that was my concern

macbook pro m3 max 15.0.1 cannot render videos

it's fine if user cannot open with quicktime but it should work with tauri app

maybe some hacks to be done on their webview thing

for example with vlc it works

so i assume we need to install some video rendering lib for h265 in tauri webview

[Neptune650]

so i assume we need to install some video rendering lib for h265 in tauri webview
@louis030195 Any suggestion to how we could do that

[louis030195]

@Neptune650

tauri-apps/tauri#11559

[Neptune650]

@Neptune650

tauri-apps/tauri#11559

@louis030195 Thanks for creating the issue

[louis030195]

chatgpt says

```screenpipe-app-tauri/components/video.tsx
// ... existing code ...
const getMimeType = (path: string): string => {
    const ext = path.split(".").pop()?.toLowerCase();
    switch (ext) {
        case "mp4":
            return "video/mp4";
        case "webm":
            return "video/webm";
        case "ogg":
            return "video/ogg";
        case "mp3":
            return "audio/mpeg";
        case "wav":
            return "audio/wav";
        case "h265": // Add support for H.265
            return "video/h265"; // Adjust MIME type if necessary
        default:
            return isAudio ? "audio/mpeg" : "video/mp4";
    }
};
// ... existing code ...

(seems wrong but need to investigate mime type)

will try when i have time

[Neptune650]

chatgpt says

```screenpipe-app-tauri/components/video.tsx
// ... existing code ...
const getMimeType = (path: string): string => {
    const ext = path.split(".").pop()?.toLowerCase();
    switch (ext) {
        case "mp4":
            return "video/mp4";
        case "webm":
            return "video/webm";
        case "ogg":
            return "video/ogg";
        case "mp3":
            return "audio/mpeg";
        case "wav":
            return "audio/wav";
        case "h265": // Add support for H.265
            return "video/h265"; // Adjust MIME type if necessary
        default:
            return isAudio ? "audio/mpeg" : "video/mp4";
    }
};
// ... existing code ...

(seems wrong but need to investigate mime type)

will try when i have time

@louis030195 I'll also carry out my own tests

[louis030195]

tried again this PR to fix macos that does not render:

• using react-player
• using plyr-react
• tweaking the mime type

does not work still

[Neptune650]

tried again this PR to fix macos that does not render:

* using `react-player`

* using `plyr-react`

* tweaking the mime type

does not work still

@louis030195 How do you feel about keeping the ffmpeg sidecar but using x264?

[louis030195]

the point of this was to enable h265

users don't have ffmpeg issues with the current build

[Neptune650]

the point of this was to enable h265

users don't have ffmpeg issues with the current build

@louis030195 Well other benefits is that it auto-installs ffmpeg if not present, and also can use x264 in windows instead of x264_mf

[louis030195]

@Neptune650 okay sure, let's do this?

lib265 for linux and windows
lib264 for macos (until we figure how to fix the rendering in tauri)

[louis030195]

/approve

[algora-pbc]

@louis030195: The claim has been successfully added to reward-all. You can visit your dashboard to complete the payment.

[Neptune650]

@Neptune650 okay sure, let's do this?

lib265 for linux and windows lib264 for macos (until we figure how to fix the rendering in tauri)

@louis030195 Sounds good to me, thanks.