Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

schema version 7 #155

Closed
5 of 11 tasks
scottlamb opened this issue Aug 29, 2021 · 8 comments
Closed
5 of 11 tasks

schema version 7 #155

scottlamb opened this issue Aug 29, 2021 · 8 comments
Labels
rust Rust backend work required schema change Requires a database schema change (see new-schema branch)

Comments

@scottlamb
Copy link
Owner

scottlamb commented Aug 29, 2021

Some easy stuff I definitely want to include:

  • flexible per-user UI settings (JSON-encoded string), as requested by @clydebarrow in JSON API for configuration #153 (comment)
  • flexible per-camera and per-stream settings (probably also a JSON-encoded string). Example things I want to include here: RTSP library to use (rather than a global flag), RTSP transport (TCP or UDP), frame timeout, audio mode (right now probably just none or use as a pacer for more accurate timestamps), force camera aspect ratio (rather than the hardcoded anamorphic sub stream sample aspect ratio overrides I have now), retry interval, any other special flags for per-camera workarounds, stream-but-don't-record mode for support "monitor mode" / live view of unrecorded streams #120.
  • flexible settings for signals and signal types.
  • support a third per-camera stream. (I think "ext" is the common name?)
  • "drop reason" for recording runs (rather than just putting this in the logfile)
  • label column in user_session, so they can give the name of the device or client agent or whatever Oh, I already added a description and forgot about it.
  • better info for fixing timestamps (recording time adjustment #9): in open boot id and monotonic time; likely a separate table with time steps observed while running. punted the latter.

Stretch goals (as in things I want but am unsure I want to hold version 7 up for their design and implementation), all punted:

  • audio support (audio support #34)
  • "events" table, for analytics that give more detail than "signals". eg tags, thumbnails/key frames.
  • object detection tracks
  • checkpoint for analytics agent
  • perfect per-frame sync between main and sub streams

My typical procedure is to start a branch, add everything in over several commits and tweak the upgrade script, and give it some more testing than usual. I want it to be pretty reliable given that the upgrade procedure can be intimidating for users and there's no downgrade procedure.

@scottlamb scottlamb added schema change Requires a database schema change (see new-schema branch) rust Rust backend work required labels Aug 29, 2021
scottlamb added a commit that referenced this issue Sep 1, 2021
This isn't yet any different than schema version 6.
scottlamb added a commit that referenced this issue Sep 1, 2021
This isn't yet any different than schema version 6.
scottlamb added a commit that referenced this issue Sep 1, 2021
scottlamb added a commit that referenced this issue Sep 1, 2021
scottlamb added a commit that referenced this issue Sep 10, 2021
for #155

TODO: the config interface is totally broken
scottlamb added a commit that referenced this issue Sep 16, 2021
for #155

The config interface code for changing cameras is quite messy but
seems to work for now.
scottlamb added a commit that referenced this issue Sep 16, 2021
for #155

The config interface code for changing cameras is quite messy but
seems to work for now.
scottlamb added a commit that referenced this issue Sep 16, 2021
@clydebarrow
Copy link
Contributor

Scott, I built the server from the new-schema branch and installed it. It would not run until I did nvr upgrade but then I was getting no data from ws streams (same in my UI and yours, recorded streams played ok.) Replaced the docker image with the distribution version but it fails because the database schema is too new, and nvr upgrade won't downgrade :-( I did not take a backup of the db. Do I have to start from scratch?

@clydebarrow
Copy link
Contributor

Hmm, starting from scratch isn't so easy. I moved the db, did nvr init, now in nvr config trying to add a sample directory it claims ENOENT
Screenshot 2021-09-28 at 1 00 14 pm

Even though that dir does exist:

root@nvr:~# ls -ld /ssd/nvr/sample/
drwx------ 2 moonfire-nvr moonfire-nvr 7569408 Sep 28 10:01 /ssd/nvr/sample/

Same result with any other path on that drive.

@scottlamb
Copy link
Owner Author

Shoot. I wasn't ready for anyone to upgrade their main installation to schema v7. I haven't fully tested it or even finalized the schema, and as you've noticed there's no downgrade code. Starting from scratch is probably best.

Hmm, starting from scratch isn't so easy. I moved the db, did nvr init, now in nvr config trying to add a sample directory it claims ENOENT

I could give that error message better context, but it was probably on the mkdir syscall. The manpage says:

   ENOENT A directory component in pathname does not exist or is a dangling symbolic link.

so possibly /ssd/nvr doesn't exist inside the docker container, even though it does on the host. Is there a --mount for that in your wrapper script?

@clydebarrow
Copy link
Contributor

Is there a --mount for that in your wrapper script?

Well, yes and no :-) I had added a mount line, but it was using a symbolic link that I must have created for reasons that I now have absolutely no recollection of. Since I'd lost the database I didn't realise I'd used the symbolic link instead of the actual mount point for the drive.

In my defence, I did re-read the installation instructions, but overlooked the note about the mount point. I assumed of course that since my script was already working it needed no changes :-(

Anyway, this has prompted me to reconfigure the hardware to swap out the SSD for a bigger high-endurance disk I'd since acquired.

I'm sure that nobody else will be stupid enough to make the same mistake - but just in case - perhaps the config ui could list the mount points available. From inside the container it would seem you can list these with mount |grep \^/dev | grep -v /etc - or maybe there is a cleaner way.

@clydebarrow
Copy link
Contributor

Oh, and I wanted to use the new user preferences schema for testing the UI. I guess I need a disposable test server for that.

@scottlamb
Copy link
Owner Author

scottlamb commented Sep 28, 2021

I'm sure that nobody else will be stupid enough to make the same mistake

Ha! No, very easy mistake to make, and I'm sure you won't be the last if we don't address it in some way.

  • but just in case - perhaps the config ui could list the mount points available. From inside the container it would seem you can list these with mount |grep ^/dev | grep -v /etc - or maybe there is a cleaner way.

That's a good idea, but I think I can do even better by #160. Then there'd be no Docker mapping to worry about at all.

[edit: although, come to think of it, displaying the mount points in the UI is probably a good idea even without Docker's extra mapping layer involved...]

@scottlamb
Copy link
Owner Author

Oh, and I wanted to use the new user preferences schema for testing the UI. I guess I need a disposable test server for that.

That's what I do when messing with schema changes. I have a few installations:

  • my main one that's exposed to the Internet running on an Intel NUC that talks to all my cameras with nice hard drives attached. It tends to run fairly close to HEAD.
  • a backup on a Raspberry Pi 4 that also talks to all the cameras and just has one older smaller hard drive. It tends to run the latest released version or even older.
  • a regularly-deleted installation on my workstation that just talks to some test camera or something as needed for whatever I'm debugging and only stores 1G or so of data. This is of course where I run code before it's committed. I do weird upgrades by hand and get it into strange states all the time.

@clydebarrow
Copy link
Contributor

Ok, all back up and running with new drive. This install is literally built on a breadboard.

IMAGE 2021-09-28 21:28:36

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
rust Rust backend work required schema change Requires a database schema change (see new-schema branch)
Projects
None yet
Development

No branches or pull requests

2 participants