Merge pull request #291 from richard-austin/release-for-ubuntu-24.04

Release for ubuntu 24.04

DEVELOPMENT.md

@@ -2,18 +2,17 @@
 Pre built .deb files for deployment on a Raspberry pi 4 are available in the Releases section.
 The requirements to build the project yourself are detailed below: -
 #### Platform for Development
-* Ubuntu 23.10 (Mantic Minotaur) on PC (Windows WSL2 or direct boot)
+* Ubuntu 24.04 (Noble Numbat) on PC (Windows WSL2 or direct boot)
 
 #### The project is verified to build with the following:-
 * go version go1.20.1
 * Angular CLI: 15.2.0 or greater
 * Node: 18.17.1
-* npm: 9.9.7
 * Package Manager: npm 9.6.7
 * Grails Version: 5.3.2
-* openjdk version "19.0.2" 2023-01-17
+* openjdk version "17.0.10" 2024-01-16
 * Gradle 7.6
-* Python 3.11.4
+* Python 3.11.6
 
 Using other versions may cause build issues in some cases.
 
@@ -23,7 +22,7 @@ git clone git@github.com:richard-austin/security-cam.git
 cd security-cam
 ```
 ### Build for deployment to Raspberry pi
-The Raspberry pi should be running Ubuntu 23.10 (Mantic Minotaur) OS.
+The Raspberry pi should be running Ubuntu 24.04 (Noble Numbat) OS.
 ```
 ./gradlew buildDebFile 
 ```

FFMPEG.md

@@ -1,69 +1,15 @@
 ### ffmpeg
 ffmpeg version 4 is used in the NVR as versions 5 and above will not correctly mux the rtsp to fmp4 when audio is present.
 The problem is due to a lack of timestamps in one or both of the streams, and whereas ffmpeg4
-deals with it satisfactorily, version 5 and above don't want to know, giving the below error messages continuously.
+deals with it satisfactorily, version 5 and above don't want to know, giving the below error messages continuously, as well
+as very poor video results in the application.
 <pre>
 pts has no value
 Packet duration: -1131 / dts: 152727 is out of range
 </pre>
 
-As ffmpeg 4 is not available as standard on Ubuntu 23.10, I have built an ffmpeg v4 executable
-for ARM64 architecture (at xtrn-scripts-and-config/ffmpeg-v4.4.4) which is deployed on installation of the
-main project deb file.
+As ffmpeg 4 is not available as standard on Ubuntu 24.04, I have built an ffmpeg v4 executable
+for ARM64 architecture (in the release area of [THIS](https://github.com/richard-austin/ffmpeg-4.4.4) repository) 
+which is automatically deployed by the .deb installer.
 
-If you are deploying to a Raspberry pi, you can skip to the *Camera Recordings Service* section.
-
-If you are deploying the NVR to a non ARM64 platform, you will need to
-build ffmpeg 4 on that platform as follows:-.
-
-
-```bash
-wget https://ffmpeg.org/releases/ffmpeg-4.4.4.tar.xz
-tar -xvf ffmpeg-4.4.4.tar.xz
-cd ffmpeg-4.4.4/
-./configure
-make
-```
-The executable ffmpeg will be in the ffmpeg-4.4.4 directory. To deploy it in the deb file with the rest of the
-NVR, copy it to the project at xtrn-scripts-and-config/ffmpeg-v4.4.4.
-##### ffmpeg Build problem
-On Ubuntu 23.10, I got the following compilation error when building ffmpeg 4: -
-<pre>
-CC      libavformat/adtsenc.o
-./libavcodec/x86/mathops.h: Assembler messages:
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error:ffmpeg-4.4.4 operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-./libavcodec/x86/mathops.h:125: Error: operand type mismatch for `shr'
-make: *** [ffbuild/common.mak:81: libavformat/adtsenc.o] Error 1
-</pre>
-
-To fix this, copy xtrn-scripts-and-config/ffmpeg-v4.4.4/mathopts.patch to
-your ffmpeg-4.4.4 build directory and (from the ffmpeg-4.4.4 build directory) run
-
-```text
-patch -u -b  ./libavcodec/x86/mathops.h mathopts.patch 
-```
-Run
-```text
-make clean
-make
-```
-
-and it should now build OK.
 

README.md

@@ -37,6 +37,8 @@ The audio and video is remultiplexed to fragmented MP4 (fMP4) for rendering on t
 * NVR includes NTP server for cameras to sync time without the need for them to connect to the internet.
 
 #### Limitations
+* The .deb file produced with ./gradlew buildDebFile is set up to install on a Raspberry pi running Ubuntu 24.04 (Noble Numbat). 
+Availability of dependencies may prevent installation on other Ubuntu versions.
 * Requires network cameras which provide H264 or H265 (HEVC) video, and optionally audio via RTSP (G711/AAC). *No video transcoding
 is done on the raspberry pi to keep CPU utilisation low*
 * The browser used must be able to display the video format used. Most browsers will support H264, but on some
@@ -46,12 +48,27 @@ with software decoding (see <a href="https://thorium.rocks/">Thorium</a> and <a
   ```--enable-features=VaapiVideoDecodeLinuxGL,VaapiVideoDecoder,VaapiVideoEncoder``` 
    in the command line to enable hevc decoding.
 This will also enable hardware decoding generally.
-* https web admin hosting and rstps (secure rtsp streaming from cameras) are not currently supported.
-* This has been tested with SV3C and ZXTech cameras and Reolink Wi-Fi doorbell.
-There might be compatibility issues with some other camera types.
+* Web admin hosting where the camera uses https and rstps (secure rtsp streaming from cameras) are not currently supported.
+* This software has been tested with SV3C and ZXTech cameras and Reolink Wi-Fi doorbell.
+There could possibly be compatibility issues with some other camera types.
 * 2 way audio (Onvif profile T) supported on the Reolink Wi-fi doorbell using firmware version v3.0.0.1996_23053101.
 The firmware from Reolink main downloads site does not fully support this functionality. 
 
+## Installation
+* Set up Ubuntu 24.04 Server (Noble Numbat) on a Raspberry pi 4 or 5
+* ```bash
+    sudo apt update
+    sudo apt upgrade
+    ```
+* Using the deb file in the [release](https://github.com/richard-austin/security-cam/releases/tag/9.0.0) section or one you have build yourself
+(see [DEVELOPMENT.md](DEVELOPMENT.md)) , Copy the deb file to the Raspberry pi.
+* ssh to the Raspberry pi.
+* ```bash
+  # Install with
+  sudo apt install ./security-cam_9.0.0_arm64.deb
+  ```
+* See [SETTING-UP.md](SETTING-UP.md) for how to set up user account, wifi, cameras etc.
+
 ## Documentation 
 You can find further documentation in these files: -
 * [STRUCTURE.md](STRUCTURE.md)

SETTING-UP.md

@@ -1,4 +1,4 @@
-## Initial Setup
+## Initial Setup (after installation)
 ### Setup for Direct Access (Browser to NVR)
 #### Set up user account
 To log into the NVR when accessing it directly,
@@ -76,11 +76,9 @@ page title).**
 
 Set the global Onvif credentials used during single and all camera Onvif discovery. Cameras successfully discovered
 will have their credentials set to those entered here. Cameras whose credentials were different from these, will
-be listed above the table*
+be listed [above the table](#wrong-onvif-creds)
 allowing the correct credentials to be entered against them before
 retrying discovery on them individually.
-
-* See *Cameras Configuration Editor Showing A Camera Whose Onvif Credentials Differ From The Global Onvif Credentials* below.
 ### Onvif
 From https://github.com/fpompermaier/onvif
 
@@ -165,9 +163,8 @@ All these parameters can be set manually, though some are auto-populated by Onvi
 | Video Width              | For motion Service, the width of the video stream in pixels (see https://motion-project.github.io/motion_config.html#width)                                                                                                                                                                                                                                                                                                    | Yes                    |
 | Video Height             | For Motion Service, the height of the video stream in pixels (see https://motion-project.github.io/motion_config.html#height)                                                                                                                                                                                                                                                                                                  | Yes                    |
 
-![config editor](README.images/config-editor2.png "Camera configuration page following Onvif discovery, showing a camera whose credentials are different from the global Onvif credentials")
-*Cameras Configuration Editor Showing A Camera Whose Onvif Credentials Differ From The Global Onvif Credentials*
-
+<a id="wrong-onvif-creds"></a> ![config editor](README.images/config-editor2.png "Camera configuration page following Onvif discovery, showing a camera whose credentials are different from the global Onvif credentials")
+*Camera Configuration Editor Showing A Camera Whose Onvif Credentials Differ From The Global Onvif Credentials*
 ## One or More Cameras Failing Onvif Authentication During Discovery
 ### Cameras in this category will be listed in the orange bordered box shown in the above screenshot.
 Enter the correct Onvif user name and password for the camera listed in the orange bordered box and click the <img src="README.images/add-circle-solid-svgrepo-com.svg" width="20"  style="position: relative; top: 5px"></img>

STRUCTURE.md

@@ -1,6 +1,6 @@
 ## NVR System Structure
 ### Run Time Platform and Installation
-The current build configuration is for Raspberry pi V4 or V5 running headless (server) version of Ubuntu 23.10 (Mantic Minotaur).
+The current build configuration is for Raspberry pi V4 or V5 running headless (server) version of Ubuntu 24.04 (Noble Numbat).
 Installation of the complete system can be done with a .deb file which you can obtain from the latest release in the Releases section or you can build yourself by following the details under the Development section below.
 
 ### Security

camera-recordings-service/src/requirements.txt

@@ -1,3 +1,3 @@
-pyftpdlib==1.5.7
+pyftpdlib==1.5.9
 pytz==2022.1
 resettabletimer==1.0.0

server/src/main/java/onvif/soap/SSLUtilities.java

@@ -21,12 +21,14 @@ public final class SSLUtilities {
    *
    * @deprecated see {@link #_hostnameVerifier}.
    */
+  @Deprecated
   private static HostnameVerifier __hostnameVerifier;
   /**
    * Thrust managers for the Sun's deprecated API.
    *
    * @deprecated see {@link #_trustManagers}.
    */
+  @Deprecated
   private static TrustManager[] __trustManagers;
   /** Hostname verifier. */
   private static HostnameVerifier _hostnameVerifier;
@@ -39,6 +41,7 @@ public final class SSLUtilities {
    *
    * @deprecated see {@link #_trustAllHostnames()}.
    */
+  @Deprecated
   private static void __trustAllHostnames() {
     // Create a trust manager that does not validate certificate chains
     if (__hostnameVerifier == null) {
@@ -55,6 +58,7 @@ private static void __trustAllHostnames() {
    *
    * @deprecated see {@link #_trustAllHttpsCertificates()}.
    */
+  @Deprecated
   private static void __trustAllHttpsCertificates() {
     SSLContext context;
 
@@ -145,6 +149,7 @@ public static void trustAllHttpsCertificates() {
    * @author Francis Labrie
    * @deprecated see {@link FakeHostnameVerifier}.
    */
+  @Deprecated
   public static class _FakeHostnameVerifier implements HostnameVerifier {
 
     /**
@@ -168,6 +173,7 @@ public boolean verify(String hostname, SSLSession session) {
    * @author Francis Labrie
    * @deprecated see {@link FakeX509TrustManager}.
    */
+  @Deprecated
   public static class _FakeX509TrustManager implements X509TrustManager {
 
     /** Empty array of certificate authority certificates. */

xtrn-scripts-and-config/apache-tomcat-9/tomcat9.service

@@ -0,0 +1,21 @@
+[Unit]
+
+Description=Apache Tomcat Web Application Container
+After=network.target
+[Service]
+Type=forking
+User=tomcat
+Group=tomcat
+Environment="JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-arm64"
+Environment="CATALINA_HOME=/var/lib/tomcat9"
+Environment="CATALINA_PID=/var/lib/tomcat9/temp/tomcat.pid"
+Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"
+ReadWritePaths=/var/log/security-cam/
+ReadWritePaths=/etc/security-cam/
+ReadWritePaths=/var/security-cam/
+ReadWritePaths=/var/lib/tomcat9/webapps/
+
+ExecStart=/var/lib/tomcat9/bin/startup.sh
+ExecStop=/var/lib/tomcat9/bin/shutdown.sh
+[Install]
+WantedBy=multi-user.target

xtrn-scripts-and-config/deb-file-creation/create-deb.sh

@@ -47,35 +47,37 @@ mkdir -p security-cam_"${VERSION}"_arm64/var/log/fmp4-ws-media-service
 
 mkdir -p security-cam_"${VERSION}"_arm64/var/log/motion
 mkdir -p security-cam_"${VERSION}"_arm64/var/log/wifimgr
-mkdir -p security-cam_"${VERSION}"_arm64/var/lib/tomcat
+mkdir -p security-cam_"${VERSION}"_arm64/var/log/tomcat9
+mkdir -p security-cam_"${VERSION}"_arm64/var/lib/tomcat9
 
 mkdir -p security-cam_"${VERSION}"_arm64/tmp
 
 mkdir -p security-cam_"${VERSION}"_arm64/lib/systemd/system/
 
 cp -r ../motion/motion.conf ../nginx.conf ../chrony.conf security-cam_"${VERSION}"_arm64/tmp
-cp ../apache-tomcat-9.0.46/conf/server.xml ../apache-tomcat-9.0.46/conf/tomcat-users.xml security-cam_"${VERSION}"_arm64/tmp
+cp ../apache-tomcat-9/conf/server.xml ../apache-tomcat-9/conf/tomcat-users.xml security-cam_"${VERSION}"_arm64/tmp
+tar -xzf ../apache-tomcat-9/apache-tomcat-9.0.89.tar.gz -C security-cam_"${VERSION}"_arm64/var/lib/tomcat9 --strip-components=1
+rmdir security-cam_"${VERSION}"_arm64/var/lib/tomcat9/logs
+ln -s /var/log/tomcat9 security-cam_"${VERSION}"_arm64/var/lib/tomcat9/logs
 cp ../tomcat9 security-cam_"${VERSION}"_arm64/tmp
 cp ../../server/build/libs/server-7.3.war security-cam_"${VERSION}"_arm64/tmp
 cp ../../initialAdmin/dist/cua.war  security-cam_"${VERSION}"_arm64/tmp
-
+cp ../apache-tomcat-9/tomcat9.service security-cam_"${VERSION}"_arm64/lib/systemd/system/
 cat << EOF > security-cam_"${VERSION}"_arm64/DEBIAN/control
 Package: security-cam
 Version: $VERSION
 Architecture: arm64
 Maintainer: Richard Austin <richard.david.austin@gmail.com>
 Description: A security camera system accessed through a secure web based interface.
-Depends: openjdk-19-jre-headless (>=19.0.2), openjdk-19-jre-headless (<< 19.9.9),
- motion (>=4.5.1-2), motion(<<5.0.0-0),
- curl (>=8.2.1), curl(<=8.3),
+Depends: openjdk-17-jre-headless (>=17.0.0), openjdk-17-jre-headless (<< 17.9.9),
+ motion (>=4.6), motion(<<5.0.0-0),
+ curl (>=8.5.0), curl(<=8.9),
  nginx (>=1.24.0), nginx(<=1.24.9),
- tomcat9 (>=9.0.43-1), tomcat9 (<= 10.0.0),
- tomcat9-admin (>=9.0.70-1), tomcat9-admin (<= 10.0.0),
  libraspberrypi-bin, chrony,
- network-manager (>= 1.42.4), network-manager (<< 2.0.0),
+ network-manager (>= 1.46.0), network-manager (<< 2.0.0),
  wireless-tools (>=30~pre9-13), wireless-tools (<< 40),
  moreutils,
- python3-pip, python3.11-venv
+ python3-pip, python3.12-venv
 EOF
 
 dpkg-deb --build --root-owner-group security-cam_"${VERSION}"_arm64