Implement Kubernetes IaC | helm | kpt - for spring boot and mysql 8 containers - statefulset and deployment #6

obriensystems · 2024-07-13T22:51:35Z

https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/
https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/
https://learn.microsoft.com/en-ca/azure/application-gateway/tutorial-ingress-controller-add-on-existing?toc=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Faks%2Ftoc.json&bc=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fbread%2Ftoc.json
https://learn.microsoft.com/en-ca/azure/application-gateway/overview

See

HA Kubernetes on 6 node 8GB Raspberry PI 5 cluster via kubeadm or Rancher k3s - bare metal simulation blog#54

mysql

michaelobrien@mbp7 kubernetes % kubectl apply -f mysql-deployment.yaml 
service/mysql created
deployment.apps/mysql created
michaelobrien@mbp7 kubernetes % kubectl get pods                      
NAME                     READY   STATUS    RESTARTS   AGE
mysql-78bc545dd6-ctsr5   0/1     Pending   0          16s

michaelobrien@mbp7 kubernetes % kubectl get storageclass
NAME                 PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
hostpath (default)   docker.io/hostpath   Delete          Immediate           false                  10m

michaelobrien@mbp7 kubernetes % kubectl describe pod mysql
 Warning  FailedScheduling  4m    default-scheduler  0/1 nodes are available: persistentvolumeclaim "mysql-pv-claim" not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

comment out the volume/pvc first to triage

working without
  Normal  Scheduled  11s   default-scheduler  Successfully assigned default/mysql-58b7767464-glldb to docker-desktop
  Normal  Pulling    11s   kubelet            Pulling image "mysql:8.0.38"
  Normal  Pulled     10s   kubelet            Successfully pulled image "mysql:8.0.38" in 764ms (764ms including waiting)
  Normal  Created    10s   kubelet            Created container mysql
  Normal  Started    10s   kubelet            Started container mysql

michaelobrien@mbp7 kubernetes % kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
mysql-58b7767464-glldb   1/1     Running   0          21s

michaelobrien@mbp7 kubernetes % kubectl get services      
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP    19m
mysql        ClusterIP   None         <none>        3506/TCP   11m

test connection to mysql

keep port forwarding up
Note: clusterIP is not set


michaelobrien@mbp7 kubernetes % kubectl get pods                                     
NAME                     READY   STATUS    RESTARTS   AGE
mysql-58b7767464-dgpf4   1/1     Running   0          6m12s
michaelobrien@mbp7 kubernetes % kubectl get services                                 
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP    34m
mysql        ClusterIP   None         <none>        3306/TCP   26m

michaelobrien@mbp7 kubernetes % kubectl port-forward mysql-58b7767464-dgpf4 3306:3306
Forwarding from 127.0.0.1:3306 -> 3306
Forwarding from [::1]:3306 -> 3306
Handling connection for 3306
Handling connection for 3306

The text was updated successfully, but these errors were encountered:

obriensystems · 2024-08-02T23:57:45Z

For

ichaelobrien@mbp7 kubernetes % kubectl get nodes
E0802 19:52:50.751864   60075 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:64151/api?timeout=32s": dial tcp 127.0.0.1:64151: connect: connection refused
The connection to the server 127.0.0.1:64151 was refused - did you specify the right host or port?

If previously running "kind" switch to docker desktop

michaelobrien@mbp7 kubernetes % kubectl get nodes    
NAME             STATUS   ROLES           AGE     VERSION
docker-desktop   Ready    control-plane   4m18s   v1.29.2

obriensystems · 2024-08-04T18:45:41Z

on mini5 - intel for pre-prod

ubuntu@mini5:~/obrienlabsdev$ docker run --name mysql-dev0 -v mysql-data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=root -d -p 3506:3306 mysql:8.0.38

ubuntu@mini5:~/obrienlabsdev$ docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS         PORTS                                                  NAMES
e14bd640e546   mysql:8.0.38   "docker-entrypoint.s…"   7 seconds ago   Up 5 seconds   33060/tcp, 0.0.0.0:3506->3306/tcp, :::3506->3306/tcp   mysql-dev0

michaelobrien@mbp7 kubernetes % docker run --name mysql-dev2 -it --network="host" --rm arm64v8/mysql:8.0.38 sh -c "mysql -h 192.168.0.105 -P 3506 -u root -p"
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 13
Server version: 8.0.38 MySQL Community Server - GPL
mysql> select count(1) from biometric.gps_record;
+----------+
| count(1) |
+----------+
| 15456722 |
+----------+
1 row in set (1.53 sec)

obriensystems · 2024-08-04T19:07:29Z

prep jdk 17 on ubuntu 22

  264  sudo apt install maven
  268  sudo apt install -y openjdk-17-jdk

ubuntu@mini5:~/obrienlabsdev$ mvn --version
Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 17.0.12, vendor: Ubuntu, runtime: /usr/lib/jvm/java-17-openjdk-amd64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.15.0-117-generic", arch: "amd64", family: "unix"
ubuntu@mini5:~/obrienlabsdev$ java --version
openjdk 17.0.12 2024-07-16
OpenJDK Runtime Environment (build 17.0.12+7-Ubuntu-1ubuntu222.04)
OpenJDK 64-Bit Server VM (build 17.0.12+7-Ubuntu-1ubuntu222.04, mixed mode, sharing)


ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ vi ./build.sh 
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ ./build.sh 
mkdir: cannot create directory ‘../../builds’: File exists
mkdir: cannot create directory ‘../../builds/10001’: File exists
Sending build context to Docker daemon  88.81MB
Step 1/5 : FROM openjdk:17
 ---> 5e28ba2b4cdb
Step 2/5 : ARG USERVICE_HOME=/opt/app/
 ---> Running in e09db7e647ec
Removing intermediate container e09db7e647ec
 ---> d525206cf146
Step 3/5 : RUN mkdir -p $USERVICE_HOME
 ---> Running in 9324b00aad2f
Removing intermediate container 9324b00aad2f
 ---> a0216433ab03
Step 4/5 : ADD biometric-nbi-*.jar $USERVICE_HOME/ROOT.jar
 ---> 034a49cecdd8
Step 5/5 : ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/opt/app/ROOT.jar"]
 ---> Running in 1ea8e2187c0c
Removing intermediate container 1ea8e2187c0c
 ---> 1bddd9f29e82
[Warning] One or more build-args [build-id] were not consumed
Successfully built 1bddd9f29e82
Successfully tagged obrienlabs/biometric-nbi:latest
The push refers to repository [docker.io/obrienlabs/biometric-nbi]
b5a74d4261aa: Pushed 
45d8487c865a: Pushed 
dc9fa3d8b576: Layer already exists 
27ee19dc88f2: Layer already exists 
c8dd97366670: Layer already exists 
0.0.1-ia64: digest: sha256:a7381c350e2a63411dc0f7f304f090909abd96b495a63c4f46fcdf73526b3101 size: 1373
biometric-nbi
biometric-nbi
WARNING: Published ports are discarded when using host network mode
88368bb68c37a5e2cbc9411af0fe3b5afe3b235fce120367efd55d73a99100cd
curl -X GET "http://192.168.0.105:8888/nbi/api/getGps?ac=0&action=u2&al=0&arx=0&ary=0&arz=0&be=0&grx=0&gry=0&grz=0&gsx=0&gsy=0&gsz=0&hr1=0&hr2=0&hrd1=0&hrd2=0&hu=0&lax=0&lay=0&laz=0&lg=0&li=0&lt=0&mfx=0&mfy=0&mfz=0&p=0&pr=0&px=0&rvx=0&rvy=0&rvz=0&s=0&te=0&ts=0&u=0&up=0" -H "accept: */*"
curl: (7) Failed to connect to 192.168.0.105 port 8888 after 0 ms: Connection refused
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker logs -f biometric-nbi

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::               (v2.5.15)

2024-08-04 19:06:58.210  INFO 1 --- [           main] d.o.b.nbi.BiometricNbiApplication        : Starting BiometricNbiApplication v0.0.1-SNAPSHOT using Java 17.0.2 on mini5 with PID 1 (/opt/app/ROOT.jar started by root in /)
2024-08-04 19:06:58.212  INFO 1 --- [           main] d.o.b.nbi.BiometricNbiApplication        : No active profile set, falling back to 1 default profile: "default"
2024-08-04 19:06:58.793  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2024-08-04 19:06:58.833  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 33 ms. Found 1 JPA repository interfaces.
2024-08-04 19:06:59.501  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2024-08-04 19:06:59.509  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-08-04 19:06:59.509  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.75]
2024-08-04 19:06:59.560  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/nbi]    : Initializing Spring embedded WebApplicationContext
2024-08-04 19:06:59.560  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1295 ms
2024-08-04 19:06:59.805  INFO 1 --- [           main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default]
2024-08-04 19:06:59.861  INFO 1 --- [           main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 5.4.33
2024-08-04 19:06:59.968  INFO 1 --- [           main] o.hibernate.annotations.common.Version   : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
2024-08-04 19:07:00.063  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2024-08-04 19:07:00.329  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2024-08-04 19:07:00.362  INFO 1 --- [           main] org.hibernate.dialect.Dialect            : HHH000400: Using dialect: org.hibernate.dialect.MySQL8Dialect
2024-08-04 19:07:00.851  INFO 1 --- [           main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2024-08-04 19:07:00.858  INFO 1 --- [           main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2024-08-04 19:07:01.289  WARN 1 --- [           main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2024-08-04 19:07:01.344 DEBUG 1 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : 10 mappings in 'requestMappingHandlerMapping'
2024-08-04 19:07:01.396  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
2024-08-04 19:07:01.445  INFO 1 --- [           main] pertySourcedRequestMappingHandlerMapping : Mapped URL path [/v2/api-docs] onto method [springfox.documentation.swagger2.web.Swagger2Controller#getDocumentation(String, HttpServletRequest)]
2024-08-04 19:07:01.523 DEBUG 1 --- [           main] s.w.s.m.m.a.RequestMappingHandlerAdapter : ControllerAdvice beans: 0 @ModelAttribute, 0 @InitBinder, 1 RequestBodyAdvice, 1 ResponseBodyAdvice
2024-08-04 19:07:01.621 DEBUG 1 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Patterns [/webjars/**, /**] in 'resourceHandlerMapping'
2024-08-04 19:07:01.626 DEBUG 1 --- [           main] .m.m.a.ExceptionHandlerExceptionResolver : ControllerAdvice beans: 0 @ExceptionHandler, 1 ResponseBodyAdvice
2024-08-04 19:07:01.731  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path '/nbi'
2024-08-04 19:07:01.732  INFO 1 --- [           main] d.s.w.p.DocumentationPluginsBootstrapper : Context refreshed
2024-08-04 19:07:01.742  INFO 1 --- [           main] d.s.w.p.DocumentationPluginsBootstrapper : Found 1 custom documentation plugin(s)
2024-08-04 19:07:01.777  INFO 1 --- [           main] s.d.s.w.s.ApiListingReferenceScanner     : Scanning for api listing references
2024-08-04 19:07:02.021  INFO 1 --- [           main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: handleUsingGET_1
2024-08-04 19:07:02.058  INFO 1 --- [           main] d.o.b.nbi.BiometricNbiApplication        : Started BiometricNbiApplication in 4.181 seconds (JVM running for 4.547)
^C

orm connects but tomcat ports are gone
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker ps
CONTAINER ID   IMAGE                                 COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
88368bb68c37   obrienlabs/biometric-nbi:0.0.1-ia64   "java -Djava.securit…"   57 seconds ago   Up 56 seconds                                                          biometric-nbi
e14bd640e546   mysql:8.0.38                          "docker-entrypoint.s…"   36 minutes ago   Up 36 minutes   33060/tcp, 0.0.0.0:3506->3306/tcp, :::3506->3306/tcp   mysql-dev0
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ 


ports are directly exposed on the host - use direct ports

ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker ps
CONTAINER ID   IMAGE                                 COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
2f4d680c0089   obrienlabs/biometric-nbi:0.0.1-ia64   "java -Djava.securit…"   8 seconds ago    Up 8 seconds                                                           biometric-nbi
e14bd640e546   mysql:8.0.38                          "docker-entrypoint.s…"   40 minutes ago   Up 40 minutes   33060/tcp, 0.0.0.0:3506->3306/tcp, :::3506->3306/tcp   mysql-dev0
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker stop mysql-dev0
mysql-dev0
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker rm mysql-dev0
mysql-dev0
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker run --name mysql-dev0 -v mysql-data:/var/lib/mysql --network="host" -e MYSQL_ROOT_PASSWORD=root -d -p 3506:3306 mysql:8.0.38
WARNING: Published ports are discarded when using host network mode
44b0c0773ba9c039ce16c816346aa3e189df7f6b73ee6713136b81a4e753ae52
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker ps
CONTAINER ID   IMAGE                                 COMMAND                  CREATED              STATUS              PORTS     NAMES
44b0c0773ba9   mysql:8.0.38                          "docker-entrypoint.s…"   5 seconds ago        Up 4 seconds                  mysql-dev0
2f4d680c0089   obrienlabs/biometric-nbi:0.0.1-ia64   "java -Djava.securit…"   About a minute ago   Up About a minute             biometric-nbi
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker run --name mysql-dev2 -it --network="host" --rm mysql:8.0.38 sh -c "mysql -h 192.168.0.105 -P 3506 -u root -p"
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.0.105:3506' (111)
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker run --name mysql-dev2 -it --network="host" --rm mysql:8.0.38 sh -c "mysql -h 192.168.0.105 -P 3306 -u root -p"
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.38 MySQL Community Server - GPL

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select count(1) from biometric.gps_record;
+----------+
| count(1) |
+----------+
| 15456722 |
+----------+
1 row in set (1.44 sec)

mysql> exit
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ vi ./build.sh 
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ vi ../main/resources/application.properties 

emf not picking up again 
CONTAINER ID   IMAGE                                 COMMAND                  CREATED          STATUS                      PORTS                                       NAMES
df6e98d9f966   obrienlabs/biometric-nbi:0.0.1-ia64   "java -Djava.securit…"   51 seconds ago   Exited (1) 46 seconds ago                                               biometric-nbi
44b0c0773ba9   mysql:8.0.38                          "docker-entrypoint.s…"   4 minutes ago    Up 4 minutes                                                            mysql-dev0

ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ vi ./build.sh 
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ vi ../main/resources/application.properties 
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker run --name mysql-dev2 -it --network="host" --rm mysql:8.0.38 sh -c "mysql -h 192.168.0.105 -P 3506 -u root -p"
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.

ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker ps -a
CONTAINER ID   IMAGE                                 COMMAND                  CREATED          STATUS                      PORTS                                                  NAMES
ccedd914fb08   obrienlabs/biometric-nbi:0.0.1-ia64   "java -Djava.securit…"   11 seconds ago   Exited (1) 5 seconds ago

try network fix

ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker network list
NETWORK ID     NAME      DRIVER    SCOPE
2776a9cd54b6   bridge    bridge    local
0864147df090   host      host      local
fdcb0d9f9c3b   none      null      local
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker network create --driver=bridge mysql
3f194fa29ef5d8808ef0cbc41406e0c6a37bc0866be864b26ba38244de8ed0ef

ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker run --name mysql-dev0 -v mysql-data:/var/lib/mysql --network="mysql" -e MYSQL_ROOT_PASSWORD=root -d -p 3506:3306 mysql:8.0.38
d2f2deae8820822363b5993f57243dd9c0195bf8fbb66549bdf8f69e9a57c0a1
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker ps 
CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS         PORTS                                                  NAMES
d2f2deae8820   mysql:8.0.38   "docker-entrypoint.s…"   4 seconds ago   Up 3 seconds   33060/tcp, 0.0.0.0:3506->3306/tcp, :::3506->3306/tcp   mysql-dev0

docker run -d -p 8888:8080 --network="mysql" --name $CONTAINER_IMAGE obrienlabs/$CONTAINER_IMAGE:$TAG

emf still busted
CONTAINER ID   IMAGE                                 COMMAND                  CREATED          STATUS                      PORTS                                                  NAMES
e9925a6c729f   obrienlabs/biometric-nbi:0.0.1-ia64   "java -Djava.securit…"   43 seconds ago   Exited (1) 38 seconds ago

obriensystems · 2024-08-04T19:39:33Z

use docker compose host name for now

sudo apt install docker-compose

compose.yaml
version: "3"
services:
  db:
    image: mysql:8.0.38
    command: 
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: root
    ports:
      - 3506:3306
    volumes:
      - mysql-data:/var/lib/mysql
  spring:
    image: obrienlabs/biometric-backend:0.0.1-ia64
    restart: always
    ports:
      - 8888:8080

Note: did not use the compose above - straight docker
However, i again forgot to rebuild my jar - ran mvn again

ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker ps -a
CONTAINER ID   IMAGE                                 COMMAND                  CREATED          STATUS                      PORTS                                                  NAMES
6d4d1a9c76ef   obrienlabs/biometric-nbi:0.0.1-ia64   "java -Djava.securit…"   31 seconds ago   Up 30 seconds               0.0.0.0:8888->8080/tcp, :::8888->8080/tcp              biometric-nbi
d2f2deae8820   mysql:8.0.38                          "docker-entrypoint.s…"   20 minutes ago   Up 19 minutes               33060/tcp, 0.0.0.0:3506->3306/tcp, :::3506->3306/tcp   mysql-dev0

just needed to wait again

working
ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ curl -X GET "http://192.168.0.105:8888/nbi/api/getGps?ac=0&action=u2&al=0&arx=0&ary=0&arz=0&be=0&grx=0&gry=0&grz=0&gsx=0&gsy=0&gsz=0&hr1=0&hr2=0&hrd1=0&hrd2=0&hu=0&lax=0&lay=0&laz=0&lg=0&li=0&lt=0&mfx=0&mfy=0&mfz=0&p=0&pr=0&px=0&rvx=0&rvy=0&rvz=0&s=0&te=0&ts=0&u=202408040&up=0" -H "accept: */*"
Record(id=15458115,uid=202408040,ssq=0,rsq=0,hr1=0,hr2=0,lat=0.0,lon=0.0,bea=0,alt=0.0,tst0,tsp=1722801083969,axx=0,acy=0,acz=0,gyx=0,gyy=0,gyz=0,tex=0,tey=0,tez=0,lax=0,lay=0,

ubuntu@mini5:~/obrienlabsdev/biometric-backend/biometric-nbi/src/docker$ docker run --name mysql-dev2 -it --network="mysql" --rm mysql:8.0.38 sh -c "mysql -h 192.168.0.105 -P 3506 -u root -p"
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 25
Server version: 8.0.38 MySQL Community Server - GPL

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select * from biometric.gps_record where userId="202408040"
    -> ;
+----------+--------+--------+--------+--------------+--------------+---------+---------+-------+-------+-------+-------+-------+-------+--------+--------+--------+--------+----------+--------------+-------+---------+---------+---------+--------------+------+----------+------+----------+---------+---------+---------+----------+-------+------+--------+--------+--------+---------+---------------+-----------+---------+
| IDENT_ID | ACCELX | ACCELY | ACCELZ | ACCURACY     | ALTITUDE     | bearing | geohash | GRAVX | GRAVY | GRAVZ | GYROX | GYROY | GYROZ | HEART1 | HEART2 | HRDEV1 | HRDEV2 | humidity | LATITUDE     | light | LINACCX | LINACCY | LINACCZ | LONGITUDE    | PRES | provider | PROX | RECV_SEQ | ROTVECX | ROTVECY | ROTVECZ | SEND_SEQ | speed | temp | teslaX | teslaY | teslaZ | tsStart | tsStop        | userId    | version |
+----------+--------+--------+--------+--------------+--------------+---------+---------+-------+-------+-------+-------+-------+-------+--------+--------+--------+--------+----------+--------------+-------+---------+---------+---------+--------------+------+----------+------+----------+---------+---------+---------+----------+-------+------+--------+--------+--------+---------+---------------+-----------+---------+
| 15458115 | 0      | 0      | 0      | 0.0000000000 | 0.0000000000 |       0 | NULL    | 0     | 0     | 0     | 0     | 0     | 0     |      0 |      0 | 0      | 0      | 0        | 0.0000000000 | 0     | 0       | 0       | 0       | 0.0000000000 |    0 | 0        | 0    |        0 | 0       | 0       | 0       |        0 |     0 |    0 | 0      | 0      | 0      |       0 | 1722801083969 | 202408040 |       0 |
+----------+--------+--------+--------+--------------+--------------+---------+---------+-------+-------+-------+-------+-------+-------+--------+--------+--------+--------+----------+--------------+-------+---------+---------+---------+--------------+------+----------+------+----------+---------+---------+---------+----------+-------+------+--------+--------+--------+---------+---------------+-----------+---------+
1 row in set (39.42 sec)

move to kubernetes now

obriensystems · 2024-08-04T19:58:29Z

fix https://github.com/ObrienlabsDev/blog/wiki/Developer-Guide#mysql-on-docker

obriensystems · 2024-08-04T20:54:18Z

testing on iphone 15

024-08-04 20:52:37.811 DEBUG 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet        : GET "/nbi/api/getGps?&u=20240804&de=iph10&pr=17.500000&lg=-75.940427&lt=45.343839&al=095.706317&ac=3.495010&be=203&s=0.106821&grx=0.000000&gry=0.000000&grz=0.000000&arx=-0.079208&ary=-0.698044&arz=-0.686600&lax=-0.079208&lay=-0.698044&laz=-0.686600&rvx=0.122784&rvy=-0.132301&rvz=0.177866&ts=1722804757797&mfx=37.345097&mfy=-37.561279&mfz=46.560547&up=11", parameters={masked}
2024-08-04 20:52:37.812 DEBUG 1 --- [nio-8080-exec-2] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to dev.obrienlabs.biometric.nbi.controller.ApiController#getGps(String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, HttpServletRequest)
2024-08-04 20:52:37.879 DEBUG 1 --- [nio-8080-exec-2] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json]
2024-08-04 20:52:37.879 DEBUG 1 --- [nio-8080-exec-2] m.m.a.RequestResponseBodyMethodProcessor : Writing ["Record(id=15458178,uid=20240804,ssq=11,rsq=0,hr1=null,hr2=null,lat=45.343839,lon=-75.940427,bea=203, (truncated)..."]
2024-08-04 20:52:37.881 DEBUG 1 --- [nio-8080-exec-2] o.s.web.servlet.DispatcherServlet        : Completed 200 OK


xcode
id:
in:
out:
nw_connection_add_timestamp_locked_on_nw_queue [C1] Hit maximum timestamp count, will start dropping events
didReceiveResponse: Code: 200
data: 322: Record(id=15458178,uid=20240804,ssq=11,rsq=0,hr1=null,hr2=null,lat=45.343839,lon=-75.940427,bea=203,alt=95.706317,tst=1722804757797,tsp=1722804757814,axx=-0.079208,acy=-0.698044,acz=-0.686600,gyx=null,gyy=null,gyz=null,tex=37.345097,tey=-37.561279,tez=46.560547,lax=-0.079208,lay=-0.698044,laz=-0.686600,geo=null,pre=null)

obriensystems · 2024-08-08T12:20:46Z

reference helm

Initialize helm

follow https://helm.sh/docs/intro/quickstart/

test with
https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx

michaelobrien@mbp7 biometric-backend % brew install helm
Running `brew update --auto-update`...
==> Downloading https://ghcr.io/v2/homebrew/portable-ruby/portable-ruby/blobs/sha256:ce5c135bf98da783bf5964e89aab587e3341c8adc379f36ace17ae7881f53e8c
######################################################################################################################################################################### 100.0%
==> Pouring portable-ruby-3.3.4_1.arm64_big_sur.bottle.tar.gz
==> Homebrew collects anonymous analytics.
Read the analytics documentation (and how to opt-out) here:
  https://docs.brew.sh/Analytics
No analytics have been recorded yet (nor will be during this `brew` run).

==> homebrew/core is old and unneeded, untapping to save space...
Untapping homebrew/core...
Untapped 3 commands and 7111 formulae (6,917 files, 646.2MB).
==> Downloading https://formulae.brew.sh/api/formula_tap_migrations.jws.json
############################################################################################
michaelobrien@mbp7 biometric-backend % helm version
version.BuildInfo{Version:"v3.15.3", GitCommit:"3bb50bbbdd9c946ba9989fbe4fb4104766302a64", GitTreeState:"clean", GoVersion:"go1.22.5"}
michaelobrien@mbp7 biometric-backend % cd ..
michaelobrien@mbp7 ObrienlabsDev % mkdir helm
michaelobrien@mbp7 ObrienlabsDev % cd helm
michaelobrien@mbp7 helm % helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories
michaelobrien@mbp7 helm % helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
"ingress-nginx" has been added to your repositories
michaelobrien@mbp7 helm % helm repo list                                                        
NAME         	URL                                       
bitnami      	https://charts.bitnami.com/bitnami        
ingress-nginx	https://kubernetes.github.io/ingress-nginx
michaelobrien@mbp7 helm % helm repo list                                                        
NAME         	URL                                       
bitnami      	https://charts.bitnami.com/bitnami        
ingress-nginx	https://kubernetes.github.io/ingress-nginx
michaelobrien@mbp7 helm % helm repo update                                                      
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ingress-nginx" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
michaelobrien@mbp7 helm % helm search repo ingress-nginx                                        
NAME                       	CHART VERSION	APP VERSION	DESCRIPTION                                       
ingress-nginx/ingress-nginx	4.11.1       	1.11.1     	Ingress controller for Kubernetes using NGINX a...
michaelobrien@mbp7 helm % 

michaelobrien@mbp7 helm % helm install dev-ingress-nginx ingress-nginx/ingress-nginx 
NAME: dev-ingress-nginx
LAST DEPLOYED: Thu Aug  8 08:14:50 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the load balancer IP to be available.
You can watch the status by running 'kubectl get service --namespace default dev-ingress-nginx-controller --output wide --watch'

An example Ingress that makes use of the controller:
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: example
    namespace: foo
  spec:
    ingressClassName: nginx
    rules:
      - host: www.example.com
        http:
          paths:
            - pathType: Prefix
              backend:
                service:
                  name: exampleService
                  port:
                    number: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
      - hosts:
        - www.example.com
        secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

https://kubernetes.io/docs/reference/kubectl/quick-reference/

michaelobrien@mbp7 helm % kubectl get pods --all-namespaces
NAMESPACE     NAME                                            READY   STATUS    RESTARTS        AGE
default       dev-ingress-nginx-controller-74c7d99fb4-qldrc   1/1     Running   0               28s
default       mysql-58b7767464-dgpf4                          1/1     Running   0               5d11h
kube-system   coredns-76f75df574-kqqxm                        1/1     Running   0               5d12h
kube-system   coredns-76f75df574-tpczt                        1/1     Running   0               5d12h
kube-system   etcd-docker-desktop                             1/1     Running   3               5d12h
kube-system   kube-apiserver-docker-desktop                   1/1     Running   3               5d12h
kube-system   kube-controller-manager-docker-desktop          1/1     Running   3               5d12h
kube-system   kube-proxy-8w62f                                1/1     Running   0               5d12h
kube-system   kube-scheduler-docker-desktop                   1/1     Running   4 (3d16h ago)   5d12h
kube-system   storage-provisioner                             1/1     Running   0               5d12h
kube-system   vpnkit-controller                               1/1     Running   0               5d12h

michaelobrien@mbp7 helm % kubectl get deployments --all-namespaces
NAMESPACE     NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
default       dev-ingress-nginx-controller   1/1     1            1           2m22s
default       mysql                          1/1     1            1           5d12h
kube-system   coredns                        2/2     2            2           5d12h

nodeports

michaelobrien@mbp7 helm % kubectl get services
NAME                                     TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
dev-ingress-nginx-controller             LoadBalancer   10.105.36.7    localhost     80:32260/TCP,443:31800/TCP   52s
dev-ingress-nginx-controller-admission   ClusterIP      10.110.25.88   <none>        443/TCP                      52s
kubernetes                               ClusterIP      10.96.0.1      <none>        443/TCP                      5d12h
mysql                                    ClusterIP      None           <none>        3306/TCP                     5d12h

michaelobrien@mbp7 helm % kubectl describe deployment dev-ingress-nginx-controller
Name:                   dev-ingress-nginx-controller
Namespace:              default
CreationTimestamp:      Thu, 08 Aug 2024 08:14:57 -0400
Labels:                 app.kubernetes.io/component=controller
                        app.kubernetes.io/instance=dev-ingress-nginx
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=ingress-nginx
                        app.kubernetes.io/part-of=ingress-nginx
                        app.kubernetes.io/version=1.11.1
                        helm.sh/chart=ingress-nginx-4.11.1
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: dev-ingress-nginx
                        meta.helm.sh/release-namespace: default
Selector:               app.kubernetes.io/component=controller,app.kubernetes.io/instance=dev-ingress-nginx,app.kubernetes.io/name=ingress-nginx
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/component=controller
                    app.kubernetes.io/instance=dev-ingress-nginx
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/part-of=ingress-nginx
                    app.kubernetes.io/version=1.11.1
                    helm.sh/chart=ingress-nginx-4.11.1
  Service Account:  dev-ingress-nginx
  Containers:
   controller:
    Image:           registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a
    Ports:           80/TCP, 443/TCP, 8443/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/dev-ingress-nginx-controller
      --election-id=dev-ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/dev-ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --enable-metrics=false
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
  Volumes:
   webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  dev-ingress-nginx-admission
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   dev-ingress-nginx-controller-74c7d99fb4 (1/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  3m16s  deployment-controller  Scaled up replica set dev-ingress-nginx-controller-74c7d99fb4 to 1




michaelobrien@mbp7 helm % kubectl describe pod dev-ingress-nginx-controller
Name:             dev-ingress-nginx-controller-74c7d99fb4-qldrc
Namespace:        default
Priority:         0
Service Account:  dev-ingress-nginx
Node:             docker-desktop/192.168.65.3
Start Time:       Thu, 08 Aug 2024 08:14:57 -0400
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=dev-ingress-nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.11.1
                  helm.sh/chart=ingress-nginx-4.11.1
                  pod-template-hash=74c7d99fb4
Annotations:      <none>
Status:           Running
IP:               10.1.0.30
IPs:
  IP:           10.1.0.30
Controlled By:  ReplicaSet/dev-ingress-nginx-controller-74c7d99fb4
Containers:
  controller:
    Container ID:    docker://b2a424dd41438d844ff1f86f83f1f005247b58fe04c27565037282e4a81f3b47
    Image:           registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a
    Image ID:        docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a
    Ports:           80/TCP, 443/TCP, 8443/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/dev-ingress-nginx-controller
      --election-id=dev-ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/dev-ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --enable-metrics=false
    State:          Running
      Started:      Thu, 08 Aug 2024 08:15:03 -0400
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       dev-ingress-nginx-controller-74c7d99fb4-qldrc (v1:metadata.name)
      POD_NAMESPACE:  default (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cdx6c (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  dev-ingress-nginx-admission
    Optional:    false
  kube-api-access-cdx6c:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From                      Message
  ----    ------     ----   ----                      -------
  Normal  Scheduled  3m44s  default-scheduler         Successfully assigned default/dev-ingress-nginx-controller-74c7d99fb4-qldrc to docker-desktop
  Normal  Pulling    3m44s  kubelet                   Pulling image "registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a"
  Normal  Pulled     3m40s  kubelet                   Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a" in 4.695s (4.695s including waiting)
  Normal  Created    3m39s  kubelet                   Created container controller
  Normal  Started    3m39s  kubelet                   Started container controller
  Normal  RELOAD     3m38s  nginx-ingress-controller  NGINX reload triggered due to a change in configuration

obriensystems self-assigned this Jul 13, 2024

obriensystems changed the title ~~Implement Kubernetes IaC | helm - for spring boot and mysql 8 containers~~ Implement Kubernetes IaC | helm - for spring boot and mysql 8 containers - statefulset and deployment Jul 21, 2024

obriensystems added a commit that referenced this issue Jul 21, 2024

#6 - kubernetes yaml start

7582841

obriensystems added a commit that referenced this issue Aug 2, 2024

#6 - mysql docker image

18c7165

obriensystems added a commit that referenced this issue Aug 3, 2024

#6 - triage PVC on docker-desktop

eb9f68f

obriensystems added a commit that referenced this issue Aug 3, 2024

#6 - triage PVC on docker-desktop

48d0c7a

obriensystems added a commit that referenced this issue Aug 4, 2024

#6 docker shared network

14fadca

obriensystems added a commit that referenced this issue Aug 4, 2024

#6 - Update application.properties

5e53ea4

obriensystems added a commit that referenced this issue Aug 5, 2024

#6 - API change

17f1d43

obriensystems changed the title ~~Implement Kubernetes IaC | helm - for spring boot and mysql 8 containers - statefulset and deployment~~ Implement Kubernetes IaC | helm | kpt - for spring boot and mysql 8 containers - statefulset and deployment Aug 8, 2024

obriensystems mentioned this issue Aug 8, 2024

Helm and Kpt for Kubernetes IaC ObrienlabsDev/blog#44

Open

obriensystems added a commit that referenced this issue Aug 29, 2024

#6 - kubernetes start

8f32751

obriensystems added a commit that referenced this issue Sep 17, 2024

#6 - adjust build for ia64

cde5119

obriensystems mentioned this issue Oct 2, 2024

HA Kubernetes on 6 node 8GB Raspberry PI 5 cluster via kubeadm or Rancher k3s - bare metal simulation ObrienlabsDev/blog#54

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Kubernetes IaC | helm | kpt - for spring boot and mysql 8 containers - statefulset and deployment #6

Implement Kubernetes IaC | helm | kpt - for spring boot and mysql 8 containers - statefulset and deployment #6

obriensystems commented Jul 13, 2024 •

edited

Loading

obriensystems commented Aug 2, 2024

obriensystems commented Aug 4, 2024

obriensystems commented Aug 4, 2024 •

edited

Loading

obriensystems commented Aug 4, 2024 •

edited

Loading

obriensystems commented Aug 4, 2024

obriensystems commented Aug 4, 2024

obriensystems commented Aug 8, 2024 •

edited

Loading

Implement Kubernetes IaC | helm | kpt - for spring boot and mysql 8 containers - statefulset and deployment #6

Implement Kubernetes IaC | helm | kpt - for spring boot and mysql 8 containers - statefulset and deployment #6

Comments

obriensystems commented Jul 13, 2024 • edited Loading

See

mysql

test connection to mysql

obriensystems commented Aug 2, 2024

obriensystems commented Aug 4, 2024

obriensystems commented Aug 4, 2024 • edited Loading

obriensystems commented Aug 4, 2024 • edited Loading

obriensystems commented Aug 4, 2024

obriensystems commented Aug 4, 2024

obriensystems commented Aug 8, 2024 • edited Loading

Initialize helm

obriensystems commented Jul 13, 2024 •

edited

Loading

obriensystems commented Aug 4, 2024 •

edited

Loading

obriensystems commented Aug 4, 2024 •

edited

Loading

obriensystems commented Aug 8, 2024 •

edited

Loading