Squashed commit of the following:

    patch: Updated CI
    patch: remove comments from dataserver Dockerfile
    patch: updated elasticsearch to 8.10.2
    patch: Updated README.md
    feat: Directrories restructure
    patch: Addresses Issue zotero/dataserver#130
    When a message has continued=false, delay is only 250ms
    Increase global-topic delays

    feat: updated create-user.sh init-mysql.sh as per latest dataserver update changes
    patch: elasticsearch logging verbosity to WARN
    Revert "Revert "Fix init-mysql.sh according to new schemas""
    This reverts commit b9627b3.

    patch: remove comments from ds.Dockerfile
    patch

    338c626ea3f252115837def54415a67df73b6f4e
    Return `Content-Type: application/zip` from S3 for compressed files

    We were returning the attachment file's content type instead of
    indicating that the response itself was a ZIP file.
    https://groups.google.com/g/zotero-dev/c/ZS7o5ymitFw/m/3RN6VxxlAQAJ

    a649154d4abbb4e39e331998eb5cd0c8bdeb71d3
    Fix storage usage calculation for users with >1 group on a shard

    When we switched to `shardLibraries.storageUsage` and removed `SUM()`,
    `WITH ROLLUP` stopped counting all but one group per shard.

    82540c48a99218aa7b5c9565957accece98e04ee
    Use MySQL's `new.` syntax in storageFileItems ON DUPLICATE KEY UPDATE

    patch: CI README.md obsolete
    patch: Added multistaging to dataserver dockerfile
    patch: update zotero client to 6.0.27
    Github Actions Update Submodule client/zotero-client
    patch: Updated stream-server submodule
    Github Actions Update Submodule dataserver
    patch: branches for submodules
    patch: change version
    patch: Updated CI
    patch: apache permissions
    Revert "Fix init-mysql.sh according to new schemas"
    This reverts commit ef3eef7.

    patch: Add user. Mariadb syntax. Email and library ID

commit 6f64386
Author: piernov <piernov@piernov.org>
Date:   Thu Nov 17 15:41:43 2022 +0100
    Fix init-mysql.sh according to new schemas

    patch: syncDeleteLogKeys and tags to utf8mb4
    Update create-user.sh for newer schemas and add custom library_id
    patch: typo in port definition for client.Dockerfile. Improve README.md
    patch: updated GKE, microk8s charts
    patch: Updated README.md
    Added Credits
    patch: updated client build to switch between linux/windows and to define api and stream server url's.
    patch: k8s updated GKE terraform, helm-chart
    patch: Updated README.md
    patch: Added SSL Certificate Manager. Updated Charts
    patch: Updated Helm Charts
    patch: Restructured secrets. Created Helm Charts
    patch: Added init.sh as postsart script into dataserver pod
    patch: removed CPU limits. Increased data storage. Updated terraform
    patch: client build update
    Pulled down update to client submodules
    patch: Connect from dataserver to mini via internal ip. Structure persistent volume names
    patch: dataserver
    patch: k8s updated liveness probes to fix GKE Ingress
    feat: For localhost server/client access minio via 10.5.5.1:9000. Removed rinetd. Update Zend/Service/Amazon/S3.php
    patch: .vscode to ignore
    patch: update k8s manifests
    patch: update k8s manifests
    feat: move dataserver ip/host config to .env. move stream server config into build
    fix: updated config.inc.php
    patch: configmap streamserver
    patch: Split manifests for microk8s and GKE
    patch: Ingress for GKE
    patch: dataserver config split between microk8s and cloud k8s
    patch: client for k8s
    patch: fix minio manifest volume issue
    patch: fix db manivfest volume issue
    patch: ignore kubeconfig
    patch: Updated terraform
    patch: Updated terraform
    patch: updated images from dockerhub. Updated terraform
    patch: refactor folders
    patch: introduced terraform
    patch: k8s refactor yaml names
    patch: k8s Added helm charts
    fix: client k8s fix
    fix: refactor miniomc entrypoint and env
    patch: k8s update folder structure. Added miniomc. update objects
    patch: Added priorityclasses
    fix: restart rinetd in case of exit
    patch: k8s update streamserver
    patch: k8s client setup
    patch: k8s objects added ingress, lb
    patch: k8s objects
    patch: k8s objects for dataserver
    fix: fix env variables for kubernetes. Remove duplicate env variables from entrypoint.sh
    patch: k8s objects for for elasticsearch updated
    fix: Added prod, dev docker-compose.yml versions, added elasticsearch RAM limit
    feat: copy dataserver during image buid and do not mount as a volume
    Reference minio/minio#5469
    fix storage bugs

    (cherry picked from commit 404ded2)

    patch: k8s objects for tinymceclean
    patch: k8s objects for streamserver; for redis updated
    patch: k8s objects for localstack; for phpmyadmin updated; for db updated
    patch: k8s objects for elasticsearch
    patch: k8s objects for phpmyadmin, memcached; for db updated
    patch: k8s objects for redis
    patch: k8s objects for mariadb; for minio pod update
    fix: fix .gitignore
    fix: fix docker compose CI
    feat: Added 1. dev, prod docker compose. 2. docker compose build CI workflow. 3. pre-built images configuration
    patch: k8s objects for minio pod workable
    patch: k8s objects for minio partial

.gitignore

@@ -1,4 +1,5 @@
 .env
+draft
 build
 logs
 **/secret*.json

.gitmodules

@@ -1,22 +1,22 @@
-[submodule "tinymce-clean-server"]
-	path = tinymce-clean-server
-	url = https://github.com/uniuuu/tinymce-clean-server
-	branch = stage
-[submodule "stream-server"]
-	path = stream-server
-	url = https://github.com/uniuuu/stream-server
-	branch = stage
 [submodule "client/zotero-client"]
 	path = client/zotero-client
 	url = https://github.com/uniuuu/zotero
+	branch = 6.0
 [submodule "client/zotero-standalone-build"]
 	path = client/zotero-standalone-build
 	url = https://github.com/uniuuu/zotero-standalone-build
 [submodule "client/zotero-build"]
 	path = client/zotero-build
 	url = https://github.com/uniuuu/zotero-build
 	branch = master
-[submodule "dataserver"]
-	path = dataserver
+
+[submodule "stack/dataserver/dataserver"]
+	path = stack/dataserver/dataserver
 	url = https://github.com/uniuuu/dataserver.git
-	branch = stage
+[submodule "stack/tinymce-clean-server/tinymce-clean-server"]
+	path = stack/tinymce-clean-server/tinymce-clean-server
+	url = https://github.com/uniuuu/tinymce-clean-server.git
+[submodule "stack/stream-server/stream-server"]
+	path = stack/stream-server/stream-server
+	url = https://github.com/uniuuu/stream-server.git
+

README.md

@@ -14,8 +14,6 @@ Table of contents
 Localhost installation is for setup when server and client will run on the same computer.  
 VM (virtual machine) installation is for setup when server and clinet are on different hosts. I.e. server is in VM and client is running on another computer.
 
-## Localhost and VM installation
-
 #### Dependencies and source code
 
 *Install latest docker compose plugin*:
@@ -42,7 +40,7 @@ DSHOST=http://localhost:8080/
 ```
 For VM Installation: 
 ```
-DSHOST=http://\<VM IP Address\>:8080/
+DSHOST=http://<VM IP Address>:8080/
 ```
 *Run*:
 ```bash
@@ -116,7 +114,7 @@ $ kubectl config get-contexts
 $ kubectl get all --all-namespaces
 ```
 **Edit ./helm-chart/values.yaml and change dsuri:, s3Pointuri:, api:, streamserver:, minios3Data:, phpmyadmin:, minios3Web: .**    
-Replace with your hostnames api (**dsuri:**, **api:**), S3 Minio Data (**s3Pointuri:**, **minios3Data:**), Stream Server (**streamserver:**), Phpmyadmin (**phpmyadmin:**) and S3 Minio Web (**minios3Web:**):  
+Replace to your hostnames api (**dsuri:**, **api:**), S3 Minio Data (**s3Pointuri:**, **minios3Data:**), Stream Server (**streamserver:**), Phpmyadmin (**phpmyadmin:**) and S3 Minio Web (**minios3Web:**):  
 - dsuri: http://api-any.yourhostname.io/  
 - s3Pointuri: s3-any.yourhostname.io  
 - api: api-any.yourhostname.io  
@@ -174,7 +172,7 @@ $ kubectl get -A ing
   HOST_DS=http://api-any.yourhostname.io/
   HOST_ST=ws://stream-any.yourhostname.io/
   ```
-- For Argument MLW=[m|w]: w=Windows, l=Linux  
+- For Argument MLW=[w|l]: w=Windows, l=Linux  
 
 Replace arguments in the respective command below and run it:  
 ```bash
@@ -187,8 +185,8 @@ $ DOCKER_BUILDKIT=1 docker build --progress=plain --file client.Dockerfile \
 ```bash
 $ ./build/staging/Zotero_VERSION/zotero(.exe))
 ```
-### Client build from Mac and Windows
-For [m|w]: m=Mac, l=Linux  
+### Client build from Mac
+For [m|l|w]: m=Mac, l=Linux, w=Windows  
 *Run*:
 ```bash
 $ git submodule update --init --recursive
@@ -223,4 +221,4 @@ Credits
 5. https://github.com/Dwarf-Planet-Project/zotero_installation
 6. https://github.com/foxsen/zotero-selfhost
 7. https://github.com/zehuanli/zotero-selfhost
-8. https://github.com/fversaci/zotero-prime
+8. https://github.com/fversaci/zotero-prime

bin/init.sh

@@ -3,7 +3,7 @@
 set -eux
 
 sudo docker compose exec zotprime-dataserver sh -cux 'cd /var/www/zotero/misc && ./init-mysql.sh'
-sudo docker compose exec zotprime-dataserver sh -cux 'cd /var/www/zotero/misc && ./db_update.sh'
+#sudo docker compose exec zotprime-dataserver sh -cux 'cd /var/www/zotero/misc && ./db_update.sh'
 sudo docker compose exec zotprime-dataserver sh -cux 'aws --endpoint-url "http://minio:9000" s3 mb s3://zotero'
 sudo docker compose exec zotprime-dataserver sh -cux 'aws --endpoint-url "http://minio:9000" s3 mb s3://zotero-fulltext'
 sudo docker compose exec zotprime-dataserver sh -cux 'aws --endpoint-url "http://localstack:4575" sns create-topic --name zotero'

client.Dockerfile

@@ -18,16 +18,16 @@ RUN git submodule update --init client/zotero-client
 
 
 WORKDIR /usr/src/app/client/zotero-client
-#RUN git checkout tags/6.0.23 -b v6.0.23
-RUN git checkout tags/6.0.26 -b v6.0.26
+#RUN git checkout tags/6.0.26 -b v6.0.26
+RUN git checkout 6.0
 RUN rm -rf *
 RUN git checkout -- .
 RUN git submodule update --init --recursive
 
 WORKDIR /usr/src/app/client/zotero-standalone-build 
-RUN git checkout tags/6.0.23 -b v6.0.23
-RUN rm -rf *
-RUN git checkout -- .
+#RUN git checkout tags/6.0.23 -b v6.0.23
+#RUN rm -rf *
+#RUN git checkout -- .
 RUN git submodule update --init --recursive
 
 WORKDIR /usr/src/app/client/zotero-build 

docker-compose-dev.yml

@@ -4,7 +4,7 @@ services:
     image: "uniuu/zotprime-db:${VER}"
 #    image: mysql:5.6
     build:
-      context: .
+      context: ./stack/db
       dockerfile: db.Dockerfile
     restart: always
 #    command: --default-authentication-plugin=mysql_native_password
@@ -25,33 +25,54 @@ services:
         ipv4_address: 10.5.5.2
   zotprime-elasticsearch:
     image: "uniuu/zotprime-elasticsearch:${VER}"
-    mem_limit: 512m
-    mem_reservation: 256M
+    mem_limit: 1024m
+    mem_reservation: 1024m
     build:
-      context: .
+      context: ./stack/elasticsearch
       dockerfile: es.Dockerfile
     environment:
       - cluster.name=zotero
       - xpack.security.enabled=false
-      - cluster.routing.allocation.disk.threshold_enabled=false 
-       #     - cluster.routing.allocation.disk.watermark.flood_stage=500mb
+#      - cluster.routing.allocation.disk.threshold_enabled=false 
       - discovery.type=single-node
-#      - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
+      - logger.level=WARN
+#      - bootstrap.memory_lock=true
+      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+#      - cluster.routing.allocation.disk.watermark.flood_stage=500mb
     volumes:
       - /etc/localtime:/etc/localtime:ro
-#   privileged: true
-   #user: root
-   #command: "sysctl -w vm.max_map_count=262144 && su elasticsearch -c bin/elasticsearch"
- #  sysctls:
- #           - vm.max_map_count=262144
-#   command: "sysctl -w vm.max_map_count=262144"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nofile:
+        soft: 65536
+        hard: 65536
+#    sysctls:
+#      - vm.max_map_count=262144
+#    cap-add:
+#      - SYS_ADMIN
+#    privileged: true
+#    user: root
+#    command: 
+#      - /bin/bash
+#      - -c
+#      - |
+#       sysctl -w vm.max_map_count=262144 &&
+#       su elasticsearch -c bin/elasticsearch
+#    command:
+#      - /bin/bash
+#      - -c
+#      - |
+#       sysctl -w vm.max_map_count=262144 && 
+#       su elasticsearch -c bin/es-docker
     networks:
       zotprime:
         ipv4_address: 10.5.5.3
   zotprime-redis:
     image: "uniuu/zotprime-redis:${VER}"
     build:
-      context: .
+      context: ./stack/redis
       dockerfile: r.Dockerfile
  #   privileged: true
  #    user: root
@@ -65,7 +86,7 @@ services:
   zotprime-memcached:
     image: "uniuu/zotprime-memcached:${VER}"
     build:
-      context: .
+      context: ./stack/memcached
       dockerfile: m.Dockerfile
     volumes:
       - /etc/localtime:/etc/localtime:ro
@@ -75,7 +96,7 @@ services:
   zotprime-localstack:
     image: "uniuu/zotprime-localstack:${VER}"
     build:
-      context: .
+      context: ./stack/localstack
       dockerfile: ls.Dockerfile
     environment:
       - SERVICES=sns,sqs,apigateway
@@ -87,14 +108,15 @@ services:
   zotprime-minio:
     image: "uniuu/zotprime-minio:${VER}"
     build:
-      context: .
+      context: ./stack/minio
       dockerfile: minio.Dockerfile
     environment:
       - MINIO_ROOT_USER=${MINIOROOTUSER}
       - MINIO_ROOT_PASSWORD=${MINIOROOTPASSWORD}
     command: server /data --console-address ":9001"
     ports:
-      - "9000:9000"
+#      - "9000:9000"
+#      - "8082:9000"
       - "9001:9001"
     volumes:
       - /etc/localtime:/etc/localtime:ro
@@ -105,7 +127,7 @@ services:
     image: "uniuu/zotprime-miniomc:${VER}"
     restart: always
     build:
-      context: .
+      context: ./stack/minio
       dockerfile: miniomc.Dockerfile
     environment:
       - MINIO_ROOT_USER=${MINIOROOTUSER}
@@ -128,7 +150,7 @@ services:
   zotprime-dataserver:
     image: "uniuu/zotprime-dataserver:${VER}"
     build:
-      context: .
+      context: ./stack/dataserver
       dockerfile: ds.Dockerfile
 #    extra_hosts:
 #      - "zotero.localhost:127.0.0.1"
@@ -169,7 +191,7 @@ services:
   zotprime-tinymceclean:
     image: "uniuu/zotprime-tinymceclean:${VER}"
     build:
-      context: .
+      context: ./stack/tinymce-clean-server
       dockerfile: tmcs.Dockerfile
     depends_on:
       - zotprime-db
@@ -187,7 +209,7 @@ services:
   zotprime-streamserver:
     image: "uniuu/zotprime-streamserver:${VER}"
     build:
-      context: .
+      context: ./stack/stream-server
       dockerfile: sts.Dockerfile
     ports:
       - "8081:81"
@@ -208,7 +230,7 @@ services:
   zotprime-phpmyadmin:
     image: "uniuu/zotprime-phpmyadmin:${VER}"
     build:
-      context: .
+      context: ./stack/phpmyadmin
       dockerfile: pa.Dockerfile
     ports:
       - "8083:80"
@@ -233,4 +255,4 @@ networks:
     ipam:
       config:
         - subnet: 10.5.5.0/28
-          gateway: 10.5.5.1
+          gateway: 10.5.5.1

docker-compose-prod.yml

@@ -16,15 +16,24 @@ services:
         ipv4_address: 10.5.5.2
   zotprime-elasticsearch:
     image: "uniuu/zotprime-elasticsearch:${VER}"
-    mem_limit: 512m
-    mem_reservation: 256M
+    mem_limit: 1024m
+    mem_reservation: 1024m
     environment:
       - cluster.name=zotero
       - xpack.security.enabled=false
       - cluster.routing.allocation.disk.threshold_enabled=false 
       - discovery.type=single-node
+      - logger.level=WARN
+      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
     volumes:
       - /etc/localtime:/etc/localtime:ro
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nofile:
+        soft: 65536
+        hard: 65536
     networks:
       zotprime:
         ipv4_address: 10.5.5.3
@@ -152,4 +161,4 @@ networks:
     ipam:
       config:
         - subnet: 10.5.5.0/28
-          gateway: 10.5.5.1
+          gateway: 10.5.5.1