-
Notifications
You must be signed in to change notification settings - Fork 12
Many Snaps on LXC
https://github.com/wekan/wekan-snap/wiki/Many-Snaps-on-LXC#lxd-init-cidr
Note: This presumes that your laptop runs newest Ubuntu or Kubuntu, and that server is Ubuntu.
-
Install Caddy2 https://github.com/wekan/wekan/wiki/Caddy-Webserver-Config
-
Optional, recommended: Encrypted VM. Idea: Bare Metal Caddy => Proxy to encrypted VM ports => Each customer separate Snap WeKan port. Snap sandbox files at /common, snap code can not access files outside of it's /common directories. Newest WeKan is Snap Candidate. Snap has automatic updates.
2.1) If your server has additional harddrives, format them:
ls /dev
mkfs.ext4 /dev/nvme0n1
2.2) Look at that above command text output, what is created disk UUID. Add those to /etc/fstab:
UUID=12345-678-90 /data ext4 errors=remount-ro 0 1
2.3) Create directories for mounting disk, and mount all drives:
sudo su
cd /
mkdir /data
mount -a
2.4) Install to your laptop and bare metal server packages for using remote desktop with KVM:
sudo apt install virt-manager qemu-system
2.5) Check that you can ssh to server with ssh public key as root. For that, create ssh key on your laptop, press enter many times until it's done:
ssh-keygen
2.6) copy laptop .ssh/id_rsa.pub content to server /root/.ssh/authorized_keys .
2.7) At server disable password login at /etc/ssh/sshd_config , password login enabled false. And sudo systemctl restart ssh
2.8) At your laptop edit .ssh/config , there add your server IP address etc. This server here is named x1.
Host x1
Hostname 123.123.123.123
User root
IdentityFile ~/.ssh/id_rsa
2.9) Check that you can ssh from laptop to server as root without prompts.
2.10) Start virt-manager GUI at your laptop:
- File / Add connection
- Hypervisor: QEMU/KVM
- Connect to remote host via SSH
- Username: root
- Machine name: x1
- Connect automatically: [X]
- Click [Connect]
2.11) AT SERVER: Create harddrive image: qemu-img create -f qcow2 web.qcow2 700G
2.12) Download newest Kubuntu desktop iso with wget.
2.13) Install kubuntu with full disk encryption. Modify amount of RAM (here 32 GB), file locations etc. Do not select to allocate encrypted disk image immediately, let image size grow.
2.14) Create growing disk with max size:
qemu-img create -f qcow2 web.qcow2 700G
2.15) Start install from .iso image. Here RAM -r 32 GB, name web, change disk and iso locations below, vcpu 20 (check your server with nproc, should be less that all):
sudo virt-install -r 32000 -n web --os-type=linux --os-variant=ubuntu16.04 \
--disk /data/VMs/web.qcow2,device=disk,format=qcow2 \
-c /data/VMs/iso/kubuntu-22.10-desktop-amd64.iso \
--vcpus=20 --vnc --noautoconsole
It will appear to your laptop virt-manager GUI.
If you have shutdown KVM VM, you can start it from virt-manager, or this way without .iso image:
sudo virt-install -r 32000 -n web --boot hd --video=vga --os-type=linux --os-variant=ubuntu16.04 \
--disk /data/VMs/web.qcow2,device=disk,format=qcow2 \
--vcpus=20 --vnc --noautoconsole
- Use Parallel Snap installs https://snapcraft.io/docs/parallel-installs
For example:
sudo snap set system experimental.parallel-instances=true
- With newest WeKan Candidate like https://github.com/wekan/wekan-snap/wiki/CentOS8 . Note: Each user has different wekan port and mongodb port.
sudo snap install wekan --channel=latest/candidate
sudo snap install wekan wekan_customer1 --channel=latest/candidate
sudo snap disable wekan
sudo snap set wekan_customer1 caddy-enabled='false'
Check that each WeKan uses candidate:
sudo snap list
If not, change to candidate:
sudo snap refresh wekan_customer1 --channel=latest/candidate
If it complains about old database version (was at stable), stop it, move old files somewhere safe, and start again. WARNING: this deletes
sudo su
mkdir old_customer1_common
snap stop wekan_customer1
mv /var/snap/wekan_customer1/common/* old_customer1_common/
snap start wekan_customer1
- Add some settings, for example Google login and AWS SES email sending:
For each customer, node and mongodb needs to be in different ports, for example:
sudo snap set wekan_customer1 port='5001'
sudo snap set wekan_customer1 mongodb-port='25001'
sudo snap set wekan_customer2 port='5002'
sudo snap set wekan_customer2 mongodb-port='25002'
For customer1:
sudo snap set wekan_customer1 port='5001'
sudo snap set wekan_customer1 mongodb-port='25001'
sudo snap set wekan_customer1 root-url='https://wekan.customer1.com'
sudo snap set wekan_customer1 mail-url='smtp://username:password@email-smtp.eu-west-1.amazonaws.com:587?tls={ciphers:"SSLv3"}&secureConnection=false'
sudo snap set wekan_customer1 mail-from='Wekan Customer1 Support <board@customer1.com>'
sudo snap set wekan_customer1 oauth2-auth-endpoint='https://accounts.google.com/o/oauth2/v2/auth'
sudo snap set wekan_customer1 oauth2-client-id='YOUR-GOOGLE-LOGIN-CLIENT_ID.apps.googleusercontent.com'
sudo snap set wekan_customer1 oauth2-secret='YOUR-GOOGLE-LOGIN-SECRET'
sudo snap set wekan_customer1 oauth2-email-map='email'
sudo snap set wekan_customer1 oauth2-enabled='true'
sudo snap set wekan_customer1 oauth2-fullname-map='name'
sudo snap set wekan_customer1 oauth2-id-map='sub'
sudo snap set wekan_customer1 oauth2-request-permissions='openid https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email'
sudo snap set wekan_customer1 oauth2-token-endpoint='https://oauth2.googleapis.com/token'
sudo snap set wekan_customer1 oauth2-userinfo-endpoint='https://openidconnect.googleapis.com/v1/userinfo'
sudo snap set wekan_customer1 oauth2-username-map='nickname'
You can check with nosqlbooster that each database has correct data.
When restoring data, stop that wekan, and restore to that port, when you have subdirectory dump:
sudo snap stop wekan_customer2.wekan
mongorestore --drop --port 25002
sudo snap start wekan_customer2.wekan
If there is errors, try again without restoring indexes:
mongorestore --drop --noIndexRestore --port 25002
To empty database, use mongodb shell:
mongosh --port 25002
show dbs
use wekan
db.dropDatabase()
To see is settings written correctly, use:
sudo snap get wekan_customer1
sudo snap get wekan_customer2
And that at Caddyfile each subdomain is proxied to correct port like 3001 etc.
And that if there is snap called wekan, it's not in use:
sudo snap disable wekan
or has different port:
sudo snap get wekan
sudo snap set wekan port='6001'
sudo snap set wekan mongodb-port='28001'
- Example backup script, that backups MongoDB databases. Although, maybe files directories needed to be added too.
Note: Here customer1 likes to get backup copy of backups, so this copies customer1 backup to separate directory that is synced only to that customer with Syncthing. Via email was asked customer syncthing ID, and added sync.
Set it as executeable:
chmod +x backup.sh
To run it:
sudo su
cd backup
./backup.sh
Here is backup.sh, using mongodb tools for mongodump/mongorestore etc:
#!/bin/bash
# Backup all MongoDB databases from different ports.
# Note: You may need to check also is there files directory,
# and also backup that, like /var/snap/wekan/common/files etc
function backup {
cd /home/wekan/backup
# >> /home/wekan/backup/backup-log.txt
mkdir -p /home/wekan/backup/new/$1
cd /home/wekan/backup/new/$1
mongodump --port $2
sudo snap get wekan_$1 > snap-settings.txt
cd ..
7z a $(date -u +$1-wekan-backup-%Y-%m-%d_%H.%M_UTC.7z) $1
mkdir /home/wekan/$3
chown wekan:wekan *.7z
mv *.7z /home/wekan/$3/
rm -rf $1
cd /home/wekan/backup
}
function backupchat {
cd /home/wekan/backup >> /home/wekan/backup/backup-log.txt
mkdir -p /home/wekan/backup/new/$1 >> /home/wekan/backup/backup-log.txt
cd /home/wekan/backup/new/$1 >> /home/wekan/backup/backup-log.txt
mongodump >> /home/wekan/backup/backup-log.txt
cd ..
7z a $(date -u +$1-backup-%Y-%m-%d_%H.%M_UTC.7z) $1
mkdir /home/wekan/$3
chown wekan:wekan *.7z
mv *.7z /home/wekan/$3/
rm -rf $1
cd /home/wekan/backup
}
# Syntax:
# backup customername port backupdir
function websync {
# Backup WeKan Kanban Snaps to different Syncthing sync directories
backup "customer1" "25001" "backup-wekan-customer1"
backup "customer1" "25001" "websync"
backup "customer2" "25002" "websync"
backup "customer3" "25003" "websync"
backup "customer4" "25004" "websync"
backup "customer5" "25005" "websync"
# Backup RocketChat Snap
backupchat "chat" "27027" "websync"
cd ~/websync
## backup the backup scripts
# 7z a $(date -u +wekan-backup-scripts-%Y-%m-%d_%H.%M_UTC.7z) ~/backup
}
(websync) >> /home/wekan/backup/backup-log.txt 2>&1
- At bare metal server is installed Caddy2.
Each customer has set in their nameserver to WeKan hosting server IP address:
A 123.123.123.123
After that, when ping wekan.company1.com shows 123.123.123.123 correctly, it's possible to get automatic Let's Encrypt SSL/TLS cert with Caddy2.
At encrypted KVM VM type ip address, it shows what is KVM VM internal IP address.
Caddy2 proxies with Let's Encrypt TLS HTTPS to encrypted VM HTTP IP address and port where WeKan (Node.js) is running.
/etc/caddy/Caddyfile . Examples when in /etc/caddy directory as root: caddy start, caddy stop, caddy validate, caddy --help
(redirect) {
@http {
protocol http
}
redir @http https://{host}{uri}
}
kanban.customer1.com {
tls {
alpn http/1.1
}
reverse_proxy 192.168.123.23:5001
}
kanban.customer2.com {
tls {
alpn http/1.1
}
reverse_proxy 192.168.123.23:5002
}
Other non-kanban examples:
# Static website that uses free CloudFlare TLS certificates
company.com {
tls {
load /data/websites/certificates/company
alpn http/1.1
}
root * /data/websites/company.com
file_server
}
# RocketChat
chat.company.com {
tls {
alpn http/1.1
}
reverse_proxy 192.168.123.23:3000
}
# Browseable files listing static website
files.company.com {
tls {
alpn http/1.1
}
root * /data/websites/files.company.com
file_server browse
}
IPv4 CIDR: 10.1.1.1/24
sudo apt install snapd
sudo reboot
sudo snap install lxd
lxd init
Would you like to use LXD clustering? (yes/no) [default=no]:
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (dir, lvm, btrfs, ceph) [default=btrfs]: dir
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 10.1.1.1/24
Would you like LXD to NAT IPv4 traffic on your bridge? [default=yes]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like the LXD server to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]:
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:
The value you have specified is the “network address”, i.e “.0” is the network address of the subnet “10.1.1.0/24”. Instead you need to specify an IP in that network, such as “.1”, e.g. “10.1.1.1/24”
https://discuss.linuxcontainers.org/t/failed-lxd-init-what-is-cidr/7181/2
Install Wekan Snap to newest Ubuntu bare metal server. Snaps have automatic updates.
For example:
sudo apt-get install snapd
reboot
sudo snap install wekan
For Let's Encrypt SSL, like this to /var/snap/wekan/common/Caddyfile. We will be proxying to inside LXD container:
boards.example.com {
tls {
alpn http/1.1
}
proxy / 10.10.10.231:3001 {
websocket
transparent
}
}
For CloudFlare SSL, like this to /var/snap/wekan/common/Caddyfile:
http://boards.example.com https://boards.example.com {
tls {
load /var/snap/wekan/common/certificates
alpn http/1.1
}
proxy / 10.10.10.231:3001 {
websocket
transparent
}
}
So when I start new lxc container:
lxc launch images:ubuntu/20.04 lxccontainername
Then I go inside container and install snapd:
lxc exec lxccontainername -- /bin/bash
apt -y install snapd
reboot
lxc exec lxccontainername -- /bin/bash
snap install wekan
snap set wekan root-url='https://boards.example.com'
sudo snap set wekan port='3001'
sudo snap set wekan mail-from='Wekan Team Boards <info@example.com>'
sudo snap set wekan mail-url='smtps://username:password@email-smtp.eu-west-1.amazonaws.com:587'
ip address
exit
That ip address command does show current IP address of container, that you can then add to bare metal /var/snap/wekan/common/Caddyfile.
You can also add more lxc containers to different subdomains and proxy to them in main Caddyfile.
lxc launch images:ubuntu/20.04 lxccontainername
lxc exec lxccontainername -- /bin/bash
lxc exec lxccontainername -- apt install p7zip-full
lxc config set lxccontainername limits.memory 4GB
lxc launch images:debian/buster mydebiancontainername
lxc launch images:centos/7/amd64 centos
lxc list -cns
Result:
+---------------+---------+
| NAME | STATE |
+---------------+---------+
| mycontainer | RUNNING |
+---------------+---------+
lxc stop mycontainer
lxc delete mycontainer