37 KiB
37 KiB
Table of Contents
- Current architecture
- Objectives
- Requirements
- Dependencies
- Suggestions/notes.
- Pi4 Storage related-specific
- Procedure
- Execution
- Transfer local network dependencies services from Pi4 to SRV.
- Update nameservers
- Transfer Gitea from Pi4 to SRV.
- "Last", change the ingress from the Pi4 Traefik ingress, to the Istio Public Ingress.
- Check for the Certificate Manager to confirm that it's generating the certificates.
- Update DNS records
- Difficulties
Current architecture
Note:
The ingress port mapping uses both port80and port443.
Public ingress port mapping: 192.168.1.2
Standalone
Gateway: 192.168.1.1
Pi4: 192.168.1.2
Srv: 192.168.1.3
Kluster
Kubernetes Cluster
A set of Orange PI 5, so far all of them are the 4GB of RAM version.
Masterk: 192.168.1.10
Slave01: 192.168.1.11
Objectives
- Set the Kubernetes Istio public Ingress as the main ingress.
- Join the Pi4 to the Kubernetes Cluster, therefore the Pi4/cluster cannot run dependencies to the same cluster. Pi4 will be the control plane
- Logs should be exported to the host
srv. - Services in kubernetes should have a health probe and prometheus configured to allow for metrics.
- Set some CI/CD pipelines -> automatically build and deploy some images that I manage/own.
Requirements
- NFS server for PVC on Kubernetes.
- Check for OWASP security
- Public and Locale Ingress.
- Configure a Cert Manager to provide public certs.
https://medium.com/@rd.petrusek/kubernetes-istio-cert-manager-and-lets-encrypt-c3e0822a3aaf
-
Pi4 USB powered storage.
-
Normalize
arm7running distribution -> choose one OS that that works on all devices. -
New
Giteaaccount with limited permissions for CI/CD.
Dependencies
- DHCP service
- DNS service
- Gitea service (and it's database)
Suggestions/notes.
-
Instead of changing the port tunnel IP from the router, could set the Istio Ingress to have the current Pi4 IP, yet this would limit some process.
-
Performing this without "downtime" will be extremely annoying, since if I need/want to use the Pi4 as the control plane device, the cluster itself should need to be recreated, therefore, until the Pi4 is not removed from the equation, the cluster cannot be created/used/enabled. Tips:
- First move the key services to the SRV.
- Remove the current drive from the Pi4, create an Ingress on the current Kubernetes Cluster with the IP set to the Pi4, configure Certificate Manager temporarily thereto allow traffic meanwhile the Pi4 is no longer available.
- Explore storage options for the Pi4, aka try the low voltage/energy consumption m.2 drive bought recently.
-
Another option would be, moving everything to the SRV and use that meanwhile, this would allow for less downtime, and I could get to do whatever I want with the Pi4 and Cluster in the meantime, could deploy the
Traefikwith the Pi4 IP (.2) to allow for shooting down the Pi4 shenanigans.
Pi4 Storage related-specific
- If I keep using a SSD, 3D print a base for it, the Pi4 doesn't need a shell as it already has one, but just a platform that allows the Pi4 to sit on the top.
Procedure
-
Move local DHCP and DNS to the host
srv. -
Move Gitea to the host
srv. -
Move registry to the host
srv. -> Not relevant for my current setup, therefore can and (probably will) be skipped. -
Update Traefik ingress configs
-
Deploy Certificate Manager on the current
Klusterwith let's encrypt certificate provisioning. -
Deploy an Istio Ingress to allow access to the Gitea service.
-
Update the router
Virtual Service Port Mappingto set the Istio Ingress deployed as the new Public Ingress. -
Ensure the Certificates are being provisioned.
-
Configure Istio to route traffic towards Jellyfin and Filebrowser services.
-
Backup the Istio/Kubernetes configurations used.
-
Explore Pi4 Storage options.
-
Consider Storage options for the OrangePi5.
-
Wipe and recreate the current
Kluster, this time using the Pi4 as a master, and the 2 Orange Pi5 as slaves (this will require updating the DNS/DHCP local services). -
Update the
Current Setupdocumentation with the new container and architecture rearrangement. -
Deploy NFS service on the
media SRV. -
Migrate some lightweight/not data heavy services from the
media SRVto theKluster. -
Update the
Current Setupdocumentation with the new container and architecture rearrangement.
Execution
Transfer local network dependencies services from Pi4 to SRV.
Install ZIP on Pi4
dnf install zip
Last metadata expiration check: 1:17:47 ago on Wed 19 Jul 2023 07:26:42 AM UTC.
Dependencies resolved.
==========================================================================================================================================================================================================================================
Package Architecture Version Repository Size
==========================================================================================================================================================================================================================================
Installing:
zip aarch64 3.0-23.el8 baseos 264 k
Installing dependencies:
unzip aarch64 6.0-46.el8 baseos 188 k
Transaction Summary
==========================================================================================================================================================================================================================================
Install 2 Packages
Total download size: 452 k
Installed size: 1.4 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): zip-3.0-23.el8.aarch64.rpm 1.7 MB/s | 264 kB 00:00
(2/2): unzip-6.0-46.el8.aarch64.rpm 1.2 MB/s | 188 kB 00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 790 kB/s | 452 kB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : unzip-6.0-46.el8.aarch64 1/2
Installing : zip-3.0-23.el8.aarch64 2/2
Running scriptlet: zip-3.0-23.el8.aarch64 2/2
Verifying : unzip-6.0-46.el8.aarch64 1/2
Verifying : zip-3.0-23.el8.aarch64 2/2
Installed:
unzip-6.0-46.el8.aarch64 zip-3.0-23.el8.aarch64
Complete!
DHCP
From Pi4
Compress DHCP service
zip -r /tmp/dhcpd.zip /scripts/docker/dhcpd
adding: scripts/docker/dhcpd/dhcpd.leases (stored 0%)
adding: scripts/docker/dhcpd/docker-compose.yaml (deflated 55%)
adding: scripts/docker/dhcpd/dhcpd.conf (deflated 61%)
adding: scripts/docker/dhcpd/Dockerfile (deflated 61%)
List compressed file
ls /tmp/dhcpd.zip -l
-rw-r--r--. 1 root root 1987 Jul 19 09:00 /tmp/dhcpd.zip
Send file to the srv.
scp /scripts/docker/dhcpd.zip user@srv:/tmp
From SRV
Unzip to the desired location, maintaining permissions.
unzip -X /tmp/dhcpd.zip -d /scripts/docker/
Archive: /tmp/dhcpd.zip
creating: /scripts/docker/scripts/docker/dhcpd/
extracting: /scripts/docker/scripts/docker/dhcpd/dhcpd.leases
inflating: /scripts/docker/scripts/docker/dhcpd/docker-compose.yaml
inflating: /scripts/docker/scripts/docker/dhcpd/dhcpd.conf
inflating: /scripts/docker/scripts/docker/dhcpd/Dockerfile
Modify the DHCP configuration to point to this server as a DNS Server.
nano /scripts/docker/dhcpd/dhcpd.conf
Replace
option domain-name-servers 192.168.1.2, 9.9.9.9, 149.112.112.112; # IBM Quad9
For
option domain-name-servers 192.168.1.3, 9.9.9.9, 149.112.112.112; # IBM Quad9
Fix the DHCP interface to listen.
nano /scripts/docker/dhcpd/Dockerfile
Replace
"-group", "dhcp", "eth0"]
For
"-group", "dhcp", "enp3s0"]
DNS
From Pi4
Compress DHCP service
zip -r /tmp/coredns.zip /scripts/docker/coredns
adding: scripts/docker/coredns/ (stored 0%)
adding: scripts/docker/coredns/config/ (stored 0%)
adding: scripts/docker/coredns/config/Corefile (deflated 31%)
adding: scripts/docker/coredns/config/example.db (deflated 37%)
adding: scripts/docker/coredns/config/filter.home.db (deflated 71%)
adding: scripts/docker/coredns/docker-compose.yml (deflated 57%)
List compressed file
ls /tmp/coredns.zip -l
-rw-r--r--. 1 root root 2284 Jul 19 09:34 /tmp/coredns.zip
Send file to the srv.
scp /tmp/coredns.zip user@srv:/tmp
From SRV
Unzip to the desired location, maintaining permissions.
unzip -X /tmp/coredns.zip -d /scripts/docker/
creating: /scripts/docker/scripts/docker/coredns/
creating: /scripts/docker/scripts/docker/coredns/config/
inflating: /scripts/docker/scripts/docker/coredns/config/Corefile
inflating: /scripts/docker/scripts/docker/coredns/config/example.db
inflating: /scripts/docker/scripts/docker/coredns/config/filter.home.db
inflating: /scripts/docker/scripts/docker/coredns/docker-compose.yml
Modify the DHCP configuration to point to this server as a DNS Server.
nano /scripts/docker/scripts/docker/coredns/config/filter.home.db
Replace
dhcp IN CNAME pi4
dns IN CNAME pi4
For
dhcp IN CNAME srv
dns IN CNAME srv
Realize that things uncompressed into /scripts/docker/scripts/docker/, fix that.
Use the mv command to fix it.
Remove the folders with rmdir to avoid surprises.
Update nameservers
From SRV
Start the DNS service
cd /scripts/docker/coredns/ && docker-compose up -d
[+] Running 3/3
⠿ coredns Pulled 3.1s
⠿ 25b7032c281a Pull complete 0.5s
⠿ 3799eae1a077 Pull complete 1.1s
[+] Running 2/2
⠿ Network coredns_default Created 0.2s
⠿ Container coredns Started 0.5s
docker-compose logs -f
coredns | .:53
coredns | filter.home.:53
coredns | CoreDNS-1.10.1
coredns | linux/amd64, go1.20, 055b2c3
Test the DNS service
From any host within the network that can reach / is supposed to use that DNS server:
dig google.es @192.168.1.3
; <<>> DiG 9.18.16 <<>> google.es @192.168.1.3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49583
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;google.es. IN A
;; ANSWER SECTION:
google.es. 99 IN A 142.250.200.67
;; Query time: 13 msec
;; SERVER: 192.168.1.3#53(192.168.1.3) (UDP)
;; WHEN: Wed Jul 19 12:30:42 CEST 2023
;; MSG SIZE rcvd: 63
Start the DHCP Service
Set static IP on the desired interface (enp3s0 in my case)
nano /etc/sysconfig/network-scripts/ifcfg-enp3s0
# Network type
TYPE=Ethernet
# Device Name #
NAME=enp3s0
DEVICE=enp3s0
# Static IP Address #
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
# Server IP #
IPADDR=192.168.1.3
# DNS SERVERS #
DNS1=127.0.0.1
DNS2=8.8.8.8
# GATEWAY ADDRESS #
GATEWAY=192.168.1.1
# SUBNET MASK #
SUBNET MASK=255.255.255.0
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
# Disable ipv6#
IPV6INIT=no
# Activate on system Boot #
ONBOOT=yes
Restart server to ensure everything is working correctly.
reboot
Start container/service
cd /scripts/docker/dhcpd/ && docker-compose up
[+] Running 0/1 ⠿ isc_dhcp Error 1.4s [+] Building 4.2s (8/8) FINISHED => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 1.03kB 0.0s => [internal] load metadata for docker.io/library/alpine:latest 1.5s => [1/4] FROM docker.io/library/alpine:latest@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 0.0s => => resolve docker.io/library/alpine:latest@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 0.0s => => sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 1.64kB / 1.64kB 0.0s => => sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70 528B / 528B 0.0s => => sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67 1.47kB / 1.47kB 0.0s => [2/4] RUN apk update && apk add --no-cache dhcp 1.7s => [3/4] RUN mkdir /var/lib/dhcpd && touch /var/lib/dhcpd/dhcpd.leases && chown dhcp:dhcp /var/lib/dhcpd -R && mkdir -p /run/dhcp 0.4s => [4/4] RUN id dhcp 0.4s => exporting to image 0.1s => => exporting layers 0.1s => => writing image sha256:07b6db561779c547d96d07fe3a995668ba459ac72fe8c5b48de5e8956e138dd9 0.0s => => naming to docker.io/library/rpi_dhcp 0.0s Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them [+] Running 1/1 ⠿ Container dhcpd Created 0.0s
docker-compose logs -f
dhcpd | Internet Systems Consortium DHCP Server 4.4.3-P1 dhcpd | Copyright 2004-2022 Internet Systems Consortium. dhcpd | All rights reserved. dhcpd | For info, please visit https://www.isc.org/software/dhcp/ dhcpd | WARNING: Host declarations are global. They are not limited to the scope you declared them in. dhcpd | Config file: /etc/dhcp/dhcpd.conf dhcpd | Database file: /var/lib/dhcpd/dhcpd.leases dhcpd | PID file: /run/dhcp/dhcpd.pid dhcpd | Wrote 0 deleted host decls to leases file. dhcpd | Wrote 0 new dynamic host decls to leases file. dhcpd | Wrote 0 leases to leases file. dhcpd | Listening on LPF/enp3s0/50:3e:aa:03:7a:37/192.168.1.0/24 dhcpd | Sending on LPF/enp3s0/50:3e:aa:03:7a:37/192.168.1.0/24 dhcpd | Sending on Socket/fallback/fallback-net dhcpd | Server starting service.
Update rest of nameservers
Modify the /etc/resolve.conf or whatever static configurations you have deployed in specific devices.
On my scenario only applies to the pi4.
Transfer Gitea from Pi4 to SRV.
Service
From Pi4
Compress Gitea service
zip -r /tmp/gitea.zip /scripts/docker/gitea
zip -r /tmp/gitea.zip /scripts/docker/gitea
adding: scripts/docker/gitea/ (stored 0%)
adding: scripts/docker/gitea/app.ini (deflated 47%)
adding: scripts/docker/gitea/docker-compose.yml (deflated 60%)
adding: scripts/docker/gitea/renderers/ (stored 0%)
adding: scripts/docker/gitea/renderers/.placeholder (stored 0%)
adding: scripts/docker/gitea/Dockerfile (deflated 49%)
List compressed file
ls /tmp/gitea.zip -l
-rw-r--r--. 1 root root 2231 Jul 19 10:39 /tmp/gitea.zip
Send file to the srv.
scp /tmp/gitea.zip user@srv:/tmp
From SRV
Unzip to the desired location, maintaining permissions.
Learn from the mistakes, use the
/path and let everything sit correctly by itself.
unzip -X /tmp/gitea.zip -d /
creating: /scripts/docker/gitea/
inflating: /scripts/docker/gitea/app.ini
inflating: /scripts/docker/gitea/docker-compose.yml
creating: /scripts/docker/gitea/renderers/
extracting: /scripts/docker/gitea/renderers/.placeholder
inflating: /scripts/docker/gitea/Dockerfile
Volumes data
From Pi4
Stop the service
docker-compose down
[+] Running 3/3 ⠿ Container gitea_cont Removed 1.6s ⠿ Container gitea-db-1 Removed 0.4s ⠿ Network gitea_gitea Removed 0.7s
Locate desired volumes
ls /var/lib/docker/volumes/gitea_d* -l
/var/lib/docker/volumes/gitea_data: total 4 drwxr-xr-x. 5 root root 4096 Dec 8 2021 _data /var/lib/docker/volumes/gitea_db: total 4 drwx------. 19 70 ssh_keys 4096 Jul 19 10:35 _data
On this scenario, we want BOTH.
Compress Gitea data volumes
zip -r /tmp/gitea_data.zip /var/lib/docker/volumes/gitea_db /var/lib/docker/volumes/gitea_data
This returns some text and can take a while so... wait.
Too much text so imagine a bunch of `adding: var/lib/docker/volumes/gitea_ etc`
List compressed file
ls /tmp/gitea_data.zip -l
-rw-r--r--. 1 root root 5486975291 Jul 19 10:59 /tmp/gitea_data.zip
Send file to the srv.
It can take a while so... wait?
scp /tmp/gitea_data.zip user@srv:/tmp
From SRV
Unzip to the desired location, maintaining permissions.
Learn from the mistakes, use the
/path and let everything sit correctly by itself.
Also, again, can take a while, let it do the needful.
unzip -X /tmp/gitea_data.zip -d /
creating: /scripts/docker/gitea/
inflating: /scripts/docker/gitea/app.ini
inflating: /scripts/docker/gitea/docker-compose.yml
creating: /scripts/docker/gitea/renderers/
extracting: /scripts/docker/gitea/renderers/.placeholder
inflating: /scripts/docker/gitea/Dockerfile
Start docker service.
cd /scripts/docker/gitea/ && docker-compose up -d
[+] Running 20/20 ⠿ db Pulled 8.5s ⠿ 4c0d98bf9879 Pull complete 0.6s ⠿ 7ff5918c11c3 Pull complete 0.7s ⠿ c393806625cd Pull complete 0.8s ⠿ 9307f3bcca3a Pull complete 3.8s ⠿ 5eee78b95230 Pull complete 3.9s ⠿ c0f2174cad0e Pull complete 3.9s ⠿ dd6b4e21c993 Pull complete 3.9s ⠿ 1011823211fa Pull complete 3.9s ⠿ gitea_cont Pulled 9.9s ⠿ 31e352740f53 Already exists 0.0s ⠿ 395889657acd Pull complete 2.8s ⠿ fca9ea837ad0 Pull complete 2.8s ⠿ 27ebbe228e73 Pull complete 2.9s ⠿ 34a463970b12 Pull complete 4.2s ⠿ 791a5d24dab3 Pull complete 4.3s ⠿ ad1a83d556b7 Pull complete 4.3s ⠿ 8da8fce0c22f Pull complete 5.2s ⠿ 559bdde1e56a Pull complete 5.2s ⠿ 40fa068ee382 Pull complete 5.3s [+] Running 5/5 ⠿ Network gitea_gitea Created 0.2s ⠿ Volume "gitea_data" Created 0.0s ⠿ Volume "gitea_db" Created 0.0s ⠿ Container gitea-db-1 Started 1.5s ⠿ Container gitea_cont Started 0.6s
docker-compose logs -f
docker-compose logs -f gitea-db-1 | gitea-db-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization gitea-db-1 | gitea-db-1 | 2023-07-19 11:42:58.664 UTC [1] LOG: starting PostgreSQL ???? on x86_64-pc-linux-musl, compiled by gcc (Alpine ????) ???? ?????, 64-bit gitea-db-1 | 2023-07-19 11:42:58.665 UTC [1] LOG: listening on IPv4 address "???????", port 5432 gitea-db-1 | 2023-07-19 11:42:58.665 UTC [1] LOG: listening on IPv6 address "::", port 5432 gitea-db-1 | 2023-07-19 11:42:58.669 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" gitea-db-1 | 2023-07-19 11:42:58.672 UTC [22] LOG: database system was shut down at 2023-07-19 10:48:03 UTC gitea-db-1 | 2023-07-19 11:42:58.676 UTC [1] LOG: database system is ready to accept connections gitea_cont | Server listening on :: port 22. gitea_cont | Server listening on 0.0.0.0 port 22. gitea_cont | 2023/07/19 13:42:59 cmd/web.go:223:runWeb() [I] Starting Gitea on PID: 20 gitea_cont | 2023/07/19 13:42:59 cmd/web.go:148:serveInstalled() [I] Gitea version: 1.20.0 built with GNU Make 4.4.1, go1.20.6 : bindata, timetzdata, sqlite, sqlite_unlock_notify gitea_cont | 2023/07/19 13:42:59 cmd/web.go:149:serveInstalled() [I] App path: /usr/local/bin/gitea gitea_cont | 2023/07/19 13:42:59 cmd/web.go:150:serveInstalled() [I] Work path: /data/gitea gitea_cont | 2023/07/19 13:42:59 cmd/web.go:151:serveInstalled() [I] Custom path: /data/gitea gitea_cont | 2023/07/19 13:42:59 cmd/web.go:152:serveInstalled() [I] Config file: /data/gitea/conf/app.ini gitea_cont | 2023/07/19 13:42:59 cmd/web.go:153:serveInstalled() [I] Run mode: prod gitea_cont | 2023/07/19 13:42:59 cmd/web.go:154:serveInstalled() [I] Prepare to run web server gitea_cont | 2023/07/19 13:42:59 routers/init.go:112:InitWebInstalled() [I] Git version: 2.40.1, Wire Protocol Version 2 Enabled (home: /data/gitea/home)
Traefik
Update SRV Traefik to accept traffic for that service.
Configure SRV Traefik to redirect the traffic towards the Gitea container.
Update Pi4 Traefik to use SRV as backend.
Configure Pi4 Traefik to use as a backend for the Gitea service, the host SRV.
Test the service is up and running
Test the Gitea service URL
curl -I https://gitea.filterhome.xyz/
Returns a status code 200, therefore we were able to receive a successful request (still it's recommended to doublecheck through the browser and ensure that the correct service was served, as it could conflict with other rules set, returning a status 200 for a different service than the desired).
HTTP/2 200
date: Wed, 19 Jul 2023 11:55:15 GMT
permissions-policy: vibrate=('self'); geolocation=('self'); midi=('self'); notifications=('self'); push=('self'); microphone=(); camera=(); magnetometer=(); gyroscope=(); speaker=(); vibrate=('self'); fullscreen=('self')
referrer-policy: same-origin
strict-transport-security: max-age=31536000; includeSubDomains
x-content-type-options: nosniff
x-frame-options: DENY
x-xss-protection: 1; mode=block
...
I am HERE right now
"Last", change the ingress from the Pi4 Traefik ingress, to the Istio Public Ingress.
This gotta be performed on the router, therefore gotta access the GUI from the ISP router and select the new IP address.