80 Refactor (#83)

* 80 Update README.md * Add prepare_dev_setup role and fix lints * 80 Move torrent stack services * 80 Refactor volumes * 80 Add duckdns domain in compose files * 80 Refactor * 80 Fix lint errors * 80 Test ansible make module for pigpio * 80 Fix yaml indentation * 80 Add ansible.posix * 80 Fix ansible lints * 80 community.docker collection --------- Co-authored-by: veerendra2 <[email protected]>
veerendra2 · Oct 8, 2023 · 3d97025 · 3d97025
1 parent 8a5b117
commit 3d97025
Show file tree

Hide file tree

Showing 64 changed files with 456 additions and 688 deletions.
diff --git a/.ansible-lint b/.ansible-lint
@@ -0,0 +1,11 @@
+---
+verbosity: 1
+
+skip_list:
+  - 'yaml'
+  - 'role-name'
+  - 'package-latest'
+  - 'latest[git]'
+  - 'no-changed-when'
+  - 'schema[meta]'
+  - 'ignore-errors'
diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
@@ -0,0 +1,30 @@
+---
+name: Lint
+'on':
+  pull_request:
+  push:
+    branches:
+      - main
+
+jobs:
+  lint:
+    name: Lint
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repo
+        uses: actions/checkout@v2
+
+      - name: Set up Python3
+        uses: actions/setup-python@v2
+        with:
+          python-version: '3.x'
+
+      - name: Install yamllint
+        run: pip3 install yamllint
+
+      - name: Run yamllint
+        run: |
+          yamllint .
+
+      - name: Run ansible-lint
+        uses: ansible/ansible-lint@main
diff --git a/.yamllint b/.yamllint
@@ -0,0 +1,7 @@
+---
+extends: default
+
+rules:
+  line-length:
+    max: 200
+    level: warning
diff --git a/README.md b/README.md
@@ -3,9 +3,10 @@
 ![GitHub forks](https://img.shields.io/github/forks/veerendra2/raspberrypi-homeserver?style=plastic)
 ![GitHub issues](https://img.shields.io/github/issues/veerendra2/raspberrypi-homeserver?style=plastic)
 ![GitHub release (release name instead of tag name)](https://img.shields.io/github/v/release/veerendra2/raspberrypi-homeserver?include_prereleases&style=plastic)
-# Raspberry Pi Home Server
+# Raspberry Pi Homeserver
+
 <p align="center">
-<b>A collection of applications and tools to make awesome Raspberry Pi homerserver</b>
+<b>A collection of self-host docker swarm stacks </b>
 </p>
 
 <table align="center">
@@ -106,6 +107,11 @@
       <img src="https://user-images.githubusercontent.com/8393701/253675772-39bd3d53-458d-479c-a49e-a6bf0a376afe.png" alt="rest-server" width="60"/> <br /> rest-server<br/>(Restic)
     </center>
   </td>
+  <td>
+    <center>
+      <img src="https://user-images.githubusercontent.com/8393701/273405568-8b3eae3f-244b-4c8c-bee0-e6aab7b751f0.png" alt="IT-Tools" width="60"/> <br /> IT-Tools
+    </center>
+  </td>
 </tr>
 </table>
 
@@ -192,10 +198,10 @@
 ![Architecture](https://user-images.githubusercontent.com/8393701/235324714-75620112-a89b-4d10-ab9d-2e44de75d36b.jpg)
 
 ## Getting Started
-> :warning: **CAUTION** Ansible playbooks are highly unstable, needs to rework in future [#54](https://github.com/veerendra2/raspberrypi-homeserver/issues/54).
->
 > Refer [Gitbook Docs](https://dust6765.gitbook.io/raspberrypi-home-server/) for more details and how to deploy manually
 
+Ansible automation works well, if all services are intended to deploy on single box. Since I have only one machine, don't have to worry about the docker volumes, all of my docker volumes stored on single SSD mounted to host.
+
 * Follow prerequisite [manual steps](https://dust6765.gitbook.io/raspberrypi-home-server/settings/manual-steps) to prepare Pi
 * Browse [vars.yml](./vars.yml) to configuration
 
@@ -206,8 +212,8 @@ $ cd raspberrypi-homeserver
 $ ansible-playbook main.yml
 ```
 ### NOTE
-* The setup created to deploy all services on single node docker swarm cluster. If you want to use this setup on multi node swarm cluster, there are some additional tweaks required like
-  * Change plancement
+* This setup is created to deploy all services on single node docker swarm cluster. If you want to use this setup on multi node swarm cluster, there are some additional tweaks required like
+  * Change placement
     ```yaml
     ...
       deploy:
@@ -216,12 +222,13 @@ $ ansible-playbook main.yml
           constraints: [node.role == manager]
     ...
     ```
-* Currently ~~I'm using IP address `192.168.0.120` to access services, change services configs if you have a domain~~ using duckdns sub-domain. Refer my [blog post](https://veerendra2.github.io/traefik-https) to get more info.
+* Currently I'm using duckdns sub-domain (Refer example -> [services/traefik/docker-stack.yml#32](./services/traefik/docker-stack.yml#32)). Refer my [blog post](https://veerendra2.medium.com/traefik-https-config-with-duckdns-for-local-homeserver-c55db9971683) to get more info.
+* Update docker volume mount paths.
 
 ## Related Blogs
-* https://veerendra2.github.io/portainer-vs-yacht
-* https://veerendra2.github.io/pihole-dhcp-relay
-* https://veerendra2.github.io/wireguard-qbittorrent-docker-swarm-1
-* https://veerendra2.github.io/wireguard-qbittorrent-docker-swarm-2
-* https://veerendra2.github.io/traefik-https
-* https://dev.to/veerendra2/how-to-deploy-rest-serverrestic-on-docker-swarm-behind-traefik-reverse-proxy-4a8h
+* [Portainer vs Yacht](https://veerendra2.medium.com/portainer-vs-yacht-316405b9e867)
+* [Pi-hole with DHCP Relay in Docker](https://veerendra2.medium.com/pi-hole-with-dhcp-relay-in-docker-cef397922e5a)
+* [Wireguard VPN and BitTorrent on Docker Swarm (Part 1)](https://veerendra2.medium.com/wireguard-vpn-and-bittorrent-on-docker-swarm-part-1-1100b4115cc0)
+* [Wireguard VPN and BitTorrent on Docker Swarm (Part 2)](https://veerendra2.medium.com/wireguard-vpn-and-bittorrent-on-docker-swarm-part-2-b92a251ba873)
+* [Traefik HTTPS Config with DuckDNS for Local Homeserver](https://veerendra2.medium.com/traefik-https-config-with-duckdns-for-local-homeserver-c55db9971683)
+* [How to Deploy rest-server(Restic) on Docker Swarm Behind Traefik Reverse Proxy](https://dev.to/veerendra2/how-to-deploy-rest-serverrestic-on-docker-swarm-behind-traefik-reverse-proxy-4a8h)
diff --git a/inventory.yml b/inventory.yml
@@ -1,11 +1,12 @@
+---
 all:
   hosts:
     atom:
       ansible_ssh_port: 22
       ansible_ssh_host: 192.168.0.120
       ansible_ssh_user: veerendra
-      #ansible_ssh_password:
+      # ansible_ssh_password:
       extra_hosts:
-        - "atom:192.168.0.130" # Wifi Address
+        - "atom:192.168.0.130"
   localhost:
-    ansible_connection: local
+    ansible_connection: local
diff --git a/main.yml b/main.yml
@@ -1,38 +1,24 @@
-# Author: Veerendra K
-# Description: An Ansible playbook to make awesome Raspberry Pi homeserver
-
-- name: Raspberry Pi Homeserver
+---
+- name: Homeserver Automation
   hosts: localhost
-  gather_facts: yes
   vars_files:
     - vars.yml
 
-  pre_tasks:
-    - name: Run update
-      apt:
-        upgrade: true
-        update_cache: yes
-      ignore_errors: yes
-      become: yes
-
-  post_tasks:
-    - name: Run autoremove
-      apt:
-        autoremove: yes
-      become: yes
+  # roles:
+  #   - role: veerendra2.prepare_dev_setup
+  #     tags: system
 
   tasks:
     - name: Prepare Pi
-      import_tasks: tasks/prepare-pi.yml
-      tags: prepare-pi
+      ansible.builtin.import_tasks: tasks/prepare-pi.yml
+      become: true
+      tags: pi
 
     - name: Deploy services
-      import_tasks: tasks/deploy-services.yml
+      ansible.builtin.import_tasks: tasks/deploy-services.yml
+      become: true
       tags: services
 
     - name: Setup firewall rules
-      import_tasks: tasks/ufw.yml
-
-    - name: Performing smoke tests
-      import_tasks: tasks/smoke-tests.yml
-      tags: never
+      ansible.builtin.import_tasks: tasks/ufw.yml
+      become: true
diff --git a/requirements.yml b/requirements.yml
@@ -0,0 +1,7 @@
+---
+roles:
+  - name: veerendra2.prepare_dev_setup
+collections:
+  - name: ansible.posix
+  - name: community.general
+  - name: community.docker
diff --git a/services/README.md b/services/README.md
@@ -1,7 +1,2 @@
 # Docker Swarm Services
-> Refer [Gitbook Docs](https://dust6765.gitbook.io/raspberrypi-home-server/services)
-
-**NOTE:** Export `DOMAIN` environmental variable before deploying services
-  ```bash
-  $ export DOMAIN=mydomain.com
-  ```
+Update your duckdns sub-domain name in traefik label in all services. For example [./traefik/docker-stack.yml#31](./traefik/docker-stack.yml#31)
diff --git a/services/databases/docker-stack.yml b/services/databases/docker-stack.yml
@@ -1,16 +1,13 @@
 ---
 # https://dust6765.gitbook.io/raspberrypi-home-server/services/databases
-version: "3.8"
+version: '3.8'
 
 networks:
   networks_public:
     external: true
   network_databases:
     external: true
 
-volumes:
-  redis:
-
 secrets:
   postgres_password:
     file: secrets/postgres_password.txt
@@ -59,17 +56,17 @@ services:
       labels:
         - "traefik.enable=false"
     hostname: redis
-    command:  [
-    "sh", "-c",
-    '
-    docker-entrypoint.sh
-    --requirepass "$$(cat $$REDIS_PASSWORD_FILE)"
-    '
+    command: [
+      "sh", "-c",
+      '
+      docker-entrypoint.sh
+      --requirepass "$$(cat $$REDIS_PASSWORD_FILE)"
+      '
     ]
     env_file:
       - .env_redis
     volumes:
-      - redis:/data
+      - /media/disk2/volumes/redis:/data
     networks:
       - network_databases
     secrets:

diff --git a/services/filebrowser/docker-stack.yml b/services/filebrowser/docker-stack.yml
@@ -1,17 +1,14 @@
 ---
 # https://dust6765.gitbook.io/raspberrypi-home-server/services/file-browser
-version: "3.8"
+version: '3.8'
 
 networks:
   network_public:
     external: true
 
-volumes:
-  filebrowser:
-
 services:
   filebrowser:
-    image: filebrowser/filebrowser:v2.24.1
+    image: filebrowser/filebrowser:latest
     deploy:
       replicas: 1
       placement:
@@ -24,18 +21,17 @@ services:
         - traefik.enable=true
         - traefik.docker.network=network_public
         - traefik.http.routers.filebrowser.tls=true
-        - traefik.http.routers.filebrowser.rule=Host(`${DOMAIN}`) && PathPrefix(`/filebrowser`)
+        - traefik.http.routers.filebrowser.rule=Host(`veeru.duckdns.org`) && PathPrefix(`/filebrowser`)
         - traefik.http.routers.filebrowser.middlewares=filebrowser-stripprefix
         - traefik.http.middlewares.filebrowser-stripprefix.stripprefix.prefixes=/filebrowser
         - traefik.http.services.filebrowser.loadbalancer.server.port=80
     hostname: filebrowser
     env_file:
       - .env_filebrowser
     volumes:
-      - /media/disk1:/srv/disk1
-      - /media/disk2:/srv/disk2
       - ./config/filebrowser.json:/.filebrowser.json
-      - filebrowser:/database
+      - /media/disk2/volumes/filebrowser:/database
+      - /media/disk2:/srv/disk2
+      - /media/disk1:/srv/disk1
     networks:
       network_public:
-
diff --git a/services/homer/assets/config.yml b/services/homer/assets/config.yml
@@ -135,6 +135,7 @@ services:
       #   subtitle: "Container management UI"
       #   url: "/portainer/"
       #   method: "get"
+
       - name: "Filebrowser"
         type: "Ping"
         logo: "assets/tools/filebrowser.png"
@@ -162,7 +163,7 @@ services:
   #       url: "/vaultwarden/"
   #       method: "get"
 
-  # - name: Search
+  # - name: Utilities
   #   icon: ""
   #   items:
   #     - name: "SearXNG"
@@ -171,3 +172,10 @@ services:
   #       subtitle: "Metasearch engine"
   #       url: "/searxng/"
   #       method: "get"
+
+  #     - name: "IT-Tools"
+  #       type: "Ping"
+  #       logo: "assets/tools/it-tools.png"
+  #       subtitle: "Useful tools for developer"
+  #       url: "/it-tools/"
+  #       method: "get"
diff --git a/services/homer/assets/tools/it-tools.png b/services/homer/assets/tools/it-tools.png
diff --git a/services/homer/docker-stack.yml b/services/homer/docker-stack.yml
@@ -1,14 +1,14 @@
 ---
 # https://dust6765.gitbook.io/raspberrypi-home-server/services/homer
-version: "3.8"
+version: '3.8'
 
 networks:
   network_public:
     external: true
 
 services:
   homer:
-    image: b4bz/homer:v23.05.1
+    image: b4bz/homer:latest
     deploy:
       replicas: 1
       placement:
@@ -20,8 +20,8 @@ services:
       labels:
         - "traefik.enable=true"
         - "traefik.docker.network=network_public"
-        - "traefik.http.routers.homer.rule=Host(`${DOMAIN}`) && PathPrefix(`/`)"
         - "traefik.http.routers.homer.tls=true"
+        - "traefik.http.routers.homer.rule=Host(`veeru.duckdns.org`) && PathPrefix(`/`)"
         - "traefik.http.services.homer.loadbalancer.server.port=8080"
     hostname: homer
     env_file:

diff --git a/services/it-tools/.env_it-tools b/services/it-tools/.env_it-tools
@@ -0,0 +1,2 @@
+#https://github.com/CorentinTh/it-tools/pull/461
+BASE_URL=/it-tools
diff --git a/services/it-tools/docker-stack.yml b/services/it-tools/docker-stack.yml
@@ -0,0 +1,29 @@
+---
+version: '3.8'
+
+networks:
+  network_public:
+    external: true
+
+services:
+  it-tools:
+    image: corentinth/it-tools:latest
+    deploy:
+      replicas: 1
+      placement:
+        constraints: [node.role == manager]
+      restart_policy:
+        condition: on-failure
+        delay: 30s
+        max_attempts: 3
+      labels:
+        - "traefik.enable=true"
+        - "traefik.docker.network=network_public"
+        - "traefik.http.routers.it-tools.tls=true"
+        - "traefik.http.routers.it-tools.rule=Host(`veeru.duckdns.org`) && PathPrefix(`/it-tools`)"
+        - "traefik.http.services.it-tools.loadbalancer.server.port=80"
+    hostname: it-tools
+    env_file:
+      - .env_it-tools
+    networks:
+      - network_public
diff --git a/services/torrent/jackett/.env_jackett → services/jacket/.env_jackett b/services/torrent/jackett/.env_jackett → services/jacket/.env_jackett
diff --git a/services/jacket/README.md b/services/jacket/README.md
@@ -0,0 +1,4 @@
+# jacket
+This stack requires `qBittorent` + `Wireguard` which should be up and running.
+
+* Deploy [qBittorrent + Wireguard](../qbittorrent/)
diff --git a/services/torrent/jackett/ServerConfig.json → services/jacket/config/ServerConfig.json b/services/torrent/jackett/ServerConfig.json → services/jacket/config/ServerConfig.json
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		#https://github.com/CorentinTh/it-tools/pull/461
		BASE_URL=/it-tools