Installation

Preparation¶

Clone https://github.com/buvis/clusters/ repository to your workstation. Please note that modifications will be necessary to make it work for you.

Tools¶

1. Install the following tools on your workstation:
2. pre-commit
3. direnv
4. gnupg and sops
5. talosctl
6. jq
7. kubectl
8. krew
9. minio kubectl plugin
10. cilium-cli
11. Install scripts requirements: pip install -r requirements.txt in clusters repository root directory

Services¶

Create accounts:

• Slack to get notifications from Flux

Gitops¶

1. Add renovate to Github
2. Export GITHUB_TOKEN environment variable into .envrc files with GitHub personal access token generated specifically for Flux
3. Export SLACK_WEBHOOK_URL environment variable into .envrc files, get incoming webhook address <SLACK_WEBHOOK_URL> from Slack

4. Enable SOPS for Flux

5. (do only once in a lifetime) Generate GPG key with no password protection. You can't protect the key with password, because Flux has no way of entering it when decrypting the secrets.

6. Get fingerprint of the key <SOPS_KEY_FINGERPRINT>

   gpg --list-secret-keys
   

7. Export SOPS_KEY_FINGERPRINT environment variable into .envrc files with the value from previous step

Network¶

1. SSH to network's router
2. Copy cluster's infrastructure/router/etc/bgpd.conf to network's router's /etc to configure bgpd to peer with the cluster
3. Enable bgpd on network's router: rcctl enable bgpd
4. Assign IP addresses to nodes and VMs in /etc/dhcpd.conf, push router's IP as DNS to them

Home cluster¶

This cluster is based on Virtual Machines provided by Proxmox.

Install Proxmox¶

1. Download Proxmox installation iso
2. Burn it to USB stick. Note: in macOS, use 1M instead of 1m as block size
3. Boot Proxmox machine from the USB stick
4. Unauthorized change message will display, confirm it
5. Set the correct date and time in BIOS
6. Disable onboard wifi and bluetooth
7. Check that disks are recognized
8. Boot - Secure Boot = Disabled
9. Exit, save, reboot
10. Install Proxmox following the installation wizard (use router's IP as DNS server)
11. Enter root's password to PM_PASS environment variable in .envrc
12. After completing the installation, SSH to Proxmox server
13. Use community repo: sed -i '1i deb http://download.proxmox.com/debian bullseye pve-no-subscription\n' /etc/apt/sources.list
14. Disable enterprise repo: sed -i 's/deb https:\/\/enterprise.proxmox.com\/debian\/pve bullseye pve-enterprise/# deb https:\/\/enterprise.proxmox.com\/debian\/pve bullseye pve-enterprise/g' /etc/apt/sources.list.d/pve-enterprise.list
15. Update the system: apt update && apt full-upgrade
16. Install temperature sensors reading tools: apt install xsensors (then use sensors to read temperature measurements)
17. Reboot: reboot now

18. Remove subscription notice:

    1. Go to UI site source: cd /usr/share/javascript/proxmox-widget-toolkit/
    2. Backup the file you'll modify: cp proxmoxlib.js proxmoxlib.js.bak

    3. Edit proxmoxlib.js: vi proxmoxlib.js

    4. Find

    Ext.Msg.show({
      title: gettext('No valid subscription'),
    

    • Replace with

    void({
      title: gettext('No valid subscription'),
    

    1. Restart Proxmox UI: systemctl restart pveproxy.service
    2. Clear browser cache and reconnect UI

19. Create storage for Persistent Volumes:

    1. Double check device name: lsblk
    2. Remove previous partitions (assuming device is /dev/sda): fdisk /dev/sda, g,<ENTER>, w, <ENTER>
    3. Connect to Proxmox management UI at https://<server_ip>:8006
    4. Create LVM Volume Group: <NODENAME> - Disks - LVM, Create: Volume Group, Name: <NODENAME>-tank on entire /dev/sda

Get Talos schematic ID¶

Currently, the Talos schematic ID of images used in home cluster is decca0d019b92170b9c7101da90f1ec0de7e9841b3d126ac4111743778d8eb1a. This corresponds the following customization:

customization:
    extraKernelArgs:
        - amd_pstate=active
    systemExtensions:
        officialExtensions:
            - siderolabs/amd-ucode
            - siderolabs/amdgpu
            - siderolabs/iscsi-tools
            - siderolabs/qemu-guest-agent
            - siderolabs/util-linux-tools

If different customization is needed, you can find the schematic ID through Talos Image Factory. Then you need to update it in .envrc and infrastructure/talos/patch-all.yaml.

Bootstrap¶

Run buvisctl bootstrap in cluster-home directory.

Office cluster¶

This cluster is based on Raspberry Pi 4 machines.

Flash SD cards¶

Flash Talos to SD cards by following Talos installation guide.

Bootstrap the cluster¶

Run make install