Docs/Proxmox Cluster

The majority of our infrastructure are VMs in the Proxmox cluster, so everything can be highly-available (meaning VMs can jump to another Proxmox node if one goes down).

In the panel for each VM in the webUI, make sure to enable the guest agent; Debian will auto install the QEMU guest agent on first install when it detects being run inside a VM.

Proxmox Nodes

The nodes in the cluster include:

[10.10.1.20] Kurisu
[10.10.1.21] Okabe (currently offline; running Windows 10 LTSC temporarily to poke around with HGST Drives)
[10.10.1.22] Daru
[10.10.1.23] Luka
[10.10.1.24] Mayuri
[10.10.1.25] MrBraun (HP Server)

Note that all these addresses are static, and must be changed manually on each host (Proxmox doesn't currently support DHCP). The process is loosely outlined by the comments here.

These are also listed in Servers since they're all physical servers in the GLRC rack.

Virtual Machines

The VMs in the cluster include:

[10.10.1.8] Huskybot IRC<->Matrix<->Discord bridge
[10.10.1.9] LUG IRC Server (running ergo)
[10.10.1.15] webserver running NGINX, hosts the lug.mtu.edu homepage and servers as a reverse-proxy for all other webservers behind our NAT.
[10.10.1.16] This mediawiki instance
[10.10.1.70] Socksproxy (so members using the split-tunneled LUG VPN have an easy way to route traffic through LUG)
[10.10.1.76] debian (noah courseproject; will eventually delete)
[10.10.1.170] hashtopolis (RedTeam Hashtopolis server for CTFs)
[10.10.1.172] badapple (parrot.live-like badapple service)
[10.10.1.202] "Main-MC" (idk; ask Allen)
[10.10.1.212] Velocity reverse-proxy for Minecraft servers (so we can offer unlimited servers to clubs/halls on campus without running out of public IPs)
[10.10.1.224] Allen's Gaming VM (runs Windows)
[10.10.1.229] "Kube-Minecraft" (idk; ask Allen)

You can see all VMs listed in the Proxmox WebUI.

Updating Nodes

Proxmox runs on top of Debian, so the updating process is the mostly same.

apt update && apt upgrade
(Optional) Remove the annoying unlicensed popup from web dashboard: sed -Ezi.bak "s/(Ext.Msg.show\(\{\s+title: gettext\('No valid sub)/void\(\{ \/\/\1/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service
(Optional) Manually migrate all VMs to other Proxmox nodes first; Proxmox doesn't do this automatically and all the VMs running on the host when it reboots will go offline until the host comes back up
(Optional but recommended) reboot the node

For the yearly major version bumps, you may need to run apt update && apt upgrade, followed by apt dist-upgrade; This is the process on Debian, but I haven't tested it on Proxmox.

Check the Proxmox wiki's 'Upgrade' category for specific instructions when the time comes.

Updating VMs

All VMs run Debian to keep things homogenous and easy to upgrade/automate, except a few Windows VMs like Allen's scuffed Win10 LTSC gaming VM; Those are presumed self-managed.

The update process is the same as any Debian system:

apt update && apt upgrade
1. If the kernel or systemd get updated, it's a good idea to reboot
For major version bumps (I think there's one each year?), you need to run the aforementioned apt update && apt upgrade, followed by apt dist-upgrade

Updates need to be automated with Ansible at some point.