Infrastructure

This page is intended as a 'hub' for all of LUGs internal documentation.

All of our documentation is intentionally public so that other student organizations or individuals can replicate aspects of our infrastructure if they so desire.

If a topic requires a significant amount of content, you may want to break it out into a new article and link it on this page.

Servers & Services

Proxmox Cluster

The majority of our infrastructure are VMs in the Proxmox cluster, so everything can be highly-available (meaning VMs can jump to another Proxmox node if one goes down).

Proxmox Nodes

The nodes in the cluster include:

• [10.10.1.20] Kurisu
• [10.10.1.21] Okabe (currently offline; running Windows 10 LTSC temporarily to poke around with HGST Drives)
• [10.10.1.22] Daru
• [10.10.1.23] Luka
• [10.10.1.24] Mayuri
• [10.10.1.25] MrBraun (HP Server)

These are also listed in Servers since they're all physical servers in the GLRC rack.

Virtual Machines

All VMs run Debian to keep things homogenous and easy to upgrade/automate (except a few Windows VMs like Allen's scuffed Win10 LTSC gaming VM)

The VMs in the cluster include:

• [10.10.1.2] PXEBoot server (inactive)
• [10.10.1.8] Huskybot IRC<->Matrix<->Discord bridge
• [10.10.1.9] LUG IRC Server (running ergo)
• [10.10.1.12] Invidious (private youtube frontend, currently inactive)
• [10.10.1.14] BookStack (alternative knowledgebase for documentation. Inactive, we're using this Wiki instead)
• [10.10.1.15] The lug.mtu.edu website and HTTP reverse-proxy for everything else behind NAT (running NGINX)
• [10.10.1.16] This Wiki
• [10.10.1.17] Netbox (network/rack-related documentation. Currently inactive, overly complicated for our needs)
• [10.10.1.70] Socksproxy (so members using the split-tunneled LUG VPN have an easy way to route traffic through LUG)
• [10.10.1.71] VM for accessvillage.net (contact Noah if any issues)
• [10.10.1.76] debian (noah courseproject; will eventually delete)
• [10.10.1.99] Noah's personal VM for random stuff
• [10.10.1.170] hashtopolis (RedTeam Hashtopolis server for CTFs)
• [10.10.1.172] badapple (parrot.live-like badapple service)
• [10.10.1.202] "Main-MC" (idk; ask Allen)
• [10.10.1.212] Velocity reverse-proxy for Minecraft servers (so we can offer unlimited servers to clubs/halls on campus without running out of public IPs)
• [10.10.1.224] Allen's Gaming VM (runs Windows)
• [10.10.1.229] "Kube-Minecraft" (idk; ask Allen)

Updating Nodes

Proxmox runs on top of Debian, so the updating process is the same.

1. apt update && apt upgrade
2. (Optional) Remove the annoying unlicensed popup from web dashboard: sed -Ezi.bak "s/(Ext.Msg.show\(\{\s+title: gettext\('No valid sub)/void\(\{ \/\/\1/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service
3. (Optional) Manually migrate all VMs to other Proxmox nodes first; Proxmox doesn't do this automatically and all the VMs running on the host when it reboots will go offline until the host comes back up
4. (Optional but recommended) reboot the node

For the yearly major version bumps, you may need to run apt update && apt upgrade, followed by apt dist-upgrade

This is the process on Debian, but I haven't tested it on Proxmox yet.

Check the Proxmox wiki's 'Upgrade' category for specific instructions when the time comes.

Mirrors

Mirrors is a standalone Dell R730xd server (3.5" drive bay variant) running FreeBSD, and all services are managed by salt.

We're in the process of rebuilding it, but in the meantime this is what we've been doing to manage it thus far:

Certificate maintenance:

put it in /usr/share/salt/<somewhere> where salt will copy it to /etc/nginx/<somewhere>

Fileserver

Coming Soon, currently unprovisioned (waiting on new PSU and drives to arrive)

Management

Time-sensitive

Email IT for new certs (example template to use, make sure to keep SubjectAltName, etc)

Install-a-thons

shirt printing

Budget

USG meetings

making presentable diagrams and representations of data