476
edits
Docs/Mirrors: Difference between revisions
→cron
(initial commit) |
(→cron) |
||
| (12 intermediate revisions by the same user not shown) | |||
|
The LUG Mirror server mirrors a number of different Linux distros.
Currently, we mirror Debian (+ISOs), Arch Linux, CentOS (+AltArch), gentoo, Fedora (+EPEL &RPMfusion), and Ubuntu (+ISOs). The full list can be seen by going to https://mirrors.lug.mtu.edu/
=== Hardware ===
Mirrors is a standalone [https://www.dell.com/en-us/shop/povw/poweredge-r730xd/1000 Dell R730xd] server (3.5" drive bay variant)
Currently iDrac is non-functional, this should be investigated.
=== Operating System ===
Mirrors runs FreeBSD.
It uses ZFS as the filesystem for the root pool and primary pool that's used for the distribution mirrors (the pool named <code>lug</code>)
It used to use salt, but it broke when upgrading from FreeBSD 12-14. Currently, all maintenance is done by hand (this is a good thing)
At its core, a mirror server performs two functions:
# Synchronizing the content from upstream mirrors to itself
# Hosting that downloaded content for end-users and other downstream mirrors to pull from
On our mirrors, this is accomplished with archvsync+cron to synchronize content with upstream, and vsftpd+rsyncd+nginx to handle hosting the content.
=== Maintenance ===
==== Certificates ====
Put the certificate (the 'intermediate' download option from our Certificate Authority) in <code>/usr/local/share/certs/mirrors_lug_mtu_edu_bundle.cer</code>, and the key in <code>/usr/local/share/certs/mirrors_lug_mtu_edu.key</code>
then run: <code>service nginx reload</code> Note: re'''load''' and NOT re'''start''', as re'''start''' kills all existing http(s) connections, while re'''load''' just applies the new settings for all future connections. It also won't kill the background daemon if the settings are not valid.
You can view the nginx configuration in <code>/usr/local/etc/nginx/nginx.conf</code> to view and change settings. This file is no longer managed by salt, and can be edited by hand.
=== Core tasks ===
At its core, a mirror server performs two functions:
# Synchronizing the content from upstream mirrors to itself
# Hosting that downloaded content for end-users and other downstream mirrors to pull from
On our mirrors, this is accomplished with archvsync+cron to synchronize content with upstream, and vsftpd+rsyncd+nginx to handle hosting the content.
==== Pulling from upstream ====
===== ftpsync =====
The ftpsync utility from the archvsync project is what we use to synchronize content on upstream servers to
archvsync is a script by the Debian project to provision mirror servers, and should be all-inclusive.
It's nice because it prevents half-sync'd files from getting served to users, so they don't pull corrupted binaries that don't verify with the package signature.
This is especially important when acting as an upstream mirror for downstream
Under the hood, ftpsync can use the ftp(?) or rsync protocols, and is determined by the ftpsync config for a distribution.
The upstream servers Mirrors pulls from, as well as what 'Tier' we are for that distribution, is as follows:
{| class="wikitable sortable"
!Distribution
!Tier
!Upstream
!Method
|-
|Arch Linux
|1
|rsync.archlinux.org
|rsync
|-
|CentOS AltArch
|1
|msync.centos.org
|rsync
|-
|CentOS Stream
|1
|rsync.stream.centos.org
|rsync
|-
|CentOS
|1
|msync.centos.org
|rsync
|-
|Debian CD
|1
|cdimage.debian.org
|rsync
|-
|Debian
|1
|syncproxy2.wna.debian.org
|rsync
|-
|EPEL
|1
|dl.fedoraproject.org
|rsync
|-
|Fedora
|1
|dl.fedoraproject.org
|rsync
|-
|gentoo
|2?
|ftp.ussg.iu.edu
|rsync
|-
|RPM Fusion
|1
|download1.rpmfusion.org
|rsync
|-
|Ubuntu Releases
|2?
|mirror.math.princeton.edu
|rsync
|-
|Ubuntu
|2?
|mirror.math.princeton.edu
|rsync
|}
It's just a repo that was <code>git pull</code>'d, and was not natively installed on the system via pkg.
===== cron =====
<code>cron</code> calls <code>ftpsync</code> to run at certain times.
This is what determines that
This should be set according to
Most distros want tier 1's to sync 4 times a day, and
Currently, this is the schedule Mirrors uses (all times in EST):
|*:43
|-
|CentOS AltArch'''*'''
|00:24 AM
|06:24 AM
|06:24 PM
|-
|CentOS
|00:24 AM
|06:24 AM
|06:24 PM
|-
|CentOS
|00:24 AM
|06:24 AM
|12:24 PM
|06:24 PM
|-
|Debian CD
|12:12 PM
|06:12 PM
|-
|Debian
|00:03 AM
|06:03 AM
|12:03 AM
|06:03 PM
|-
|EPEL'''***'''
|06:15 PM
|-
|Ubuntu
|00:45 AM
|06:45 AM
|12:45 PM
|06:45 PM
|-
|Ubuntu
|00:30 AM
|06:30 AM
|12:30 PM
|06:30 PM
|}
'''* = Deprecated, should be removed'''
Keep in mind this is when syncing starts, it may take a moment before it's fully up-to-date with upstream.
==== Serving to downstream ====
<code>vsftpd</code> is the ftp daemon running on port <code>21</code>, and allows all recursive content inside <code>/lug</code> to be downloaded by anonymous users.
It has a hardcoded <code>if</code> block pointing to each distro's dataset path, I'm almost certain could just be replaced with <code>root /lug</code> in the <code>server</code> block.
=== Salt ===
Salt used to administer these services, but it's half-broken at the moment and ''should not be reinstalled'' (in my opinion).
As such, I think the way Mirrors is setup is essentially perfect, sans salt.
salt.tar.gz contains all the configuration for salt, and the config files it uses to overwrite the config files located in the standard location (as well as the template files it uses to 'build' configs for services like rsyncd and archvsync when a new distro is added to the primary salt config)
| |||