Storage management - Revision history

imported>Sjwhitak at 13:12, 20 April 2022

2022-04-20T13:12:16Z

← Older revision		Revision as of 13:12, 20 April 2022
Line 205:		Line 205:

	Here's my computer's setup:		Here's my computer's setup:
	<syntaxhighlight lang="~~fstab~~">		<syntaxhighlight lang="text">
	/dev/lvm/lv_root / ext4 defaults 0 1		/dev/lvm/lv_root / ext4 defaults 0 1
	/dev/lvm/lv_home /home ext4 defaults 0 2		/dev/lvm/lv_home /home ext4 defaults 0 2

imported>Sjwhitak: /* Logical volume management (LVM) */

2022-04-20T13:10:23Z

Logical volume management (LVM)

← Older revision		Revision as of 13:10, 20 April 2022
Line 115:		Line 115:

	I'd highly, highly recommend following along with [https://www.howtoforge.com/linux_lvm this tutorial] within an LVM. You will be very comfortable with all of these commands after this.		I'd highly, highly recommend following along with [https://www.howtoforge.com/linux_lvm this tutorial] within an LVM. You will be very comfortable with all of these commands after this.

			{{Warning\|You destroy all data when running these commands, so if you're doing this on your system just move everything off the drives before running these commands.}}

	=== Demo setup ===		=== Demo setup ===

imported>Sjwhitak: /* Demo setup */

2022-04-20T13:10:03Z

Demo setup

← Older revision		Revision as of 13:10, 20 April 2022
Line 70:		Line 70:
	</syntaxhighlight>		</syntaxhighlight>

	{{Tip\|I'd recommend playing with this in [https://www.virtualbox.org/ VirtualBox] because it is very difficult to change these on your filesystem once you have data on it. In fact, ~~you destroy all data when running these commands, so you'd just move everything off the drives before doing this. It's a big pain.~~}}		{{Tip\|I'd recommend playing with this in [https://www.virtualbox.org/ VirtualBox] because it is very difficult to change these on your filesystem once you have data on it. In fact, }}
			{{Warning\|You destroy all data when running these commands, so if you're doing this on your system just move everything off the drives before running these commands.}}

	== Logical volume management (LVM) ==		== Logical volume management (LVM) ==

imported>Sjwhitak: /* Demo setup */

2022-04-20T13:09:03Z

Demo setup

← Older revision		Revision as of 13:09, 20 April 2022
Line 62:		Line 62:

	=== Demo setup ===		=== Demo setup ===
	Follow [https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Disks#Partitioning_the_disk_with_GPT_for_UEFI this] setup to test. ~~I'd recommend playing with this in [https://www.virtualbox.org/ VirtualBox] because it is very difficult to change these on your filesystem.~~		Follow [https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Disks#Partitioning_the_disk_with_GPT_for_UEFI this] setup to test.
	<syntaxhighlight lang="text">		<syntaxhighlight lang="text">
	Device Start End Sectors Size Type		Device Start End Sectors Size Type
Line 69:		Line 69:
	/dev/sda3 2623488 19400703 16777216 8G Linux filesystem		/dev/sda3 2623488 19400703 16777216 8G Linux filesystem
	</syntaxhighlight>		</syntaxhighlight>

			{{Tip\|I'd recommend playing with this in [https://www.virtualbox.org/ VirtualBox] because it is very difficult to change these on your filesystem once you have data on it. In fact, you destroy all data when running these commands, so you'd just move everything off the drives before doing this. It's a big pain.}}

	== Logical volume management (LVM) ==		== Logical volume management (LVM) ==

imported>Sjwhitak: /* Mounting commands */

2022-04-20T13:07:26Z

Mounting commands

← Older revision		Revision as of 13:07, 20 April 2022
Line 37:		Line 37:
	{{RootCmd\|umount /mnt}}		{{RootCmd\|umount /mnt}}
	If you plug in a USB and you want to access it, use <code>lsblk -f</code>, then <code>mount /dev/sdXN /mnt</code> or wherever you want the USB files. Then, <code>umount /mnt</code> to safely remove.		If you plug in a USB and you want to access it, use <code>lsblk -f</code>, then <code>mount /dev/sdXN /mnt</code> or wherever you want the USB files. Then, <code>umount /mnt</code> to safely remove.
	{{Note\|If you're testing this on [https://www.virtualbox.org/ VirtualBox], your virtual drives have no format, so just run ~~mkfs.ext4 /dev/sdX for every drive and you can mount them.}}~~		{{Note\|If you're testing this on [https://www.virtualbox.org/ VirtualBox], your virtual drives have no format, so just run
			{{RootCmd\|mkfs.ext4 /dev/sdX}} for every drive and you can mount them.}}

	== Bare-metal drives ==		== Bare-metal drives ==

imported>Sjwhitak: .

2022-04-20T13:06:43Z

← Older revision		Revision as of 13:06, 20 April 2022
Line 9:		Line 9:
	to make the partition a swap file, then run		to make the partition a swap file, then run
	{{RootCmd\|swapon /dev/sda2}}		{{RootCmd\|swapon /dev/sda2}}
	to enable swap. Run top or htop to see your swap and memory usage.		to enable swap. Run <code>top</code> or <code>htop</code> to see your swap and memory usage.

	== Looking at things ==		== Looking at things ==

imported>Sjwhitak: Copied from https://lug.mtu.edu/minutes/fall2021/meeting09.html

2022-04-20T13:04:24Z

Copied from https://lug.mtu.edu/minutes/fall2021/meeting09.html

New page

== Introduction ==

Save large files like code, binary files, movies, etc. The storage media are things like HDDs, SSDs, magnetic tape.

Storage is not memory, where memory caches memory for quick access, things like any type of RAM: SRAM, SDRAM, VRAM, etc.

Swap storage is fake memory that is actually storage, but pretends to be memory. You need a partition for this, and so you run
{{RootCmd|typemkswap /dev/sda2}}
to make the partition a swap file, then run
{{RootCmd|swapon /dev/sda2}}
to enable swap. Run top or htop to see your swap and memory usage.

== Looking at things ==
* [https://linux.die.net/man/8/lsblk lsblk] - "list block devices", shows partitions
** <code>lsblk -f</code> gives same info as <code>blkid</code>
* [https://linux.die.net/man/8/blkid blkid] - "block device ids", shows storage UUIDs
* [https://linux.die.net/man/8/fdisk fdisk] - list all your storage media using <code>fdisk</code>
* [https://linux.die.net/man/1/df df -h] - shows you the free space on each drive.
{{Tip|When trying to save space, use <code>du -sh ./*</code> to look at folder sizes and work your way down to delete the things that aren't needed.}}

== Mounting systems ==
=== [https://linux.die.net/man/8/findmnt findmnt] ===
Finds all the mounts in your system.

Anything in [https://www.thegeekdiary.com/understanding-the-sysfs-file-system-in-linux/ /sys] or [https://www.geeksforgeeks.org/proc-file-system-linux/ /proc] are kernel stuff. The kernel opens up files for you to look at. I don't know anything else about it! Look it up yourself, cuz I don't know what it means. It's not important to working with storage systems.

/dev is your stuff... mostly. Some system stuff that you don't use: [https://unix.stackexchange.com/questions/93531/what-is-stored-in-dev-pts-files-and-can-we-open-them /dev/pts], [http://www.csl.mtu.edu/cs3331.ck/www/Home.html /dev/shm], [https://users.pja.edu.pl/~jms/qnx/help/watcom/clibref/mq_overview.html /dev/mqueue].

=== Mounting commands ===
You're working with storage media, so every single command will always be run as root.

<code>/dev/sdX</code> are your drives! <code>/dev/sda</code> is your first, <code>/dev/sdb</code> is your second, etc.

Mount a drive to look at:
{{RootCmd|mount /dev/sdb1 /mnt}}
Unmount it when you're done:
{{RootCmd|umount /mnt}}
If you plug in a USB and you want to access it, use <code>lsblk -f</code>, then <code>mount /dev/sdXN /mnt</code> or wherever you want the USB files. Then, <code>umount /mnt</code> to safely remove.
{{Note|If you're testing this on [https://www.virtualbox.org/ VirtualBox], your virtual drives have no format, so just run mkfs.ext4 /dev/sdX for every drive and you can mount them.}}

== Bare-metal drives ==
We use [https://linux.die.net/man/8/fdisk fdisk].
{{RootCmd|fdisk /dev/sda}}
{| class="wikitable" style="margin:auto"
|+ [https://linux.die.net/man/8/fdisk fdisk] commands
|-
! Command !! What it does
|-
| g || sets disk as GUID Partition Table (GPT). This supports UEFI and up to 8 ZB (it goes GB, TB, PB, EB, ZB.. so a lot).
|-
| o || Sets disk as a Master Boot Record MBR. This supports legacy boot, NO UEFI, and up to 2 TB size. Easier to set up than GPT.
|-
| n || New partition.
|-
| p || Primary partition.
|-
| t || Sets the file type. Key ones are: (MBR) 82=Linux swap, 83=Linux, fd=RAID, 8e=LVM. (GPT) 19=Linux swap, 20=Linux, 29=Linux RAID, 30=Linux LVM, 1=EFI. The default on fdisk is usually Linux.
|-
| w || Write your changes.
|}

=== Demo setup ===
Follow [https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Disks#Partitioning_the_disk_with_GPT_for_UEFI this] setup to test. I'd recommend playing with this in [https://www.virtualbox.org/ VirtualBox] because it is very difficult to change these on your filesystem.
<syntaxhighlight lang="text">
Device Start End Sectors Size Type
/dev/sda1 2048 526335 524288 256M EFI System
/dev/sda2 526336 2623487 2097152 1G Linux swap
/dev/sda3 2623488 19400703 16777216 8G Linux filesystem
</syntaxhighlight>

== Logical volume management (LVM) ==
LVMs abstract away drives quite a bit.
<syntaxhighlight lang="text">
/---------------\ /-----\ /--------\
Logical volumes | movies | code | home |
\---------------/ \-----/ \--------/

/----------------------------------\
Volume groups | lvm |
\----------------------------------/

/----------\ /---------\ /----------\
Physical volumes | /dev/sda1 | /dev/sdb1 | /dev/sdc1 |
\----------/ \---------/ \----------/
</syntaxhighlight>

The main commands are:
{| class="wikitable" style="margin:auto"
|+ LVM commands
|-
! Command !! What it does
|-
| pvcreate || Add a drive to the physical volumes.
|-
| pvdisplay || View physical volumes in the LVM.
|-
| vgcreate || Create a logical group with specified physical volumes.
|-
| vgextend || Add physical volumes to the volume group.
|-
| vgdisplay || View logical groups in the LVM.
|-
| lvcreate || Create a logical volume within the volume group.
|-
| lvextend || Expand a logical volume within the volume group.
|-
| lvreduce || Decrease a logical volume within the volume group.
|-
| lvdisplay || View all logical volumes.
|}

I'd highly, highly recommend following along with [https://www.howtoforge.com/linux_lvm this tutorial] within an LVM. You will be very comfortable with all of these commands after this.

=== Demo setup ===

Let us assume we have <code>/dev/sda3</code>, <code>/dev/sdb1/</code>, <code>/dev/sdc1</code>, 3 drives ready to be added to an LVM group. We will name our volume group <code>lvm</code> because why not? This will create a folder, <code>/dev/lvm</code> that contains our logical volumes.

First, add your drives to the physical volume list:
{{RootCmd|pvcreate /dev/sda3 /dev/sdb1 /dev/sdc1}}
Then create a volume group with them all:
{{RootCmd|vgcreate lvm /dev/sda3 /dev/sdb1 /dev/sdc1}}
Say you want your home directory in a different volume:
{{RootCmd|lvcreate -L 30G lvm -n lv_home}}
And then use the rest of the logical volume:
{{RootCmd|lvcreate -l 100%FREE lvm -n lv_root}}
You got a new hard drive? Cool. It's <code>/dev/sdd1</code>. Add it to the volume group:
{{RootCmd|vgextend lvm /dev/sdd1}}
You ran out of storage in a volume group? Resize it:
{{RootCmd|lvextend -L 35G /dev/lvm/lv_home}}

== RAID ==
[https://wiki.gentoo.org/wiki/User:SwifT/Complete_Handbook/Software_RAID RAID] is for data protection. This is not a <em>backup</em>. It is just for protection. Instead of one drive dying and you lose all your data, you can use a RAID system so you have protection against it dying.

Instead of a table of the different RAID systems, here's a [https://www.cru-inc.com/table-raid-levels/ list of them]. I'm going to assume we use RAID 1 or RAID 10.

RAID is done all under [https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm mdadm]. It's straightforward comapred to LVM.
{{RootCmd|mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1}}
{{RootCmd|mdadm --create /dev/md0 --level=10 --raid-devices=2 /dev/sdb1 /dev/sdc1}}

== ZFS ==
You can use LVM and RAID by manually making sure you don't go over 50% or whatever RAID you want requires, then using mdadm on your logical volumes. Or just use ZFS to create a pool. Even less work.
{{RootCmd|zpool create pool /dev/sdb /dev/sdc}}
RAID 1:
{{RootCmd|zpool create pool mirror /dev/sdb /dev/sdc}}
RAID 10:
{{RootCmd|zpool create pool mirror /dev/sdb /dev/sdc mirror /dev/sdd /dev/sde}}
Check out what you did:
{{RootCmd|zpool status}}
Here is [https://mirrors.lug.mtu.edu/ our Linux mirrors] configuration:
<syntaxhighlight lang="text">
pool: lug
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
lug ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
label/lug-HGST-4TB-01 ONLINE 0 0 0
label/lug-HGST-4TB-02 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
label/lug-HGST-4TB-03 ONLINE 0 0 0
label/lug-HGST-4TB-04 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
label/lug-HGST-4TB-05 ONLINE 0 0 0
label/lug-HGST-4TB-06 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
label/lug-HGST-4TB-07 ONLINE 0 0 0
label/lug-HGST-4TB-08 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
label/lug-HGST-4TB-09 ONLINE 0 0 0
label/lug-HGST-4TB-10 ONLINE 0 0 0

errors: No known data errors

pool: zroot
state: ONLINE
scan: scrub repaired 0B in 00:01:47 with 0 errors on Sun Sep 26 16:42:05 2021
config:

NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da10p3 ONLINE 0 0 0
da11p3 ONLINE 0 0 0

errors: No known data errors
</syntaxhighlight>

== fstab ==
Automatically mount your file systems at start up and tells you which file system corresponds to which drive, etc.

<code><fs> <mountpoint> <type> <opts> <dump/pass>.</code>

[https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/System#Creating_the_fstab_file It is recommended] you use UUIDs for non-LVM non-ZFS. [https://xan.manning.io/2017/05/29/best-practice-for-mounting-an-lvm-logical-volume-with-etc-fstab.html It is recommended] you use the device mapper symlink for LVM or ZFS. I am not smart enough to know the benefits for either.

Options are seen in "Filesystem independent mount options" and "Filesystem dependent mount options" in the [https://linux.die.net/man/8/mount mount(8)] man page. I'd just use <code>defaults</code>. Use <code>_netdev</code> for a network drive (cough David cough).

The numbers at the end are not too necessary, but read the [https://linux.die.net/man/5/fstab man page] on the 5th and 6th fields if you really want to know. [https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/System#Creating_the_fstab_file Gentoo] recommends having / at 0 1 and everything else as 0 2.

Here's my computer's setup:
<syntaxhighlight lang="fstab">
/dev/lvm/lv_root / ext4 defaults 0 1
/dev/lvm/lv_home /home ext4 defaults 0 2
UUID=46ef4b37-b2bc-4a3a-b821-604c9fb64787 /boot ext4 defaults,noatime 0 2
UUID=61518d0a-1a81-4936-85e0-6bb7e1d69155 none swap sw 0 0
</syntaxhighlight>