Linux devices

1. Types of devices
2. Hardware discovery and detection
3. Persistent storage : blocks vs sectors
4. Manage storage devices

• Devices are special files stored in /dev that provide interfaces to the physical hardware.
• There are two main types of devices under all Unix systems, character and block devices.

• Provide unbuffered direct access to the corresponding hardware.

• Reads and writes are always performed synchronously.

• Examples of character devices are :

  path	description
  /dev/mem	System RAM
  /dev/fb0	Graphics card frame buffer
  /dev/sg0	Generic SCSI layer
  /dev/input/*	Input peripherals (keyboard, mouse, etc)
  /dev/tty*	Native terminals
  /dev/pts*	Emulated terminals (ssh, etc)

• Provide buffered/cached access to the corresponding hardware.

• As a result reads and writes can be asynchronous.

• Examples of block devices are :

  path	description
  /dev/sda	Hard disk (SCSI driver)
  /dev/sda1	System boot partition (fat32)
  /dev/sda2	System and data partition (ext4)
  /dev/sda3	Swap partition
  /dev/sdb	Second hard disk (unpartitioned)

Note : in Unix-like systems, devices can exist without physical hardware (pseudo-devices).

• The kernel provides an abstract interface to the system hardware through device drivers which implement low-level interactions.
• Device drivers are part of the linux kernel, and make up for more than half of its source code.
• They allow hardware discovery and hardware addition / removal event detection among other things.
• Once a device driver is loaded, the kernel sends hardware events to the userspace daemon udevd which maintains the virtual filesystem mounted at /dev.
• Processes can then access the hardware through system calls or by reading/writing to the device files.

Note : many shell commands are in fact wrappers around system calls.

• Linux always creates block devices for storage hardware (file systems can only be mounted if they exist on a block device).
• It is important to understand the concept of blocks and sectors and how they relate to block devices :

• It is the minimal number of contiguous bytes that can be read / written on a physical storage disk.

  sector size	media
  512 bytes	HDD
  2048 bytes	CD-ROM/DVD-ROM
  4096 bytes	SSD/newer HDD (AF - advanced format)

• It depends on storage media manufacturers.

• Each sector stores metadata (address, error correction information, flaw indicator ...) as well as user data.

• AF uses 4096 bytes sectors because of the increase in disks size (less sector metadata to store).

• For backwards compatibility, AF disks emulate 512 bytes sectors to the operating system (logical vs physical sectors size).

• If a file does not fill a whole sector, the remainder of the sector is filled with zeroes.

• It is the minimal number of contiguous disk sectors that can be read/written when accessing a file system.
• It depends on file system options (for instance in ext4 it is a group of sectors between 1KiB and 64KiB).
• Block size cannot be smaller than the corresponding disk sector size and is often a multiple of it.
• If a file size is not a multiple of the file system block size, its last block will only be partially filled (fragmentation).

# print storage media sector size
sudo fdisk -l /dev/sda | grep "Sector size"

# print block size for a block device
sudo blockdev --getbsz /dev/sda2

# print the number of blocks used by a file or a directory (recursive)
bash -c 'du -a -B "$(stat -f --format="%S" "$0")" "$0"' "$HOME/some_file" | sort -gr

# print hex + ASCII dump of blocks 65536 to 131072 from /dev/sda2 (4096 equals 0x1000)
# it's funny to scroll down and see your files contents appear on the file system blocks
sudo xxd -c 32 -seek 0x10000000 -l 0x10000000 /dev/sda2 | less

# you can also use the same command to see which blocks contain specific text values
# (remember that utf8 and utf16 are supersets of ASCII ...)
sudo xxd -c 32 -seek 0x10000000 -l 0x10000000 /dev/sda2 | grep confidential

Notes :

• Linux always assume 512 bytes (logical) block size by default regardless of the filesystem block size.
• For example, if the disk uses AF, any non empty file is reported by stat as using at least 8 blocks.
• Filesystems with a small block size use space more efficiently, but are slower.

• The kernel detects new storage devices and creates the corresponding block devices at /dev/sd* :

# search kernel messages for all detected SCSI hard drives
sudo dmesg -kHxL=always | grep -E "sd[a-z].+SCSI"

• Run fdisk on the block device to partition : sudo fdisk /dev/sdb
• Run command F to list unpartitioned space on the device :

# output
Unpartitioned space /dev/sdb: 5 GiB, 5367660544 bytes, 10483712 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
Start      End  Sectors Size
2048 10485759 10483712   5G

• Run command g to create the device partition table :

# output
Created a new GPT disklabel (GUID: 13DEDBC4-CAC8-4ED3-9D72-A7AD218CEDCE)

• Run command n to create a new partition :

• All sizes are in bytes expressed as powers of 2 (see also this practical conversion table).
• For instance, the last sector of partition 1 (2 Gib) on a 5 Gib disk is 2048 + (2147483648 / 512) - 1 = **_4196352_**.
• Once done, print, verify and save the partition table using p, v and w.
• List the newly created partitions using sudo fdisk -x /dev/sdb :

# truncated output
Device       Start      End Sectors Type-UUID                            UUID                                 Name Attrs
/dev/sdb1     2048  4196351 4194304 0FC63DAF-8483-4772-8E79-3D69D8477DE4 A5D1FA5C-8CEF-46F1-A203-AA2893F7DC78
/dev/sdb2  4196352 10483711 6287360 0FC63DAF-8483-4772-8E79-3D69D8477DE4 486C7828-2E1B-44DB-A565-8469A2801E00

• ext4 is the filesystem of choice for modern linux ditributions :

# set block group size and root owner, create a random UUID for the partition
sudo mkfs.ext4 -cv -G 4096 -E root_owner=0:0 -U random /dev/sdb1
sudo mkfs.ext4 -cv -G 4096 -E root_owner=0:0 -U random /dev/sdb2

• View details on file systems options and defaults : man mke2fs.conf
• Edit /etc/fstab and add mount points for the new partitions :

# truncated fstab file
UUID=62a060c0-e3d6-41df-a027-cf27ea326ee8  /                 ext4  errors=remount-ro        0  1
UUID=C967-D80F                             /boot/efi         vfat  umask=0077               0  1
UUID=9783f895-adc7-496d-a1a3-19c04c54b606  none              swap  sw                       0  0
/dev/sr0                                   /media/cdrom0           udf,iso9660 user,noauto  0  0
# mounts for new partitions (lowercase UUIDs can be used as well)
/dev/sdb1                                  /new_partition_1  ext4  defaults                 0  1
/dev/sdb2                                  /new_partition_2  ext4  defaults                 0  1

• Important : the mount points must exist and the mount will overwrite whatever happens to be in it.
• At this stage, persist the mounts with systemctl daemon-reload and mount the partitions with sudo mount -a.