Hardware for Data Storage

Over the past few months you have worked with SQL, interacted with data via Python and R, and looked at the impacts of indexing. It is now worth discussing the hardware that runs these applications and stores the data as well as how we ensure the integrity of the data stored on the machines and how we reduce risk by working to protect ourselves from hardware failure.

How we connect to our storage devices

We won't spend to much time on the history of drives, so we will start with what is actually in use today. One of the most common interfaces is SATA (Serial Advanced Technology Attachment) which was introduced in 2003. This is the connection used to connect storage devices such as hard disk drives (HDD), optical drives, and solid state drives (SSD) to the motherboards of servers and desktops.

With SATA, there are three revisions you may encounter, with SATA III as the most common seen today. The largest difference between the revisions is the speed of the interface.

These speeds show above is the maximum speed that the interface supports, it is not the speed that will be achieved by the drives. Depending on the drive type, the drive may be the bottleneck for the system.

Available storage devices

Hard Disk Drives (HDDs) and Solid State Drives (SSDs) are the two primary types of storage devices you are likely to encounter. Lets compare:

1. Technology:
   • HDDs use mechanical components to read and write data. They consist of spinning magnetic disks (platters) that store data, and a read/write head that moves across the platters to access or modify data. Data is stored magnetically on the platters.
   • SSDs use NAND flash memory to store data. They have no moving parts, and data is stored electronically on memory chips. SSDs are more akin to USB drives or memory cards in terms of their technology.
2. Speed and Performance:
   • HDDs are slower compared to SSDs. The physical movement of the read/write head and the spinning platters introduce latency, which results in slower data access times. This makes HDDs less suitable for tasks that require fast data transfer rates, such as booting up an operating system or loading large applications or datasets.
   • SSDs are significantly faster than HDDs. They offer nearly instantaneous data access because there are no moving parts. This results in quicker boot times, faster application loading, and improved overall system responsiveness.
3. Durability and Reliability:
   • HDDs are less durable and reliable due to their mechanical components. They are sensitive to shock, vibration, and temperature variations. Over time, the mechanical parts can wear out, leading to potential data loss or drive failure.
   • SSDs are more durable and reliable. Since they have no moving parts, they are not as vulnerable to physical shocks or temperature fluctuations. However, they have a limited number of write cycles (program/erase cycles) for each memory cell, which can lead to cell degradation and, eventually, drive failure. Modern SSDs use wear-leveling algorithms to mitigate this issue.
4. Size and Form Factor:
   • HDDs are typically larger and heavier due to their mechanical components and the need for space to accommodate spinning platters and read/write heads.
   • SSDs are smaller and lighter because they lack mechanical parts. This makes them suitable for slim laptops and other compact devices.
5. Noise and Power Consumption:
   • HDDs generate noise when the platters spin and the read/write head moves. They also consume more power due to the mechanical components.
   • SSDs are completely silent and consume less power, which can extend battery life in laptops and reduce the heat generated by the storage device.
6. Cost:
   • HDDs are generally more affordable in terms of cost per gigabyte of storage.
   • SSDs are more expensive than HDDs, but the cost has been decreasing over time.

Since most of our systems will exist in data centers far away from the users, we will probably want to focus most of our thoughts on cost and speed. Reliability is also really important, but we actually have other ways of dealing with that!

Pooling drives together

Drive pooling allows you to combine multiple physical storage drives into a single logical storage pool. This pool appears as a single, unified storage volume to the operating system, making it easier to manage. Drive pooling is often used in situations where you have multiple hard drives or SSDs and want to create a larger, more flexible storage solution.

One of the best part of pooling drives together is that we can use many smaller, less expensive, and/or slower drives, to create a single and possibly faster volume. Once the pool is created, the data is distributed across the pooled drives. This can be done in various ways, including:

• Striping: Data is split into blocks and distributed across multiple drives, improving read and write performance. However, this method does not provide data redundancy, so a drive failure can result in data loss.
• Mirroring or Redundancy: Data is duplicated or mirrored across multiple drives to ensure data integrity. If one drive fails, the data remains accessible from a duplicate drive.
• Combination: Some pooling solutions offer a combination of striping and redundancy, allowing you to balance performance and data protection.

Pooling in Practice - RAID

RAID, or Redundant Array of Independent (sometimes called inexpensive) Disks, is a method that combines multiple physical hard drives into a single logical unit to enhance data storage, performance, and redundancy. RAID configurations are commonly used in server environments and high-capacity storage systems. The way RAID works depends on the specific RAID level, each of which offers different features and trade-offs. Here's an overview of how RAID works:

1. Selection of Drives: To set up a RAID array, you need two or more physical hard drives. The specific configuration and the number of drives used depend on the desired RAID level.
2. Data Striping: Data striping is a key feature of RAID that involves dividing data into blocks or stripes and distributing those blocks across the drives in the array. This improves read and write performance because multiple drives can work in parallel. The size of the stripes and how data is distributed vary depending on the RAID level.
3. Data Redundancy (not required but recommended): Some RAID levels incorporate redundancy mechanisms to protect against data loss in case of drive failures. These mechanisms include mirroring, parity, or a combination of both.
   • Mirroring: Each data block is duplicated on a separate drive. If one drive fails, the data is still accessible from the mirror drive.
   • Parity: Use computed parity information to reconstruct data if a drive fails. Parity is a mathematical calculation based on the data blocks, and it's stored on one or more dedicated drives in the array. If a drive fails, the parity data can be used to rebuild the lost data.
4. Common RAID Levels: There are several common RAID levels:
   • RAID 0: Provides data striping for improved performance but no redundancy. It uses multiple drives to store data without mirroring or parity. If one drive fails, all data in the RAID 0 array is lost.
   • RAID 1: Provides data mirroring for redundancy. It duplicates data on two or more drives to ensure data is still accessible if one drive fails.
   • RAID 5: Uses data striping and parity for a balance of performance and redundancy. It can tolerate a single drive failure without data loss.
   • RAID 6: Similar to RAID 5 but with double parity, allowing for the simultaneous failure of two drives without data loss.
   • RAID 10 (or 1+0): Combines RAID 1 (mirroring) and RAID 0 (striping). It provides both redundancy and improved performance by striping data across mirrored pairs of drives.

Speed at Scale

We talked about the SATA interface earlier, specifically because SATA/SAS is the most common drive type used in storage systems using hardware RAID. If we were to pool together 24 SATA drives, you could imagine that the throughput could become as high as 24 * 600MB/s = 14.4GB/s. But in practice you are most likely not going to achieve these ideal numbers. Lets do a quick comparison between Hard Drives and Solid State Drives again:

1. Data Access Time: SSDs have virtually instantaneous data access times since they store data electronically on NAND flash memory chips. This means there is very low latency. HDDs, on the other hand, have mechanical components, including spinning platters and a read/write head that must physically move to the right track on the platter to access data. This mechanical movement introduces significant latency, making data access times relatively slow.
2. Data Transfer Speed: SSDs provide for faster data transfer speeds than HDDs. Sequential read and write speeds for SATA SSDs can be sustained at 500 MB/s or even higher. SATA HDDs in contrast, the sequential read and write speeds for SATA HDDs typically range from 100 MB/s to 200 MB/s.
3. Random Access Speed: SSDs excel in random read and write operations, making them exceptionally fast when it comes to small, random data requests.

What we see in practice is that we can take a large number of more inexpensive drives, and by using RAID, we can increase the overall speed of the array and provide for some redundancy in the event of failure. If we were using an array of hard drives, it would be common to see numbers in the order of 1-2GBs for a raid with 20 hard drives, and closer to 5GBs for a similar number of SSDs. Typically we would use hardware raid controllers that have battery-backed-up memory, these cards would further cache the material that was waiting to be written to disk. This provide a bit more speed, while also allowing some protection against significant events like power failures because that data will remain on the card for up to 24 hours and allow administrators to fix larger problems without loosing data.

Faster Interfaces

We have looked at Solid State Drives that are connected via SATA/SAS interfaces, but what would happen if we were to connect a solid state drive directly to the main bus of a computer?

Modern computers use the PCIe (Peripheral Component Interconnect Express) interface to connect hardware components such as network or video cards via a direct link to the CPU. Each of these PCIe interfaces are connected via a number of lanes to the CPU (or similar controller chip). For terms of reference, the 3 primary PCIe standards in use now, PCIe Version 3, 4, and 5 have speeds of 1GB/s, 2GB/s, and 4GB/s per lane. And most slots are rated at X1, X4, X8, or X16 meaning that they have either 1, 4, 8, or 16 lanes available.

What if we connected the flash storage within a solid state drive directly via a PCIe interface? This is actually the pupose of NVMe (Non-Volatile Memory Express). The NVME standard is a X4 connect via either PCI 3, 4, or 5. Lets just do some tabletop math before we move forward! Thats 4 x 1GBs, 4 x 2GBs, or 4 x 4GBs per drive! Although in practice the rates are closer to 3500MBs, 7500MBs, and 13500MBs, but again, this is per drive.

Currently with these drive setups, we can achieve considerably faster input/output operations, but it becomes much harder to use hardware RAID. Typically with NVMe storage, software RAID is used to combine the drives at the expense of CPU usage.

Local Examples

The Computer Science Department's main file server uses hardware RAID6 against nine 4TB SATA SSDs. While our virtualization servers use 24 NVMe Drives.

Practice Problems

1. The Data Science file server nfs.cs.usna.edu, has a /home drive that was constructed from 9 drives in RAID6, this drive is mounted as /home on every lab workstation as well as ssh.cs.usna.edu and desktop.cs.usna.edu.
   • Bring up a terminal on your workstation (or ssh into ssh.cs.usna.edu) and type the following to show all of the mounted drives on the machine.
     
     

     df -BG

   • You will see something like
     
     

     Filesystem                    1G-blocks  Used Available Use% Mounted on
     tmpfs                                2G    1G        2G   1% /run
     /dev/sda2                          234G   41G      181G  19% /
     tmpfs                                8G    1G        8G   1% /dev/shm
     tmpfs                                1G    1G        1G   1% /run/lock
     /dev/sda1                            1G    1G        1G   2% /boot/efi
     nfs.cs.usna.edu:/home            ?????G  ???G    ?????G  ??% /home

   • Under the 1G-blocks column you will see the total available space in this drive pool. Since this system has 9 drives in RAID6, what is the size of each of the 9 drives?
2. Each of the drives on nfs.cs.usna.edu is a solid state drive, so lets assume that we can get a sustained write speed of 500MB/s, lets imagine that our RAID controller adds no latency and can stripe across the data drives perfectly. What is the maximum possible write speed to this array?
3. The Data Science backup server, backups.cs.usna.edu, which you can see mounted on ssh.cs.usna.edu as
   
   

   Filesystem                           1G-blocks   Used Available Use% Mounted on
   backups.cs.usna.edu:/snapshot/shared   204081G 13097G   190984G   7% /backups      

   Has an array of 12 Hard Disk Drives (with about 150MB/s sustained write speeds) in RAID6. What are the sizes of the individual drives and whats the total possible throughput on the system?
4. The Data Science major has multiple virtualization servers, you may not have ever connected to them at this point, their drive configuration is a bit more since they are using NVMe (PCI Gen 3) drives. These machines have 24 U.2 NVMe Drives in the equivalent of a RAID0! What is the maximum possible throughput across this array?
5. Our file server is connected to the Academy Network via a 1Gb/s network port, Are we able to fully utilize the throughput of our server? What about when the network gets upgraded to 10Gb/s in the next year? Our file server is connected to our backup server via a seperate 40Gb/s link, how does that compare?