This HOWTO describes the (networked) storage performance measurements methodology and tips.
As the members of TERENA's TF-Storage we try to collect general informations HOW TO
prepare and perform the storage performance testing, basing on our own experience and
knowledge as well as the material accessible in the Internet.
This HOWTO re-uses the presentation prepared by IBBT/Ghent University,
Storage Benchmarking Cookbook.
Obviously, the world is too complicated to be described in a single work.
However, we hope that both background and practical, hands-on informations included in this HOWTO
can be useful for the community.
HOWTO does not cover all possible aspects, problems and approaches to the
storage benchmarking process. We try to describe the most important issues,
most popular approaches and tools used to evaulate the storage systems.
The organisation of the sections of this HOWTO follows the real-world, networked storage systems layers layout (see table below).
We provide general and layer-by-layer comments on how to perform storage benchmarking
along with some background information, practical benchmarking tips and examples
as well as links to the external sources.
Typical storage environment is composed of layers similar to presented above.
It is important to understand, that each of these layers can have important influence of the data access performance.
In following subsections we shortly discuss the layers and mention the main factors,
that have the most important impact on the efficiency of the whole storage system.
Client applications run in the client hosts. These hosts are the NAS clients. NAS services include file system-level
services, basic user authentication and access control.
Following application details may influence the data access performance:
Also the hardware and software components of the client systems such as:
Therefore, the networked storage benchmarking procedure should include examining the features
of the hardware and software components of the client system.
Note, that client hosts can use block-level services provided by the storage system: DAS (Directly Attached Storage) or SAN (Storage Area Network) instead of NAS services. This case is discussed in details in one of
the following sections.
Client network interconnects the clients and the filesystem cluster, called also 'NAS cluster'.
The efficiency of data access performed by the client network may depend on:
Most popular client network is the IP connectivity implemented over Gigabit Ethernet.
In the HPC clusters IP over Inifinband is often used. Concerning IB networks, some
solutions to encapsulate high-level NAS protocol traffic directly in the IB frames exist, e.g.
NFS over RDMA.
As the client-NAS server network can introduce data transmission delays and
significant performance limits, its features must be carefull examined
in the process of benchmarking the networked storage solution.
Filesystem and cluster nodes implement the NAS (Network Attached Storage) functionality,
including filesystem services, users authentication and data access control.
Physically, there are two most typical ways of implementing the NAS services:
The physical and logical characteristics of the implementation of NAS services
have very important impact on the performance of the networked storage.
Therefore, their features must be carefully tested in the confines of the networked storage bemchmarking evaluation.
The NAS elements may consitute a bottle-neck, among others due to:
Note, that while being a NAS servers, NAS gateways act as the block-level services clients in the same time. Block-level services are provided by storage controllers accesible through storage front-end network, e.g.
a FC-based SAN.
The performance of file system-level services strongly depends on the performance
of the block-level volumes. Therefore, the evaluation of the performance of the
networked storage environment should include examining the block storage components.
This is possible in open systems, where we can have access to the block-level storage directly.
In typical networked storage setups also the application/client systems can be connected to the storage network directly, exploiting the block-level storage. In such case, storage benchmarking can be focused
on the performance evaluation of SAN compoments, SAN network interfaces (so called HBAs)
(Host Bus Adapters) and SAN driver characteristics.
The block-level storage can be served directly, locally (DAS - Direct Attached Storage) or by the Storage Area Network. In both cases, the performance evaluation of the block-level storage can be performed using similar methods and tools, as the virtualisation mechanismes implemented in modern operating systems hide the fact that the SAN storage is not local.
SAN network may constitute a performance bootleneck, depending on:
Beside the FC technology, iSCSI and FCIP, iFCP and SRP (over IB) protocols are used to implement the data transport in the front-end storage networks.
Storage controllers implement the basic virtualisation functions such as RAID structures, volumes (LUNs) and volume
mapping to hosts and volumes access control. They are crucial elements of the storage system, so their features may determine its performance characteristics.
Advanced functions of modern controllers can have the significant impact on the data access efficiency.
They should be taken into account while planning the storage controllers testing. Examples are:
Solid benchmarking of the storage controllers should include (but should not be limited to):
RAID structures are the basic virtualisation technique used in storage controllers. The techniques and tricks used to optimise particular RAID implementations can be a differentiating factor of various storage controllers available on the market.
Therefore, controller settings that relate to RAID structures should be carefully examined in the confines
of storage system testing. Possible configuration settings include:
Also the logical volumes can be configured in different ways. The settings that should be taken into account while performing the storage testing are:
Particular storage controllers may include advanced configuration parameters, that can be used
for performance tunnig. Examples of such parameters are:
Significant amount of time should be spent in order to get a realistic picture
of a given controller performance characteristics.
Back-end storage network interconnects the storage controllers with he disks drives.
Various type of the technologies are used for that purpose. Traditionally, Fibre Channel
loops span disk drives and storage controller back-end ports. In enterprise solutions,
switched back-end topologies are used. Currently, SAS technology is often used
for the back-end connectivity as it provides high bandwidth at optimal cost.
The storage back-end links can be a bootle-neck, especially in a sustained transfer
operations. Therefore its characteristics such as latency and bandwidth should be taken into account while
running the performance tests.
As the disk drives hold the actual data, their characteristics are very important
component of storage system performance.
The efficiency of accessing data held on disk drives may depend on:
As the data processed in storage systems ultimately goes to disk drives, it might be useful to examine the performance of the individual disk drives (if possible), in order to determine the realistic performance that can be achieved after combining multiple disks into RAID arrays.
In above sections we showed that networked storage systems are typically composed of multiple layers.
Each of this layers has its own features, complexity and performance characteristics.
Solid storage benchmarking should take into account these facts. Appropriate testing metodology
should be used in order to reveal the real characteristics of particular components
or to find a source of the actual or potential bottle-neck observed in the system.
Overview of the benchmarking methodologies is provided in the next section of the HOTWO.
The complexity of the networked storage system makes solid storage benchmarking a difficult task.
One of the basic decisions that must be taken while preparing the benchmarking procedure
is the PURPOSE of running the benchmark and the EVALUATION CRITERIA for the test results.
There are many possible benchmarking purposes. Here are some examples of them:
In order to evaluate the benchmark results, the EVALUATION CRITERIA must be set up.
The criteria can differ significantly, depending on the TARGET APPLICATION of the storage system.
For instance, when considering home directories storage, both sequential and random access pattern performance
are important. For high-performance computing, rich-media and backup/archive applications,
the sequental data access performance matters the most. Databases storage requires good
IOPS characteristics of the storage system.
Benchmarking procedure and tools should be able to reveal the characteristics of the examined
storage system that matches the defined criteria.
Depending on the purpose, the benchmarking process should be organised in different ways.
For instance, the evaluation of the storage system efficiency in a tunned configuration
require running multiple rounds of tests, while changing configuration parameters on single
or multiple storage system layers beetween rounds and doing the analysis phase after each test.
While planning the benchmarking process we shoul try to:
Both approaches has some advantages and disadvantages. The accepted tactics should fit the purpose
of the test.
If we want to quickly determine the overall performance of the storage system, we can
start from the top, e.g. by running a file-system level benchmark (more on benchmarks in the BENCHMARKING
TOOLS section), and go down in case the performance is lower than expected. This is most probably less
time consuming approach, however we may get an incomplete picture of the actual system efficiency.
Going bottom-up is more costly, but gives as the knowledge about the efficiency of each system layer and the overhead it introduces comparing to lower layers. This approach is suitable if we want to find a bottle-neck in the
system or we want to determine the realistic maximum performance in a tunned setup.
Having in mind the complexity of multi-layer storage system, we may be forced to:
Benchmarking the multi-layered system we have to make sure that we are measuring the correct
component of the system.
Typical issue faced while benchmarking on the file-system level is the influence of the data
caching which can take place on the client side (file system-level, operating system-level), NAS gateway side,
or on the storage controller.
In order to measure the selected storage system element, e.g. the storage backend network bandwidth,
we may want to disable caching or plan the benchmark in the way, where the influence of the caching
will be eliminated.
Some practical methods to achieve that are:
Another side effect of the data caching is that, the speed of writing the the buffer
can be measured instead of the actual tranmission speed. This can be avoided by using the data set large enought to fill up the buffers
and transmit enough data over the real communication channels. We may also use external monitoring tools,
such as Gigabit Ethernet network or SAN switches performance analysers, host ports usage monitoring tools to
analise the actual data traffic taking place in the network.
While doing the benchmark we have to make sure, that we are able to determine the reason of the
the performance bottle-neck visible in the results. On the other hand, we should try to avoid
incorrect intepretation of the results in case if some caching/buffering effect makes the results
better than actual system features. We can use monitoring tools in addition to the benchmarking tools for that purpose.
Example monitoring tools we could while benchmarking the storage systems are:
Note however, that the decision to switch off or eleminate the effects of caching should be taken carefully.
Disabling the cache, can make the operation of some optimisation technologies implemented in storage system elements
impossible. For instance, if we disable the write cache in the RAID controller, full-stripe write technique cannot
be exploited.
Switching the caching off, we also make the testing environment a bit artificial – the results we get
in such setup, can be useless in the real-life configuration.
Getting the realistic picture of the storage system performance, requires that we apply
the benchmark that generates the workload that matches our needs. Againt, the kind
of the testing workload should fit the purpose of performing the benchmark.
Possible purposes of the storage testing are:
If we have a specific application in mind, we should make sure to be able to prepare
the benchmark that reflects the application characteristics.
This can be achieved in some ways:
There is a lot of benchmarks types available for free or under the license.
They differ in:
The correct benchmark should generate the workload that is similar to the target application needs.
Some application-level benchmarks simulate the behaviour of typical computing systems applications.
The table below provides some examples of benchmarks along with the links to Internet resources related to them.
Level | Workload generator / benchmark | Auxiliary monitoring tools |
|---|---|---|
applications-level | real application (FTP,NFS client, DBMS) | top |
| SPC (seq/random R/W) |
|
| SPECsfs2008 (CIFS, NFS) |
|
| DVDstore (SQL) |
|
| TPC (transactions) |
|
network level | iperf | dstat, ethreal/wireshark, ntop |
| smartbits appliance | optiview link analyser |
filesystem level | dd |
|
| xdd |
|
| iozone |
|
| bonnie/bonnie++ |
|
device level | dd | dstat |
| iometer | iostat, vmstat |
| xdd |
|
| diskspeed, hdtune, hdtach, zcav | Linux's procfs directories |
| own tools |
|
In this section we discuss some benchmarks in details. We selected them basin on our experience
and interests. Therefore, the selection may be not optimal for every situation.
Storage Performance Council (SPC) try to standardize storage system evaluation.
The organisation defines industry-standard storage workloads.
This "forces" vendors to publish a standardized performance of their storage systems.
SPC-1 and SPC-2 benchmarks evaluate complete storage systems, while SPC-1C and SPC-2C evaluate storage subsystems,
e.g. individual disk drives, HBAs, storage software (e.g. LVM, ...)
The table below summarizes SPC-1 and SPC-2 specifics.
| SPC-1 | SPC-2 |
|---|---|---|
Typical applications |
|
|
Workload | Random I/O | Sequential I/O (1+ streams) |
Workload variations |
|
|
Reported metrics |
|
|
If we decide to perform layer-by-layer testing of the storage system or we want to find a source
of the bottleneck observed in the system we may want to examine a single element of the system and
avoid the influence of the other elements.
Example problem is to examine the network ability to carry the data traffic using a given protocol,
without the influence of the disk access latency. To do so, we may use some tricks, for instance:
As mentioned before, the client network can be a source of significant performance bootlenecks.
However, the degree the network parameters impact the performance observed by the application
is related to the fact, what is the application sensitive for.
In case, the application is bandwidth intensive, we should examine mainly the bandwidth of the network, while
for I/O intensive application the network delay should be carefully tested.
The parameters of the network link we should evaluate depend also on the kind of the protocol used
for data transmission. For instance, in case of NFS, both bandwidth and delay of the network link matters,
as typicall NFS performs a synchronous data transmission. Another example is the GridFTP protocol,
which can exploit multiple parallel tranmission streams. Therefore, the network benchmark should
be able to examine the network bandwidth using multiple parallel transmission streams (for instance, we
can use iperf benchmark with multiple TCP/IP streams).
This section of the HOTWO contains the more detailed informations about benchmarking selected storage system
compoments. Each of the subsections contains some background informations as well as practical informations
on testing the storage system element.
This part of the HOWTO describes how to perform clients network testing. Some background information is
provided along with the practical informations related to TCP network testing using iperf tool.
Iperf is a open-source benchmark working both under Linux and Windows operating system.
In this HOWTO we focus on testing the network under Linux, i.e. between two machines running Red Hat Linux Enterprise.
As mentioned in the introduction, the efficiency of data access performed by the client network may depend on
network topology, technology, transmission protocol and network delay which in turn can result from both the physical distance between communication parties and the feature of communication equipment that processes the network traffic.
As the most popular client network in storage systems is the IP connectivity implemented over Gigabit Ethernet
we focus on this technology in this part of the HOWTO. We will show, how to examine data transmission delays
present in the network and the bandwidth available in the link. As some data transmission protocols are able to exploit
multiple parallel data streams (TCP connections), we also show how to examine the influence of the number
of the TCP streams used in parallel on the actual bandwidth observed by the application.
Because Iperf is an client-server application, you have to install iperf on both machines involved in tests. Make sure that you use iperf 2.0.2 using p-threads or newer due to some multi-threading issues with older versions. You can check the version of this installed tools with a following command:
[:userathostname ~]$ iperf -v iperf version 2.0.2 (03 May 2005) pthreads |
Because in many cases the low network performance is caused by high CPU load, you should measure CPU usage at both link ends during every test round. In this HOWTO we use an open-source vmstat LINK-! tool which you probably have already installed on your machines.
Before we start the tests, we should take a look on our network link setup. First, we should ensure that we can use MTU larger than the standard Ethernet MTU. We should try to use MTU 9000. Using jumbo frames is recommended especially in reliable and fast networks, as bigger frames boost network performance due to a better header-to-body-frame ratio. But we should remember that, it is possible to use MTU 9000 only if all network hardware between tested hosts (routers, switches, NICs etc.) support jumbo frames.
In order to enable MTU 9000 on the machine network interfaces you may use ifconfig command.
[root@hostname ~]$ ifconfig eth1 mtu 9000 |
Alternatively, you can put this settings into interface configuration scripts, e.g. /etc/sysconfig/network-scripts/ifcfg-eth1 (on RHEL, Centos, Fedora etc.).
If jumbo frame are working properly, you should be able to ping one host from another using large MTU:
[root@hostname ~]$ ping 10.0.0.1 -s 8960 |
In the example above, we use 8960 instead of 9000 because ping tool option -s needs frame size minus frame header which lenght is equal to 40 bytes. If you cannot use jumbo frames set the mtu to default value 1500.
To tune your link you should measure the average Round Trip Time (RTT) between machines. RTT can be obtained by multiplying the value returned by a ping command by 2. When you have RTT measured, you can set TCO read and write buffers sizes. There are three values you can set: minimum, initial and maximum buffer size. The theoretical value (in bytes) for initial buffer size is BPS / 8 * RTT, where BPS is the link bandwidth in bits/second. Example commands that set these values for the whole operating system are:
[root@hostname ~]# sysctl -w net.ipv4.tcp_rmem="4096 500000 1000000" [root@hostname ~]# sysctl -w net.ipv4.tcp_wmem="4096 500000 1000000" |
Probably, it is best if you start with values computed using the formula mentioned above and then tune these values according to the tests results.
You can also experiment with maximum socket buffer sizes:
[root@hostname ~]# sysctl -w net.core.rmem_max=1000000 [root@hostname ~]# sysctl -w net.core.wmem_max=1000000 |
Another options that should boost performance are:
[root@hostname ~]# sysctl -w net.ipv4.tcp_no_metrics_save=1 [root@hostname ~]# sysctl -w net.ipv4.tcp_moderate_rcvbuf=1 [root@hostname ~]# sysctl -w net.ipv4.tcp_window_scaling=1 [root@hostname ~]# sysctl -w net.ipv4.tcp_moderate_rcvbuf=1 [root@hostname ~]# sysctl -w net.ipv4.tcp_sack=1 [root@hostname ~]# sysctl -w net.ipv4.tcp_fack=1 [root@hostname ~]# sysctl -w net.ipv4.tcp_dsack=1 |
COMMENT: the meaning of these parameters is explained in the Linux documentation (sysctl command) LINK-!.
After setting up the network link parameters, we are ready to run the test. Obviously, configuring
the network settings can be an iterative process, where we check different settings by running the
tests and evaluating their results.
To perform the test, we should run iperf in server mode in one host:
[root@hostname ~]# iperf -s -M $mss |
On the other host we should run command like this:
[root@hostname ~]# iperf -c $serwer -M $mss -P $threads -w $\{window\} -i $interval -t $test_time
|
There is a description of names and symbols used in the command line:
As we want to perform several actions before, during and after a single test. These actions include:
To simplify the remote machine access (i.e. to avoid providing the password each time we want to execute a command on a remote host,
we may setup ssh keys properly.
To generate key-pair we use a following command:
[root@hostname ~]# ssh-keygen -t dsa |
Then we copy the public key to the remote server and add it to authorized keys file:
[root@hostname ~]# cat identity.pub >> /home/sarevok/.ssh/authorized_keys |
Now we can login from the remote serve to bug without password. So can also run programs remotely, e.g. from the bash script.
Below, we provide some simple scripts to run iperf and vmstat on both machines. The iperf and vmstat runs in rounds for various number of threads.
Note, that after each round both vmstat and iperf are killed and restarted to obtain identical testing environment.
The test is performed for different number of threads (up to 128), because the worse the link RTT, the greater performance boost can be achieved
by using multiple transmission threads. We may expect that, if RTT of the link is poor, we will achieve performance boost when
many threads are used.
There is a simple shell script to run iperf test:
#!/bin/sh file_size=41 dst_path=/home/stas/iperf_results script_path=/root curr_date=`date +%m-%d-%y-%H-%M-%S` serwer="10.0.1.1" user="root" test_time=60 interval=1 mss=1460 window=1000000 min_threads=1 max_threads=128 |
for threads in 1 2 4 8 16 32 64 80 96 112 128 ; do
ssh $user@$serwer $script_path/run_iperf.sh -s -w $\{window\} -M $mss &
ssh $user@$serwer $script_path/run_vmstat 1 vmstat-$window-$threads-$mss-$curr_date &
vmstat 1 > $dst_path/vmstat-$window-$threads-$mss-$curr_date &
iperf -c $serwer -M $mss -P $threads -w $\{window\} -i $interval -t $test_time >> $dst_path/iperf-$window-$threads-$mss-$curr_date
|
ps ax | grep vmstat | awk '\{print $1\}' | xargs -i kill \{\} 2&>/dev/null
ssh $user@$serwer $script_path/kill_iperf_vmstat.sh
done
|
Script run_iperf.sh can look like this:
#\!/bin/sh |
iperf $1 $2 $3 & |
run_vmstat.sh script can contain:
#\!/bin/sh vmstat $1 > $2 & |
kill_iperf_vmstat.sh may look like this:
#\!/bin/sh
ps -elf | egrep "iperf" | egrep -v "egrep" |awk '\{print $4\}' | xargs -i kill -9 \{\}
ps -elf | egrep "vmstat" | egrep -v "egrep" |awk '\{print $4\}' | xargs -i kill -9 \{\}
|
To start test script that can ignoring hangup signals, you can use nohup command.
[:stasatworm]$ nohup script.sh & |
This command keeps the test running when you close the session with server.
To obtain reliable and recurrent test results be sure that both machines are not performing any other tasks when the test is running. Run a number of tests and count an average the values acquired from all of them.
To present obtained results you may use the open-source gnuplot LINK-! program, as the graphical results presentation may help yuo to interpret them. Generating the plots can also be automated.
...
One of the most known and important storage virtualisation techniques is the RAID (Redundant Array of Independent Disks) technology. It allows to combine independent disk resources into structures that can provide advanced reliability and performance features, which are not possible to deliver using individual disk resources (e.g. individual drives).
In this section we describe standard RAID levels including RAID 0, 1, 5 and 6 as well as nested RAID structures such as 10, 50 and 60. At the end of the section we summarize fault tolerance, performance and storage efficiency characteristics of particular RAID structures.
RAID0 (striping) does not provide any data redundancy. The main purpose of using RAID0 structures is to distribute the data traffic load over RAID components. Each file is split into blocks of a certain size and those are distributed to the various drives. This block is called chunk. A number of chunks for whom one parity chunk was counted is called stripe. Chunk size is a user-defined parameter characteristic to array. The chunks are send to all disk in the array in a way that one chunk is written to only one disk. Distributing the I/O operations among multiple drives allow to accumulate the performance of particular RAID components.
RAID1 is implemented as mirroring. The data that are written to both RAID components (disks) are ideal copies of each other. This means that every write operation performed on array must be done on both RAID components (drives). Variant of this technology is duplexing , where two independent RAID controllers are used to perform parallel writes on both mirrors.
Parity mechanism provides mechanism to write redundant data in more sophisticated way than in case of mirroring. This mechanism is used to im- plement RAID5 and RAID6 structures. If there are N drives in array, the disk controller splits data into N-1 stripe-sized chunks to write simultaneously on N-1 disks. Additionally the controller counts additional stripe-sized data called parity, and writes it to N-th disk. The parity is calculated using XOR operation. While recovering the broken data in the array (e.g. when one stripe is damaged), XOR operation is performed on the parity stripe and valid N-2 stripes. The result equals the broken stripe.
Parity information can be written to one special disk (RAID3 and RAID7structures) or can be spread across all the drives in the array (RAID5 and RAID6). The latter approach has some performance advantages, as the parity-writing load is distributed over all RAID components, opposite to the former case in which parity stripe writing can be a bottleneck. The known limitation of parity mechanism is that calculating the parity data can affect the write performance. RAID5 uses single parity stripe, while RAID6 uses double parity stripes, which provides extra fault tolerance - the array can deal with two broken drives. However, double-parity calculation can affect the write performance.
Standard RAID structures have contradictory performance and redundancy features, e.g. mirroring provides high data redundancy, while limiting the writing performance. In order to provide both redundancy and performance, nested RAID structures are used. RAID10 combines some number of RAID1 structures by striping the data over them RAID0. In that way, superior fault tolerance can be achieved. The array can deal with 50% broken drives if for every broken drive there is his mirror drive working. RAID10 can achieve performance similar or or even better (in random read case) than RAID0. Another commonly used nested RAID structure is RAID50. It is RAID0 made of some number of RAID5 structures. RAID50 improve the performance of RAID5 thanks to the fact that the I/O traffic is distributed over particular RAID5 structures. This approach is effective especially for write operations. It also provides better fault tolerance than the single RAID level does. The drawback of nested RAID structures is that they require a relatively high number of drives to implement a given storage space.
Taking into account all theoretical information shown above, we summarize the storage efficiency information in Table 1
Table below contains a RAID levels comparison. The notes range is 0 - the worst to 5 - the best. The notes are based on http://www.pcguide.com/ref/hdd/perf/raid/levels/comp.htm and modified according to gathered experience and the fact that we are considering the same amount of drives in every RAID structure. To compare performance we assume that we make each RAID structure using the same number of drives and we use one thread to read or write data from or to the RAID structure.
RAID Level | Capacity | Storage efficiency | Fault tolerance | Sequential read perf | Sequential write perf |
RAID0 | S*N | 100% | 0 | 5 | 5 |
|---|---|---|---|---|---|
RAID1 | S*N/2 | 50% | 4 | 2 | 2 |
RAID5 | S * (N - 1) | (N - 1)/N | 3 | 4 | 3 |
RAID6 | S * (N - 2) | (N - 1)/N | 4,5 | 4 | 2,5 |
RAID10 | S * N/2 | 50% | 4 | 3 | 4 |
RAID50 | S * N0 * (N5 - 1) (N5 - 1)/N5 | (N 5 - 1)/N | 3,5 | 3 | 3,5 |
This part of the HOWTO explains the methodology and tips for measuring performance of client's Block-level storage configured on certain types of RAID storage.
Note, that the RAID storage used in this section is the software RAID implemented using the Linux MD mechanism LINK-!, over the disk drives directly attached
to the server (DAS storage).
The purpose of this part of the HOWTO is to:
For the purpose of this howto we also suppose that:
Here, we present how to make software raid structure using Linux md tool. To create simple raid level from devices sda1 sda2 sda3 sda4 you should use following command:
mdadm --create --verbose /dev/md1 --spare-devices=0 --level=0 --raid-devices=4 /dev/\{sda1, sda2, sda3, sda4\}
|
Where:
When you create a RAID structure you should be able to see some RAID details similar to the informations shown below:
[root@sarevok bin]# mdadm --detail /dev/md1
/dev/md1:
Version : 00.90.03
Creation Time : Mon Apr 6 17:41:43 2009
Raid Level : raid0
Array Size : 6837337472 (6520.59 GiB 7001.43 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 3
Persistence : Superblock is persistent
|
Update Time : Mon Apr 6 17:41:43 2009
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
|
Chunk Size : 64K |
Rebuild Status : 10% complete |
UUID : 19450624:f6490625:aa77982e:0d41d013
Events : 0.1
|
Number Major Minor RaidDevice State
0 65 16 0 active sync /dev/sda1
1 65 32 1 active sync /dev/sda2
2 65 48 2 active sync /dev/sda3
3 65 64 3 active sync /dev/sda4
|
When performance is considered the chunk size of the md device may be important parameter to tune. There is -c option of the mdadm command, that cen be usedto specify chunk size of kilobytes. The default is 64 kB, however it should be setup up according to some factors such as:
Important!! When you want to run any benchmark on a new RAID structure you should wait until the RAID structure (re)building is finished. When you run a test when the RAID is being (re)building you achieve poor performance results, because system resources and disks are involved in (re)build RAID structure.
Because iozone benchmark is used in the next part of the HOWTH, we should firstly make a file system on the newly created md device.
The file system choice is not the topic of this part of the HOWTO. However, we assume that ext2 or ext3 file system is a good choice for our test.
When using ext2 and ext3 we may be interested in tunnning some basic file system parameters such as journaling type. We can do it using tune2fs Linux command.
To create the file system on md device and then mount it to some directory we use command like this:
mkfs.ext3 /dev/md1 mount /dev/md1 /mnt/md1 |
Again, there are some filesystem parameters that are interesting from the performance point of view. One of them is
the blocksize. It should be set taking into account the application features and the underlying storage components.
One of the rules of thumb is to use the block size equal to the size of the RAID stripe. You can set the the blocksize
using mkfs's -b parameter. There is also possibility to influence the filesystem behaviouds by using using mount command options.
To examine the performance of md device we normally use iozone tool. However, for making a quick test (for example to receive fast results) we may use dd tool.
The idea of dd is to copy file from 'if' location to 'of' location. Using this tool to measure disk devices requires some trick. To measure write speed you read data from /dev/zero to file on the tested device. For measuring the read performance you should read the data from the file on tested device and write it to /dev/zero.
In that way we avoid measuring more that one storage system at a time. To measure time of reading or writing the file we use time tool. The example commands to test write and read 32 GB of data are:
for writing performance (please note using the sync command before and during the benchmark, so you are not measuring your operating system's cache performance) :
[root@sarevok ~]# sync; time (dd if=/dev/zero of=/mnt/md1/test_file.txt bs=1024M count=32; sync) |
and for reading performance:
[root@sarevok ~]# time dd if=/mnt/md1/test_file.txt of=/dev/zero bs=1024M count=32 |
where:
For more precise tests we use iozone tool. Iozone allows to run test in many modes, including:
In order to evaluate storage system features you should use a mode that fits best the application data access pattern. However, for all tests you have to run the write/rewrite mode. benchmark first (in order to generate the files for reading).
The other parameters which are interesting is the amount of Linux p-threads which are used for simultaneous read or write operations. Thanks to that feature we can realize how the performance changes while using more than 1 benchmark thread.
We can also use the block-size parameter to set the read/write chunk size (compare with md and file system chunk size).
Another important options allow to decide if the file close and flush operations are included in the measured time. Including flush time is especially significant, because this is the time when the cache buffers are written to disks.
To perform one round of the test we can use command:
iozone -T -t $threads -r $\{blocksize\}k -s $\{file_size\}G -i 0 -i 1 -i 2 -c -e
|
where:
It is also vital thing to realize that we measure the performance of software raid. This means that all tasks of the RAID controller (for example parity counting and checking) is performed by machine CPU. So it is necessary to measure CPU load when you perform a md device test, since CPU can be the performance bottleneck in such system. To measure CPU load we use vmstat tool.
To automate the testing we can write some simple SH script like this:
#\!/bin/sh dst_path=/home/sarevok/wyniki_test_iozone curr_date=`date +%m-%d-%y-%H-%M-%S` |
file_size=128 min_blocksize=1 max_blocksize=32 |
min_queuedepth=1 max_queuedepth=16 |
mkdir $dst_path cd /mnt/sdaw/ |
blocksize=$min_blocksize
while [: _blocksize -le $max_blocksize ]; do
queuedepth=$min_queuedepth
while [: _queuedepth -le $max_queuedepth ]; do
|
vmstat 1 > $dst_path/vmstat-$blocksize-$queuedepth-$curr_date
/root/iozone -T -t $queuedepth -r $\{blocksize\}k -s $\{file_size\}G -i 0 -i 1 -c -e > $dst_path/iozone-$blocksize-$queuedepth-$curr_date
|
ps ax | grep vmstat | awk '\{print $1\}' | xargs -i kill \{\} 2&>/dev/null
|
queuedepth=`expr $queuedepth \* 2`
file_size=`expr $file_size \/ 2`
done
blocksize=`expr $blocksize \* 2`
done
|
To start test script that can ignore hangup signals, you can use nohup command:
: root@sarevok$ nohup script.sh &
This command keeps the test running when we close the session with server. To present obtained results we use the open-source gnuplot program.
When you perform any disk device or file system benchmark you should have in mind that there are many levels of cache in the system, e.g. file system cache, operating system cache, disk drive cache etc. The simplest way to avoid cache influence is to use such amount of data to fill all cache levels buffers. To do this we use the amount of data that is at least is equal to the machine RAM size doubled. Such data size should successfully eliminate the caching influence on measured md device performance.
After every round of tests, when we want to change some RAID (mD) or the file system parameters it is recommended to make the fresh file system in md device, in order to avoid the influence of the filesystem state on the test results.
This section of the HOTWO discusses the details of particular benchmarking tools and provides
practical information about theirs usage, automation, interpretation of the results and so on.
TO BE EXPANDED.
One of the most common usages of storage benchmarking is making sure that the storage systems you buy meets your requirements. As always there are practical limits how complex the benchmark can be. This section lists benchmark procedures actually used in tenders.
Brief disk array requirements:
Performance requirements:
Random:
Same setup of the volume as in the sequential test, but for this test, it will be connected without any filesystem (on a block level). The following test will be run on the connected LUN using fio v1.4.1 with this test definition:
[global] description=CESNET_test [cesnet] # change it to name of the block device used filename=XXXX rw=randrw # 70% rand read, 30% rand write rwmixread=70 size=10000g ioengine=libaio bs=2k runtime=8h time_based numjobs=32 group_reporting # --- end --- |
The result of the test is sum of write and read IO operations divided by total elapsed time of the test in seconds.
Minimum required performance 9000 IOPs.
Results of the tests required as a part of proposal: YES
Notes after evaluation: the tests themselves were OK, but the test architectures could be defined a bit better: The tests actually measured only performance of the FC disks (candidates obviously configured the volume in such a way that it was faster), performance of SATA volumes was not evaluated at all. Also, the winner used RAID5 as required but there was a big RAID0 volume above the 20 individual RAID5s (thus creating RAID50) which was allowed but not used in production afterwards.
The text re-uses the material presented by Stijn Eeckhaut in Espoo, Finland,
during the 1st TF-Storage Meeting, 8 April 2008.
It also includes the material prepared by Maciej Brzezniak from Poznan Supercomputing
and Networking Centre, Poland and Stanislaw Jankowski, student at Poznan University of Technology.
If you want to contact the HOWTO authors directly, send the email to: