Introduction

General informations

HOWTO content

This HOWTO describes the (networked) storage performance measurements methodology and tips.
As the members of TERENA's TF-Storage we try to collect general informations HOW TO
prepare and perform the storage performance testing, basing on our own experience and
knowledge as well as the material accessible in the Internet.

This HOWTO re-uses the presentation prepared by IBBT/Ghent University,
Storage Benchmarking Cookbook.

HOWTO limitations

Obviously, the world is too complicated to be described in a single work.
However, we hope that both background and practical, hands-on informations included in this HOWTO
can be useful for the community.
HOWTO does not cover all possible aspects, problems and approaches to the
storage benchmarking process. We try to describe the most important issues,
most popular approaches and tools used to evaulate the storage systems.

HOWTO organisation

The organisation of the sections of this HOWTO follows the real-world, networked storage systems layers layout (see table below).

We provide general and layer-by-layer comments on how to perform storage benchmarking
along with some background information, practical benchmarking tips and examples
as well as links to the external sources.

System layers

Typical storage environment is composed of layers similar to presented above.
It is important to understand, that each of these layers can have important influence of the data access performance.

In following subsections we shortly discuss the layers and mention the main factors,
that have the most important impact on the efficiency of the whole storage system.

Clients and applications layer

Client applications run in the client hosts. These hosts are the NAS clients. NAS services include file system-level
services, basic user authentication and access control.

...

Note, that client hosts can use block-level services provided by the storage system: DAS (Directly Attached Storage) or SAN (Storage Area Network) instead of NAS services. This case is discussed in details in one of
the following sections.

Client network

Client network interconnects the clients and the filesystem cluster, called also 'NAS cluster'.

...

As the client-NAS server network can introduce data transmission delays and
significant performance limits, its features must be carefull examined
in the process of benchmarking the networked storage solution.

Filesystem and cluster nodes... and block-level storage clients

Filesystem and cluster nodes implement the NAS (Network Attached Storage) functionality,
including filesystem services, users authentication and data access control.

NAS services implementation

Physically, there are two most typical ways of implementing the NAS services:

...

• a lack of scalability of data processing performance,
• overhead of the file-level services compared to block-level storagem
• specifics of the NAS gateways hardware and software including:
• file system settings: caching, journaling, prefetching, redundancy,
• operating systems limits on NAS gateways,
• transport protocol parameters in both client and storage front-end network.

Block-level storage clients

Note, that while being a NAS servers, NAS gateways act as the block-level services clients in the same time. Block-level services are provided by storage controllers accesible through storage front-end network, e.g.
a FC-based SAN.

...

In typical networked storage setups also the application/client systems can be connected to the storage network directly, exploiting the block-level storage. In such case, storage benchmarking can be focused
on the performance evaluation of SAN compoments, SAN network interfaces (so called HBAs)
(Host Bus Adapters) and SAN driver characteristics.

Storage front-end connectivity (SAN or DAS)

The block-level storage can be served directly, locally (DAS - Direct Attached Storage) or by the Storage Area Network. In both cases, the performance evaluation of the block-level storage can be performed using similar methods and tools, as the virtualisation mechanismes implemented in modern operating systems hide the fact that the SAN storage is not local.

...

Beside the FC technology, iSCSI and FCIP, iFCP and SRP (over IB) protocols are used to implement the data transport in the front-end storage networks.

Storage controllers and RAID structures

Storage controllers implement the basic virtualisation functions such as RAID structures, volumes (LUNs) and volume
mapping to hosts and volumes access control. They are crucial elements of the storage system, so their features may determine its performance characteristics.

...

Significant amount of time should be spent in order to get a realistic picture
of a given controller performance characteristics.

Storage back-end connectivity

Back-end storage network interconnects the storage controllers with he disks drives.
Various type of the technologies are used for that purpose. Traditionally, Fibre Channel
loops span disk drives and storage controller back-end ports. In enterprise solutions,
switched back-end topologies are used. Currently, SAS technology is often used
for the back-end connectivity as it provides high bandwidth at optimal cost.

The storage back-end links can be a bootle-neck, especially in a sustained transfer
operations. Therefore its characteristics such as latency and bandwidth should be taken into account while
running the performance tests.

Disk drives

As the disk drives hold the actual data, their characteristics are very important
component of storage system performance.

...

As the data processed in storage systems ultimately goes to disk drives, it might be useful to examine the performance of the individual disk drives (if possible), in order to determine the realistic performance that can be achieved after combining multiple disks into RAID arrays.

Storage system layers summary

In above sections we showed that networked storage systems are typically composed of multiple layers.
Each of this layers has its own features, complexity and performance characteristics.
Solid storage benchmarking should take into account these facts. Appropriate testing metodology
should be used in order to reveal the real characteristics of particular components
or to find a source of the actual or potential bottle-neck observed in the system.

Overview of the benchmarking methodologies is provided in the next section of the HOTWO.

Storage Benchmarking methodologies overview

The complexity of the networked storage system makes solid storage benchmarking a difficult task.
One of the basic decisions that must be taken while preparing the benchmarking procedure
is the PURPOSE of running the benchmark and the EVALUATION CRITERIA for the test results.

Benchmarking purposes and evaluation criteria

There are many possible benchmarking purposes. Here are some examples of them:

...

Benchmarking procedure and tools should be able to reveal the characteristics of the examined
storage system that matches the defined criteria.

Benchmarking tactics and organisation

Depending on the purpose, the benchmarking process should be organised in different ways.

...

• plan the correct sequence of the benchmarking actions, e.g.:
• work bottom-up to know the efficiency of each system layer and the overhead it introduces comparing to lower layers;
• work top-down to determine see the limit on the highest level and try to find its source on the lower layers.
• choose the appropriate set of the examined configuration parameters and determine the set of the values used for them,
• eliminate the influence of unwanted optimisation mechanisms (in case we want it), e.g.:
• storage controller-side caching,
• NAS gateway filesystem-side caching,
• client-level, filesystem-side and application-side caching
• make sure, that the benchmarking load reflects the real load of the target applications and the target environment,

Benchmarking tactics: bottom up or top-down?

Both approaches has some advantages and disadvantages. The accepted tactics should fit the purpose
of the test.
If we want to quickly determine the overall performance of the storage system, we can
start from the top, e.g. by running a file-system level benchmark (more on benchmarks in the BENCHMARKING
TOOLS section), and go down in case the performance is lower than expected. This is most probably less
time consuming approach, however we may get an incomplete picture of the actual system efficiency.
Going bottom-up is more costly, but gives as the knowledge about the efficiency of each system layer and the overhead it introduces comparing to lower layers. This approach is suitable if we want to find a bottle-neck in the
system or we want to determine the realistic maximum performance in a tunned setup.

Benchmarking parameters selection

Having in mind the complexity of multi-layer storage system, we may be forced to:

• use heuristics,
• trust our experience and intuition
  in order to determine (and limit!!) the list of the configuration parameters modified
  during the benchmarking process and their examined values. Choosing the correct parameters
  and their values is a difficult task and can be performed basing on the deep knowledge
  of the storage systems, networks and computers systems architectures and characteristics.

Measuring the correct bottle-neck

Benchmarking the multi-layered system we have to make sure that we are measuring the correct
component of the system.

Data caching influence

Typical issue faced while benchmarking on the file-system level is the influence of the data
caching which can take place on the client side (file system-level, operating system-level), NAS gateway side,
or on the storage controller.

...

Another side effect of the data caching is that, the speed of writing the the buffer
can be measured instead of the actual tranmission speed. This can be avoided by using the data set large enought to fill up the buffers
and transmit enough data over the real communication channels. We may also use external monitoring tools,
such as Gigabit Ethernet network or SAN switches performance analysers, host ports usage monitoring tools to
analise the actual data traffic taking place in the network.

Monitoring the system compoments

While doing the benchmark we have to make sure, that we are able to determine the reason of the
the performance bottle-neck visible in the results. On the other hand, we should try to avoid
incorrect intepretation of the results in case if some caching/buffering effect makes the results
better than actual system features. We can use monitoring tools in addition to the benchmarking tools for that purpose.

...

• network interfaces monitoring tools: ntop, ethreal, tcpdump
• CPU monitoring tools:: top, dstat, vmstat
• system statistics collectors: vmstat, sar, dstat
• /proc filesystem in linux
• virtual machines monitoring tools: xm top, virt-top, virt-manager for Xen;
• and many others.

Avoiding the artificial setup

Note however, that the decision to switch off or eleminate the effects of caching should be taken carefully.
Disabling the cache, can make the operation of some optimisation technologies implemented in storage system elements
impossible. For instance, if we disable the write cache in the RAID controller, full-stripe write technique cannot
be exploited.

Switching the caching off, we also make the testing environment a bit artificial – the results we get
in such setup, can be useless in the real-life configuration.

Benchmarking workloads

Getting the realistic picture of the storage system performance, requires that we apply
the benchmark that generates the workload that matches our needs. Againt, the kind
of the testing workload should fit the purpose of performing the benchmark.

...

• by finding the benchmark that generates the workload that is similar to the application data access pattern (micorbenchmarks or macrobenchmarks); the similarity should include:
• access pattern: sequential, random or mixed,
• read/write ratio,
• temporal and spatial locality of storage access requests,
• number of simultaneous access requests (concurrent application/benchmark threads).
• by choosing the test that focus on the storage characteristic that are crucial for application:
• throughput-intensive (performance measured e.g. in MB/s),
• I/O intensive (performance measured e.g. in IOPS or in request serving time).

Workload generators and benchmark types

There is a lot of benchmarks types available for free or under the license.
They differ in:

...

Selected benchmarks discussion

In this section we discuss some benchmarks in details. We selected them basin on our experience
and interests. Therefore, the selection may be not optimal for every situation.

SPC benchmarks

Storage Performance Council (SPC) try to standardize storage system evaluation.
The organisation defines industry-standard storage workloads.
This "forces" vendors to publish a standardized performance of their storage systems.

...

• database operations
• mail servers
• OLTP |
• large files processing
• large database queries
• video on demand |

  Workload	Random I/O	Sequential I/O (1+ streams)
  Workload variations

• addressrequestdistribution: uniform + sequential
• R/W ratio |
• transfer size
• R/W ratio
• number of outstanding I/O requests |

  Reported metrics

• I/O rate (IOPS)
• total storage capacity
• price-performance |
• Data rate (MB/s)
• Total storage capacity
• Price-performance |

Good practices and tips for benchmarking

Storage system components benchmarking

If we decide to perform layer-by-layer testing of the storage system or we want to find a source
of the bottleneck observed in the system we may want to examine a single element of the system and
avoid the influence of the other elements.

Network-only benchmarking

Example problem is to examine the network ability to carry the data traffic using a given protocol,
without the influence of the disk access latency. To do so, we may use some tricks, for instance:

• measure the network link features with a dedicated tool, that performs RAM to RAM transfers, e.g. iperf,
• configure a RAM disk and export it, e.g. using NFS (say we want to test NFS transmission efficiency).

Network benchmarking tips

As mentioned before, the client network can be a source of significant performance bootlenecks.
However, the degree the network parameters impact the performance observed by the application
is related to the fact, what is the application sensitive for.
In case, the application is bandwidth intensive, we should examine mainly the bandwidth of the network, while
for I/O intensive application the network delay should be carefully tested.

The parameters of the network link we should evaluate depend also on the kind of the protocol used
for data transmission. For instance, in case of NFS, both bandwidth and delay of the network link matters,
as typicall NFS performs a synchronous data transmission. Another example is the GridFTP protocol,
which can exploit multiple parallel tranmission streams. Therefore, the network benchmark should
be able to examine the network bandwidth using multiple parallel transmission streams (for instance, we
can use iperf benchmark with multiple TCP/IP streams).

Benchmarking the storage systems compoments

This section of the HOTWO contains the more detailed informations about benchmarking selected storage system
compoments. Each of the subsections contains some background informations as well as practical informations
on testing the storage system element.

Clients network

This part of the HOWTO describes how to perform clients network testing. Some background information is
provided along with the practical informations related to TCP network testing using iperf tool.
Iperf is a open-source benchmark working both under Linux and Windows operating system.
In this HOWTO we focus on testing the network under Linux, i.e. between two machines running Red Hat Linux Enterprise.

Background information

As mentioned in the introduction, the efficiency of data access performed by the client network may depend on
network topology, technology, transmission protocol and network delay which in turn can result from both the physical distance between communication parties and the feature of communication equipment that processes the network traffic.

As the most popular client network in storage systems is the IP connectivity implemented over Gigabit Ethernet
we focus on this technology in this part of the HOWTO. We will show, how to examine data transmission delays
present in the network and the bandwidth available in the link. As some data transmission protocols are able to exploit
multiple parallel data streams (TCP connections), we also show how to examine the influence of the number
of the TCP streams used in parallel on the actual bandwidth observed by the application.

Test preparation

Because Iperf is an client-server application, you have to install iperf on both machines involved in tests. Make sure that you use iperf 2.0.2 using p-threads or newer due to some multi-threading issues with older versions. You can check the version of this installed tools with a following command:

...

Because in many cases the low network performance is caused by high CPU load, you should measure CPU usage at both link ends during every test round. In this HOWTO we use an open-source vmstat LINK-! tool which you probably have already installed on your machines.

Link properties

Before we start the tests, we should take a look on our network link setup. First, we should ensure that we can use MTU larger than the standard Ethernet MTU. We should try to use MTU 9000. Using jumbo frames is recommended especially in reliable and fast networks, as bigger frames boost network performance due to a better header-to-body-frame ratio. But we should remember that, it is possible to use MTU 9000 only if all network hardware between tested hosts (routers, switches, NICs etc.) support jumbo frames.

...

COMMENT: the meaning of these parameters is explained in the Linux documentation (sysctl command) LINK-!.

Iperf tool description

After setting up the network link parameters, we are ready to run the test. Obviously, configuring
the network settings can be an iterative process, where we check different settings by running the
tests and evaluating their results.

...

• s - server
• c - client
• $server - address of machine on which iperf server is running
• M - MSS=MTU-40
• P number of threads which are sending data through tested link simultaneously
• w - TCP initial buffer size
• i - interval between two test rounds in seconds
• t - test time

Testing methodology

As we want to perform several actions before, during and after a single test. These actions include:

...

To present obtained results you may use the open-source gnuplot LINK-! program, as the graphical results presentation may help yuo to interpret them. Generating the plots can also be automated.

RAID structures

...

RAID background information

One of the most known and important storage virtualisation techniques is the RAID (Redundant Array of Independent Disks) technology. It allows to combine independent disk resources into structures that can provide advanced reliability and performance features, which are not possible to deliver using individual disk resources (e.g. individual drives).
In this section we describe standard RAID levels including RAID 0, 1, 5 and 6 as well as nested RAID structures such as 10, 50 and 60. At the end of the section we summarize fault tolerance, performance and storage efficiency characteristics of particular RAID structures.

Standard RAID structures

RAID0 (striping) does not provide any data redundancy. The main purpose of using RAID0 structures is to distribute the data traffic load over RAID components. Each file is split into blocks of a certain size and those are distributed to the various drives. This block is called chunk. A number of chunks for whom one parity chunk was counted is called stripe. Chunk size is a user-defined parameter characteristic to array. The chunks are send to all disk in the array in a way that one chunk is written to only one disk. Distributing the I/O operations among multiple drives allow to accumulate the performance of particular RAID components.

...

Parity information can be written to one special disk (RAID3 and RAID7structures) or can be spread across all the drives in the array (RAID5 and RAID6). The latter approach has some performance advantages, as the parity-writing load is distributed over all RAID components, opposite to the former case in which parity stripe writing can be a bottleneck. The known limitation of parity mechanism is that calculating the parity data can affect the write performance. RAID5 uses single parity stripe, while RAID6 uses double parity stripes, which provides extra fault tolerance - the array can deal with two broken drives. However, double-parity calculation can affect the write performance.

Nested RAID structures

Standard RAID structures have contradictory performance and redundancy features, e.g. mirroring provides high data redundancy, while limiting the writing performance. In order to provide both redundancy and performance, nested RAID structures are used. RAID10 combines some number of RAID1 structures by striping the data over them RAID0. In that way, superior fault tolerance can be achieved. The array can deal with 50% broken drives if for every broken drive there is his mirror drive working. RAID10 can achieve performance similar or or even better (in random read case) than RAID0. Another commonly used nested RAID structure is RAID50. It is RAID0 made of some number of RAID5 structures. RAID50 improve the performance of RAID5 thanks to the fact that the I/O traffic is distributed over particular RAID5 structures. This approach is effective especially for write operations. It also provides better fault tolerance than the single RAID level does. The drawback of nested RAID structures is that they require a relatively high number of drives to implement a given storage space.

...

RAID benchmarking assumptions

This part of the HOWTO explains the methodology and tips for measuring performance of client's Block-level storage configured on certain types of RAID storage.

...

• we have the same number of drives to build each RAID structure we examine,
• we create all tested RAID structures using the same identical pool of disks,
• we use the same type and size of the filesystem for filesystem-level testing.
  In that way we assure that we measure and compare only the RAID structures performance differences.

RAID structures preparation

Here, we present how to make software raid structure using Linux md tool. To create simple raid level from devices sda1 sda2 sda3 sda4 you should use following command:

...

Again, there are some filesystem parameters that are interesting from the performance point of view. One of them is
the blocksize. It should be set taking into account the application features and the underlying storage components.
One of the rules of thumb is to use the block size equal to the size of the RAID stripe. You can set the the blocksize
using mkfs's -b parameter. There is also possibility to influence the filesystem behaviouds by using using mount command options.

Methods and tools for RAID levels benchmarking

To examine the performance of md device we normally use iozone tool. However, for making a quick test (for example to receive fast results) we may use dd tool.

dd tool

The idea of dd is to copy file from 'if' location to 'of' location. Using this tool to measure disk devices requires some trick. To measure write speed you read data from /dev/zero to file on the tested device. For measuring the read performance you should read the data from the file on tested device and write it to /dev/zero.
In that way we avoid measuring more that one storage system at a time. To measure time of reading or writing the file we use time tool. The example commands to test write and read 32 GB of data are:

...

• if - input file/device path
• of - output file/device path
• bs - size of a chunk of data to copy
• count - how many times a chunk defined by bs is copied

iozone tool

For more precise tests we use iozone tool. Iozone allows to run test in many modes, including:

...

This command keeps the test running when we close the session with server. To present obtained results we use the open-source gnuplot program.

Remarks

When you perform any disk device or file system benchmark you should have in mind that there are many levels of cache in the system, e.g. file system cache, operating system cache, disk drive cache etc. The simplest way to avoid cache influence is to use such amount of data to fill all cache levels buffers. To do this we use the amount of data that is at least is equal to the machine RAM size doubled. Such data size should successfully eliminate the caching influence on measured md device performance.

After every round of tests, when we want to change some RAID (mD) or the file system parameters it is recommended to make the fresh file system in md device, in order to avoid the influence of the filesystem state on the test results.

Benchmarking tools discussion

This section of the HOTWO discusses the details of particular benchmarking tools and provides
practical information about theirs usage, automation, interpretation of the results and so on.

TO BE EXPANDED.

Links:

Practical information:

File system benchmarking examples:

• Filesystem contest
• Benchmarking Filesystems

Independent storage benchmarking organisations:

Storage Performance Council (SPC)

• Storage Performance Council
• SPC-1 Benchmark Results
• SPC-2 Benchmark Results

Enterprise Strategy Group (ESG)

• Enterprise Strategy Group

Howto authors:

The text re-uses the material presented by Stijn Eeckhaut in Espoo, Finland,
during the 1st TF-Storage Meeting, 8 April 2008.
It also includes the material prepared by Maciej Brzezniak from Poznan Supercomputing
and Networking Centre, Poland and Stanislaw Jankowski, student at Poznan University of Technology.

...