Is Amazon PIOPS Really Better than Standard EBS?

- Ofer Bengal, founder & CEO at Garantia Data (https://garantiadata.com)

Last month, Amazon launched its EBS Provisioned IOPS (PIOPS) for running high

performance databases in the cloud, and introduced EBS-Optimized instances to deliver

dedicated throughput between EC2 and EBS. As the provider of the Redis Cloud service, we

wanted to know whether the PIOPS EBS with EBS-Optimized instances would really perform

better than Standard EBS with non-optimized instances for Redis.

Running Redis on AWS has been a challenge for Redis users, mainly due to EBS’ slow and

unpredictable network and I/O throughput. Though Redis is a pure in-memory datastore

system and serves everything from RAM, it also has two persistent-storage mechanisms for

better durability: AOF (Append Only File) and Snapshotting. These mechanisms are very

efficient when interfacing with persistent storage, as they only use sequential writes with

no seek time. As a result, Redis can run pretty well with data-persistence even on non-SSD

hardware.

That being said, we decided to conduct some benchmarks for Amazon’s EBS PIOPS because

slow storage devices can still be very painful for Redis. We see this when the AOF “fsync

every second” policy blocks an entire Redis operation due to slow disk I/O. In addition,

background saving for point-in-time snapshots or AOF rewrites takes longer, and therefore

more memory is used for copy-on-write — leaving less memory available for your app.

You may have seen the detailed post from Strattalux comparing the performance of PIOPS

EBS vs. Standard EBS using EBS-Optimized and non-Optimized instances.

While their tests were interesting, we thought we’d dig a little deeper and examine whether the performance of a simple Redis application is influenced by different storage architecture. Therefore, we were very specific and only tested the performance of sequential writes, ignoring other scenarios like random read/write and sequential read. For our benchmark, we also reviewed cost implications, and used what we believe is a more noptimal performance configuration of Standard EBS (1 x 1TB EBS volume vs. 10 x 100GB RAID0 EBS, per a very educational blog about Standard EBS performance by Adrian Cockcroft from Netflix). So, without further ado, here are the details of our tests...

Benchmark Setup

We used the following setup: PIOPS EBS with Optimized-Instance (i.e. m2.4xlarge with

EBS-Optimized attribute set to true) vs. Standard EBS with non-Optimized Instance

(m2.4xlarge with EBS-Optimized attribute set to false).

We also tested another setup in which we compared PIOPS EBS vs. Standard EBS on a non

Optimized-Instance (m2.2xlarge). Our intention was to see whether AWS users who cannot

use the EBS Optimized-Instance for various reasons (such as cost) can get benefit from the

new PIOPS EBS. Those results were very similar to the setup described above.

For all tests, we used 2 x m1.xlarge instances that run our memtier_benchmark load

generation tool (memtier_benchmark is an advanced load generator tool we developed for

testing Redis and Memcached datastores, which we will soon share in our github account).

We ran 8 tests on each configuration to simulate a Redis application, then ran each of these

tests 3 times and calculated the average results.

Our tests combinations included:

• 100 and 500 connections on each load-generator machine (200 and 1000 connections total).
• 100B and 1KB object size
• 50%/50% read/write scenario, and 100% write scenario (for imposing load on EBS)

Benchmark Results

Note - we have chosen to show the results from the 1KB object size tests, which impose

higher load on the EBS.

The RPS and Latency results aggregated from our two memteir_benchmark tools are shown

above. Our latency tests take into account the network round-trip, the Redis processing

time, and the time it takes for the memtier_benchmark tool to parse the result. As you

can see, RPS and latency are approximately the same on all tests, though there are some

differences which might be due to the fluctuations in the AWS infrastructure during the test

time. Needless to say, Redis runs better with fewer disk accesses.

The results above are taken from iostats commands executed while the tests were running

(here’s a useful AWS forum post that explains how to interpret Linux iostats). We ran

iostats every 2.5 seconds and aggregated results from both EBS volumes. These charts

only present relevant performance parameters for Redis — “read” parameters were not

presented as they were constantly zero during these tests. As you can see, except from

few glitches here and there, iostats show approximately the same results for both EBS

configurations.

Conclusion

After 32 intensive tests with Redis on AWS (each run in 3 iterations for a total of 96 test

iterations), we found that neither the non-optimized EBS instances nor the optimized-

EBS instances worked better with Amazon’s PIOPS EBS for Redis. According to our results,

using the right standard EBS configuration can provide equal if not better performance

than PIOPS EBS, and should actually save you money. Keep in mind that these findings

represent a specific storage usage pattern (sequential write only), so if you have different

storage access patterns, you may want to review the Stratalux test results (while taking

into account the different Standard EBS configuration we suggest).

Given these results, we plan to continue running Garantia Data’s clusters on a Standard EBS

configuration. We welcome comments from the Amazon EBS Team on our benchmark, and

will be sure to share any new information we discover.