Storage cluster performance on Exoscale

This page documents the performance of the VSHN Managed Storage Cluster on Exoscale.

The performance numbers presented on this page have been obtained from a storage cluster instance deployed on OpenShift 4.7. All performance numbers on this page have been measured on an otherwise idle storage cluster instance.

Block storage performance

Block storage is provided by Ceph RBD.

Read IOPS

Block storage provides 36303 read IOPS on average with a standard deviation of 2254IOPS over 10 benchmark iterations.

ceph rbd read iops

Read bandwidth

Block storage provides an average read bandwidth of 1.425 GB/s with a standard deviation of 60MB/s over 10 benchmark iterations.

ceph rbd read bandwidth

Write IOPS

Block storage write IOPS vary depending on the frequency of fsync calls. The graph below shows write IOPS for different fsync frequencies. The shaded areas in the graph represent the standard deviation of the measured IOPS for each series of data points.

ceph rbd write iops

The results show that block storage provides:

  • 6292 write IOPS on average with a standard deviation of 425 IOPS over 10 iterations without calling fsync during the benchmark ("no fsync").

  • 225 write IOPS on average with a standard deviation of 12 IOPS over 10 benchmark iterations with an fsync call after each operation ("fsync=1").

  • 2179 write IOPS on average with a standard deviation of 256 IOPS over 10 benchmark iterations with fsync calls after every 32 operations ("fsync=32").

  • 3264 write IOPS on average with a standard deviation 179 IOPS over 10 benchmark iterations with fsync calls after every 128 operations ("fsync=128").

Write bandwidth

Block storage write bandwidth exhibits the same performance variation based on fsync frequency as block storage write IOPS.

ceph rbd write bandwidth

The results show that block storage provides a write bandwidth of:

  • 310MB/s on average with a standard deviation of 63MB/s over 10 iterations without fsync calls during the benchmark ("no fsync").

  • 23MB/s on average with a standard deviation of 1MB/s over 10 iterations with an fsync call after each operation ("fsync=1").

  • 203MB/s on average with a standard deviation of 31MB/s over 10 iterations with fsync calls after every 32 operations ("fsync=32").

  • 226MB/s on average with a standard deviation of 39MB/s over 10 iterations with fsync calls after every 128 operations ("fsync=128").

File storage performance

File storage is provided by CephFS.

Read IOPS

File storage provides 36343 read IOPS on average with a standard deviation of 1235IOPS over 10 benchmark iterations.

cephfs read iops

Read bandwidth

File storage provides an average read bandwidth of 1.405 GB/s with a standard deviation of 92MB/s over 10 benchmark iterations.

cephfs read bandwidth

Write IOPS

File storage provides up to 6000 write IOPS on average with a standard deviation of 250 IOPS over 10 benchmark iterations.

cephfs write iops

CephFS doesn’t provide the same guarantees for fsync as Ceph RBD. This difference is easily visible in the results, which show much smaller differences for different fsync frequencies. Additionally, there’s no clear correlation between fsync frequencies and achieved write IOPS.

The following results have been measured for file storage:

  • 6020 IOPS on average with a standard deviation of 251 IOPS over 10 iterations without fsync calls during the benchmark ("no fsync")

  • 5004 IOPS on average with a standard deviation of 152 IOPS over 10 iterations with an fsync call after each operation ("fsync=1")

  • 4659 IOPS on average with a standard deviation of 704 IOPS over 10 iterations with an fsync call after every 32 operations ("fsync=32")

  • 4202 IOPS on average with a standard deviation of 212 IOPS over 10 iterations with an fsync call after every 128 operations ("fsync=128")

Write bandwidth

File storage provides a write bandwidth of up to 306MB/s on average with a standard deviation of 9MB/s over 10 benchmark iterations.

cephfs write bandwidth

As already stated, CephFS doesn’t provide the same guarantees for fsync as Ceph RBD. Again, there’s no clear correlation between fsync frequencies and achieved write bandwidth.

The following results have been measured for file storage write bandwidth:

  • 278MB/s on average with a standard deviation of 49MB/s over 10 iterations without fsync calls during the benchmark ("no fsync")

  • 291MB/s on average with a standard deviation of 17MB/s over 10 iterations with an fsync call after each operation ("fsync=1")

  • 299MB/s on average with a standard deviation of 8MB/s over 10 iterations with an fsync call after every 32 operations ("fsync=32")

  • 305MB/s on average with a standard deviation of 8MB/s over 10 iterations with an fsync call after every 128 operations ("fsync=128")