Maximizing Scylla PerformanceCopy

Use a Representative EnvironmentCopy

• Execute benchmarks on an environment that reflects your production.

• Benchmarking on the wrong environment can easily lead to an order of magnitude performance difference. For example, on a laptop, you could do 20K OPS while on a dedicated server you could easily do 200K OPS. So unless you have your production system running on a laptop, do not benchmark on a laptop.

• It is recommended to automate benchmarking with tools such as Terraform/Ansible to ensure repetitive tests run smoothly.

• If you are using shared hardware in a containerized/virtualized environment, note that a single guest can increase latency in the other guests.

• Make sure that you do not underprovision your load generators to prevent the load generators from becoming the bottleneck.

Use a Representative Data ModelCopy

Tools, such as cassandra-stress, use a default data model that does not reflect the actions that you will perform in production. For example, the cassandra-stress default data model has a replication factor set to 1 and uses the LOCAL_ONE as a consistency level.

Although cassandra_stress is convenient to get some initial performance impressions, it is critical to benchmark the same/similar data model that is used in production. Therefore we recommend that you use a custom data model. For more information refer to the user mode section.

Use Representative DatasetsCopy

If you run the benchmark with a dataset that is smaller than your production data, you may have misleading or incorrect results due to the reduced number of I/O operations. Therefore, it is critical to configure the size of the dataset to reflect your production dataset size.

Use a Representative LoadCopy

Run the benchmark using a load that represents, as closely as possible, the load you anticipate will be used in production. This includes the queries submitted by the load generator. The read/write ratio is important due to the overhead of compaction and finding the right data on disk.

Proper Warmup and DurationCopy

When benchmarking, it is important that the system is given time to warm up. This allows the database to fill the cache. In addition, it is critical to run the benchmarks long enough so that at least one compaction is triggered.

Latency Tests vs. Throughput TestsCopy

When performing a load test you need to differentiate between a latency test and a throughput test. With a throughput test, you measure the maximum throughput by sending a new request as soon as the previous request completes. With a latency test, you pin the throughput at a fixed rate. In both cases, latency is measured.

Most engineers will start with a throughput test, but often a latency test is a better choice because the desired throughput is known e.g. 1M op/s. Especially if your production depends on meeting the needs of the SLA For example, the 99.99 percentile should have a latency less than 10ms.

Coordinated OmissionCopy

A common problem when measuring latencies is the coordinated omission problem that causes the worst latencies to be omitted from the metrics. As a result, it renders the higher percentiles useless. A tool such as cassandra-stress prevents coordinated omissions from occurring. For more information, read this article.

Don’t Average PercentilesCopy

Another typical problem with benchmarks is that when a load is generated by multiple load generators, the percentiles are averaged. The correct way to determine the percentiles over multiple load generators is to merge the latency distribution of each load generator and then to determine the percentiles. If this isn’t an option, then the next best alternative is to take the maximum (the p99, for example) of each of the load generators. The actual p99 will be equal or smaller than the maximum p99. For more information on percentiles, read this blog.

Use Proven Benchmark ToolsCopy

Instead of rolling out custom benchmarks, use proven tools like cassandra-stress. It is very flexible and takes care of coordinated omission. Yahoo! Cloud Serving Benchmark (YCSB) is also an option, but needs to be configured correctly to prevent coordinated omission. TLP-stress is not recommended because it suffers from coordinated omission. When benchmarking make sure that cassandra-stress that is part of the Scylla distribution is used because it contains the shard aware drivers.

Use the Same Benchmark ToolCopy

When benchmarking with different tools, it is very easy to run into an apples vs. oranges comparison. When comparing products, use the same benchmark tool if possible.

Reproducible ResultsCopy

Make sure that the outcomes of the benchmark are reproducible; so execute your tests at least twice. If the outcomes are different, then the benchmark results are unreliable. One potential cause could be that the old data set of a previous benchmark has not been cleaned and this can make a performance difference for writes.

Correct Data ModelingCopy

The key to a well-performing system is using a properly defined data model. A poorly structured data model can easily lead to an order of magnitude performance difference compared to that of a proper model.

A few of the most important tips:

• Choose the right partition key and clustering keys. Reduce or even eliminate the amount of data that needs to be scanned.

• Add indexes where appropriate.

• Partitions that are accessed more than others (hot partitions) should be avoided because it causes load imbalances between CPUs and nodes.

• Large partitions, large rows and large cells should be avoided because it can cause high latencies.

Use Prepared StatementsCopy

Prepared statements provide better performance because: parsing is done once, token/shard aware routing and less data is sent. Apart from performance improvements, prepared statements also increase security because it prevents CQL injection. Read more about Stop Wasting Scylla’s CPU Time by Not Being Prepared.

Use Paged QueriesCopy

It is best to run queries that return a small number of rows. However, if a query can return many rows, then the unpaged query can lead to a huge memory bubble. This will eventually cause Scylla to kill the query. With a paged query, the execution collects a page’s worth of data and new pages are retrieved on demand, leading to smaller memory bubbles. Read about More Efficient Query Paging.

Don’t Use Reverse QueriesCopy

When using a query with an ORDER BY clause, you need to make sure that the order is the same as the order in the data model. Otherwise you run into a problem called reverse queries which can cause unbound memory usage and killed queries.

Use Workload PrioritizationCopy

In a typical application there are operational workloads that require low latency. Sometimes these run in parallel with analytic workloads that process high volumes of data and do not require low latency. With workload prioritization, one can prevent that the analytic workloads lead to an unwanted high latency on operational workload. Workload prioritization is only available with Scylla Enterprise.

Bypass CacheCopy

There are certain workloads, e.g. analytical workloads, that scan through all data. By default ScyllaDB will try to use cache, but since the data won’t be used again, it leads to cache pollution: i.e. good data gets pushes out of the cache and replaced by useless data,

As a consequence it can lead to bad latency on operational workloads due to increased rate of cache misses. To prevent this problem, queries from analytical workloads can bypass the cache using the ‘bypass cache’ option.

See Bypass Cache for more information.

BatchingCopy

Multiple CQL queries to the same partition can be batched into a single query. Imagine a query where the round trip time is 0.9 ms and the service time is 0.1 ms. Without batching the total latency would be 10x(0.9+0.1)=10.0 ms. But if you created a batch of 10 instructions, the total time would be 0.9+10*0.1=1.9 ms. This is 19% of the latency compared to no batching.

CPU Core Count guidelinesCopy

Scylla, by default, will make use of all of its CPUs cores and is designed to perform well on powerful machines and as a consequence fewer machines are needed. The recommended minimum number of CPU cores per node for operational workloads is 20.

The rule of thumb is that a single physical CPU can process 12.5 K queries per second with a payload of up to 1 KB. If a single node should process 400K queries per second, at least 32 physical CPUs or 64 hyper-threaded cores are required. In cloud environments hyper-threaded cores are often called virtual CPUs (vCPUs) or just cores. So it is important to determine if a virtual CPU is the same as a physical CPU or if it is a hyper threaded CPU.

Scylla relies on temporarily spinning the CPU instead of blocking and waiting for data to arrive. This is done to reduce latency due to reduced context switching. The drawback is that the CPUs are 100% utilized and you might falsely conclude that Scylla can’t keep up with the load. Read more about Scylla System Requirements.

Memory GuidelinesCopy

During startup, Scylla claims nearly all of the available memory for itself. This is done for caching purposes to reduce the number of I/O operations. So the more memory available, the better the performance.

Scylla recommends at least 2 GB of memory per core and a minimum of 16 GB of memory for a system (pick the highest value). This means if you have a 64 core system, you should have at least 2x64=128 GB of memory.

The max recommended ratio of storage/memory for good performance is 30:1. So for a system with 128 GB of memory, the recommended upper bound on the storage capacity is 3.8 TB per node of data. To store 6 TB of data per node, the minimum recommended amount of memory is 200 GB.

Read more about Scylla System Requirements or Starting Scylla in a Shared Environment.

Storage GuidelinesCopy

Scylla utilizes the full potential of modern NVMe SSDs; so the faster drive, the better the performance. If there is more than one SSD, it is recommended to use them as RAID 0 for the best performance. This is configured during scylla_setup and Scylla will create the RAID device automatically. If there is limited SSD capacity, the commit log should be placed on the SSD.

The recommended file system is XFS because of its asynchronous appending write support and is the primary file system ScyllaDB is tested with.

As SSD’s wear out over time, it is recommended to re-run the iotune tool every few months. This helps Scylla’s IO scheduler to make the best performing choices.

Read more about Scylla System Requirements.

Networking GuidelinesCopy

For operational workloads the minimum recommended network bandwidth is 10 Gbps. The scylla_setup script takes care of optimizing the kernel parameters, IRQ handling etc.

Read more about Scylla Network Requirements.

Image GuidelinesCopy

Use the Scylla provided AMI on AWS EC2 or the Google Cloud Platform (CGP) image, if possible. They have already been correctly configured for use in those public cloud environments.

AWSCopy

AWS EC2 i3, i3en and cd5 bare metal instances are highly recommended because they are optimized for high I/O.

Read more about Scylla Supported Platforms.

If bare metal isn’t an option, use Nitro based instances and run with ‘host’ as tenancy policy to prevent the instance being shared with other VM’s. If Nitro isn’t possible, then use instance storage over EBS. If instance store is not an option, use an io2 IOPS provisioned SSD for best performance. If there is limited support for instance storage, place the commit log there. There is a new instance type available called r5b that has high EBS performance.

GCPCopy

For GCP we recommend n1/n2-highmem with local SSDs.

Read more at: https://docs.scylladb.com/getting-started/system-requirements/#google-compute-engine-gce

AzureCopy

For Azure we recommend the Lsv2 series. They feature high throughput and low latency and have local NVMe storage. Read more about Azure Requirements.

DockerCopy

When running in Docker platform, please use CPU pinning and host networking for best performance. Read more about The Cost of Containerization for Your Scylla.

KubernetesCopy

Just as with Docker, on a Kubernetes environment CPU pinning should be used as well. In this case the pod should be pinned to a CPU so that no sharing takes place.

Read more about Exploring Scylla on Kubernetes.