Was this page helpful?
Caution
You're viewing documentation for a previous version of ScyllaDB Open Source. Switch to the latest stable version.
Lower than expected performance can be a result of many factors, from HW (storage, CPU, network) to data modeling to the application layer. As a first step, make sure to have Scylla Monitoring Stack in place. Looking at the Scylla dashboard is the best way to look for bottlenecks. If you need our help, please follow How to Report a Performance Problem to share data securely.
Scylla uses available memory to cache your data. Scylla knows how to dynamically manage memory for optimal performance; for example, if many clients connect to Scylla, it will evict some data from the cache to make room for these connections; when the connection count drops again, this memory is returned to the cache.
The --smp option (for instance, --smp 2) will restrict Scylla to a smaller number of CPUs. It will still use 100 % of those CPUs, but at least won’t take your system out completely. An analogous option exists for memory: -m.
It is not possible to turn off the Scylla cache. But cache problems do not arise in normal operation. Scylla can use up to 50% of memory for cache, and will dynamically evict rows from the cache if they are too large. So the only possibility of getting an out of memory error is if a single row is bigger than 50% of the memory for a shard. This is (total_machine_memory / (number_of_lcores * 2)).
For a 64GB machine with 16 cores and hyperthreading enabled, you have 2GB per shard, of which the cache can use 1GB per shard. With such large rows, you will have other problems. We recommend staying with rows that are less than a few megabytes maximum size.
Scylla tries to utilize all available resources (processor cores, memory, storage, and networking) by always operating in parallel and never blocking. If Scylla needs to read a disk block, it initiates the read and immediately moves on to another task. Later, when the read completes Scylla resumes the original task from where it left off. By never blocking, a high degree of concurrency is achieved, allowing all resources to be utilized to their limit. Read more on Scylla Architecture:
Seastar creates an extra thread per core for blocking syscalls (like open()/ fsync() / close() ); this allows the Seastar reactor to continue executing while a blocking operation takes place. Those threads are usually idle, so they don’t contribute to significant context switching activity.
Yes, for more than one reason:
each shard (core) will run its compactions independently, often at the same time,
each table will run its compactions independently, often at the same time
depending on the compaction strategy, more than one compaction can run in parallel. For example in Sized Tier Compaction Strategy (STCS), large sstable compaction can take time, allowing smaller sstable to be compacted at the same time
As part of the Scylla setup process, iotune runs a short benchmark of your storage. When completed, it generates the /etc/scylla.d/io.conf configuration file. Note that iotune has known issues benchmarking HDD storage.
Note
This section is not relevant in 2.3
Therefore, when using Scylla with HDD storage, it is recommended to use RAID0 on all of your available disks, and manually update the io.conf configuration file max-io-request parameter. This parameter sets the number of concurrent requests sent to the storage. The value for this parameter should be 3X (3 times) the number of your disks. For example, if you have 3 disks, you would set max-io-request=9.
As a rule of thumb, for Scylla’s best performance, each client needs at least 1-3 connection per Scylla core. For example, a cluster with three nodes, each node with 16 cores, each client application should open 32 (2x16) connections to each Scylla node.
swap on a Scylla node?¶Yes, configuring swap on a Scylla node is recommended.
swap size should be set to either total_mem/3 or 16GB - lower of the two.
total_mem is the total size of the nodes memory.
For example:
If the node total_mem is 18GB swap size should be set to 6GB.
If the node total_mem is 240GB swap size should be set to 16GB.
Swap can be set up in several ways. One way to set up swap is detailed in the KB Article How to Set up a Swap Space.
Scylla 2.2 enables the compaction automatic controller which was not present prior to version 2.2. What this means is that in Scylla 2.1 (and earlier) the system waits for 4 SSTables to be present in the same tier before starting a compaction. In Scylla 2.2 (and later), compactions can be controlled to not impact the workload. This means that workloads which have been considered as backlog in Scylla 2.1 and earlier, in Scylla 2.2 and later are not.
If you are using a time range in the query, refer to the solution in the troubleshooting document, Time Range Queries Do Not Return Some or All of the Data.
As SI is built ontop of MV, you can expect to see MV in your schema. There is nothing wrong with your system. More information on Global Secondary Indexes.
The Java driver’s SimpleStatement is token unaware by default. This means that requests sent out will reach the Controller node before it is known which shard it’s supposed to access. We suggest using PreparedStatements instead.
scylla.yaml includes an auto_snapshot parameter; when true (it is by default), Scylla creates a snapshot for a table just before dropping it, as a safety measure.
You can find the snapshot in the snapshots directory, under the table SSTable. For example, for dropped table users in keyspace mykeyspace:
/var/lib/scylla/data/mykeyspace/users-bdba4e60f6d511e7a2ab000000000000/snapshots/1515678531438-users
As the snapshot take the same space as the dropped table, disk usage will remain the same. You can clean snapshots by using nodetool clearsnapshot. Read more on snapshot and clearsnapshot
You need to add the line experimental: true to your scylla.yaml file.
Launch the file in a text editor: $ vi /etc/scylla/scylla.yaml. (Alternately, on docker, it’s $ docker exec -it your_node vi /etc/scylla/scylla.yaml);
Add the line experimental: true;
Save the file and exit.
Stop and restart the node.
On RedHat Enterprise Linux, CentOS or Ubuntu:
$ sudo systemctl restart scylla-server
On Docker:
$ docker stop <your_node> && docker start <your_node>
Alternately, starting from Scylla 2.0, you can start Scylla for Docker with the experimental flag as follows:
$ docker run --name <your_node> -d scylladb/scylla --experimental 1
You should now be able to use the experimental features available in your version of Scylla.
On a regular system or VM (running Ubuntu, CentOS, or RedHat Enterprise): $ scylla --version
Check the Operating System Support Guide for a list of supported operating systems and versions.
On a docker node: $ docker exec -it Node_Z scylla --version
See ScyllaDB and Apache Cassandra Compatibility for compatibility matrix.
See ScyllaDB and Apache Cassandra Compatibility for current and future Apache Cassandra release compatibility.
The new “mc” SSTable format is supported in Scylla 3.0 and later. Scylla only starts using the newer format when every node in the cluster is capable to generate it. Therefore, only when all nodes in the cluster are upgraded the new format is used.
Apache Cassandra 4.0 includes breaking changes in the DESCRIBE command by moving its implementation from the client (cqlsh) to the server. As a result, using DESCRIBE in cqlsh from Cassandra 4.0 package with Scylla will result in an error. As a remedy, you can do one of the following:
Use Scylla cqlsh:
On Linux, you can install the
scylla-toolspackage from the official ScyllaDB repository on GitHub. The package contains cqlsh and other Apache Cassandra compatible tools for Scylla.On Linux, Windows, or Mac, you can run a Scylla container.
Downgrade your cqlsh to a version based on Cassandra 3.x, which supports DESCRIBE commands.
Note
The Scylla roadmap includes moving DESCRIBE to server side, similarly to Cassandra 4.0. See https://github.com/scylladb/scylla/issues/9571 for information about progress.
Running Scylla on Ubuntu 14.04 requires kernel 3.15 or later
To check your kernel version: $ uname -a
If your kernel is older than 3.15 then:
Check for available kernels: $ sudo apt-cache search linux-image
Install: $ sudo apt-get install linux-image-your_version_choice, for example linux-image-3.16.0
restart: $ sudo reboot now
Connectivity problems may occur if you are trying to connect to the ScyllaDB nodes with their Docker internal IP addresses.
If you need to reach your nodes from outside the internal Docker network, you must expose the appropriate ports to the Docker host. See Error connecting Java Spring application to ScyllaDB Cluster in Docker for more information and an example.
Scylla comes with its own version of the Apache Cassandra client tools, in the package scylla-tools. Trying to install it on a server with Cassandra already installed may result in something like:
Unpacking scylla-tools (1.0.1-20160411.b9fe89b-ubuntu1) ...
dpkg: error processing archive /var/cache/apt/archives/scylla-tools_1.0.1-20160411.b9fe89b-ubuntu1_all.deb (--unpack):
trying to overwrite '/usr/bin/nodetool', which is also in package cassandra 2.1.4
We recommend uninstalling Apache Cassandra before installing scylla-tools.
The APT package manager used for Ubuntu, Debian, and image installations of ScyllaDB installs the latest patch release (x.y.z) of a given major release (x.y). To remain on a ScyllaDB patch release that is not the latest, you can use pinning as a workaround.
The following example shows pinning ScyllaDB Enterprise version 2021.1.0-0.20210511.9e8e7d58b-1:
$ cat <<EOF | sudo tee /etc/apt/preferences.d/99scylla-enterprise
Package: scylla-enterprise*
Pin: version 2021.1.0-0.20210511.9e8e7d58b-1
Pin-Priority: 1001
EOF
Pinning may be particularly useful when you want to downgrade ScyllaDB or upgrade to a version that is not the latest available version. See this article for details about pinning on Debian-based systems.
Alternatively, you can explicitly install all the ScyllaDB packages for the desired non-latest version. For example:
sudo apt-get install scylla-enterprise{,-server,-jmx,-tools,-tools-core,-kernel-conf,-node-exporter,-conf,-python3}=2021.1.0-0.20210511.9e8e7d58b-1
sudo apt-get install scylla-enterprise-machine-image=2021.1.0-0.20210511.9e8e7d58b-1 # only execute on AMI instance
If you are creating a production cluster or if your cluster is going to have more than one data center you need to use a DC-aware snitch, e.g. GossipingPropertyFileSnitch or Ec2MultiRegionSnitch. You will also need to use a DC-aware replication strategy, e.g. NetworkTopologyStrategy.
However, if you are going to create your first cluster or want to try something simple, if your cluster is going to have a single data center then you may use a SimpleSnitch and then use a SimpleStrategy for your keyspaces.
Our general recommendation is to always use a NetworkTopologyStrategy and use Ec2XXX snitches on AWS based clusters and GossipingPropertyFileSnitch in all other cases.
A description of all snitch options we support may be found here: Snitches.
Note: trying to mix a SimpleSnitch with a DC-aware strategy or a DC-aware snitch with a SimpleStrategy may cause your cluster not to work as intended therefore we strongly discourage these types of configurations in general.
Not using a proper snitch-strategy combination may cause different types of errors.
For instance:
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level for cl LOCAL_ONE. Requires 1, alive 0" info={'required_replicas': 1, 'alive_replicas': 0, 'consistency': 'LOCAL_ONE'}
If you see this error you should always check that you are not using a SimpleSnitch in your cluster configuration in conjunction with some DC-aware replication strategy for a keyspace of a table you are failing to query.
When working with GossipingPropertyFileSnitch or Ec2MultiRegionSnitch you should edit the cassandra-rackdc.properties
For node using GossipingPropertyFileSnitch, the file should look like the following:
dc=asia_datacenter
rack=rack1
prefer_local= true
When the node is the Asia data center, on rack1 and to minimize BW usage for inter-datacenter, use the prefer_local
For Ec2MultiRegion the file should include the following information
dc_suffix=my_dc
This will create a suffix for the node location for example:
us-east1_my_dc
The problem may also arise if you are using some DC-aware snitch, e.g. Ec2MultiRegionSnitch, and a SimpleStrategy in a multi-DC cluster.
Please, make sure that both a snitch and a replication strategy of a keyspace are either both of a Simple kind or both are DC-aware.
After that, if you are using a DC-aware configuration, make sure that the replication strategy uses the proper data centers’ names. Verify the data centers names in your cluster using a nodetool status command.
Yes, but it will require running a full repair (or cleanup) to change the replica count of existing data:
Alter the replication factor for desired keyspace (using cqlsh for instance).
If you’re reducing the replication factor, run nodetool cleanup <updated Keyspace> on the keyspace you modified to remove surplus replicated data.
Cleanup runs on a per-node basis.
If you’re increasing the replication factor, refer to How to Safely Increase the RF
Note that you need to provide the keyspace namr. If you do not, the cleanup or repair operation runs on all keyspaces for the specific node.
listen_address to listen to 0.0.0.0 (all my addresses)?¶Scylla is a gossip-based distributed system and listen_address is the address a node tells other nodes to reach
it at. Telling other nodes “contact me on any of my addresses” is a bad idea; if different nodes in the cluster pick
different addresses for you, Bad Things happen.
If you don’t want to manually specify an IP to listen_address for each node in your cluster (understandable!), leave
it blank and Scylla will use InetAddress.getLocalHost() to pick an address. Then it’s up to you or your ops team
to make things resolve correctly (/etc/hosts/, dns, etc).
If using three node cluster, with RF=3, each node located on a different availability zone (AZ).
For example:
Datacenter: DC1
Status=Up/Down
State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.1.201 118.82 KB 256 33.6% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c A1
UN 192.168.1.202 111.82 KB 256 33.1% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c B1
UN 192.168.1.203 114.82 KB 256 33.3% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c C1
All nodes holds 100% of the data. If needed to add a single node to the cluster (scale out), the cluster will become imbalance. Because the single additional node will split the tokens only with the existing node in the same AZ.
Note
This is only an example, if having more nodes or different RF the number of nodes may be different.
The token distribution will be:
AZ A1 node A: 100% of the data
AZ B1 node B: 100% of the data
AZ C1 node C: 50% of the data
AZ C1 node D: 50% of the data
The solution is to add a node in each AZ.
Datacenter: DC1
Status=Up/Down
State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.1.201 118.82 KB 256 16.6% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c A1
UN 192.168.1.202 111.82 KB 256 16.1% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c B1
UN 192.168.1.203 114.82 KB 256 16.3% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c C1
UN 192.168.1.204 118.82 KB 256 16.6% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c A1
UN 192.168.1.205 111.82 KB 256 16.1% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c B1
UN 192.168.1.206 114.82 KB 256 16.3% 8d5ed9f4-7764-4dbd-bad8-43fddce94b7c C1
ScyllaDB Community Forum: Discuss using ScyllaDB and developing client applications.
scylladb-dev: Discuss the development of ScyllaDB itself.
Data you write to Scylla gets persisted to SSTables. Since SSTables are immutable, the data can’t actually be removed when you perform a delete, instead, a marker (also called a “tombstone”) is written to indicate the value’s new status. Never fear though, on the first compaction that occurs between the data and the tombstone, the data will be expunged completely and the corresponding disk space recovered.
Seeds are used during startup to discover the cluster. They are referred by new nodes on bootstrap to learn about other nodes in the ring. When you add a new node to the cluster, you must specify one live seed to contact.
In ScyllaDB versions earlier than Scylla Open Source 4.3 and Scylla Enterprise 2021.1, a seed node has an additional function: it assists with gossip convergence. See Scylla Seed Nodes for details.
We recommend updating your ScyllaDB to version 4.3 or later (Open Source) or 2021.1 or later (Enterprise).
The ring can operate or boot without a seed; however, you will not be able to add new nodes to the cluster. It is recommended to configure multiple seeds in production systems.
No, it is not required, but it is highly recommended when using Scylla with more than one drive. Scylla requires one drive for its data file and one drive for commit log (can be the same). If you want to take advantage of more than one drive, the easiest way to do so is set RAID0 (striped) across all of them. If you choose, scylla_setup will setup RAID0 for you on your selected drive, as well as XFS file system (recommended). Similarly, Scylla AMI on EC2 will automatically mount all available SSD drives in RAID0.
You can, but it is not recommended. Scylla clustering architecture already provides data replication across nodes and DCs. Adding another layer of replication in each node is redundant, slows down I/O operation and reduces available storage. Want a higher level of replication? Increase the Replication Factor (RF) of relevant Keyspaces.
JBOD is not supported by Scylla.
JBOD may be a reasonable solution for Cassandra because it rebuilds nodes very slowly. As this is not an issue for Scylla, it’s more efficient to use RAID.
Explanation: There are two types of deployment when multiple disks exist. In the JBOD case, each disk is an isolated filesystem. I/O isn’t stripped and thus performance can be slower than that of RAID. In addition, as the free space isn’t shared, a single disk can be full while the others are available.
The benefit of JBOD vs RAID is that it isolates failures to individual disk and not the entire node. However, Scylla rebuilds nodes quickly and thus it is not an issue when rebuilding an entire node.
As a result, it is much more advantageous to use RAID with Scylla
Nodetool Repair a Local (One Node) Operation or a Global (Full Cluster) Operation?¶When running nodetool repair on a node, it performs a repair on every token range this node owns; this will also repair other nodes that share the same range.
If you wish to repair the entire cluster, it is recommended to run nodetool repair -pr on each node in the cluster, sequentially, or use the ScyllaDB Manager.
You can restrict the number of items in the IN clause with the following options:
--max-partition-key-restrictions-per-query - Specifies the maximum number of distinct partition keys restrictions per query. This limit places a bound
on the size of IN tuples, especially when multiple partition key columns have IN restrictions. The default is 100.
--max-clustering-key-restrictions-per-query - Specifies the maximum number of distinct clustering keys restrictions per query. This limit
places a bound on the size of IN tuples, especially when multiple clustering key columns have IN restrictions. The default is 100.
Warning
We recommend that you use these options with caution. Changing the maximum number of IN restrictions to more than 100 may result in server instability.
The options can be configured on the command line, passed with SCYLLA_ARGS in /etc/default/scylla-server or /etc/sysconfig/scylla-server,
or added to your scylla.yaml (see Scylla Configuration).
You can make MV table an in-memory, as you would with every other table. It effects the total memory size allocation just like any other table. Make sure before you create any kind of in-memory table that its use case warrants the creation.
No. In Scylla Enterprise 2018.1.7, in-memory tables are always persistent using an on-disk mirror file.
Yes, by edit sysctl.d.
Procedure
Create /etc/sysctl.d/99-scylla-coredump.conf (this file exists by default in Scylla AMI).
Open the 99-scylla-coredump.conf file.
Add the following line kernel.core_pattern=|/<path>/<coredump_directory> %p %u %g %s %t %e"
For example:
kernel.core_pattern=|/home/centos/core/ %p %u %g %s %t %e"
Run sysctl -p /etc/sysctl.d/99-scylla-coredump.conf
upgradesstables when moving to a new format?¶Unlike Apache Cassandra, Scylla does not ship with upgradesstables, a tool that converts SSTables to newer formats. When upgrading to a new table format, Scylla can still continue to read from the old format. Having this option, ensures a smoother transition and upgrade. New writes use the new format and reads will use both formats until the old tables are removed. If you want to purge all of the old SSTables in a single step, generate a compaction with nodetool compact follow by nodetool cleanup to remove no longer needed token ranges that belong to that node.
Copyright
© 2016, The Apache Software Foundation.
Apache®, Apache Cassandra®, Cassandra®, the Apache feather logo and the Apache Cassandra® Eye logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.
Was this page helpful?
On this page
swap on a Scylla node?listen_address to listen to 0.0.0.0 (all my addresses)?Nodetool Repair a Local (One Node) Operation or a Global (Full Cluster) Operation?upgradesstables when moving to a new format?