Upgrade Guide - ScyllaDB 5.0 to ScyllaDB Enterprise 2022.1 for UbuntuCopy

Check the cluster schemaCopy

Make sure that all nodes have the schema synched before the upgrade. The upgrade will fail if there is a schema disagreement between nodes.

nodetool describecluster

Drain the nodes and backup the dataCopy

Before any major procedure, like an upgrade, it is recommended to backup all the data to an external device. In ScyllaDB, backup is done using the nodetool snapshot command. For each node in the cluster, run the following command:

nodetool drain
nodetool snapshot

Take note of the directory name that nodetool gives you, and copy all the directories having this name under /var/lib/scylla to a backup device.

When the upgrade is completed on all nodes, the snapshot should be removed with the nodetool clearsnapshot -t <snapshot> command, or you risk running out of space.

Backup the configuration filesCopy

for conf in $(cat /var/lib/dpkg/info/scylla-*server.conffiles /var/lib/dpkg/info/scylla-*conf.conffiles /var/lib/dpkg/info/scylla-*jmx.conffiles | grep -v init ) /etc/systemd/system/{var-lib-scylla,var-lib-systemd-coredump}.mount; do sudo cp -v $conf $conf.backup-5.0; done

Gracefully stop the nodeCopy

sudo service scylla-server stop

Download and install the new releaseCopy

Before upgrading, check what version you are running now using dpkg -l scylla\*server. You should use the same version in case you want to rollback the upgrade. If you are not running a 5.0.x version, stop right here! This guide only covers 5.0.x to 2022.1.y upgrades.

To upgrade:

1. Update the Scylla Enterprise Deb repo to 2022.1.

2. Install:

   sudo apt-get clean all
   sudo apt-get update
   sudo apt-get remove scylla\*
   sudo apt-get install scylla-enterprise
   for conf in $(cat /var/lib/dpkg/info/scylla-*server.conffiles /var/lib/dpkg/info/scylla-*conf.conffiles /var/lib/dpkg/info/scylla-*jmx.conffiles | grep -v init ) /etc/systemd/system/{var-lib-scylla,var-lib-systemd-coredump}.mount; do sudo cp -v $conf.backup-5.0 $conf; done
   sudo systemctl daemon-reload
   

   Answer ‘y’ to the first two questions.

   If you use a cloud image with a preinstalled version of ScyllaDB (for example, AMI), you need to install an additional package scylla-enterprise-machine-image:

   sudo apt-get install scylla-enterprise-machine-image
   sudo systemctl daemon-reload
   

Start the nodeCopy

New io.conf format was introduced in ScyllaDB 2.3 and 2019.1. If your io.conf doesn’t contain –io-properties-file option, then it’s still the old format. You need to re-run the io setup to generate new io.conf.

sudo scylla_io_setup

sudo service scylla-server start

ValidateCopy

1. Check cluster status with nodetool status and make sure all nodes, including the one you just upgraded, are in UN status.

2. Use curl -X GET "http://localhost:10000/storage_service/scylla_release_version" to check the ScyllaDB version.

3. Check scylla-server log (by journalctl _COMM=scylla) and /var/log/syslog to validate there are no errors.

4. Check again after 2 minutes to validate no new issues are introduced.

Once you are sure the node upgrade is successful, move to the next node in the cluster.

More on ScyllaDB Metrics Update - ScyllaDB 5.0 to 2022.1

Gracefully shutdown ScyllaDBCopy

nodetool drain
sudo service scylla-server stop

Download and install the old releaseCopy

1. Remove the old repo file.

   sudo rm -rf /etc/apt/sources.list.d/scylla.list
   

2. Update the ScyllaDB deb repo to 5.0.

3. Install:

   sudo apt-get clean all
   sudo apt-get update
   sudo apt-get remove scylla\* -y
   sudo apt-get install scylla
   

Answer ‘y’ to the first two questions.

Restore the configuration filesCopy

for conf in $(cat /var/lib/dpkg/info/scylla-*server.conffiles /var/lib/dpkg/info/scylla-*conf.conffiles /var/lib/dpkg/info/scylla-*jmx.conffiles | grep -v init ) /etc/systemd/system/{var-lib-scylla,var-lib-systemd-coredump}.mount; do sudo cp -v $conf.backup-5.0 $conf; done
sudo systemctl daemon-reload

Restore system tablesCopy

Restore all tables of system and system_schema from the previous snapshot because 2022.1 uses a different set of system tables. See Restore from a Backup and Incremental Backup for reference.

cd /var/lib/scylla/data/keyspace_name/table_name-UUID/
sudo find . -maxdepth 1 -type f  -exec sudo rm -f "{}" +
cd /var/lib/scylla/data/keyspace_name/table_name-UUID/snapshots/<snapshot_name>/
sudo cp -r * /var/lib/scylla/data/keyspace_name/table_name-UUID/
sudo chown -R scylla:scylla /var/lib/scylla/data/keyspace_name/table_name-UUID/

Start the nodeCopy

sudo service scylla-server start

ValidateCopy

Check the upgrade instruction above for validation. Once you are sure the node rollback is successful, move to the next node in the cluster.