How to Report a ScyllaDB Problem¶

ScyllaDB Doctor¶

ScyllaDB Doctor is a troubleshooting tool that checks the node status regarding system requirements, configuration, and tuning. The collected information is output as a .vitals.json file and an archive file with ScyllaDB logs. You need to run the tool on every node in the cluster.

1. Download ScyllaDB Doctor as a Linux package or a generic tarball:

   • Ubuntu/Debian (DEB): https://downloads.scylladb.com/downloads/scylla-doctor/deb/

   • RHEL/Rocky (RPM): https://downloads.scylladb.com/downloads/scylla-doctor/rpm/

   • Tarball: https://downloads.scylladb.com/downloads/scylla-doctor/tar/

2. Run ScyllaDB Doctor on every node in the cluster.

   • If you installed ScyllaDB Doctor with DEB or RPM, you can run it with the scylla-doctor command.

   • If you downloaded the tarball, extract the scylla_doctor.pyz file and copy the file to all nodes in the cluster. Next, execute the following command from the directory where you copied scylla_doctor.pyz on each node:

     sudo ./scylla_doctor.pyz --save-vitals <unique-host-id>.vitals.json
     

     Make sure you provide a unique host identifier in the filename, such as the host IP.

   Running ScyllaDB Doctor will generate:

   • <unique-host-id>.vitals.json - ScyllaDB Doctor vitals

   • scylla_logs_<timestamp>.tar.gz - ScyllaDB logs

   Authenticated Clusters

   If CQL authentication is enabled on the cluster, you need to additionally provide CQL credentials with permissions to perform the DESCRIBE SCHEMA command using the following parameters:

   -sov CQL,user,<CQL user name> -sov CQL,password,<CQL password>
   

   ScyllaDB Doctor employs cqlsh installed on a given node using the provided credentials. Make sure to set up any additional configuration required to use cqlsh, such as TLS-related information, in the .cqlshrc file.

3. Collect the .vitals.json and log files from each node into a local directory with a name identifying your cluster and compress them into an archive. In the following example, the Linux tar command is used to compress the files in the my_cluster_123 directory:

   tar czvf my_cluster_123_vitals.tgz my_cluster_123
   

4. Upload the archive using the instructions in the Send files to ScyllaDB support section.

Core Dump¶

When ScyllaDB fails, it creates a core dump which can later be used to debug the issue. The file is written to /var/lib/scylla/coredump. If there is no file in the directory, see Troubleshooting Core Dump.

xz -z core.21692

Troubleshooting Core Dump¶

In the event the /var/lib/scylla/coredump directory is empty, the following solutions may help. Note that this section only handles some of the reasons why a core dump file is not created. It should be noted that in some cases where a core dump file fails to create not because it is in the wrong location or because the system is not configured to generate core dump files, but because the failure itself created an issue where the core dump file wasn’t created or is not accessible.

Report a performance problem¶

If you are experiencing a performance issue when using ScyllaDB, let us know and we can help. To save time and increase the likelihood of a speedy solution, it is important to supply us with as much information as possible.

Include the following information in your report:

• Complete ScyllaDB Doctor Vitals

• A Server Metrics Report

• A Client Metrics Report

• The contents of your tracing data. See Collecting Tracing Data.

scyllatop *gauge-load

sar -P ALL

perf top -C0

ostat -x 1`` to observe the disk utilization.

sudo perf record --call-graph dwarf -C 0 -F 99 -p $(ps -C scylla -o pid --no-headers) -g sleep 10

#!/bin/bash -e

mkdir report
rpm -qa > ./report/rpm.txt
journalctl -b > ./report/journalctl.txt
df -k > ./report/df.txt
netstat > ./report/netstat.txt

sar -P ALL > ./report/sar.txt
iostat -d 1 10 > ./report/iostat.txt
sudo perf record --call-graph dwarf -C 0 -F 99 -p $(ps -C scylla -o pid --no-headers) -g --output ./report/perf.data sleep 10

export report_uuid=$(uuidgen)
echo $report_uuid
tar c report | xz > report.tar.xz
curl --request PUT --upload-file report.tar.xz "scylladb-users-upload.s3.amazonaws.com/$report_uuid/report.tar.xz"
echo $report_uuid

docker ps

sudo docker cp a64bf3ba0b7f:/prometheus /tmp/prometheus_data

sudo tar -zcvf /tmp/prometheus_data.tar.gz /tmp/prometheus_data/

Send files to ScyllaDB support¶

Once you have collected and compressed your reports, send them to ScyllaDB for analysis.

Procedure

1. Generate a UUID:

export report_uuid=$(uuidgen)
echo $report_uuid

2. Upload all required report files:

curl -X PUT https://upload.scylladb.com/$report_uuid/yourfile -T yourfile

For example with the Scylla Doctor’s vitals:

curl -X PUT https://upload.scylladb.com/$report_uuid/my_cluster_123_vitals.tgz -T my_cluster_123_vitals.tgz

The UUID you generated replaces the variable $report_uuid at runtime. yourfile is any file you need to send to ScyllaDB support.

Open a support ticket or GitHub issue¶

If you have not done so already, supply ScyllaDB support with the UUID. Keep in mind that although the ID you supply is public, only ScyllaDB support team members can read the data you share. In the ticket/issue you open, list the documents you have uploaded.

Procedure

1. Do one of the following:

• If you are a ScyllaDB customer, open a Support Ticket and include the UUID within the ticket.

• If you are a ScyllaDB user, open an issue on GitHub and include the UUID within the issue.