4.7. Troubleshooting CouchDB 3 with WeatherReport¶
WeatherReport is an OTP application and set of tools that diagnoses
common problems which could affect a CouchDB version 3 node or cluster
(version 4 or later is not supported). It is accessed via the
weatherreport command line escript.
Here is a basic example of using
weatherreport followed immediately
by the command’s output:
$ weatherreport --etc /path/to/etc [warning] Cluster member email@example.com is not connected to this node. Please check whether it is down.
For most cases, you can just run the
weatherreport command as
shown above. However, sometimes you might want to know some extra
detail, or run only specific checks. For that, there are command-line
weatherreport --help to learn more about these
$ weatherreport --help Usage: weatherreport [-c <path>] [-d <level>] [-e] [-h] [-l] [check_name ...] -c, --etc Path to the CouchDB configuration directory -d, --level Minimum message severity level (default: notice) -l, --list Describe available diagnostic tasks -e, --expert Perform more detailed diagnostics -h, --help Display help/usage check_name A specific check to run
To get an idea of what checks will be run, use the –list option:
$ weatherreport --list Available diagnostic checks: custodian Shard safety/liveness checks disk Data directory permissions and atime internal_replication Check the number of pending internal replication jobs ioq Check the total number of active IOQ requests mem3_sync Check there is a registered mem3_sync process membership Cluster membership validity memory_use Measure memory usage message_queues Check for processes with large mailboxes node_stats Check useful erlang statistics for diagnostics nodes_connected Cluster node liveness process_calls Check for large numbers of processes with the same current/initial call process_memory Check for processes with high memory usage safe_to_rebuild Check whether the node can safely be taken out of service search Check the local search node is responsive tcp_queues Measure the length of tcp queues in the kernel
If you want all the gory details about what WeatherReport is doing,
you can run the checks at a more verbose logging level with
$ weatherreport --etc /path/to/etc --level debug [debug] Not connected to the local cluster node, trying to connect. alive:false connect_failed:undefined [debug] Starting distributed Erlang. [debug] Connected to local cluster node 'firstname.lastname@example.org'. [debug] Local RPC: mem3:nodes()  [debug] Local RPC: os:getpid()  [debug] Running shell command: ps -o pmem,rss -p 73905 [debug] Shell command output: %MEM RSS 0.3 25116 [debug] Local RPC: erlang:nodes()  [debug] Local RPC: mem3:nodes()  [warning] Cluster member email@example.com is not connected to this node. Please check whether it is down. [info] Process is using 0.3% of available RAM, totalling 25116 KB of real memory.
Most times you’ll want to use the defaults, but any syslog severity
name will do (from most to least verbose):
debug, info, notice,
warning, error, critical, alert, emergency.
Finally, if you want to run just a single diagnostic or a list of specific ones, you can pass their name(s):
$ weatherreport --etc /path/to/etc nodes_connected [warning] Cluster member firstname.lastname@example.org is not connected to this node. Please check whether it is down.