Chapter 7. Administering WombatDialer

7.1. Installing

WombatDialer is easily installed using yum or it can be installed manually. It can be installed on the same server as Asterisk is (for small systems or testing) but on larger production systems we recomment that it should be installed on a dedicated system (even a virtual machine will do).

WombatDialer can share a Tomcat instance with QueueMetrics for smaller systems (up to 50 agents/50 lines) but it should be really installed on a separate machine for larger systems, as it might use large quantities or RAM and CPU.

[Tip]

WombatDialer can also be easily deployed using Docker, as explained in Docker installation Section 7.1.5, “Installing on Docker”. This is perfect for installing a test system; and although currently experimental, we have run real-life workloads on it.

7.1.1. Installing using yum

On a CentOS or derived Linux distribution, just run the following commands:

wget -P /etc/yum.repos.d http://yum.loway.ch/loway.repo

yum install wombat

This will in turn:

  • install Java 8
  • install Tomcat 8.5
  • install MariaDB or MySQL server (if missing)
  • install the dialer

At the moment we support CentOS 6 and 7. If you run CentOS 5, it is time to upgrade - if you really cannot, please see https://www.queuemetrics.com/faq.jsp?uid=faq-110-java8-centos5

7.1.2. Manual installation

You can install WombatDialer within any working servlet container version 2.5 or newer - we suggest Tomcat 8.5. You will also need a working MariaDB or MySQL server. OpenJDK 8+ is fine.

  • Download the latest version of WombatDialer as a .tar.gz package from the website
  • Unpack the file in your servlet containers’s webapps directory and rename the folder to "wombat"
  • Restart your servlet container

Example: manual installation on Debian / Ubuntu

The following example was tested on an Ubuntu 18.04 system. Debian systems should be very similar.

First make sure you are running your shell as as root.

The main dependencies are OpenJDK and MariaDB. WombatDialer works with MySQL as well.

apt-get install tomcat8 mariadb-server

Then let’s download WombatDialer:

cd /var/lib/tomcat8/webapps/
wget ..../WombatDialer-18.08.1-215.tar.gz
tar zxvf WombatDialer-18.08.1-215.tar.gz
mv wombatdialer-18.08.1/ wombat/

Now we install the initial database schema - we will create a database called "wombat" on the local DB.

cd /var/lib/tomcat8/webapps/wombat/WEB-INF/DB
mysql < wombat_create.sql
mysql -uwombat -pdials wombat < wombat_schema.sql

Now we restart Tomcat so chnges will be picked up:

service tomcat8 restart

Now connect to http://127.0.0.1:8080/wombat and follow the on-line wizard to create/update the database.

The default installers for both Tomcat 8 and MariaDB start their services automatically on boot.

7.1.3. On installation complete….

When installation is done, connect to http://myserver.address:8080/wombat - you will be asked for the MySQL root password to create a local database When done, login using as user demoadmin password demo.

WombatDialer ships with a free demo key that lets you use up to two lines - no limit on the number of calls you make. You may want to get a free demo key so you can run a full evaluation of the software in a real-life scenario.

License keys can be installed from the License page Section 4.8, “The License page” as soon as you log in.

7.1.4. Configuring e-mail

If you plan to use e-mail notifications, you should edit the tpf.properties file to tell WD about your SMTP servers and the e-mail address it is supposed to use.

SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_AUTH=yes
SMTP_USER=your-gmail-account@gmail.com
SMTP_PASSWORD=wombat
SMTP_USE_SSL=no
SMTP_FROM="WombatDialer" <your-gmail-account@gmail.com>
SMTP_DEBUG=yes

As e-mail sending happens on the notification thread, a very slow SMTP server might delay HTTP notifications. In this case, we suggest setting up a local SMTP relay that will immediately accept e-mail and will forward it to the "true" SMTP host.

We suggest leaving SMTP_DEBUG on during the initial setup as it writes a verbose log of all SMTP activity on the logs and lets you spot errors and issues quickly.

7.1.5. Installing on Docker

Docker is an open-source project that automates the deployment of applications inside software containers, by providing an additional layer of abstraction and automation of operating system-level virtualization on Linux.

When you have Docker installed, starting a temporary instance of WombatDialer is as simple as typing:

docker run -p 8080:8080 -d loway/wombatdialer

This will download the current WombatDialer (already pre-configured for you) and will run it by exposing it on the local port 8080.

The problem with this approach is that when the image is terminated, all data is lost.

A better approach would be to create a data-only container to store the data. At this point, every time you start WombatDialer, it looks for its own data into the data-only container.

You start by creating a named data-only container:

docker run --name=MYWBT loway/data true

At this point, you can run WombatDialer using that container, as in:

docker run --volumes-from MYWBT -p 8080:8080  -d loway/wombatdialer

WombatDialer will recognize whether the data container is blank and will initialize it properly. The container also monitors the health of the running WombatDialer process and will restart it as needed.

If you prefer, you may use a local folder instead of a data-only container, as in:

docker run -v /root/WBT1DATA:/data -p 9090:8080 -d loway/wombatdialer

This command mounts the local folder /root/WBT1DATA for WombatDialer to store its data in, and - for a change - will expose WombatDialer on port 9090.

You can also control the local time-zone and the amount of memory used, by passing them as parameters when starting up the image:

docker run -e CFG='{"memory":400,"timezone":"GMT"}' -p 8080:8080 -d loway/wombatdialer

When you need to upgrade WombatDialer - you simply stop the current image and start a new one using the same data container. Your dialer is updated!

7.2. Upgrading

Upgrading WombatDialer is meant to be very easy - it is in any case worthwhile to make a complete backup (see below) before attempting an upgrade. If you run WombatDialer on a virtual machine, the easiest way might be taking a snapshot of the whole box before attempting the upgrade.

[Warning]

Before attempting an upgrade, always make sure that no calls are in progress and that the dialer is turned OFF. If you have running campaigns, you should pause them and wait for relevant calls to terminate.

If you update from a version of WombatDialer before 18 to version 18+, the database driver was changed. Look for your tpf.properties file and replace the entry:

JDBC_DRIVER=com.mysql.jdbc.Driver
JDBC_URL=jdbc:mysql://127.0.0.1/wombat?user=wombat&password=dials&useUnicode=true&characterEncoding=UTF-8

With:

JDBC_DRIVER=org.mariadb.jdbc.Driver
JDBC_URL=jdbc:mariadb://127.0.0.1/wombat?user=wombat&password=dials&autoReconnect=true

And restart.

7.2.1. Upgrading via yum

If you originally installed via yum, this should be enough:

yum update wombat

System settings should be automatically transferred.

When the update is complete, on the first connection you might be asked to upgrade the database - the dialer will not start until the database is up to date. Database upgrades will be applied automatically.

[Tip]

If after an upgrade you get an error like java.util.MissingResourceException: Can't find bundle for base name org.eclipse.jdt.internal.compiler.problem.messages, this means that there are cached classes created with a different version of the JDK.

You can have them flushed by issuing the commands:

service qm-tomcat6 cleancache
service qm-tomcat6 start

7.2.2. Manual upgrade

  • Stop the servlet container
  • Download the latest version of WombatDialer as a .tar.gz package from the website
  • Move the current "wombat" webapp to a storage place (do not delete it!)
  • Unpack the file in your servlet containers’s webapps directory and rename the folder to "wombat"
  • Copy the tpf.properties file from the old wombat webapp to wombat/WEB-INF
  • Restart your servlet container

When the update is complete, on the first connection you might be asked to upgrade the database - the dialer will not start until the database is up to date. Database upgrades will be applied automatically.

7.3. Starting and stopping

If you installed using yum, you should be able to run:

/etc/init.d/qm-tomcat6 stop
/etc/init.d/qm-tomcat6 start

to stop and start the Tomcat servlet container and thence WombatDialer.

If you installed manually, see the user manual for your servlet container.

[Warning]

Before stopping, always make sure that no calls are in progress and that the dialer is turned OFF. If you have running campaigns, you should pause them and wait for all relevant calls to terminate. If not, those calls will end up in status LOST.

7.3.1. The dialer status at startup

The dialer will usually start automatically when the webapp is loaded. Exceptions to this behavior happen if:

  • WombatDialer was updated and the database required updating. In this case the dialer will be turned off until manually activated.
  • There is no valid license installed.

If you prefer to avoid the dialer starting automatically, you can do this by creating a flag file called 'donotstart' to be placed under WEB-INF, like e.g.

touch ./WEB-INF/donotstart

In order to revert to the previous behavior, simply remove the flag file.

7.4. Backing up

All the information WombatDialer requires to run is contained withn the wombat MySQL database. Backing it up should suffice in most cases.

If you use a different database that the defsult MySQL on localhost, then you need to back-up the file wombat/WEB-INF/tpf.properties that contains the database connection information.

For ease of recovery, you may also want to back up the whole "wombat" webapp.

7.5. Logs and logging

User-serviceable logs are available on the syslog viewer page. This tracks errors, agent logins, campaign activity and many common user-facing conditions.

A log of severe errors is also held in the Tomcat logs directory, usually with file names of the format wombat.2013-01-05.txt or similar.

This behavior can be customized by editing the file WEB-INF/classes/logging.properties in the WombatDialer webapp.

org.apache.juli.FileHandler.level = SEVERE
org.apache.juli.FileHandler.directory = ${catalina.base}/logs
org.apache.juli.FileHandler.prefix = wombat.

java.util.logging.ConsoleHandler.level = SEVERE
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

You can turn off logging by setting the logging level to NONE, or create extra-detailed and huge logs by setting the log level to FINE.

An extensive tutorial on Tomcat logging can be found at http://wiki.apache.org/tomcat/Logging_Tutorial

[Caution]

Please note that running very detailed logs on a production system may create huge files and may render the system unusable by generating too many I/O requests.

7.6. Troubleshooting a running instance

Problems with a WD instance are usually either on the Java side or on the database side. We present a few useful techniques to gather important data to be sent over for inspection.

7.6.1. Creating a thread dump

In order to diagnose problems with WombatDialer, it may be needed to take thread dumps by running:

/etc/init.d/qm-tomcat6 threaddump > dump_2013-01-18.txt

These dumps prints what the threads are doing at a point in time. It may be worthwhile to take multiple ones, a few minutes apart from each other.

7.6.2. Creating a TCP dump

In order to diagnose problems with WombatDialer, it may be needed to take TCP dumps:

First off you can check your network interfaces by typing:

tcpdump -D

the result will be something like

1.eth0
2.any (Pseudo-device that captures on all interfaces)
3.lo

Issue the dump on port 5038 (remember to add the -s option to avoid capturing only packet’s headers)

tcpdump -i lo -s 65535 -w dump.pcap port 5038

This will generate a file called dump.pcap which can be analized using Wireshark.

7.6.3. Creating a heap dump

On request, you may need to create a heap dump. This is a large file containing the full contents of the live JVM.

[root@wombat1 ~]# /usr/local/queuemetrics/java/bin/jps
1900 Jps
30808 Bootstrap

The line containing Bootstrap contains the PID of the Java VM running Tomcat.

[root@wombat1 ~]# /usr/local/queuemetrics/java/bin/jmap -F -dump:live,format=b,file=heap.bin 30808
Attaching to process ID 30808, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 17.1-b03
Dumping heap to heap.bin ...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Heap dump file created

The resulting file (called heap.bin as specified on the command line) might be very large and it might take a while to create on busy systems with large Java heaps.

Before sending the file over it is better to compress it:

[root@wombat1 ~]# ls -l heap.bin
-rw-r--r-- 1 root root 40779016 Jan 10 08:45 heap.bin
[root@wombat1 ~]# bzip2 heap.bin
[root@wombat1 ~]# ls -l heap.bin.bz2
-rw-r--r-- 1 root root 8055474 Jan 10 08:45 heap.bin.bz2

7.6.4. Logging all MySQL queries

Edit the my.cnf file (usually in /etc/mysql/my.cnf). Look for the section called "Logging and Replication".

#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.

log = /var/log/mysql-all-queries.log

Just uncomment the "log" entry to turn on logging. Restart MySQL with this command:

service mysql restart

7.6.5. Logging of "slow queries" in MySQL

Edit file my.cnf and add the following parameters:

[mysqld]
log-slow-queries=/var/log/mysql-slow-queries.log
long_query_time = 2
log-queries-not-using-indexes

This will log all queries taking more than two seconds or running without a defined index to the slow queries log.

Then restart MySQL.

7.6.6. Database Clean-up Script

In order to manage WombatDialer’s database size, a clean-up script is included in WombatDialer’s installation package. The script file can be found in the application folder at the following path:

WEB-INF/DB/cleanup_wbt.sh

Before you run this script make sure you give it the appropriate permissions it needs to be executed.

This script, when executed, searches for old data that is no longer useful or has been hidden by the user. To decide how old must this data be in order for it to be deleted, the user must input a date that will be considered its clean-up treshold.

There are two areas where data can be deleted:

  • Campaign runs that are hidden, or happened before the given threshold date. This means that the script will search for all runs of campaigns that are currently hidden that happened before the input date. The runs will be removed; if the campaign has no other runs, then it will be removed as well. If a campaign is deleted, all logs associated with the campaign will be deleted as well. All System logs records before the given date are erased as well.
  • Unused lists. If a list is hidden, and is used by no campaign or runs, then the list and all numbers and attributes are removed. If you run this after cleaning old campaigns, lists used by those campaigns will be removed.

When all the data has been erased, the script will initiate a procedure to rebuild all tables and indexes, thus optimizing and improving database access times.

[Caution]

Please note that in order to succesfully run this script, WombatDialer must be turned off before running it. On a multi-gigabyte database, plan a few hours of possible downtime.

[Warning]

Running this script will cause some data to be deleted permanently. It is possible for the script to leave the database in an inconsistent state. It is mandatory that you make a backup copy of your database before running this script.

7.6.7. Running the clean-up script

After navigating to the correct folder and setting the appropriate permissions to the script file, you can execute it by typing:

./cleanup_wbt.sh -u wombat -p dials -d wombat -t 2017-12-31 -m all

As you can see, we need to input five parameters to execute the script. These parameters are all mandatory except for one:

  • -u: MySQL user parameter (Mandatory)
  • -p: MySQL password parameter (Mandatory)
  • -d: MySQL WombatDialer database name (Mandatory)
  • -t: Date threshold to identify old data (yyyy-mm-dd). Data more recent than this date will not be considered for elimination. (Mandatory)
  • -m: Mode selection. Available modes are: list, delete, optimize, preview, all. The default mode is all.

There are five different modes for the script:

  • list: Does not delete any data, only shows hopper elements that would be removed in a normal execution.
  • preview: Does not delete any data, only shows hopper elements and call lists that would be removed in a normal execution.
  • delete: Deletes all old data, without table optimization.
  • optimize: Optimizes all the tables.
  • all: Default mode. Executes all the passages in order.

We suggest manually running the script as preview first; then delete and then optimize.