Rebooting CUCM servers, verifying system status and database replication

To preempt potential fallout from UC server reboots, I’ll outline the process for rebooting the servers, as well as verifying services states and database replication between the Publisher and subscriber servers.

Here’s the high level process:
1. Reboot the publisher
2. Check services status on publisher
3. Reboot subscriber(s)
4. Check services status on subscriber(s)
5. Verify dbreplication status on Publisher

To reboot CUCM servers, always start with the Publisher.   On each server, at the appropriate time, you will execute the following command via the CLI:

admin:utils system restart

Once the Publisher has rebooted and you’re able to login, execute the following command intermittently to verify all required services have started successfully:

admin:utils service list

Optionally, to generate a paginated output of the services status, run execute the command below:

admin:utils service list page

Once the publisher and all subscriber servers have been rebooted and the appropriate services have started successfully, execute the two commands below to verify database replication is established between servers and running successfully:

admin:utils dbreplication status

Executing the ‘utils dbreplication status’ command above will trigger a process to validate db replication between the publisher and subscriber servers. Here’s sample output from the command:

admin:utils dbreplication status 

Replication status check is now running in background. 

Use command 'utils dbreplication runtimestate' to check its progress 
The final output will be in file cm/trace/dbl/sdi/ReplicationStatus.2017_11_03_21_17_52.out 

Please use "file view activelog cm/trace/dbl/sdi/ReplicationStatus.2017_11_03_21_17_52.out" command to see the output

As noted in the output of the ‘utils dbreplication status’ command, you can copy and paste the command ‘file view activelog cm/trace/dbl/sdi/ReplicationStatus.2017_11_03_21_17_52.out’ to view the sync results of the individual tables in the database. This is useful once synchronization is complete if errors are reported during validation. Note that the log filename will differ each time you run the status command as the filename is generated based on the system time at the moment the command is executed. The log file is quite large and will take some time to review; as an alternative, you can execute the command below to view a summary of the synchronization process:

admin:utils dbreplication runtimestate

The ‘utils dbreplication runtimestate’ command provides a summary of the validation process. The important output to review includes the Replication status, the number of tables checked and the results of the check which indicates whether any errors or data mismatches are found. Additionally you will want to check RTMT value (2 is good) to determine whether replication setup is completed and data is synchronizing successsfully.

admin:utils dbreplication runtimestate 

Server Time: Fri Nov  3 21:19:02 EDT 2017 

Cluster Replication State: Replication status command started at: 2017-11-03-21-17 
Replication status command in PROGRESS. Checked 412 tables out of 692 
Last Completed Table: aardialprefixmatrix 
No Errors or Mismatches found. 

Use 'file view activelog cm/trace/dbl/sdi/ReplicationStatus.2017_11_03_21_17_52.out' to see the details 

DB Version: ccm11_0_1_23900_5 
Repltimeout set to: 300s 
PROCESS option set to: 1 

Cluster Detailed View from dpccmpub (2 Servers): 
                                      PING      DB/RPC/   REPL.    Replication    REPLICATION SETUP 
SERVER-NAME         IP ADDRESS        (msec)    DbMon?    QUEUE    Group ID       (RTMT) & Details 
-----------         ----------        ------    -------   -----    -----------    ------------------ 
dpccmpub            172.31.0.210      0.031     Y/Y/Y     0        (g_2)          (2) Setup Completed 
dpccmsub1           172.31.3.110      1.157     Y/Y/Y     1038     (g_3)          (2) Setup Completed