Access Keys:
Skip to content (Access Key - 0)









How-to Use Rhino Housekeeping

Print this page

Introduction

Rhino has housekeeping commands for interrogating the SLEE about application state, and for cleaning up or removing unwanted state. Application state may be unwanted in some dropped call scenarios caused by failures external to the SLEE. The housekeeping interface allows for activities and SBB's to be located using time based searches to isolate old or stale application state.

Most Rhino housekeeping API's are provided by the RhinoHousekeepingMBean and can be accessed using JMX along with other management functions. There are also commands available in rhino-console for accessing the housekeeping API's. This guide will focus on the use of the interactive rhino-console for housekeeping tasks rather than programmatically using the API's.

Main Body

The available housekeeping commands in rhino-console can be loosely broken down into three categories: Query - searching, query - detail, and cleanup/removal.

Query - searching

  • findactivities
  • findsbbs
  • findtimers
  • findactivitybindings
  • findlongrunningtransactions
  • listactivealarms

Query - examining state in detail

  • getactivityinfo
  • getsbbinfo
  • getattachedactivities
  • getclusterstate

Commands for removing or clearing state

  • removeactivity
  • removesbb
  • removeallactivities
  • removeallsbbs
  • rollbacklongrunningtransaction
  • clearalarm
  • clearalarms
  • canceltimer
  • removeactivitybinding

Example usage: findactivities and getactivityinfo

The findactivities command provides the following options to narrow the search for activities:

  • By RA entity, using the -ra option activities from only a single RA entity can be selected.
  • By last update time, using the -updated-after and -updated-before options activities that were last updated before and/or after certain times can be selected.
  • By creation time, using the -created-before and -created-after options activities that were originally created before and/or after certain times can be selected.
  • By Rhino cluster node using the -node option to select only activities belonging to a particular node.

Time based options like -created-before and -updated-before will accept arguments either as an absolute time and date (e.g. 'YYYY/mm/dd HH:mm:ss') or a relative time expressed as a number of days, hours, minutes, and seconds prior to the current time. Any combination of 'd', 'h', 'm' and 's' can be used. (e.g. '1d' for 1 day ago, or '2h30m' for 2 hours 30 minutes ago).

Finally there is a -max option to limit the number of results returned. For performance reasons housekeeping find commands return a finite number of results, by default at most 100 entries per SLEE node. The -max option can be used to override this and provide a higher (or lower) limit. Note the -max option applies to every cluster member queried, so the number of results may be up to max * number of cluster members.

The following example locates activites belong to the RA entity 'sipra' created more than 1 hour ago:

[Rhino@localhost (#6)] findactivities -ra sipra -created-before 1h
pkey                       handle                           ra-entity   replicated   submission-time     update-time         
-------------------------  -------------------------------  ----------  -----------  ------------------  ------------------  
 65.6.479DBE10.0.61D2D6EE   SIPDialogAH[101,1,-1070693210]       sipra        false   20080128 19:39:48   20080128 19:39:49  
 65.5.479DBE10.2.7C9FCFB3   SIPDialogAH[101,0,-1070693210]       sipra        false   20080128 19:39:48   20080128 19:39:49  
 65.A.479DBE10.2.35B5D666   SIPDialogAH[101,2,-1070693210]       sipra        false   20080128 19:39:48   20080128 19:39:49  
 65.9.479DBE10.2.6B5B477D   SIPDialogAH[101,3,-1070693210]       sipra        false   20080128 19:39:48   20080128 19:39:49  

4 rows

Four activities are found. The first column gives the primary key of the activity. This uniquely identifies the activity throughout the SLEE cluster and is used as the argument for other activity related commands like getactivityinfo, removeactivity and removeactivitybinding.

To get additional information about an activity, including the SBBs it is attached to use the getactivityinfo command:

[Rhino@localhost (#12)] getactivityinfo 65.6.479DBE10.0.61D2D6EE
pkey             : 65.6.479DBE10.0.61D2D6EE
busy             : false
ending           : false
events-processed : 1
handle           : SIPDialogAH[101,1,-1070693210]
processing-node  : 101
queue-size       : 0
ra-entity        : sipra
reference-count  : 1
replicated       : false
sbbs-invoked     : 1
submission-time  : 20080128 19:39:48
submitting-node  : 101
update-time      : 20080128 20:20:47
attached-sbbs    : 
       > pkey                 replicated   sbb-component-id              service-component-id   
       > -------------------  -----------  ----------------------------  ---------------------  
       >  101:102008760977:0        false   UASDialogSbb^Open Cloud^1.1   B2BUA^Open Cloud^1.1  

       > 1 rows

events           : 
no rows

Searching for SBB's

The findsbbs command is used for searching for SBB's. Unlike findactivities, findsbbs has one mandatory parameter the component id of the service to search within (-service option). It is also possible to search only for SBB's of a particular type within a service using the -sbb option. The other arguments for findsbbs are the same as for findactivities, except findsbbs does not support searching by last update time only creation time.
The following example shows searching for all SBB's created by the B2B UA service (one of the SIP example services provided with Rhino):

[Rhino@localhost (#10)] findsbbs -service B2BUA\ 1.1,\ Open\ Cloud 
pkey                 creation-time       parent-pkey          replicated   sbb-component-id               service-component-id   
-------------------  ------------------  -------------------  -----------  -----------------------------  ---------------------  
 101:102008761146:0   20080128 19:39:48   101:102008760917:0        false   ACLocationSbb^Open Cloud^1.6   B2BUA^Open Cloud^1.1  
 101:102008760917:1   20080128 19:39:48   101:102008760851:0        false        ProxySbb^Open Cloud^1.7   B2BUA^Open Cloud^1.1  
 101:102008761020:0   20080128 19:39:48   101:102008760851:1        false    UACDialogSbb^Open Cloud^1.1   B2BUA^Open Cloud^1.1  
 101:102008761020:1   20080128 19:39:48   101:102008760851:0        false    UACDialogSbb^Open Cloud^1.1   B2BUA^Open Cloud^1.1  
 101:102008760977:1   20080128 19:39:48   101:102008760851:0        false    UASDialogSbb^Open Cloud^1.1   B2BUA^Open Cloud^1.1  
 101:102008760977:0   20080128 19:39:48   101:102008760851:1        false    UASDialogSbb^Open Cloud^1.1   B2BUA^Open Cloud^1.1  
 101:102008760851:0   20080128 19:39:48                             false           B2BUA^Open Cloud^1.1   B2BUA^Open Cloud^1.1  
 101:102008760917:0   20080128 19:39:48   101:102008760851:1        false        ProxySbb^Open Cloud^1.7   B2BUA^Open Cloud^1.1  
 101:102008760851:1   20080128 19:39:48                             false           B2BUA^Open Cloud^1.1   B2BUA^Open Cloud^1.1  
 101:102008761146:1   20080128 19:39:48   101:102008760917:1        false   ACLocationSbb^Open Cloud^1.6   B2BUA^Open Cloud^1.1  

10 rows

The above 10 rows represent two entity trees in the B2BUA service. Each entity tree has a B2BUA SBB at its root (as it is the root SBB for the service), and 4 children (the ACLocationSbb is an indirect child via the proxy SBB).

The first column returned, pkey, is the SBB primary key. Unlike the activity primary key the SBB primary key does not uniquely identify an SBB entity within the entire cluster - SBB primary keys are allocated uniquely across the entire cluster but within a single SBB component. For this reason other commands that accept an SBB primary key as an argument such as getsbbinfo or removesb}}b also require the service and SBB component id's as mandatory arguments. The {{parent-pkey column is the primary key of the SBB's parent SBB (and is null for the root SBB).

The following example extracts additional information about one of the SBB's returned above:

[Rhino@localhost (#14)] getsbbinfo B2BUA\ 1.1,\ Open\ Cloud B2BUA\ 1.1,\ Open\ Cloud 101:102008760851:1
parent-pkey          : 
pkey                 : 101:102008760851:1
convergence-name     : :::::56:2-19749@fe80::217:f2ff:fee8:da99%en1:z9hG4bK-19749-2-0:1
creating-node-id     : 101
creation-time        : 20080128 19:39:48
priority             : 0
replicated           : false
sbb-component-id     : B2BUA^Open Cloud^1.1
service-component-id : B2BUA^Open Cloud^1.1
attached-activities  : 
no rows

The above SBB is a root SBB (it's parent-pkey is null), so removing it will remove an entire entity tree. Below we use the removesbb command to remove the entity tree, leaving only 5 SBB's:

[Rhino@localhost (#15)] removesbb B2BUA\ 1.1,\ Open\ Cloud B2BUA\ 1.1,\ Open\ Cloud 101:102008760851:1 
[Rhino@localhost (#16)] findsbbs -service B2BUA\ 1.1,\ Open\ Cloud                                     
pkey                 creation-time       parent-pkey          replicated   sbb-component-id               service-component-id   
-------------------  ------------------  -------------------  -----------  -----------------------------  ---------------------  
 101:102008760917:1   20080128 19:39:48   101:102008760851:0        false        ProxySbb^Open Cloud^1.7   B2BUA^Open Cloud^1.1  
 101:102008761020:1   20080128 19:39:48   101:102008760851:0        false    UACDialogSbb^Open Cloud^1.1   B2BUA^Open Cloud^1.1  
 101:102008760977:1   20080128 19:39:48   101:102008760851:0        false    UASDialogSbb^Open Cloud^1.1   B2BUA^Open Cloud^1.1  
 101:102008760851:0   20080128 19:39:48                             false           B2BUA^Open Cloud^1.1   B2BUA^Open Cloud^1.1  
 101:102008761146:1   20080128 19:39:48   101:102008760917:1        false   ACLocationSbb^Open Cloud^1.6   B2BUA^Open Cloud^1.1  

5 rows

Bulk Removal

It is possible to perform bulk removal of state, either removing all activities belonging to a single RA entity with the removeallactivities command, or all SBB's belonging to a certain service with the removeallsbbs command. These commands are intended to help the system to drain stale state and cannot be used while the RA entity or service are active.

In the below example the B2BUA service is deactivated and all SBB's are removed instantly:

[Rhino@localhost (#19)] deactivateservice B2BUA\ 1.1,\ Open\ Cloud 
Service[B2BUA 1.1, Open Cloud] transitioned to the Deactivating state
[Rhino@localhost (#20)] removeallsbbs B2BUA\ 1.1,\ Open\ Cloud     
[Rhino@localhost (#21)] findsbbs -service B2BUA\ 1.1,\ Open\ Cloud 
no rows

When performing build removal of activities Rhino stages the actual activity removal to spread out system load. The actual activity removal is done by the query liveness mechanism. With query liveness Rhino periodically queries an activities resource adaptor to see if the activity is still live. Query liveness is configurable but by default every activity should be queried at least every 6 minutes, unless events are actively being processed on it. Bulk removal of activities circumvents this process causing all candidate activities to be automatically removed next time they are due to be queried for liveness. Because of this it is necessary to wait some time (up to 6 minutes using the default settings) for all activities to drain when using the removeallactivities command.

The below example deactivates the RA entity sipra and removes all its remaining activities:

[Rhino@localhost (#22)] deactivateraentity sipra 
Deactivated resource adaptor entity sipra
[Rhino@localhost (#23)] removeallactivities sipra 

After some time there should be no activities remaining:

[Rhino@localhost (#24)] findactivities -ra sipra
no rows

A Word About Tab Completion

In rhino-console many things are completable using tab completion. This can be used to simplify the arguments to many commands especially those concerning SBB's that require service or SBB component id's which are typically longish strings requiring exact formatting. Where possible the available options such as -created-before or -created-after are tab complete-able also.

Adaptavist Theme Builder Powered by Atlassian Confluence