Tuesday, 5 August 2014

The Effects of Agile Development on BPM

As a BPM designer I come across the term "Agile Development" all the time. The meaning of the term seems to change from project to project. Often a misconception is that if you use an Agile Development methodology, you can do the same amount of work in less time, this is obviously not the case! This post is not about the details of agile development methodologies as there are many more qualified people to talk on this subject. This post is more to do with how some of these development practices can affect the way BPM projects are implemented. This post does have a focus on IBM BPM (formally Lombardi TeamWorks) as this is my area of expertise, but I think the issues discussed will affect most BPM platforms.

Agile software development is a group of software development methods based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams - Wikipedia

Unfinished Processes

Because of the iterative nature of Agile Development, this often results in functionality not fully developed by the time it reaches a production environment. On several of the projects I have worked on, the time scales were such that this would result in a process making it to production where only the first half of the process was complete. This would result in a window of time to complete the development for the second half of the process while in-flight instances were making their way through the first half. Obviously, some process instances would fly through the process reaching the unfinished code before the second piece of development was completed. To stop the process instances falling off the edge into oblivion, we used a simple holding pattern to halt the instances until the code was completed.

The "Wait?" decision gateway allows us to switch the holding functionality on or off depending on the sate of code on the other side. This is usually controlled by an EPV so it is editable live without the need of snapshot release. If the holding functionality is switched on the token will wait on the "Wait" task until either the task is completed or the attached message event is triggered. The message event uses the same correlation ID for ALL process instances so that we only need to fire the message event once to move on all cases. The reason we use a message event attached to a task is so that we have the opportunity to progress individual cases by completing the task. It also gives us the ability to easily tell how many cases are waiting at this point in the process. It is important that the message event does not use Durable Subscription for the message event or your process instances will not be held again if you need to switch the holding functionality back on.

Changing Tasks

Another side affect of iterative development methodologies is that functionality my change between deployments. What was originally a simple task may have evolved into a much more complex beast which requires more data inputs/outputs and more complex processing. Often it is simple to develop the new functionality on top of what is currently available but usually the in-flight instances may need more care. For example, if the new improved task requires more data, this cannot be simply added as a new input as instances which are currently sitting on this step will not have that data available. In this situation I prefer to pass only the ID relating to the business data into each task so that this information can be loaded fresh from the system of record each time. This also makes sure that the data being used is up-to-date at all times throughout the process and doesn't rely on passing large amounts of business data around the process. The obvious downside of this is that tracking business data at the BPD layer is then impossible and also driving decision gateways from business data is also more difficult. There are ways of getting around these issues but I am not going to discuss them here. If this approach still leaves the in-flight instances in danger because the task has changed beyond recognition, we use a more extreme approach. In this situation the safest thing to do is to leave all current tasks on the old code, and only allow new tasks onto the latest version of the code. We do this by disconnecting the flow into the old task but leaving it in the process. The flow is then attached to the new task. This allows all tokens which are currently sat on the old code to use that task, but all new tasks will use the new code.
The original task which is now disconnected can be safely deleted when all tokens have moved on. This approach also works well for changing sub processes where you want old in-flight instances to use the same sub-process. I know that some people will be thinking "why just not migrate the instances on to the latest snapshot if you want them not to use the latest version of the code?". Unfortunately this isn't always possible as the latest snapshot may include fixes/changes to other areas of the process which are required.

Dashboard Driven Development

The ability to provide smart effective dashboards is often one of the main selling points of a BPM platform. Dashboards (or scoreboards if you’re from the Lombardi/IBM BPM world) provide the visibility across your process needed to turn what is essentially workflow into Business Process Management. Without this visibility it is impossible for the business to effectively manage the process let alone allow you to prove any ROI on the developed solution.

Unfortunately, dashboards and reports often find their way down the list of priorities when developing the final solution. This is due to the fact they aren't seen as functionally integral but something that can be added at a later date.

This approach will provide you with a functional workflow system but will not provide the “Management” aspect or the ability to perform process improvement. Both of which are key BPM concepts that a business will buy into when purchasing a BPM platform.

Developing the dashboards at a later date isn’t always as easy as it seems. Careful thought is needed to ensure you are collecting all of the data required to provide the business with metrics that will enable them to manage their process effectively. If this is not put in place upfront, you will find it difficult to add it retrospectively as the information will often have already been lost for in flight process instances. Also, the implementation of the process will often be different if you understand fully the metrics that the business requires.

To provide the business with this information, more emphasis needs to be given to the dashboards during requirements gathering. Even if this just results in ensuring that all required data is tracked.

Calling a Service from a Report

The reporting framework in IBM BPM provides a good platform for building real-time reports & dashboards but this is often not what I use them for. The Report & Scoreboard components provide a container for serving any custom HTML pages to the user via the portal. Frequently, I end up using reports to build custom inboxes or custom landing pages instead of going to the default inbox.

To be able to build these pages you will need to know how to use the report & JavaScript APIs to your advantage. These will allow you to utilise many of the components and services already constructed in IBM BPM. One of the core pieces of functionality is being able to call services from a report page.

There are 2 main ways to call a service from a report.

Calling a Service from Server Side

By using the <#= #> notation, you can call a service from the Server Side by using the JavaScript API.

The following code snippet, is an example of calling a service to return a list of cities. This is then used to build options which are later added to a dropdown list to be rendered on the page.
// Execute Service "Retrieve Cities" with no params (empty map)
var outVars = tw.system.model.findServiceByName("Retrieve Cities").execute(new tw.object.Map());

//Retrieve the output variable "cities" which contains a list of NameValuePairs
var cities = outVars.get("cities");

var cityOptions = "";

//Loop through each city and create an "option" appended to the cityOptions String
for(var i = 0; i < cities.length; i++){
    cityOptions += "" + cities[i].name + "";

This will only execute when the page loads, so it is useful for content that will stay the same throughout the life of the page. To execute it again you would need to reload the page again. For this to use user customisable parameters (filters for example) you would need to use report filters in the URL which can often be fiddly. These are often a necessity, especially if you changing report pages.

Calling a Service using AJAX

The second way to call a service from a report in IBM BPM is to use AJAX. This allows you to call a service and return the result without having to reload the page. Built into the product is a wrapper JavaScript function called "tw.coach.callService" which performs the AJAX call to a service. This is used by the AJAX controls in the coach designer. Although this is not publicised as a supported API, it is fairly widely used to call services via AJAX in both reports and from coaches. The following code snippet is an example of calling the service “Save Input” passing the parameters “input1” and “input2”. These represent the input variables of the service being called.
//Setup Inputs
var vars = ""
    + "" + document.getElementById('input1').value + ""
    + "" + document.getElementById('input2').value + ""
    + "";

//Make AJAX call to Service "Save Input"
tw.coach.callService("Save Input", vars,
    //Call Back Function
        if(data.returnValue == undefined){
            alert("Error calling 'Save Input' Service");
            //Process Values
    '<#= tw.system.model.processAppSnapshot.id #>');
The function specifies a return function which is called with the output values of the service (if any). The <#= tw.system.model.processAppSnapshot.id #> as the last parameter of the AJAX function call specifies the snapshot ID of the process app where the service resides. In the example above, this is assuming that that service is in the same process app. If the service is in a different process app, you will need to use this, where "ACRONYM" is the short name given to the process app:

Wednesday, 23 July 2014

IBM BPM Caching Using WebSphere DynaCache

The nature of BPM often requires integrations to other systems to retrieve information. Depending on the nature of the integration these can often be slow. Where the same data is being retrieved many times (e.g. reference data) and across user sessions, an application can achieve a performance boost by introducing a caching mechanism.

This article will show you how to use WebSphere's inbuilt DynaCache caching utility to cache IBM BPM objects. I have tested this in 7.5.1 but there is no reason why this would not work on other version of IBM BPM (or the older Lombardi TeamWorks).

This isn't going to be a full introduction to WebSphere's DynaCache utility so if you need more information on setting up and configuring caches you will need to look at the IBM documentation.

Setting Up The Cache

DynaCache provides the ability to perform 2 types of caching: Servlet Caching and Object Caching. For this exercise we are interested in the Object Caching. This provides the ability to cache Java objects. The configuration for the Object cache instances can be found in the WebSphere admin console under Resources > Cache instances > Object cache instances.

You will need to create a new Object cache instance to cache your BPM objects. The cache instances are referred to by JNDI name, so you can create as many cache instances as you require. For this example I have created an Object cache instance called "bpmCache" with a JNDI name of "services/cache/bpmCache" (this is the standard naming convention for JNDI names for cache instances). The rest of the fields can be left as their defaults but you may need to refer to the IBM documentation for DynaCache to get your required configuration.

Accessing the Cache using Java

The Object cache instance is exposed to Java via a com.ibm.websphere.cache.DistributedMap object which is retrieved using the JNDI name. Here is an example of retrieving the DistributedMap (Object cache instance) using the JNDI name:

String jndiName = "services/cache/bpmCache"; 
DistributedMap map = null; 
try {
    InitialContext context = new InitialContext(); 
    map = (DistributedMap) context.lookup(jndiName); 
} catch(NamingException e){
    throw new BPMCacheException("Failed to retrieve the Object Cache Instance with JNDI: " + jndiName); 

The DistributeMap is an implementation of the java.util.Map, so items can be added a retrieved using the familiar put and get methods. DisputedMap has 2 versions of the put method: one with the familiar key, value pair and another with some extra parameters for more granular control over the items being cached. The JavaDocs explain what each of these items provide:
public java.lang.Object put(java.lang.Object key,
                            java.lang.Object value,
                            int priority,
                            int timeToLive,
                            int sharingPolicy,
                            java.lang.Object[] dependencyIds)

Exposing the Cache to IBM BPM

To expose the caching functionality to IBM BPM you need to create a JAR file with a class which exposes the relevant methods. To each I pass the JNDI name of the Object cache instance to use (as you may have many).

Here are some of the examples (for a JAR and source code see the Resources section):

* Put an item on the Distributed Map (cache)
* @param jndi              String  JNDI name of the Object Cache Instance
* @param key               String  String value representing the key for the item
* @param value             Object  Value to be cached.
* @param priority          int     The priority of the cached item (higher priority items are disposed of last)
* @param timeToLive        int     The number of seconds until the item is invalid
* @param inactivityTime    int     The number of seconds unused before the item is invalid
* @throws BPMCacheException
public void put(String jndi, String key, Object value, int priority, int timeToLive, int inactivityTime) throws BPMCacheException {

    // Put the value in the Distributed Map
    map.put(key, value, priority, timeToLive, inactivityTime, DEFAULT_SHARING_POLICY, null);

* Get an item out of the Distributed Map (cache)
* @param jndi      String  JNDI name of the Object Cache Instance
* @param key       String  String value representing the key for the item to retrieve
* @return          The Object relating the the key or null if the key does not exist
* @throws BPMCacheException
public Object get(String jndi, String key) throws BPMCacheException {
    // Call the initialise incase its the first access
    // Return the value retrieved from the key
    return map.get(key);

* Check to see if the provided key is in the Distributed Map
* @param jndi      String  JNDI name of the Object Cache Instance
* @param key       String  String value representing the key for the item to retrieve
* @return          true if the key is found, false if the key is not found
* @throws BPMCacheException
public boolean containsKey(String jndi, String key) throws BPMCacheException {
    // Call the initialise incase its the first access
    // Return if the Distributed Map contains the provided key
    return map.containsKey(key);

* Manually invalidate the value represented by the key passed in
* @param jndi      String  JNDI name of the Object Cache Instance
* @param key       String  String value representing the key for the item to retrieve
* @throws BPMCacheException
public void invalidate(String jndi, String key) throws BPMCacheException {
    // Call the initialise incase its the first access
    // Invalidate the cached value represented by the key value

* Clear the cache
* Empty cache of all elements
* @param jndi      String  JNDI name of the Object Cache Instance
* @throws BPMCacheException
public void clearCache(String jndi) throws BPMCacheException {
    // Call the initialise incase its the first access
    // Clear the map

Finally, you can add the JAR as a managed file and expose the caching functionality through Java Integration Components. For the put and get methods which take/return type of java.lang.Object, you can use the BPM type of ANY. This will allow you to store any complex variable (as they are of type com.lombardisoftware.core.TWObject) in the cache. When the object is retrieved from the cache it will be instantiated automatically into the correct BPM variable type.


A copy of the JAR and source code can be found on Github.