Insights

In large enterprise environments, queue-based systems are often used to provide efficient, asynchronous communication between data consumers and data providers. WebSphere MQ is a very popular queuing technology from IBM. It is platform independent, allowing a variety of platform technologies (Unix, Linux, Windows, etc.) to interoperate. WebSphere MQ provides assured, once-only delivery of messages. If the receiving application or the communication channel to the receiving application is unavailable, WebSphere MQ automatically stores the message and forwards it at a later time.

Getting started with WebSphere MQ can be a daunting task. While there is information available on the internet, it is neither centralized nor comprehensive, making it difficult for developers when starting a project. In this paper, we reflect on our own project experience when looking for resources to help us get started with an enterprise integration project involving WebSphere MQ and .NET. We discuss what we went through when faced with a requirement to integrate our .NET solution with MQ. This paper highlights our decisions, as well as the factors supporting them, and also provides code samples and references. We examine “native” WebSpehere MQ protocol and the use of MQ messages with available .NET libraries. (This article does not discuss WebSphere MQ transport for SOAP.)

Choosing the Right MQ Library

Choosing the right MQ library is important because the right selection can ease compatibility issues with other applications and speed up the development effort. WebSphere MQ offers many integration points on many different platforms, but not all of them are equal. One has to consider what type of messages are going to be used what messaging protocol needs to be supported as the transport mechanism by different end points, and what messaging model is required.

This paper discusses the native MQ protocol, which is the messaging protocol used by most implementations. The messaging models available include a publish/subscribe model or point-to-point messaging. In a publish/subscribe model, multiple consumers can subscribe to and consume the same message published by the provider. In this model, WebSphere MQ also supports topic-based subscriptions, which allow consumers to only subscribe to topics they are interested in.

In a point-to-point messaging model, both the sender and the receiver must agree on a pre-defined queue beforehand. In this model, each message will be forwarded to one particular queue and can be retrieved by one consumer only. Different library implementations can choose to support either both or only one messaging model. It is important to determine if your library of choice supports the messaging model required by your application.

You can enable applications to use message queuing by using a programming interface known as the MQI (Message Queue Interface). This is a cross-platform API, so application calls on one platform can be ported to another. WebSphere MQ provides both a Java and .NET Framework implementation of the MQI functionality. When you want to work with a higher level of abstraction than MQI on the .NET platform, there are two libraries available to .NET programmers: WebSphere MQ classes for .NET and Message Service Clients for C/C++ and .NET, commonly known as XMS. These options are discussed in the following sections.

a. WebSphere Message Queue Interface (MQI)

MQI is a set of low level libraries provided by IBM for communicating with the message queue. It is a good choice if are looking for flexibility and fast access, you favor a procedural style of coding. Unfortunately, MQI places limitations on the language you can use for development, forcing your team to work with the previous generation of languages such as C, C++, Visual Basic, COBOL, RPG, PL/I and assembler. This means that your message queue cannot be accessed directly from your managed .NET environment, but instead would have to be accessed from unmanaged code. If you are working on a .NET platform and are using a managed language such as C# or VB.NET, then you have to choose between the WebSphere MQ Classes for .NET or XMS.

b. WebSphere MQ Classes for .NET

The WebSphere MQ Classes for .NET is another library offered by IBM to access the message queue. It provides a high level, object oriented Application Program Interface (API) for applications written in .NET framework that wish to take advantage of the “native” facilities of WebSphere MQ. The API is developed and controlled by IBM; therefore, decisions regarding flexibility vs. usability trade-offs have already been made. There are some limitations which may or may not be a problem for you. The API supports point-to-point messaging but does not support the publish/subscribe model. It also does not support connection pools.

c. XMS

XMS, officially known as IBM Message Service Clients for C/C++ and .NET, is a collection of new messaging APIs that render the Java Messaging Service (JMS) in C/C++ and C#. They bring the benefits of JMS, a standard, abstracted messaging API, publish/subscribe and point-to-point messaging to the non-Java world.

JMS is a Java API that defines the way an application talks to the messaging infrastructure. It was designed to define a single independent API for any queue, similar to JDBC/ODBC for data and databases. IBM implemented the JMS spec with WebSphere MQ JMS. In JMS, there is an envelope called RFH2 that is supplied with a message. The message, effectively, looks like this:

Figure 1

The envelope takes care of some fundamental messaging concepts, concepts already present in the MQ message. For example, the correlation id and message id are replicated in the JMS message header and are independent of the MQ equivalents. However, this comes at a price. If you try to dequeue a message from MQ using a non-JMS based API, you will not get the desired results if the message was enqueued with JMS.

The solution is XMS. By implementing the JMS spec, it helps to work with messages that originated from JMS based systems on non-Java platforms, and it creates new integration possibilities by extending WebSphere messaging to those who have, for example, used .NET as their client platform, or who want to use messaging to integrate legacy C++ applications with new J2EE applications.

d. Watch Out for Compatibility Issues

Because there are different libraries targeting different MQ implementations (“native” MQ vs. JMS), there will be conflicts when mixed messages are used. It is important to identify the end points in your MQ integration scenarios to help you in in choosing the correct MQ library. So what can you do to limit compatibility issues?

1. Understand which type of message you need to work with (“native” MQ or JMS).
2. Isolate and abstract your queuing code, perhaps into a pluggable module, so that you can easily switch between the two modes of operation.
3. Don’t try to receive both types of messages on the same queue. A queue should either accept all “native” WebSphere MQ messages or all JMS messages.

While working on different platforms, each library provides the ability to serialize application objects out to the queue and place primitive types or strings. On each platform, the standard serialization mechanism is used. This must be used with caution, however. Each platform implements the serialization mechanism differently, which means you cannot put objects in a queue from one platform and take them out from the other. Interoperability at the object level is not possible. If you want to achieve interoperability, you must consider creating a canonical message format that each platform can understand and serialize your data as an XML string.

Our Experience

We were approached by a large national health plan provider organization (Client) to develop a solution that would allow their consumer applications to use Service Oriented Architecture (SOA) and communicate with different backend systems in a ubiquitous way by channeling service requests via an Enterprise Service Bus. Overall system context diagram can be seen in Figure 1.

Figure 2

Our solution, which we’ll refer to as Netsoft Service Provider for the purposes of this paper, was a service provider hub, acting as a broker, distributor and data aggregator. It was designed with speed, stability and scalability in mind, and implemented some core principles of parallelism, caching and layering.

The main objective was to receive service requests from the ESB, which used WebSphere MQ for its messaging sub-system. Requests were then analyzed by the service broker component and routed to a participating service for execution. The service executing the request would aggregate data from different data sources, including Oracle, DB2 and mainframe legacy systems, and execute the corresponding business logic. Communication with the mainframe system was achieved by connecting to WebSphere MQ through point-to-point messaging. The generated service response was sent back to the ESB for further consumption within the enterprise.

When it came to MQ integration, we identified two integration points.

1. Integration with the ESB
2. Integration with mainframe systems

Many of the existing provider and consumer applications within the Client’s enterprise had been developed with Java. Our goal was to plug into the ESB and provide an integration and extensibility point for the .NET platform.

Our Decision

During systems integration analysis, we looked at other systems in the enterprise and identified various connectivity options for MQ integration. One point of integration was between our service broker component and the ESB, where we were pulling the messages for further distribution to one of our services. The ESB was built with the JMS spec; in order for us to comply with other systems and avoid compatibility issues with JMS formatted messages, it was apparent that we had to use the XMS library.

Another point of integration was between our services and the mainframe legacy systems. It required direct communication which could be achieved using point-to-point messaging with “native” MQ messages. We had two options from which to choose. The question was: “Should we choose WebSphere MQ classes for .NET or XMS?” It seemed that using XMS, with all its extensive options and support for publish/subscribe messaging, point-to-point messaging, and wide systems integration, would be overkill in this particular case. There was no need to integrate with different platforms. We only had one consumer interested in the message and we needed to achieve maximum performance in doing that. For these reasons, we decided to use WebSphere MQ classes for .NET.

For code samples and references on how to use XMS and WebSphere MQ classes for .NET,
please see Parts 2 and 3 of this article.

Issues to Watch For

During our MQ integration experience, we faced some issues that can be classified into two categories: versioning and compatibility.

a. Versioning Issues

When developing for MQ, it becomes important to know and keep track of MQ versions used by the client and server components in different environments and try to stay in sync. Some MQ libraries have a hard check in them and will not work with all versions. XMS, for example, does not run on versions prior to 7.1. We discovered this issue the hard way. In our development environment, we had MQ server 7.1 while the staging environment had MQ server 6.0. Everything appeared to be working fine in the development environment but failed to run successfully in staging. It was only after we disassembled the MQ components that we learned about the version check.

b. Compatibility Issues

As mentioned previously, a compatibility issue may come up when different MQ libraries are used to process different MQ message types. It is important to evaluate different end points in the integration scenario to choose the right MQ library.

Some issues may be caused by various MQ options that one needs to set when using a particular MQ library. The problem here is that some MQ options are not maintained consistently by IBM, thereby affecting backward compatibility. For example, something that may be set by default in one version has to be explicitly specified in another version. Such behavior causes runtime errors and makes code migrations more difficult.

References

IBM WebSphere MQ Information Center

Introducing XMS – The IBM Message Service API

Interoperability Technologies: Data Tier

MQSeries Publish/Subscribe Applications

WebSphere MQ Solutions in a Microsoft .NET Environment