My client is trying to use Com-Plete OLTP to access IBM MQSeries and is running into multi-threaded concurrency issues with the MQ Batch adapter. They assembled a set of MQSeries assembler stubs that call the various MQ APIs which they then call from their Natural code. The problem is when the workload reaches a point that two MQ calls try to use the MQ connection at the same time they get MQ reason codes 2009/2219 indicating that the connection is already in use or there maybe some issue with the connection/handle.
We are looking at other options for making MQ API calls, such as thru a CICS/MQ interface that does support multiple threads and are wondering if there is any Com-plete/Natural interfaces for making connections to CICS.
Warning: answer tainted by it being a long time since I worked deeply with Complete.
Usually when I have seen these types of issues in the past with Complete, it’s because the control blocks used by the service (not just limited to WMQ) were in thread storage, and the blocks were relocated to a different thread. A lot of the connection tokens are actually the addresses of data items in the control blocks, thus if the blocks are moved, the fields don’t pass the basic sanity checks, and errors are reported. (Or, worse, sometimes it goes boom.)
A long time ago, Complete didn’t intercept GETMAIN/STORAGE OBTAIN requests above subpool 127. I have exploited this in the past by getting storage from subpool 230, which is private (but requires authorization). Since then, subpools 129-132 were added. These subpools are owned by the job step TCB, not the TCB of the issuer of the request. Subpools 131-132 do not require authorization by problem state programs, which is very nice
If you can exercise control, or modify source of the stubs, you may be able to get storage out of subpool 131, which would allow the control blocks to relocate but the targets of the fields to not relocate.
Again, it’s been a long time since I dealt with Complete at a deep level. But hopefully this can open up the discussion, and people can correct my old thinking, or this can give you something to play with. Also, maybe we can brainstorm ideas that would minimize having to bring in an additional address space and making things more inefficient.
Hello Allen,
the MQ Batch adapter probably hooks into the TCB, but Complete normally does TCB pooling. This means, more than one Natural session may run on the same TCB, and also the same Natural session may run one transaction on one TCB and the next transaction on a different TCB.
I can think of two possible ways to get it to work:
1)
Set sysparm RESOURCE-MANAGER=YES
This will have Complete switch user contexts when sessions switch TCBs. It’s a must when you talk to DB2 from Complete, maybe it works for MQ as well.
If (1) doesn’t work, then try to avoid TCB pooling:
- In the TASK-GROUP sysparm, allocate enough TCBs for the number of users you’re expecting in Complete, and
- In ULIB turn on “OS task affinity” for your Natural transaction(s).
Please, let us know if either one works or not.
Steffen Kuhnert
Complete Development
Hallo Steffen, wie geht’s?
The MQ interface was taken from DB2, in fact, they share what’s called the ERLY code. This should work.
Thanks… we have the client’s SAG client engineer contacting his SAG/Complete engineer and they will discuss with SAG developers and the client’s Complete system admin’s to see if this configuration change is possible to do in their test environments and see if they can recreate this issue.
I am not sure I follow the comment about the MQ/DB2 ERLY code… they are different LPA modules… though the IEFSSNxx subsystem entries have very similar formats.
When MQ was first developed, they “borrowed” a lot of the subsystem infrastructure (the “ERLY” code) from DB2 (which was across the hall in the IBM Toronto office at the time). There are a lot of similarities in how DB2 and WMQ requests are handled internally.
One usually finds that if one encounters strange system issues with one, the same or very similar solution will fix the same problem with the other
They had already set the RESOURCE-MANAGER=YES enabled in their Com-Plete sysparms, so that was not the issue nor the answer.
I have one more item to pursue… whether the Com-plete Natural user programs that make the MQ API calls would or should be linked with the MQ RRS Batch adapter stubs (CSQBRRSI or CSQBRSTB) to see if the application would behave better if it was relinked with RRS stubs that could handle mult-tasking or threading. for this Com-plete MQ TCP/context switching to work. The following “quote” in cyan/blue is a response I had gotten from an IBM MQ developer… but at the time I did not know how Com-plete handled TCB/context switching with the sysparm RESOURCE MANAGER=YES. Does anyone know if Com-Plete also handles the IBM RRS context switching too?
[color=blue]
[/color]
RRS Begin_Context, Switch_Context, etc. is what RESOURCE MANAGER=YES does, so using the RRS stubs definitly sounds like a good idea.
Complete does have a WLM interface, just add sysparm WLM-SUBSYSTEM-TYPE=COMP if you want to try this.
Thanks for the quick reply… one more question…there is a Com-plete sysparm for DB2…quoted from the System Programmer documentation. Does there need to be a similar SERVER definition, except for MQ? That would have the MQ SSID… and then of course the appropriate MQ load-libraries concatenated to the COMPLIB? Is this a ndw feature/function… something that could be added fairly easily to Com-plete?
[color=darkblue]
[/color]