Geeks With Blogs

News View Michael Stephenson's profile on BizTalk Blog Doc View Michael Stephenson's profile on LinkedIn
Michael Stephenson keeping your feet on premise while your heads in the cloud

We had a situation last week where BizTalk really showed some of its value and it was nice to get some good feedback from one of the application service managers. I thought id share to brighten all of your days.

So the situation was that we had a production incident where the volume of messages the BizTalk environment had been processing was well in excess of what had been performance tested. The performance tests had been done against assumptions for one of our processes which had been incorrect and we processing a lot more messages on a day to day basis. This wasn't really too big a deal as BizTalk was handling it fine, however one particular morning there had been a backlog in the batch processing in one of the systems which seemed to have been flushed out and caused a flood of messages to come into BizTalk within a very short period of time.

The result of this flood of messages was that we ended up with a lot of orchestration instances running at the same time and we ended up with a memory based throttling situation for the host which was running these orchestrations. This normally wouldn't have been an issue as the 3 main orchestrations which were running are all asynchronous processes and would just work their way through and no external systems would be affected by a delay in these getting processed.

What was a problem however was that we had some other orchestrations where a delay in their processing would be an issue. These other orchestrations share the same host and for a long time had never experienced any issues. However since the latest release of one of the applications this unexpected increase in load was now a problem and this was the first incident.

The good news starts with the fact that because BizTalk could be monitored easily we had a good heads up on this issue starting to happen and we were able to begin doing something about it before it became a big problem. It's always good to be seen as being proactive and being able to deal with things without the business feeling any pain. So we were getting alerts from our HP Openview setup about this happening. We were able to quickly get on top of the issue by creating a new host and moving a couple of orchestrations to this new host so they wouldn't be affected in large floods of messages from other processes. The second thing we did was to increase the throttling settings for the first BizTalk host so that we could clear the backlog of messages a little quicker. It took about 10 mins to modify the BizTalk settings and to get things running nicely again. Although it took a couple of hours for BizTalk to work through the backlog of messages.

When we went over this issue with some of the Application Service Managers it was very pleasing to see their reaction to how we dealt with this. From other applications they had been used to expecting to require code changes, lots of analysis and testing and probably a number of weeks before they could implement changes to correct problems properly. They were also used to a significant cost in terms of having to implement workarounds to deal with the fall out of these type of situations.

In our case however BizTalk really showed its strength as an execution platform. Its flexibility and manageability are really powerful features.

Posted on Sunday, October 25, 2009 2:26 PM BizTalk | Back to top


Comments on this post: A nice BizTalk day

# re: A nice BizTalk day
Requesting Gravatar...
Tengo 17 años y le tengo Paycheck Calculator | Online Calculator | Paycheck Tool mucha pasion a este bonito deporte,me gustaria dedicarme a nivel profesional.
Left by rouge on Jun 12, 2017 8:27 AM

Your comment:
 (will show your gravatar)


Copyright © Michael Stephenson | Powered by: GeeksWithBlogs.net