BizTalk Server Host throttling

We faced a performance issue in BizTalk some time back. As a result of bulk operations due to financial closing at year-end, a lot of messages were in a dehydrated state and the processing was too slow. As usual Googling led to Performance tips and tricks provided by Microsoft. The first step in such investigations, as rightly pointed out, is obviously identifying the bottleneck, which in our case was the BizTalk Tier. To identify bottlenecks, we monitored a few performance counters. As the article suggests, we started with Message Delivery Throttling State and the Message Publishing Throttling State performance counters and immediately we caught hold of the culprit! Host Throttling Performance Counters gives a complete listing of all performance counters that can be monitored to identify such bottlenecks.

Message Publishing Throttling State was 4 which means Throttling due to process memory pressure. The possible values are as follows:
0: Not throttling
2: Throttling due to imbalanced message publishing rate (input rate exceeds output rate)
4: Throttling due to process memory pressure
5: Throttling due to system memory pressure
6: Throttling due to database growth
8: Throttling due to high session count
9: Throttling due to high thread count
11: Throttling due to user override on publishing

I will discuss the measures taken to overcome this issue later in this article, but, what we first need to look at is, what is host throttling. This is what Microsoft says:
To manage the use of resources by a host instance process, BizTalk Server utilizes an adjustable throttling mechanism that governs the flow and processing of messages through a host instance. The throttling mechanism moderates the workload of the host instance to ensure that the workload does not exceed the capacity of the host instance or any downstream host instances. The host throttling mechanism also detects when available resources are being underutilized. It helps to ensure that the system operates at an optimal and sustainable level. This mechanism continually monitors for a throttling condition, calculates the severity of the throttling condition, and applies host throttling progressively depending on the calculated severity. The throttling mechanism is self tuning and the default configuration options are suitable for the majority of BizTalk Server 2006 processing scenarios. For example, if inbound throttling is applied to a receive adapter then the receive adapter may stop receiving messages until the throttling condition is mitigated. For details, read What Is Host Throttling? Host throttling configuration parameters are set on a per host basis in the BizTalk Server Administration console.

How BizTalk Server Implements Host Throttling provides a mitigation strategy for each of the above throttling conditions. Since throttling was due to process memory pressure, we increased Process memory usage threshold for the host, as suggested in the article. By default, it is 25%. What this really means, is that, a throttling condition will be triggered when process memory usage exceeds 25% of the total available process memory (The maximum available process memory is capped at an address space size of 2 GB) i.e., 500MB. We increased it to 35%, which means that, throttling condition will be triggered only when process memory usage exceeds around 700 MB. In our case, the actual process memory usage was 630MB, which is more that 500MB, which resulted in a throttling condition. Increasing the threshold resulted in mitigation of throttling condition and the messages were immediately proccessed.
This also meant that, with increasing number of BizTalk applications, one host instance would not be enough and such issues would reoccur. This prompted us to introduce an additional host instance. Now we have two host instances, one that is used for real-time applications and another, that is used for scheduled applications.