Exchange 2007 Back Pressure and SMTP 452 4.3.1 Insufficient resources SMTP error

I have been running Exchange 2007 'lab' servers since January of 2006. I installed it in to my main lab system in May, but it was not until Beta 2 was released that I got brave enough to actually move some mailboxes over to it. These are my @somorita.com and @directory-update.com mailboxes, so it is still essentially a 'lab' environment. I'm not crazy enough to move a real company over to Beta software without direct Microsoft involvement and support.

Anyway, today I came across a feature that I really felt was blog worthy. All inbound mail still comes in to an E2K3 server. There is a routing group connector that connects to my E2K7 lab servers.

Yesterday, I started noticing that the E2K3's queue to E2K7 was consistently in retry and the "Additional Queue Information" box showed "The Connection Was Dropped By The Remote Host". After several reboots, I finally checked the event viewer and the protocol logs. The protocol log on the E2K3 server included this text:
2006-11-25 21:32:14 192.168.254.19 OutboundConnectionResponse - - 452+4.3.1+Insufficient+system+resources

The Application event viewer on the E2K7 server had a couple of interesting and educational events. I wanted to share these because they help to provide some insight in to how E2K7 was designed and works.
_________
Event Type: Warning
Event Source: MSExchangeTransport
Event Category: ResourceManager
Event ID: 15002
Date: 11/25/2006
Time: 11:38:17 AM
User: N/A
Computer: HNLEX03
Description: The resource pressure is constant at High. Statistics:

Queue database and disk space ("C:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue\mail.que") = 75% [High] [Normal=70% MediumHigh=72% High=74%]

Queue database logging disk space ("C:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue\") = 76% [Normal] [Normal=92% MediumHigh=94% High=96%]

Version buckets = 1 [Normal] [Normal=40 MediumHigh=60 High=100]

Private bytes = 14% [Normal] [Normal=71% MediumHigh=73% High=75%]

Physical memory load = 52% [limit is 94% to start dehydrating messages.]

Inbound mail submission from other Hub Transport servers, the Internet, the Pickup directory, the Replay directory, and the Mailbox server, if it is on a Hub Transport server, has stopped. Loading of e-mail from the queuing database, if available, continues.

__________
Event Type: Warning
Event Source: MSExchangeMailSubmission
Event Category: MSExchangeMailSubmission
Event ID: 1009
Date: 11/25/2006
Time: 11:39:10 AM
User: N/A
Computer: HNLEX03
Description: The Microsoft Exchange Mail Submission service is currently unable to contact any Hub Transport servers in the local Active Directory site. The servers may be too busy to accept new connections at this time.
__________
Why is this happening? The Exchange 2007 Hub Transport server role has a new feature called Back Pressure. As a Hub Transport server starts to get low on particular resource, it will reject connections (rather than just stopping service). If the resource that is low (such as disk space) starts to get better, then Exchange will not exert back pressure and will start processing inbound messages again.

Notice this part of Event ID 15002?
Queue database and disk space ("C:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue\mail.que") = 75% [High] [Normal=70% Medium High=72% High=74%]

The disk that hosts this server's queue database is above the threshold of 74% and thus Exchange stops accepting inbound mail. However, the services did not stop and as soon as I correct the problem Exchange will start processing mail again. Pretty cool, eh?