I recently took part in a major upgrade for a customer, who moved from Exchange 2003 to 2010, a final step was to migrate devices relaying mail through Exchange and one of these was the Dell EqualLogic Storage Array in which this customer has heavily invested in.

The EqualLogic was configured to use two Exchange 2010 Hub Transport servers at each site for SMTP service but when sending email would receive the following message "WARNING  11/14/11  3:22:32 PM  XX "diag' or "update' was unable to send output using e-mail.  SMTP returned the following error: TimeOut  "

When the SAN sends it's diagnostic reports to a mailbox on Exchange 2007/2010 the message will fail to be delivered when EQ reaches it's timeout threshold and ends the session. The EqualLogic array sends messages in small packets with the BDAT command.

Exchange 2007/2010, by default, will tarpit any BDAT command against Exchange that is not the last chunk command. Tarpit is Microsoft terminology that means to "artificially slow down". Look at the above log, any BDAT command like BDAT 218, will be delayed for 5 seconds before Exchange will confirm to the sender the chunk of data has been received and the next chunk can be sent.

Though if the command BDAT 218 LAST is received then it will not be tar-pitted and the message will be accepted without delay and queued for delivery, as this indicates the end of the data stream. There are many of these BDAT commands being sent per message, each with a 5-second delay per chunk of data and doing the math quickly adds up to several minutes. The EqualLogic and many devices like it ie: printers, scanners have a predetermined timeout to prevent the session from hanging forever, the EqualLogic's timeout is 2 minutes. As you can see this isn't enough time to complete mail delivery and the session will eventually end before it's completed because the timeout threshold has been reached.

The tarpit situation is a design change from older versions of Exchange which only tar-pitted based on the RCPT TO command, and did not have the ability to tarpit based on the BDAT command.

Microsoft has offered two workarounds for this behavior:

  • Create a new Receive Connector for the EqualLogic mail traffic and set the -TarpitTimeInterval to 00:00:00; instead of the 5 seconds which is default. This will functionally disable tar pitting.
  • set-ReceiveConnector "XXXXAnonymous" -TarpitTimeInterval 00:00:00
  • Create a new Receive Connector for the EqualLogic mail traffic. Disable the advertising of Chunking and Binary Mime on this connector. This should prevent any sending server from using the BDAT verb against Exchange, and should cause them to default to the DATA command instead. This is the preferred method in an environment where you don't want to create a dedicated Receive Connector for the EqualLogic, rather have one dedicated to Anonymous traffic for example. You don't really want to disable tar pitting.
  • set-ReceiveConnector "XXXXAnonymous" -BinaryMimeEnabled $false -ChunkingEnabled $false

Additional Research:

http://technet.microsoft.com/en-us/library/bb125140.aspx