There's a point in time where on one side, chaos reigns and the fundamental laws of order seem to be suspended. Then on the other side is the light, the complete understanding of something that now seems simple. This is a story of a boy, an HTTP Send Port and his trust in Microsoft's 'calculations.' It all started with a MOM alert:
Severity: Error
Status: New
Source: BizTalk Server 2006http://localhost/xxxx/sendPoWS/sendPoWS.asmx/upDatePOData
Name: Error: An outbound message is being suspended by the adapter.
Description: A message sent to adapter "HTTP" on send port "scosPoOut" with URI "http://localhost/xxxx/sendPoWS/sendPoWS.asmx/upDatePOData" is suspended.
Error details: The HTTP send adapter cannot complete the transmission within the specified time.
Destination: http://localhost/xxxx/sendPoWS/sendPoWS.asmx/upDatePOData
MessageId: {DA614303-8AA6-444E-8312-4C41B4BBD29C}
InstanceID: {954FAA15-BAA0-453A-9AD5-C62E186AF05E}
Cannot Complete the Transmission? It's a request-response HTTP port off of an orchestration... the webservice is on the local host... why can't it complete sending the XML message to the service? Let's look at this IIS Logs...
#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
2007-06-10 10:59:18 W3SVC1 127.0.0.1 POST /xxxx/services_1/sendDemandWS.asmx/upDateData - 80 - 127.0.0.1 Microsoft+(R)+BizTalk+(R)+Server+2006+3.0.1.0 200 0 64
It has a sc-status of 200? 200 - OK? Checked the web service logs and it successfully submitted the data. But wait... what is this sc-win32-status of 64...type net helpmsg 64 at cmd prompt gets "The specified network name is no longer available". This is localhost, why is it having problems talking to itself? We've had server build issues in the past but how could the TCPIP stack be hosed? That's impossible. OK. Let's back up and go to BizTalk.
BizTalk is sending the message to the web service via the orchestrations HTTP send port. It retries 5 times per the configuration and then fails. Each time it tries, the message is received and processed by the web service. BizTalk is just not getting back that final ACK. But it's local Host...
After some Google'ing on sc-win32-status and 64, the collective wisdom said it's an indication that one side of the TCP connection is closing prematurley. So am I throwing errors in this web service somebody else wrote that I have to maintain? No... there's a 200 returned by IIS - that's success. So where's the other end of the connection... BizTalk!
The HTTP send port's Request Timeout is set to '0' which is default. When set to zero, the time-out is calculated based on the request message size. Should be fine. But it isn't. How do they calculate the size? I found one posting with the answer:
Timeout = Min((180sec + ((MessageSize* 3)/1000)), 3600sec)
So, Ihave a message that is 6.4 Megs in size. I do the calculation and it's going to top out at 3600 secs. I turn on additional logging in IIS and see that the actual IIS run takes 5900 seconds (I know this is a poor design and is way too long for a TCP connection to hang open - v2 will change the architecture). So back to the beginning. BizTalk is giving up after an hour and retrying. This why I see the retries going at intervals of 1 hour and 5 minutes. 1 Hour to timeout and 5 minutes for the retry interval. Overriding the send port's default value of '0' to '8000' solves the problem.
It's really crazy that the solution was one of the first items that I dismissed. And I hold no malice toward Microsoft's calculation as it is reasonable and this is not a reasonable circumstance. I was really disapointed that the calculation method was not there in bold print. Instead I meandered down many dead ends before I regrouped and went deep. I am a wiser mand for having made the journey.