Blog Home  Home Feed your aggregator (RSS 2.0)  
Implements IVillage - Monday, September 10, 2007
It takes a village to keep up with .Net
 
 Monday, September 10, 2007

We've been having a heck of a time lately with the BizTalk working queues filling up with ghost messages.  The level of blocking in the Database had become unbearable at certain times.  We had no idea where it was coming from as our activity levels have remained relatively constant (we hadn't found out about the queues yet).  We worked with MS support to do some cleanup of our produciton environment.  This seemed to help but then the queues kept filling up again.  After some focuse digging, we managed to stear ourselves and MS support to the solution.  A new hotfix just off the presses - KB936536.  We've applied it today and will see if it works.  I will post a more complete write up in a few days.

Monday, September 10, 2007 8:54:13 PM (Eastern Daylight Time, UTC-04:00)  #    Comments [1]    | 

The Jeff Barnes & Joe Healy travelling MSDN Quarterly Briefing Redux will be in Melbourne tomorrow at the Space Coat Credit Union Headquarters building. Link. I was hopeful to ba able to attend the full day, but despite the time savings, the meetings keep piling up.  I am sure I will be able to stop in for a bit, but not the whole day.  Looking forward to seeing everyone there!

Monday, September 10, 2007 8:47:23 PM (Eastern Daylight Time, UTC-04:00)  #    Comments [0]    |   | 
 Thursday, August 16, 2007

The meeting was a success.  Before leaving work to go to the meeting I checked the clicktoattend roster.  14 people signed up.  I was hoping for a smaller group for my first presentation but was none the less suprised and delighted by the interest.  The actual turnout was solid and all went fairly well.  Lots of good questions and even got corrected on one or two points.  The presentation went for an hour and a half and I only counted a few people dozing off.  Here is the slide deck for those who are interested.

The User Group has had strong turn out the last two meetings and I think we will be having a strong next meeting with Cory Foy on Ruby for C# Developers.  Used to be a half dozen folks sitting in the room.  I am hopeful a combination of good technical content and community are really driving the groups success.  I made a strong push to have more members deliver content and try to trim down on the Microsoft bleeding edge speakers.  Not that I don't enjoy the Microsoft bleeding edge content, but most people won't get to implement Silverlight or LINQ for a while.  The talks on subjects like WCF, Enterprise Library, Ruby, Extensibility Patterns, etc. seem to generate more genuine interest and leave attendees with value.

One more word of thanks for all who attended.  If you have any feedback, please feel free to email me!

Thursday, August 16, 2007 4:12:27 PM (Eastern Daylight Time, UTC-04:00)  #    Comments [1]    | 
 Sunday, August 12, 2007

I will be be giving my first technical presentation at the Space Coast .Net User Group on this Wednesday night.  If you are coming, don't forget to clicktoattend.  Follow the link for directions too.  I hope to see everyone there and you will be frisked for rotten vegetables at the door.

Monday, August 13, 2007 6:35:59 AM (Eastern Daylight Time, UTC-04:00)  #    Comments [1]    |   | 
 Tuesday, July 10, 2007

We've had a problem with our BizTalk Servers for quite a while.  Receive ports to a particular file server would seem to shutdown almost at random.  After much searching of the internet, there was a consenus to build a port watcher application using BizTalk WMI.  We did this and it worked, ports would stop and the port watcher would kick them in the butt to get them going again.  But this was just a band-aid to a huge sore under the surface.  After almost a year of this, a re-researching of the problem finally gave way to a helpful post and then a solution

It turns out that when a BizTalk Server is checking LOTS of directories on the same server, the network BIOS command limit can be reached:

"This issue may occur if the client computer submits simultaneous, long-term requests against a file server that uses the Server Message Block (SMB) protocol. An example of a long-term request is when a client computer uses the FindFirstChangeNotification function to monitor a server share for changes."

"This issue may occur if the MaxCmds registry value setting on the client is less than 50, or the MaxMpxCt registry value setting on the server is less than 50."

So, per the kb article, an increase of the MaxCmds regsitry entry to 500 solves the problem (after a reboot)!  Not a BizTalk problem at all.  The limitation in the Windows 2003 OS designed to throttle its workload (for good reason) was causing the receive location to shutdown.  The error message wasn't very helpful from BizTalk as it really didn't provide a good error code or reason to point to the command limit that was eventually found to be the problem.  Now if I can only figure out why my File Recieve loactions to DFS shares do the same thing!

Tuesday, July 10, 2007 10:09:27 PM (Eastern Daylight Time, UTC-04:00)  #    Comments [0]    |   | 
 Wednesday, June 13, 2007

There's a point in time where on one side, chaos reigns and the fundamental laws of order seem to be suspended.  Then on the other side is the light, the complete understanding of something that now seems simple.  This is a story of a boy, an HTTP Send Port and his trust in Microsoft's 'calculations.'  It all started with a MOM alert:

Severity: Error
Status: New
Source: BizTalk Server 2006http://localhost/xxxx/sendPoWS/sendPoWS.asmx/upDatePOData
Name: Error: An outbound message is being suspended by the adapter.
Description: A message sent to adapter "HTTP" on send port "scosPoOut" with URI "http://localhost/xxxx/sendPoWS/sendPoWS.asmx/upDatePOData" is suspended.
Error details: The HTTP send adapter cannot complete the transmission within the specified time.
Destination: http://localhost/xxxx/sendPoWS/sendPoWS.asmx/upDatePOData
MessageId: {DA614303-8AA6-444E-8312-4C41B4BBD29C}
InstanceID: {954FAA15-BAA0-453A-9AD5-C62E186AF05E}

Cannot Complete the Transmission?  It's a request-response HTTP port off of an orchestration... the webservice is on the local host... why can't it complete sending the XML message to the service?  Let's look at this IIS Logs...

#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
2007-06-10 10:59:18 W3SVC1 127.0.0.1 POST /xxxx/services_1/sendDemandWS.asmx/upDateData - 80 - 127.0.0.1 Microsoft+(R)+BizTalk+(R)+Server+2006+3.0.1.0 200 0 64

It has a sc-status of 200?  200 - OK?  Checked the web service logs and it successfully submitted the data.  But wait... what is this sc-win32-status of 64...type net helpmsg 64 at cmd prompt gets "The specified network name is no longer available".  This is localhost, why is it having problems talking to itself?  We've had server build issues in the past but how could the TCPIP stack be hosed?  That's impossible.  OK. Let's back up and go to BizTalk.

BizTalk is sending the message to the web service via the orchestrations HTTP send port.  It retries 5 times per the configuration and then fails.  Each time it tries, the message is received and processed by the web service.  BizTalk is just not getting back that final ACK.  But it's local Host...

After some Google'ing on sc-win32-status and 64, the collective wisdom said it's an indication that one side of the TCP connection is closing prematurley.  So am I throwing errors in this web service somebody else wrote that I have to maintain? No... there's a 200 returned by IIS - that's success.  So where's the other end of the connection... BizTalk!

The HTTP send port's Request Timeout is set to '0' which is default.  When set to zero, the time-out is calculated based on the request message size.  Should be fine.  But it isn't.  How do they calculate the size?  I found one posting with the answer:

Timeout = Min((180sec + ((MessageSize* 3)/1000)), 3600sec)

So, Ihave a message that is 6.4 Megs in size.  I do the calculation and it's going to top out at 3600 secs.  I turn on additional logging in IIS and see that the actual IIS run takes 5900 seconds (I know this is a poor design and is way too long for a TCP connection to hang open - v2 will change the architecture).  So back to the beginning.  BizTalk is giving up after an hour and retrying.  This why I see the retries going at intervals of 1 hour and 5 minutes.  1 Hour to timeout and 5 minutes for the retry interval.  Overriding the send port's default value of '0' to '8000' solves the problem.

It's really crazy that the solution was one of the first items that I dismissed.  And I hold no malice toward Microsoft's calculation as it is reasonable and this is not a reasonable circumstance.  I was really disapointed that the calculation method was not there in bold print.  Instead I meandered down many dead ends before I regrouped and went deep.  I am a wiser mand for having made the journey.

 

 

 

Thursday, June 14, 2007 6:04:42 AM (Eastern Daylight Time, UTC-04:00)  #    Comments [0]    | 
Copyright © 2008 Christian M Loris. All rights reserved.
DasBlog 'Portal' theme by Johnny Hughes.