Morning techs/devs,

Quick one here; xeams on ws2012r2 ---> exchange 2013, , 5990 build, no config has changed.

All of a sudden periodically we are getting:

2018-07-13 11:50:33,402 ERROR connector.d - Unable to start SMTP server. Cannot bind to port 25/192.168.254.6

every 2 or 3 hours and having to restart the service.  It works for another 3 or so hours then fails again.

Logs show nothing more that that.   
netstat -abo shows xeams is running when we restart it then when it stops running netstat -abo no longer shows it in the list.

Nothing else is binded to 25..?

Would really appreciate your help on this one.   cheers.

You're not the only one running into this issue. We have received calls from 3 other clients running into the same issue. Apparently, it started happening after a Windows update. For some reason the port gets blocked and the only way to release it to restart the machine. We even tried starting a different process (other than Xeams) to listen on port 25. That runs into the same issue.

We are investigating this problem.

Perfect - thanks.

Just to update you on this:

KB4340558 and KB4054566 fail to install which are the July patch Tuesday files - this may also have something to do with it.

Need to keep a few servers up so I've punched in some specific firewall rules on the network adapter per application and not per port and things seem to have calmed down a little. I'll watch it.

thanks

Sorry forgot to add name on last post.

Rgds, James

Nop, still stops after a while.  Do we have any updates to this..?

Can't leave them like this over the weekend..

Thanks,

James

Do you have an option of uninstalling the Windows Updates? If yes, do that.

I am not sure how long is it going to take before a solution is found. Therefore, a quick fix temporarily is to uninstall the MS Updates

We have a spare VM's for general issues like this that are a few Windows Updates behind - could we just copy the entire directory and just run the daemon without installing?

If so, what is the recommended practise for doing this?

James

p.s. I understand SSL will have a few issues but we don't need to worry too much about that.

You can certainly do that. However, a better option is to install Xeams as a service rather than running it as an application. Here are the steps:

• Install Xeams as if it's a brand new install
• Patch that to the latest build
• Copy the entire C:\Xeams\config folder from your older to new 
• Start Xeams
• You will not have any problem with SSL if the public host name of your network does not change.

Sounds good - please keep us updated on any progress.

These VM's are going to be just temp.

Thanks!

Just out of interest; do you know exactly which updates caused the issue so others can uninstall if required..?

Thanks, James

James,

We are not sure exactly which update caused it. So far about 12 to 15 customers have reported this issue. Here is what we've done on our end.

• We have tried to reproduce this problem on two separate machines without success. One is running Windows 10, and other is running Windows 2012 R2. Both are patched with the latest updates from Microsoft. We've sent more than 10,000 messages within a short period and they both accepted them normally.
• The number of clients using Xeams on Windows is in thousands. The fact that only a few clients are running into this issue indicates the problem is happening due to a combination of patches on certain machines. This makes debugging a lot harder.
• Here is what's happening:
  • Every incoming email into Xeams creates a new TCP/IP connection. This connection stays open for a very small duration, typically less than a few seconds.
  • After opening a few thousand connections, the operating system blocks future connections and Xeams is no longer able to accept new emails.
  • The OS continues to block these connections even AFTER the process is terminated - meaning even if Xeams is restarted. This blockade does not end until you restart the machine. Normally, the operating system should release every resource (network connections, file handles, etc...) once a process terminates.
• We created a small program that tries to open a TCP/IP connection to any port. Stop Xeams service once the problem starts and run this program instead to recreate the issue, confirming that the OS is not releasing the connections and continues to block them. You can download this test program from here along with its source code.

Unfortunately, at this time the only recommendation we can make is to uninstall the updates that were pushed in the last few days. We will continue to work on this problem and will let you know once a solution is found. 

A Request

It is very difficult for us to troubleshoot this problem since we are not able to recreate it in our labs. If you're running Xeams on a VM and don't mind sharing your machine with us, let us know via email. Trying different solutions on a machine that is running into this issue will help us tremendously. We will send you an upload link so you can upload the image.

Right, I have some interesting updates for you;

The issue is indeed due to the usual Microsoft Patch Tuesday. I have removed KB4338815 and the problem has completely subsided on a few boxes.

Since the number of clients varies as you have described we now have a fully patched 2016 test machine running Xeams and we are watching this very carefully.

The 2012R2 servers are running fine being patched completely until JUNE 2018.  Attempting to install the security roll up KB4338815 from July will cause these problems.

We have also noticed that KB4054566 (update for .Net) is also causing general installation problems with many clients as is KB4054566 (New .Net 4.7.2)

Either way, removing the July security rollup KB4338815 will mitigate the SMTP binding problem so you can go ahead and inform those clients to start an uninstall.

My advice would be to firstly take a checkpoint if a VM or a backup if physical.

I'll be sure to update you on the 2016 boxes as soon as the config is complete.

Please also keep me updated on your side.

James.

Thanks for that info James.

I was facing the same issue with a Xeams install on a Win7 Enterprise box.  Turns out that uninstalling the KB438818 update (I assume it's the same update as the one you're referring to for 2012 R2) fixed the issue.

James,

Thank you very much for letting everyone know about the exact KB number. Your input will help many users. Quite a few users have confirmed, via email as well as on this forum, that uninstalling KB mentioned by James solves the problem. 

Your Help

We need to report this to Microsoft. However, it is very difficult to explain this behavior unless we are able to reproduce this on our end. Can someone please give us a VM or access to a machine via RDP that is running into this issue so we can have Microsoft look at it.

No worries - it's team effort and I hope we have managed to reach out to all that have the problem.

Moving forward, I can also confirm that if Xeams is running on WS2016 on the LTSB ring this issue does not happen.
The issue seems to only effect the Kernels related to Windows 7 / Server 2008 and Windows 8 / Server 2012.
Windows 10 and Server 2016 currently do not seem to have this issue.

With regards to uploading a VM, I would need to get permission from the client's site and that might involve Non-Disclosure Contracts and Agreements which would be a pain for all of us.

The only thing I can attempt to do is get a trial of Windows 2012R2 trial and do the full install there and then upload the VM.
In the mean time, you could potentially download a trial of Windows 2012R2 as a VM from here: https://www.microsoft.com/en-gb/evalcenter/evaluate-windows-server-2012-r2 and patch using WSUS to see how far you go and where it 'breaks' so to say.

Let me know how you get on.

Regards,
James

James,

We have created a brand new machine on MS Azure running Server 2012 R2. So far it's been running fine. We will keep an eye on it to see if it fails later today.

any update on this? 

I've read the thread and see the issue is being taken care of. I just wanted to add that I had the same problem on an SBS 2011 server. Before finding this thread, I uninstalled, reinstalled, and then moved Xeams to a Windows 7 Professional box in an attempt to see if the problem would go away. The same thing happened again.

After reading these comments, I uninstalled KB4338815 and the issue was resolved. However, this is temporary, so hopefully this will be resolved soon.

+1 for the issue on 2k8 R2

We have a bit of good news. We have been able to recreate the problem on one of our Windows 2012 R2 servers. As a result, we were also able to contact Microsoft and have created a support ticket. Below is a snippet from Microsoft's tech support.

As we discussed the issue you are experiencing might happen because of the KB4338815.

This has not been identified as a bug from Microsoft yet.

But we recommend to uninstall the KB4338815 as of now. We are trying our best to get this issue resolved.

Please refer below article for more information.

 

July 10, 2018—KB4338815 (Monthly Rollup)

https://support.microsoft.com/en-us/help/4338815/windows-81-update-kb4338815

 

Thank you for your patience!

Some more info about the problem:

• Microsoft is aware of this issue and is working on it. The link above confirms a similar problem with SQL Server and IIS. We were able to do a remote screen share session with MS support and showed them the problem.
• The problem occurs when a TCP/IP server runs for a few hours and is restarted. The restart always fails because the OS thinks the port is in use. There are several reasons for the SMTP server in Xeams to restart:
  • Someone changes the configuration requiring restart
  • Entire Xeams service is restarted
  • A built-in watchdog service in Xeams restarts the SMTP server. This watchdog service attempts to restart the SMTP server if no emails are received in 10 minutes. You can configure the watchdog to wait longer before a restart occurs. This will mitigate the problem to a certain extent because the port is never released by Xeams. To do this add the following line in server.properties file.
    
    smtp.watchdog.timeout.minutes=1440
    
    This value is specified in minutes. A value of 1,400 translates into 24 hours.
• The problem is NOT restricted to port 25. It could happen on any port that is being actively used.

so for 2008 R2 would it possibly be related to  KB4338823 since it reports the same sql issue?

https://support.microsoft.com/en-us/help/4338823

MS have released a potential fix: https://support.microsoft.com/en-gb/help/4345459

I shall apply it and see what happens.

Lesson learnt:disable auto update for Windows

Good news: Microsoft got back to us and said there was be an update around 2 PM PST on July 18 that takes care of this issue. 

Just to confirm this latest patch resolves the issue for me.

Morning techs/devs,

hope your all well.

Is the patch that is being released to be applied after the problem KB or is the original KB now repackaged with the fix?
Also, do you have the URL for the patch or is it delivered via Windows Update automatically?

Thanks,
James

The new patch is available via regular Windows Update. It is comes up as an optional update (go figure why). See image below.

An "Optional update" - hilarious those folks at Redmond are sometimes.

Thanks guys.

regards,
James

Yes they do make you Smile more so as this not only affected the Server that I have Xeams installed on but olso the Server that I ahve hosting Exchange.... :-)

Chris, out of curiosity, which version of Exchange server are you having the problems with?

Thanks,
James

Windows 2008 R2 wih Exchange 2010

As far as I know, this bug will happen with ANY TCP/IP server as long as a restart occurs. If you don't restart the service, no problem. The minute service is restarted, you will not be able to bind again until the machine is rebooted. Well, the good news is the patch fixed it.