The Tangled Web of Troubleshooting
As the connectivity between various systems becomes more intricate and complex, the risk of tricky configuration problems grows considerably. When such problems finally manifest themselves, they are usually harder to identify and resolve. They also have a tendency to remain dormant for a longer period of time.
My home network sports a fairly complex messaging infrastructure that entails an anti-spam server, a mailing list server, and my Exchange 2007 server – all managing 5+ email domains. The mail flow requires its own network diagram, which I will probably update sometime in early January 2009.
Here’s what it looked like while my Exchange 2003 server was in the mix:
Mail came in to the anti-spam server, and from there, it went to the Mailing List server. If the addresses were not local on the mailing list server, they were then sent to Exchange, otherwise they were handled on the mailing list server.
All outbound mail went out from the mailing list server via my ISP…
Back in May 2007, I started using Microsoft Exchange 2007, and I went through some interesting configuration challenges with public folders. Most of these issues were not cleared up until Service Pack 1 for Exchange 2007 was released, providing a reasonable GUI for managing Public Folders. The problem was, that my Exchange 2003 server had already died (at least I was able to migrate the mailboxes before its final demise). The primary reason for me moving to Exchange 2007 in the first place was to escape the imminent hardware failure that the Exchange Server was facing.
For the most part, everything was fine, because external mail was flowing properly. I left the public folder issues alone and attended to other things.
Then, in November 2007, I began testing an anti-spam service for use at work. I didn’t have any spare domains to use at work, and I didn’t feel like subjecting my organization’s employees (and our clients) to weird email issues, so I had the service configured against my personal email domains.
Immediately, I encountered some issues with the service, but that is because, in my estimation, the product was not designed to deal with the kind of configuration I have at home or at work. Over the course of the next 3 or 4 months, there was quite a bit of tweaking that I went through with the support team. They were very helpful, but the way the product operated did not quite suit my home or work needs. (For one thing, it rewrote all headers such that mail for an alias address showed as it if had been sent to the primary mailbox name. This masked my own problem with the one public folder).
Over the next few months, I started to recreate the necessary public folders and straighten out the email addresses, but I made two mistakes:
- I accidentally generated an email address for one of the public folders that was the same as the the email address for one of the mailing lists that it was supposed to receive mail from.
- I saw occasional weird behavior when posting to that one specific mailing list from my Outlook account vs. Gmail or a POP3 client, but I did not pursue the issue for many months.
While I was testing the anti-spam solution, my email configuration was adjusted as per the following diagram. Because of how it masked email addresses, I had to configure separate destinations for emails headed to my Exchange Server, vs. emails headed to my Mailing List server. It was during this time that I made the ill-fated configuration change on one of the public folders. While the anti-spam service was in place, mail went where it was supposed to, except from my own mailbox. I got around that issue by creating an email contact that went straight from Exchange to the mailing list server, but I didn’t bother to investigate the issue in more details.
The vendor and I were supposed to get back to testing the configuration after a product update in May or June 2008, but they didn’t contact me for quite some time, and I got busy dealing with other issues. In the meantime, I went out and obtained a really cool anti-spam appliance from Sendio.com which addressed my anti-spam needs at work.
When the anti-spam service vendor finally reached out to me at the end of October 2008 to have the service shut off, I returned to using my standalone anti-spam software, but I made a minor change to the mail flow because Exchange 2007 provides much more control over incoming and outgoing mail than its predecessor did.
Here’s what it looks like now that the anti-spam service is no longer in the mix:
Mail comes in to the anti-spam server, and from there, it goes to Exchange 2007. Any addresses in the domain not handled by Exchange are then passed on to Mailing List Server. A number of different mail connectors are used.
All outbound mail can leave from either location to go to the ISP…
What should have been a relatively simple change, instead exposed my months-old configuration error. After the issue was brought to light by several mailing list subscribers, it took me a couple of hours of message tracking and other troubleshooting, to find out what had happened and remediate it. And I have no one to blame but myself, mostly because I should have done the following:
- Documented my initial changes from the beginning
- Taken the time to investigate my weird email issues when they first surfaced
- Paid attention to what I was doing :)
The cool thing about technology is that there are always learning opportunities. This lesson focuses on addressing problems in a timely fashion, and being aware that expedience generates a higher level of risk.