Recently, we had an outage due to heat in some of our switch closets. Not only did this generate a lot of alarm traffic for IT Systems…but it also generated a lot of alarm traffic for HVAC and even other systems as well. In the end, we sent out so many automated notifications (emails) to pagers and cell phones that we got rate controlled and then subsequently blacklisted due to the massive volume of emails heading out of our SMTP IP Address. This blacklist meant that email was brought to a screeching halt in the enterprise. I had to have our Network Team fail us over to our secondary SMTP IP Address in order to get mail flowing again. This of course, meant we were no longer using our primary network circuit and there were some DNS hiccups…and really, we didn’t need anything else to make the IT department look bad…DNS was the icing on the cake.
So my question to all the Email Administrators out there is…how do you handle automated notifications in your enterprise? Do you have secondary SMTP servers with different IP Addresses that you use to send out automated notifications? Do you use the same Exchange server but filter traffic using Exchange and virtual SMTP Servers to an external smart host? How do you do it?
I hope someone has some examples for me and I appreciate any help you can give…we’re just venturing into territory I haven’t been in before and it’s always good to get perspective of those who have been there already.
This almost ranks right up there with my Ghost Delegates head scratcher of an issue. Now we’re talking about ghost messages. In my case, an automated system was sending out a notification email to a user in our enterprise environment. I could track the message through our SPAM Firewall showing the hand-off (RECEIVE stamp) of the message to our Exchange 2007 Transport Server and then a DELIVER notification of the message being placed inside the mail store.
In other words: The message was actually being delivered into the user inbox…but they never appeared there to the end user. No pop-up notification of a new message in Outlook 2010 on the desktop. Nothing.
So how do we troubleshoot this problem? Luckily, I approached this the correct way right away so I saved myself tons of time and I’m going to share with you how I did it.
Where to Start – Cache Corruption
First and foremost, we use Outlook Cached Mode in our environment. My first thoughts were that the cache file had become corrupt. I’ve seen in the past where a user has not been able to send or receive email to individual distribution groups or they are unable to update an appointment…just really odd singular stuff like this. So I backed up the user .ost file by closing Outlook and Windows Search tool and renaming it to username.ost.bak. Then just startup Outlook and it thinks this is the first time it is starting up…so it re-establishes the .ost from scratch. If there is any corruption, it will be inside of the old .ost file.
For some of you, this might fix your issue. For me, it was back to the drawing board.
Second Step – Establish Where the Problem Exists
The second step is to isolate the problem. Does it exist on the server or is it a problem with Outlook? You can do this by shutting down Outlook and using Outlook Web Access (OWA). Is the message delivered in OWA or to a mobile device when Outlook is closed? If the answer is no, then the issue is NOT an Outlook issue. If yes, congratulations…your issue is an Outlook Issue. Switch it away from cached mode and try using Online mode…this should fix any issues you have. Unfortunately, for my user, the issue did not go away…the message still did not show up. This tells me the issue has to be server side.
Third Step – Check Spam/Junk Folder & Verify Message Delivery
The next step is to check spam folders and junk folders to make sure that the message isn’t getting held up there. I made sure to whitelist the email address on both the client and the server side of things and I tested again. Nothing was going into the spam folders.
I also wanted to make sure, not only with deliver stamps from Exchange, but with absolute proof that the mail message was being delivered. You can do this by adding yourself as a journal recipient on the user account. It is my understanding that Journal receipts process before the filters/rules/spam processing do inside an inbox. So we can prove with 100% certainty that the message is arriving in the user mailbox by adding yourself as journal receipt and watching the mail come in. This should also tell us that it absolutely is a rule that is the culprit versus anything else.
To do this, open up the Exchange Management Console and right click the user that is having a problem. Choose properties. Go to the Mail Flow Settings tab and choose ‘Delivery Options’. Click ‘Properties’ and under the ‘Forward To’ section, choose yourself. Don’t forget to check ‘Deliver message to both forwarding address and mailbox’ or your user won’t any mail. In my case, after I chose these options and had a test message sent out…I received the email in MY inbox but the user still didn’t. So, we now have 100% absolute proof that the message is making it to the inbox and that the user has a rule that is preventing it from being viewed. Let’s see how we can fix this issue in the next step.
Fourth Step – Mail Rules
I know the issue isn’t existing on the client side (Outlook) but I went ahead and started Outlook using the /cleanrules flag from a command prompt. This is supposed to clean out any mail processing rules from the end user.
[box type=”warning”] USE CAUTION! This will remove ALL rules from the user mailbox and the end user should be aware that they will need to recreate any that were previously in place.[/box]
In my case, the user had on 2 rules which were disabled and thus, even with the clean rules switch enabled…the problem was not fixed.
What to do now? I knew something was grabbing the message from the inbox BEFORE it could synchronize to Outlook. So I decided to check the rules in OWA to see if there was a difference (there shouldn’t be). Upon looking in OWA, I found 3 rules present instead of 2. All three were disabled. This was different than the 2 disabled rules I found inside of Outlook. When I looked at the rule, it was taking the ‘ghost message’ from the automated system and moving them to the ‘Inbox’…so there was some kind of loop. They were arriving in the inbox, and then marking themselves as read and moving themselves to the Inbox again…somehow, they were disappearing from view when this happened.
I figured now it would be a simple case of deleting the rule right? Exchange had other ideas. I couldn’t delete the rule…and as I said, it was disabled but somehow still working. I realized this was going to require more tools than I had on hand.
Last Step – Pesky Rules Managed by MFCMapi Tool
The last effort I could use on this problem is to look and see what was going on with the MFCMapi Tool. This tool is available from Microsoft Codeplex and is a small executable that allows you to see a bit more going on in a mailbox than what you could normally see using standard tools.
On the client machine where Outlook is installed (the user who’s rules you want to change) open up MFCMapi.exe and choose the profile the user selects when opening their email. This should give you something similar to the following screenshot:
Open the rules section by going to QuickStart > Inbox Rules:
Search for the rule that you want to delete. In my case, since all the rules were disabled already…I could delete all 3 rules. Your situation may be different…the screenshot is actually my inbox rules so there are quite a few more than 3. However, I was able to delete the rule using this interface where I couldn’t delete it from OWA and I couldn’t see it via Outlook.
After deleting the rule, the emails began to show inside the user inbox and the mystery of the disappearing mail messages was solved. I treated myself to some Scooby Snacks and Old Man Winters would have gotten away with it if it weren’t for those meddling kids.
Hopefully, this helps you out. It is by no means a comprehensive troubleshooting guide…it’s just one that I often use to figure out where the issue lies. I start with the client and move on to the server eliminating possible culprits along the way. If you have any feedback or questions, please let me know in the comments section below. Thanks for reading!
[box type=”info”] Side Note: Interestingly enough, you can use this tool, MFCMapi.exe, to remove ghost delegates as well which is something I’ve spoken about in the past. To do this, use MFCMapi.exe on the client PC where the delegate issue exists and look for the rule whose PR_RULE_PROVIDER is SCHEDULE+ EMS Interface and delete it. Then have he user remove all delegates and then add back the ones they want. Things will be magically repaired.[/box]
I had a problem since our migration from Exchange 2003. The problem was a handful of users were not appearing on our Global Address List.
Tracking down what was going on has taken me quite a long time despite Microsoft KB articles published on the matter. The solution posted by Microsoft in this instance just didn’t work for me. I ran that powershell command until I was blue in the face and it returned zero results. To find out how this happened and why they’re not displaying, I had to dig around for many weeks and create a custom powershell scriptblock until I FINALLY found the answer. Hopefully, passing on this info here will save readers time and effort in their search for resolution of the same thing.
Why Is It Happening?
The reason this is happening is that when you first move from Exchange 2003 to Exchange 2007 or 2010, you have a “Default Policy” in Email Address Policy that should be automatically created (expecially if you run in tandem like I did). This policy is carried over from Exchange 2003. Microsoft changed the way 2007 filters work…they no longer use LDAP as they did with Exchange 2003, they now use OPATH. For a more detailed explanation on this, see this blog post. This default policy can be found in the Exchange Management Console >> Organization Configuration >> Hub Transport. The default policy will place the ‘user alias’ in front of the domain that you have set for your receive connector. When you first apply the policy, it appears like this:
default policy
Where can you view the screen pictured? You can click edit on the default policy found in the location above and press next a couple of times and then cancel out (you don’t want to apply the policy another time do you?).
This information lets us know that the default policy sets up our domain (edited to show domain.com in the picture) with the user network name, aka alias, so that users can immediately send and receive based on login names. Often times, if you are in a small organization, users may login with firstname.lastname or firstinitial.lastname and this may be their email address as well. In this case, you wouldn’t implement a secondary email address policy like I had to do.
Our organization wanted to receive email with firstname.lastname. They’ve implemented a network name (aka alias) policy of first two initials of first and last name followed by last three of employee number. So if my name were John Smith and my employee number were 12345, the network name would become josm345. This is great for not having repeat names on our network but it’s bad for emailing people. Enter my secondary email policy.
secondary policy
The secondary email address policy I created after we switched off Exchange 2003 set up the domain to be able to send and receive with firstname.lastname@domain.com. I then applied this secondary policy and the SMTP address it created as the primary SMTP address for all users and gave this policy a higher priority over the default policy noted above. You can see from the picture the policy takes firstname and lastname in front of the domain.
I also had to update all address lists as shown in this helpful blog post at MXExchangeteam.com. After this, I updated the offline address book by going to Organization Configuration >> Mailbox >> Offline Address Book >> Right-click Update. If you need to create a new global address list, you can’t do it from the EMC, you have to do it via Exchange Management Shell.
It’s a long process, but once you know that you have TWO email address policies that are applied, it makes sense that if a user doesn’t have both of the criteria to match these policies they will not appear on the Global Address List nor the Offline Address Book. So now we have found out who this has happened to and then we’ll know why users are not showing up in these lists. We also need to know what we need to do in order to fix it.
Finding Users Without Both SMTP Addresses
If we find users without both SMTP Addresses present on their mailbox, we will find the users who are NOT appearing in our Global Address List. These users either didn’t have both email address policies applied to them or they had one SMTP address deleted. Either way, if both policies (and thus both SMTP addresses) were not applied to them they will not be included. You can fix this by adding in the SMTP address that is lacking making sure to appropriately set the one you want to be default.
I tried to use Microsoft’s KB article to find the users without both addresses but the query they wanted me to run via Exchange Management Shell returned nothing back for me. I looked up what the query was doing and rewrote it how I thought it should be written. Here’s the finished product and I’ll go through what it’s doing below:
First, with ‘Get-Mailbox -ResultSize Unlimited’ we are getting ALL mailboxes in the enterprise. We’re piping the results using the ‘|’ and then selecting an object based on criteria. The object criteria we are selecting: DisplayName so we can tell who the person is, Primary SMTP address and then we’re creating something called a scriptblock with the third ‘Select-Object’ criteria…I’ve taken it out of the command above to help explain it below:
This command grabs the email address and matches it to the DisplayName we selected previously. It then searches through each Object and ‘Where-Object’ matches the string “smtp” it displays that information. Notice it is in lowercase. The primary SMTP address are stored in AD as capital letters and the secondary SMTP address (or third or fourth, etc) are stored in lower case…so we’re matching ANY other SMTP address (lowercase) that appears for DisplayName. The results should display similar to the following:
DisplayName ServerName PrimarySmtpAddress EmailAddresses
Nologist, Tek ExchServer tekn.nologist@teknologist.net teno321@teknologist.net
The last portion of the code, ‘export-csv c:\export.csv‘, exports the report to the C: drive as a CSV so we can manipulate it in Excel. The -NoTypeInformation removes the type information line from the top of our results…mainly because it’s not needed.
Using this query to find the users that are not appearing on your GAL is now as easy as finding blank entries in a CSV 🙂 Scan that CSV and where an SMTP address cell is missing (primary as noted in ‘PrimarySmtpAddrss’ or secondary as noted under ‘EmailAddresses’) you can note the DisplayName as a user who is not appearing in the GAL.
After you’re done with the list, you will need to add in the missing SMTP information on each of the users that you noted. If you’re wondering WHERE to add it in, open the Exchange Management Console >>Recipient Configuration >> Mailbox >> Right-click user >> Properties >> Email Addresses Tab. From there, you should only see one SMTP address…if you have two policies, you’ll need to add in another SMTP address that matches said policy. As you can see displayed in the picture, some of my users had only my secondary policy applied. So when I went to the Email Addresses Tab, some users appeared to only have firstname.lastname@domain.com.I needed them to also have alias@domain.com to match the default policy I carried over from 2003. To do this, just click the “Add” button and create another SMTP address that matches the user alias@domain.com. Make sure the checkbox at the bottom is checked so that any lists containing this user are updated.
When I first discovered the solution to this, I thought “Why do I need that default policy after I’ve powered down my Exchange 2003 server?” Then it hit me! Users probably still receive mail sent to their alias! Some of the employees have been here for 30+ years and their alias is still a first initial and last name…who’s to say they don’t still give that out as their email address? So I had to keep this policy in play whether I wanted to or not.
After you have found all the users that needed another SMTP server added to their Email Addresses area, you’ll need to go back and update the Offline Address Book again (shown above when I spoke about address lists). The change should be instantaneous for all your OWA users but Outlook users may have to wait for a day before they appear depending on how you have Outlook 2007 setup. If you’d like your Outlook users to immediately have the change, click on the “Send/Receive” arrow and choose “Download address Book”. Select the appropriate address book and click “Ok” Allow it to update and then check to make sure the users that did NOT appear before are actually appearing now. Congratulations! You’ve fixed the problem!
I hope that this post does a good job explaining WHY this occurs and how you can find those users and how you can get them to appear in the GAL and OAB again.
