Client Access Server Archives

How Do You Handle Automated Notifications in Your Enterprise?

Recently, we had an outage due to heat in some of our switch closets. Not only did this generate a lot of alarm traffic for IT Systems…but it also generated a lot of alarm traffic for HVAC and even other systems as well. In the end, we sent out so many automated notifications (emails) to pagers and cell phones that we got rate controlled and then subsequently blacklisted due to the massive volume of emails heading out of our SMTP IP Address. This blacklist meant that email was brought to a screeching halt in the enterprise. I had to have our Network Team fail us over to our secondary SMTP IP Address in order to get mail flowing again. This of course, meant we were no longer using our primary network circuit and there were some DNS hiccups…and really, we didn’t need anything else to make the IT department look bad…DNS was the icing on the cake.

So my question to all the Email Administrators out there is…how do you handle automated notifications in your enterprise? Do you have secondary SMTP servers with different IP Addresses that you use to send out automated notifications? Do you use the same Exchange server but filter traffic using Exchange and virtual SMTP Servers to an external smart host? How do you do it?

I hope someone has some examples for me and I appreciate any help you can give…we’re just venturing into territory I haven’t been in before and it’s always good to get perspective of those who have been there already.

Email Shows Delivered in Message Tracking But Not in User Inbox

This almost ranks right up there with my Ghost Delegates head scratcher of an issue. Now we’re talking about ghost messages. In my case, an automated system was sending out a notification email to a user in our enterprise environment. I could track the message through our SPAM Firewall showing the hand-off (RECEIVE stamp) of the message to our Exchange 2007 Transport Server and then a DELIVER notification of the message being placed inside the mail store.

In other words: The message was actually being delivered into the user inbox…but they never appeared there to the end user. No pop-up notification of a new message in Outlook 2010 on the desktop. Nothing.

So how do we troubleshoot this problem? Luckily, I approached this the correct way right away so I saved myself tons of time and I’m going to share with you how I did it.

Where to Start – Cache Corruption

First and foremost, we use Outlook Cached Mode in our environment. My first thoughts were that the cache file had become corrupt. I’ve seen in the past where a user has not been able to send or receive email to individual distribution groups or they are unable to update an appointment…just really odd singular stuff like this. So I backed up the user .ost file by closing Outlook and Windows Search tool and renaming it to username.ost.bak. Then just startup Outlook and it thinks this is the first time it is starting up…so it re-establishes the .ost from scratch. If there is any corruption, it will be inside of the old .ost file.

For some of you, this might fix your issue. For me, it was back to the drawing board.

Second Step – Establish Where the Problem Exists

The second step is to isolate the problem. Does it exist on the server or is it a problem with Outlook? You can do this by shutting down Outlook and using Outlook Web Access (OWA). Is the message delivered in OWA or to a mobile device when Outlook is closed? If the answer is no, then the issue is NOT an Outlook issue. If yes, congratulations…your issue is an Outlook Issue. Switch it away from cached mode and try using Online mode…this should fix any issues you have. Unfortunately, for my user, the issue did not go away…the message still did not show up. This tells me the issue has to be server side.

Third Step – Check Spam/Junk Folder & Verify Message Delivery

The next step is to check spam folders and junk folders to make sure that the message isn’t getting held up there. I made sure to whitelist the email address on both the client and the server side of things and I tested again. Nothing was going into the spam folders.

I also wanted to make sure, not only with deliver stamps from Exchange, but with absolute proof that the mail message was being delivered. You can do this by adding yourself as a journal recipient on the user account. It is my understanding that Journal receipts process before the filters/rules/spam processing do inside an inbox. So we can prove with 100% certainty that the message is arriving in the user mailbox by adding yourself as journal receipt and watching the mail come in. This should also tell us that it absolutely is a rule that is the culprit versus anything else.

To do this, open up the Exchange Management Console and right click the user that is having a problem. Choose properties. Go to the Mail Flow Settings tab and choose ‘Delivery Options’. Click ‘Properties’ and under the ‘Forward To’ section, choose yourself. Don’t forget to check ‘Deliver message to both forwarding address and mailbox’ or your user won’t any mail. In my case, after I chose these options and had a test message sent out…I received the email in MY inbox but the user still didn’t. So, we now have 100% absolute proof that the message is making it to the inbox and that the user has a rule that is preventing it from being viewed. Let’s see how we can fix this issue in the next step.

Fourth Step – Mail Rules

I know the issue isn’t existing on the client side (Outlook) but I went ahead and started Outlook using the /cleanrules flag from a command prompt. This is supposed to clean out any mail processing rules from the end user.

[box type=”warning”] USE CAUTION! This will remove ALL rules from the user mailbox and the end user should be aware that they will need to recreate any that were previously in place.[/box]

In my case, the user had on 2 rules which were disabled and thus, even with the clean rules switch enabled…the problem was not fixed.

What to do now? I knew something was grabbing the message from the inbox BEFORE it could synchronize to Outlook. So I decided to check the rules in OWA to see if there was a difference (there shouldn’t be). Upon looking in OWA, I found 3 rules present instead of 2. All three were disabled. This was different than the 2 disabled rules I found inside of Outlook. When I looked at the rule, it was taking the ‘ghost message’ from the automated system and moving them to the ‘Inbox’…so there was some kind of loop. They were arriving in the inbox, and then marking themselves as read and moving themselves to the Inbox again…somehow, they were disappearing from view when this happened.

I figured now it would be a simple case of deleting the rule right? Exchange had other ideas. I couldn’t delete the rule…and as I said, it was disabled but somehow still working. I realized this was going to require more tools than I had on hand.

Last Step – Pesky Rules Managed by MFCMapi Tool

The last effort I could use on this problem is to look and see what was going on with the MFCMapi Tool. This tool is available from Microsoft Codeplex and is a small executable that allows you to see a bit more going on in a mailbox than what you could normally see using standard tools.

On the client machine where Outlook is installed (the user who’s rules you want to change) open up MFCMapi.exe and choose the profile the user selects when opening their email. This should give you something similar to the following screenshot:

Open the rules section by going to QuickStart > Inbox Rules:

Search for the rule that you want to delete. In my case, since all the rules were disabled already…I could delete all 3 rules. Your situation may be different…the screenshot is actually my inbox rules so there are quite a few more than 3. However, I was able to delete the rule using this interface where I couldn’t delete it from OWA and I couldn’t see it via Outlook.

After deleting the rule, the emails began to show inside the user inbox and the mystery of the disappearing mail messages was solved. I treated myself to some Scooby Snacks and Old Man Winters would have gotten away with it if it weren’t for those meddling kids.

Hopefully, this helps you out. It is by no means a comprehensive troubleshooting guide…it’s just one that I often use to figure out where the issue lies. I start with the client and move on to the server eliminating possible culprits along the way. If you have any feedback or questions, please let me know in the comments section below. Thanks for reading!

[box type=”info”] Side Note: Interestingly enough, you can use this tool, MFCMapi.exe, to remove ghost delegates as well which is something I’ve spoken about in the past. To do this, use MFCMapi.exe on the client PC where the delegate issue exists and look for the rule whose PR_RULE_PROVIDER is SCHEDULE+ EMS Interface and delete it. Then have he user remove all delegates and then add back the ones they want. Things will be magically repaired.[/box]

User Appears in “All Users” list but not in Global Address List (GAL)

Ever had a new user appear in the “All Users” Address list that you can access via your Address Book but that DO NOT APPEAR inside the Global Address Book (GAL)?

These are most likely users that have only recently been added…because the minute that Exchange does it’s standard maintenance window, it will most likely update. But what if, like me, you need to have it update right away due to something like a BlackBerry server?

Through trial and error, I’ve found out WHY the user is not put into the Global Address List right away and how you can force it there. Please note that I’m using Exchange 2007 with a separate CAS, HT, and Mailbox Server. First, let’s go over what is happening and why it is and then we’ll go over how we can force the OAB (Offline Address Book) or GAL to update with these users.

What is Happening and Why?

So, you added Joe Smith to the company and you’ve right-click updated the “All Users” address list so he can be included in distribution lists and so that he receives emails on dynamic distribution lists. Then you right clicked the OAB and selected ‘Update’. Excellent! Welcome to the company Joe! But wait, Joe is not appearing inside the GAL/OAB you just updated! Why is this problem happening!? If you’re using the web distribution of the offline address book, the Exchange Client Access Server waits for a ‘polling time’ to arrive before it updates…just like any standard DFS (distributed file system) in Active Directory. The default time is 480 minutes and of course, we don’t want to wait that long. You can read all about how OAB via DFS works by visiting this link about Exchange polling times and OAB distribution.

Now that we know it isn’t working by design, how do we fix it? We manually force it to poll. We do this through the Exchange Management Shell:

update-filedistributionservice -identity Servername

Make sure you substitute in your Client Access Server where the OAB is distributed with ‘servername’ above. A warning will appear if you do not have Unified Messaging installed on your server. If you don’t, it is safe to ignore that warning.

After you’ve forced the update, manually download the address book in any Outlook client and the user will magically appear in the Global Address List. Hope this helps someone…I know it took me a while to figure out what was happening.

Unstable Update to BlackBerry Desktop Software

The recent update to BlackBerry Desktop Software (version 6.0) is hazardous and will mess up your install causing you to have to manually scrape it from your computer and revert back to the version you have on disk (most likely, 4 or 5). Be warned, you’ll get errors when launching the application that will prevent it from starting.

black bury it in the ground — go go gadget error reporting!

If you have earlier versions of the software and are prompted to update…my advice is to NOT update it. Unless you’re a fan of fail…then by all means give it a go.

Why You Can’t Trust Mobile Device Reports

The Problem

Let’s say you’re an exchange administrator and you use EAS (Exchange Active Sync) to plug smart phones into your Exchange environment. You’re probably one of the many out there who do this…I do it in my environment.

Then you stumble across a handy powershell script that will allow you to query your environment for mobile phones that have synched on user accounts and report back the last successful synch as well as any phone details that are reported. I found this one somewhere (can’t remember):

$devices = @()
$mailboxes = Get-CASMailbox -ResultSize:Unlimited | Where-Object {$_.HasActiveSyncDevicePartnership -eq $true -and $_.ExchangeVersion.ExchangeBuild -ilike "8*"}

foreach ($m in $mailboxes)
{
$devices += Get-ActiveSyncDeviceStatistics -Mailbox $m.Identity
}

$devices | Export-Csv DeviceStats.csv

You get a nice little list of stuff right? This should be every single EAS enabled account that has a phone synching with Exchange right? Wrong. It absolutely is NOT an accurate list of phones that have synched with your Exchange server.

When I executed that powershell scriptlet above on my system it reported back quite a few phones…but my phone, a HTC Evo, was not on the list. In fact, my account wasn’t reported on the list. This despite my device synching just fine every 15 minutes. The integrity of this ‘report’ has been challenged. So what’s the deal?

Workaround Solution

A quick investigatory glancing at google brings little results…However, I did notice when using the EMC to highlight my account and going to ‘Manage mobile device” I was in for a suprise….because that link wasn’t there on my account. That’s right, “manage mobile device” is missing on my account. This despite having Active Synch enabled under the ‘mailbox features’ tab on my account and despite having my device pair up with Exchange in 15 minute intervals.

Searching google for this problem yeilds much better results. It seems that a flag embedded for a value in Active Directory doesn’t toggle correctly OR isn’t added into AD at all (perhaps bad replication). Either way, it means you’ll be whipping out trusty adsiedit.msc (per the forum thread linked to above). The setting that needs toggled is “msExchMobileMailboxFlags” which should be set to 1. After setting this, you’ll be able to manage the mobile device AND your synch will be picked up by the powershell report above.

The Real Crux

The real crux of the matter is that this toggle shouldn’t have to be set at all and regardless if it is set or not, powershell should be able to report if a device is synching with Exchange. There is a reason why RIM is #1 when it comes to businesses and smartphones…it’s because you can have 100% accountability for what is on your network at any given time. With Exchange 2007 and above, it is unfortunately, much like a ring toss at a circus…sometimes you get a ring on that bottle and win a stuffed giraffe but other times you’re going home without plush pals.

Microsoft has a problem here because you can’t trust your own powershell queries. If it doesn’t work for this individual area (phones, EAS) then what else doesn’t work? Do we call into question the entire integrity of all powershell commands and commandlets? What do you think? I know this workaround is pretty silly on a huge environment with multiple user containers in AD or even mutlipe forests. Is the workaround something that is tolerable or should Microsoft do more to fix this broken part of their product?