How Do You Handle Automated Notifications in Your Enterprise?

Recently, we had an outage due to heat in some of our switch closets.  Not only did this generate a lot of alarm traffic for IT Systems…but it also generated a lot of alarm traffic for HVAC and even other systems as well.  In the end, we sent out so many automated notifications (emails) to pagers and cell phones that we got rate controlled and then subsequently blacklisted due to the massive volume of emails heading out of our SMTP IP Address.  This blacklist meant that email was brought to a screeching halt in the enterprise.  I had to have our Network Team fail us over to our secondary SMTP IP Address in order to get mail flowing again.  This of course, meant we were no longer using our primary network circuit and there were some DNS hiccups…and really, we didn’t need anything else to make the IT department look bad…DNS was the icing on the cake.

So my question to all the Email Administrators out there is…how do you handle automated notifications in your enterprise?  Do you have secondary SMTP servers with different IP Addresses that you use to send out automated notifications?  Do you use the same Exchange server but filter traffic using Exchange and virtual SMTP Servers to an external smart host?  How do you do it?

I hope someone has some examples for me and I appreciate any help you can give…we’re just venturing into territory I haven’t been in before and it’s always good to get perspective of those who have been there already.

Disappearing Favorite Calendars in Outlook 2007 and 2010

Some of my users will submit tickets where they are missing ‘favorite’ calenders when they go into Outlook.  This is not a huge problem in itself because they can just browse to it via Public Folders (or whatever resource they add it from).  The main problem is that a user cannot add a favorite back after it has disappeared.  It’s more a frustration…because it’s not like the calendar is gone…it’s just not one click accessible.

So, I did quite a bit of research to solve this the first time it reared its head with no fix found.  I’ve had this happen enough times now that I decided to take the time to figure out exactly what was happening and how I could solve it.  There are three solutions I’ve found with first solution (#1 below) being a last resort.

  1. You can create a new mail profile. This resets everything you’ve customized unfortunately, so it is, in my opinion, a last resort.
  2. You can navigate to the folder that Outlook stores settings in. In Windows XP it defaults to:
    C:\Documents and Settings\<username>\Application Data\Microsoft\Outlook
    In this folder you will find a file called Outlook.xml. Very innocuous sounding, don’t you think? This is the file causing all of the woes! You should exit Outlook completely then rename the file or delete it. Restart Outlook and you will notice all of your favorites and calendars are gone. Add them back the way you would like, exit Outlook again and restart. *Poof* they are back and there to stay!
  3. This solution is hit or miss…it’s fixed it for some people and not for others.  Launch Outlook via command line with the /resetnavpane flag afterwards:  outlook.exe /resetnavpane

The problem of course is a corrupt file.  This Outlook.xml file for whatever reason becomes fragmented/corrupt and causes calendars to evaporate from your listing in Outlook with the inability to add them back in as persistent through start/exit of Outlook.

http://blog.palehorse.net/2007/09/26/outlook-ate-my-favorite-shortcuts/

I hope this helps someone because it took me a lot of digging to figure out what was going on.  In the blog post I linked to above, the person’s shortcuts were disappearing….different terminology that hid it from my search results.  I’ve taken the solutions from the blog post and posted them here because I don’t know how long other blog posts will be accessible.  I’ve had my blogs up and running since 2003 so I know they’re not going anywhere…so I record handy information for myself (and others?) here.  Thanks for readin!

Delegate Management of Distribution Groups

owner of group

Ever wanted to stop managing a distribution group because you get 90 million requests every other day to add someone or remove someone from said group?  Ok, maybe not 90 million but it has to be close to 80 million right?  Sometimes you will run across a distribution group that changes its membership frequently.  The best solution I’ve found is to find a point of contact for that group who will be able to manage the membership.  This means less requests to you in the future.

Normally, what I do is add the person as the ‘owner’ under the properties of the distribution group.  While this does nothing to give the person rights to the group, it does allow me to remember which member of the group is the point of contact.  In the example above, I added Scooby Doo as the owner of the 4th Floor NAs (nursing assistants) distribution group.  This allows me to remember the person (or cartoon dog) I am granting write permissions to manage the group to.

Next you have to do some powershell magic to grant write permissions to that very same person:

Add-ADPermission -Identity:'Group Display Name’ -User:domain\username -AccessRights ReadProperty, WriteProperty -Properties 'Member'

Now, if you wanted to grant permissions to a group of people…you might not be able to add an owner…but you can fill out the ‘notes’ section shown in the picture above and drop yourself a line to remember which Active Directory group has permissions to write membership of the group.  The command here would be:

Add-ADPermission -Identity:'Group Display Name’ -User:'Display Name of Permissions Group’ -AccessRights ReadProperty, WriteProperty -Properties 'Member'

I know this has been covered in countless other blogs and other nooks and crannies of the interwebs…I’m sure I’m not telling anyone anything new.  Please remember though that this blog is not only a tool for people to find on the internet…but also a knowledge repository for myself.  I can find the things that are most useful to me simply because I write about them.  I know where to look after I blog about them…and I can guarantee that this blog will be up indefinitely since I host it myself.  That’s more than I can claim for most blogs/resources of information out there covering these topics…most blogs dry up after a few years.

Hopefully, this information will help a few searching souls out there looking to decrease their distribution list management burden.  Thanks for reading!

Dear Microsoft, Public Folders

Dear Microsoft,

Public Folders are way to hard to manage in Exchange 2007/2010.  Adding/Removing folders is a pain the neck; even if I’ve got the right permissions, I still don’t have the right permissions.  Please fix this immediately.

Yours Truly,

Teknologist

PS:  Do something about getting Local Continuous Replication (LCR) in 2010 too?  Thanks!

Exchange 2003 to 2007 Global & Universal Groups

I hit a snag that was a result of my conversion from Exchange 2003 to Exchange 2007.  The snag was with global groups.  The problem is that global groups cannot be expanded by Exchange…so if you have a distribution list that is a member of a global group it will be grayed out in the Exchange Management Console and you will not be able to edit it.  You also won’t be able to right click and convert it to universal because it will give you the error of:

A global group cannot have a universal group as its member

So what to do?  How does one convert this group to a universal one?  The problem doesn’t lie with the group  itself but with the membership of the groups.  In order to convert one, you have to convert them all.

The EXPTA blog has an excellent post regarding what is going on with solutions to boot:

As you may know, Exchange Server 2007 and Exchange Server 2010 force you to create all new distribution groups as universal distribution groups.
The reason for this is that Exchange 2007/2010 requires a local Global Catalog (GC) server in the Active Directory site where Exchange resides to query for group expansion. A GC can expand domain local, global, and universal groups. However, domain local groups (and sometimes global groups) can only be expanded within the domain local scope. If the GC is a member of the companyabc.com domain, it will be unable to expand a domain local group in the sales.companyabc.com subdomain.

I’m going to be posting how they solved their problem for posterity but you can head over to the EXPTA blog for the full post which contains much more meat than this post will.

To solve the problem, query the groups and look at how many you have that are global.  Just open up a command window on a domain administrator account and remember that this command may take a while on a large enterprise:

dsquery group -limit 0 | dsget group -samid -scope -secgrp > Groups.txt

Next run the command to convert the global groups to universal.  There are no adverse problems that will result from this…the conversion doesn’t mess with permissions.

dsquery group -limit 0 | dsmod group -c -q -scope u

You WILL have to run this 2nd command many, many times.  See, each time it cycles through the groups it finds the top level one it has not converted and converts it to universal…there may be 3 more subgroups that need converted but it won’t convert them until the parent is a universal.

For more explanation, see the EXPTA blog link above and hopefully this helps someone out!

How to Fix Ghost Delegates and List Delegates in Exchange 2007

The Problem of Ghost Delegates

There is a problem with Microsoft Exchange 2003 and delegates.  When users are deleted, their delegate or send on behalf attributes are not deleted with them.

This problem has been around since Exchange 5.5 and possibly before and it’s REALLY annoying.  Microsoft has a hotfix available for Exchange 2003…but the problem is, you have to know WHICH mailboxes the fix needs to be run on.  Top that off with my environment actually being Exchange 2007 and NOT 2003 and one might discover an even larger problem.

Let’s backtrack and define the problem with a little story:

Say you work for Acme Corp which has 1500 employees in a single forest environment running Exchange 2007.  John Smith, an employee of 10 years at your corporation has left for greener pastures over at Emca Corp.  John was a delegate with send rights on Frank Doe’s mailbox as he was Frank Doe executive assistant for almost all of those years.

When HR notified your Active Directory administrator that John Smith had left, they promptly disabled and later deleted the user…this may have happened while you were still using Exchange 2003.

Now you have a problem…John Smith has left the company and was a delegate on Frank Doe’s mailbox.  When someone sends an invitation to a meeting with Frank Doe or mail to a group that Frank Doe is a part of…they may get an Non Delivery Notice (NDR) similar to the following:

Smith, John
The recipient’s e-mail address was not found in the recipient’s e-mail system. Microsoft Exchange will not try to redeliver this message for you. Please check the e-mail address and try resending this message, or provide the following diagnostic text to your system administrator.

_____

Sent by Microsoft Exchange Server 2007

This glitch is EXCELLENT for annoying users everywhere!  It’s also a potential problem as someone could mass email Frank Doe and potential generate a NDR in addition to each email they send.  If they were to identify a few more people who may have had this problem, the possibility of having twice the amount of email generated and thus, twice the amount of email traffic, is something that large corporations may not want to have in their existing Exchange environments.

Microsoft realized that they had a problem on their hands and released a fix that you apply to mailboxes that suffer from the ‘ghost delegate’ “feature”.  Great right?  This fix is of course for Exchange 2003…so if you’re using that, you’re set.  You still will need to know a major detail before proceeding though:

  1. How many mailboxes have this problem? (usual response)

Therein lies the major problem.  How do you identify which mailboxes have old ghost delegate cruft still floating around generating NDR’s all over the place?  Especially in smaller environments where EVERYONE was a delegate of everyone else?

It’s impossible.

You can try to identify which mailbox has the problem but you would have to use the process of elimination…not a very time saving process and definitely not something I want to do (imagine visiting 50+ users!).  Instead,  I elected to identify all current users with delegates on their mailbox because chances are that the ghost delegate was once a delegate of these identified users.  I plan on cross referencing these users with a distribution list that is generating the NDR to properly identify the problem.  So, onward with how to identify the potential problem with a powershell query.

Identifying Problem Mailboxes with Powershell

Search the web, I’ve found many different ‘solutions’ to this problem with none of them getting things exactly how I needed them to be.  Some of the solutions would run a query and return information that was valuable…but not valuable enough.  Other queries should have worked but didn’t.

I’m by no means a powershell expert so I’m absolutely sure this little scriptlet can be improved.  It does the job though by returning anyone who has a delegate and listing who each delegate is.  So without further fanfare, the powershell command:

Get-Mailbox -resultsize unlimited | Where {$_.GrantSendOnBehalfTo -ne $null} | select Name, @{Name='GrantSendOnBehalfTo';Expression={[string]::join(";", ($_.GrantSendOnBehalfTo))}} | Export-CSV C:\SendOnBehalfTo.csv -noTypeInformation

Let’s break this command down.  The first portion Get-Mailbox -resultsize unlimited retrieves all user mailboxes and doesn’t limit to 1000 as is default with Exchange 2007.  We pipe this command into the next one with the ‘|’ symbol.  Here we use a conditional Where and select the attribute GrantSendOnBehalfTo returning only mailboxes from the first part of the command where said attribute is NOT null or empty.

Next, out of these mailboxes that don’t have the GrantSendOnBehalfTo attribute as empty, we select the Name of the user.  Now we should be able to stop here, however, we’re working with a string attribute here and returning Name will return a Multivalued property instead of the proper DN or user name we’re looking for (something like this Microsoft.Exchange.Data.MultiValuedProperty`1[Microsoft.Exchange.Data.Directory.ADObjectId] ).  To get what we need, we have to join the Multivalued property (string) with the results from the first half of the query.  To do this we create a commandlet and use the join command to bind the first part of our command to the multivalued property and translate that property into something readable for us.

It’s in the string conversion section that most people get frustrated with.  I know I got frustrated with it and used a portion of a commandlet for allowing a user or group to only accept messages from certain mailboxes…I just adapted that string conversion and join for my own purposes.  Like I said, I’m not extremely fluent in the ways of the powershell script…I just know that adapting it worked.

Identifying the Problem Mailboxes with Active Directory Queries

Of course, after spending 6-8 hours constructing my query in powershell I found a simple little tool from Joeware.net called “ADFind” that allowed me to execute a simple AD query and output all of the same information into a much more readable format than my powershell command.  If you’d like to do the same thing, head over to the ADFind page, enter your email and download the tool.  Extract it to your desktop and then open a command prompt and change directory to your desktop.  Next, enter a proper AD query with the adfind.exe prepended:

AdFind.exe -default -f "&((homeMDB=*)(publicdelegates=*))" publicdelegates > C:\delegates.txt

The output from Joe’s tool is MUCH nicer than what I was able to accomplish using powershell.  I’m sure someone will be able to take what I did with powershell and improve on it…but when there are handy tools like joeware provides, what need is there?

Implementing the Microsoft Fix

The ‘official’ fix from Microsoft is to install a hotfix if you’re running Exchange 2003.  Supposedly, Exchange 2007 doesn’t suffer from this problem…but there is a caveat there.  If, like me, you’ve upgraded from an Exchange 2003 environment that suffered from this problem…you migrate the problems with it!  So, I have an Exchange 2007 environment suffering from ghost delegates and NDR’s.

How to remedy this?  There are 3 directions provided.  The first from Microsoft is to go inside the tools >> options menu in Outlook and delete the delegate.  The problem with ghost delegates is that they aren’t there…which renders this fix completely useless.  The second solution is to launch Outlook with the /cleanrules switch.  The third solution out there is to use the MFCMapi Editor on the user mailbox that has the problem, find the hidden bits, and delete them.  The problem with this solution is most people AREN’T comfortable with editing the profile inside a snap-in…it’s much like using ADSIEdit to edit AD manually and can be overwhelming.

Examining the 3 options, it’s obvious that /cleanrules switch is probably the easiest to implement.  The problem is that it will delete ALL rules from the mailbox and if you have users who have been around for quite some time it means tons of rules.  The other problem is that it has to be run manually on EACH mailbox.  This of course is bad if you have an elaborate delegate system.

When we identified who has delegates with our queries above, we most likely found a culprit for who is having the problem and generating the NDR.  For me it was rather easy because I cross referenced my list of those with delegates with a distribution list.  When one of the unit secretaries sent email to this distribution group, it would also generate the dreaded NDR.  This means that a member of the distribution group has a ghost delegate.  Cross referencing gave me a list of 6 people to run the /cleanrules switch on versus the over 50 delegate mailboxes from the queries above…a much smaller number to visit.

I hope my confusing process can help someone out there and please improve on anything you see here…as I said, I’m by no means an expert…I just like to document my learning process to help others and also to remind myself of the fixes I find 🙂  Thanks for reading!

Fowler, Carol
The recipient’s e-mail address was not found in the recipient’s e-mail system. Microsoft Exchange will not try to redeliver this message for you. Please check the e-mail address and try resending this message, or provide the following diagnostic text to your system administrator.

 

_____
<!–[if !supportLineBreakNewLine]–>
<!–[endif]–>

Sent by Microsoft Exchange Server 2007

User Not Appearing in GAL 2007 Fix

I had a problem since our migration from Exchange 2003.  The problem was a handful of users were not appearing on our Global Address List.

Tracking down what was going on has taken me quite a long time despite Microsoft KB articles published on the matter.  The solution posted by Microsoft in this instance just didn’t work for me.  I ran that powershell command until I was blue in the face and it returned zero results.  To find out how this happened and why they’re not displaying, I had to dig around for many weeks and create a custom powershell scriptblock until I FINALLY found the answer.  Hopefully, passing on this info here will save readers time and effort in their search for resolution of the same thing.

Why Is It Happening?

The reason this is happening is that when you first move from Exchange 2003 to Exchange 2007 or 2010, you have a “Default Policy” in Email Address Policy that should be automatically created (expecially if you run in tandem like I did).  This policy is carried over from Exchange 2003.  Microsoft changed the way 2007 filters work…they no longer use LDAP as they did with Exchange 2003, they now use OPATH.  For a more detailed explanation on this, see this blog post.  This default policy can be found in the Exchange Management Console >> Organization Configuration >> Hub Transport.  The default policy will place the ‘user alias’ in front of the domain that you have set for your receive connector.  When you first apply the policy, it appears like this:

default policy

Where can you view the screen pictured?  You can click edit on the default policy found in the location above and press next a couple of times and then cancel out  (you don’t want to apply the policy another time do you?).

This information lets us know that the default policy sets up our domain (edited to show domain.com in the picture) with the user network name, aka alias, so that users can immediately send and receive based on login names.  Often times, if you are in a small organization, users may login with firstname.lastname or firstinitial.lastname and this may be their email address as well.  In this case, you wouldn’t implement a secondary email address policy like I had to do.

Our organization wanted to receive email with firstname.lastname.  They’ve implemented a network name (aka alias) policy of first two initials of first and last name followed by last three of employee number.  So if my name were John Smith and my employee number were 12345, the network name would become josm345.  This is great for not having repeat names on our network but it’s bad for emailing people.  Enter my secondary email policy.

secondary policy

The secondary email address policy I created after we switched off Exchange 2003 set up the domain to be able to send and receive with firstname.lastname@domain.com.  I then applied this secondary policy and the SMTP address it created as the primary SMTP address for all users and gave this policy a higher priority over the default policy noted above.  You can see from the picture the policy takes firstname and lastname in front of the domain.

I also had to update all address lists as shown in this helpful blog post at MXExchangeteam.com.  After this, I updated the offline address book by going to Organization Configuration >> Mailbox >> Offline Address Book >> Right-click Update.  If you need to create a new global address list, you can’t do it from the EMC, you have to do it via Exchange Management Shell.

It’s a long process, but once you know that you have TWO email address policies that are applied, it makes sense that if a user doesn’t have both of the criteria to match these policies they will not appear on the Global Address List nor the Offline Address Book.  So now we have found out who this has happened to and then we’ll know why users are not showing up in these lists.  We also need to know what we need to do in order to fix it.

Finding Users Without Both SMTP Addresses

If we find users without both SMTP Addresses present on their mailbox, we will find the users who are NOT appearing in our Global Address List.  These users either didn’t have both email address policies applied to them or they had one SMTP address deleted.  Either way, if both policies (and thus both SMTP addresses) were not applied to them they will not be included.  You can fix this by adding in the SMTP address that is lacking making sure to appropriately set the one you want to be default.

I tried to use Microsoft’s KB article to find the users without both addresses but the query they wanted me to run via Exchange Management Shell returned nothing back for me.  I looked up what the query was doing and rewrote it how I thought it should be written.  Here’s the finished product and I’ll go through what it’s doing below:

Get-Mailbox -ResultSize Unlimited |Select-Object DisplayName,PrimarySmtpAddress, @{Name=“EmailAddresses”;Expression={$_.EmailAddresses |Where-Object {$_.PrefixString -ceq “smtp”} | ForEach-Object {$_.SmtpAddress}}} | export-csv c:\export.csv –NoTypeInformation

First, with ‘Get-Mailbox -ResultSize Unlimited’ we are getting ALL mailboxes in the enterprise.  We’re piping the results using the ‘|’ and then selecting an object based on criteria.  The object criteria we are selecting:  DisplayName so we can tell who the person is, Primary SMTP address and then we’re creating something called a scriptblock with the third ‘Select-Object’ criteria…I’ve taken it out of the command above to help explain it below:

@{Name=“EmailAddresses”;Expression={$_.EmailAddresses |Where-Object {$_.PrefixString -ceq “smtp”} | ForEach-Object {$_.SmtpAddress}}} | export-csv c:\export.csv –NoTypeInformation

This command grabs the email address and matches it to the DisplayName we selected previously.  It then searches through each Object and ‘Where-Object’ matches the string “smtp” it displays that information.  Notice it is in lowercase.  The primary SMTP address are stored in AD as capital letters and the secondary SMTP address (or third or fourth, etc) are stored in lower case…so we’re matching ANY other SMTP address (lowercase) that appears for DisplayName.  The results should display similar to the following:

DisplayName    ServerName    PrimarySmtpAddress                EmailAddresses
Nologist, Tek  ExchServer    tekn.nologist@teknologist.net     teno321@teknologist.net

The last portion of the code, ‘export-csv c:\export.csv‘, exports the report to the C: drive as a CSV so we can manipulate it in Excel.  The -NoTypeInformation removes the type information line from the top of our results…mainly because it’s not needed.

Using this query to find the users that are not appearing on your GAL is now as easy as finding blank entries in a CSV 🙂  Scan that CSV and where an SMTP address cell is missing (primary as noted in ‘PrimarySmtpAddrss’ or secondary as noted under ‘EmailAddresses’) you can note the DisplayName as a user who is not appearing in the GAL.

After you’re done with the list, you will need to add in the missing SMTP information on each of the users that you noted.  If you’re wondering WHERE to add it in, open the Exchange Management Console >>Recipient Configuration >> Mailbox >> Right-click user >> Properties >> Email Addresses Tab.  From there, you should only see one SMTP address…if you have two policies, you’ll need to add in another SMTP address that matches said policy.  As you can see displayed in the picture, some of my users had only my secondary policy applied.  So when I went to the Email Addresses Tab, some users appeared to only have firstname.lastname@domain.com. I needed them to also have alias@domain.com to match the default policy I carried over from 2003.  To do this, just click the “Add” button and create another SMTP address that matches the user alias@domain.com.  Make sure the checkbox at the bottom is checked so that any lists containing this user are updated.

When I first discovered the solution to this, I thought “Why do I need that default policy after I’ve powered down my Exchange 2003 server?”  Then it hit me!  Users probably still receive mail sent to their alias!  Some of the employees have been here for 30+ years and their alias is still a first initial and last name…who’s to say they don’t still give that out as their email address?  So I had to keep this policy in play whether I wanted to or not.

After you have found all the users that needed another SMTP server added to their Email Addresses area, you’ll need to go back and update the Offline Address Book again (shown above when I spoke about address lists).  The change should be instantaneous for all your OWA users but Outlook users may have to wait for a day before they appear depending on how you have Outlook 2007 setup.  If you’d like your Outlook users to immediately have the change, click on the “Send/Receive” arrow and choose “Download address Book”.  Select the appropriate address book and click “Ok”  Allow it to update and then check to make sure the users that did NOT appear before are actually appearing now.  Congratulations!  You’ve fixed the problem!

I hope that this post does a good job explaining WHY this occurs and how you can find those users and how you can get them to appear in the GAL and OAB again.