Exchange Server EMail Routing – Accepted Domains and Send Connectors

Exchange Server (And Exchange Online) can be a little confusing at times, particularly when we’re dealing with mail routing. Internal mail routes are handled almost automatically (especially if you keep all your Exchange servers in the same AD Site, which I recommend), but how do you get it to route email to mail servers *outside* your organization? What about partner companies, business departments with their own AD forests, or between on-prem and cloud mail platforms? Most environments don’t have to mess with complicated mail routing issues, but if you’re a consultant, or if you are working with a large Exchange deployment with multiple partner organizations, you will need to understand how mail routing works in Exchange. There are 3 pieces to this; Exchange Organizations, Accepted Domains, and Connectors.

Exchange Organizations

This portion of Exchange Mail routing is more about terminology than function. Put Simply, an Exchange Organization is an number of Exchange mail servers that exist in a single Active Directory (AD) forest. Exchange is heavily integrated with Active Directory, which is Microsoft’s technology that allows central control of usernames, passwords, and computers/devices. If you’re trying to learn Exchange Server and you don’t know anything about Active Directory, stop and learn that first, or you will have a lot of problems understanding what is going on.

An Exchange Organization covers every single Exchange server that exists in the same AD forest. You can’t have two Exchange Organizations in the same forest, and this is an important concept, because email routing in each Exchange Organization is controlled with Active Directory Sites and not traditional email routing techniques. If two users in the same AD Forest want to send email to one another, it doesn’t matter what their @domain.com email address is (As long as it’s an accepted domain for that Organization), mail routing will be done automatically by Exchange based on which AD Site each each user’s mailbox is located in.

The need for clear terminology is because Exchange deployments in different Forests have to be routed properly to work right. This is especially complicated if multiple Exchange Organizations have users with the same email domain (@domain.com).

Accepted Domains

Accepted domains are the core component of Exchange email routing. Each domain represents the portion of an email address after the @ sign. So for my email address, adam@acbrown-it.com, the Exchange environment that managed my email has acbrown-it.com as an accepted domain. In Exchange Server (On-prem) there are also three types of accepted domains. For Exchange Online, there are two types of accepted domains. The accepted domain types are:

  1. Authoritative
  2. Internal Relay
  3. External Relay (On-prem only)

Each type of accepted domain functions differently and, depending on circumstances, can be used to route email. It’s worth noting here that the name of each type doesn’t necessarily make its function obvious. Here’s how they work…

Authoritative Domain

You might think that an Authoritative domain would be a central server for a specific email domain. For instance, if you had an environment that had multiple Exchange organizations, the name “Authoritative” would make you think that you would set the @domain.com domain as authoritative on the main Exchange server that receives email for this domain. This is not how it works. When a domain is set as authoritative, that tells Exchange that all mail routing for the domain will STOP at this organization. If you were set up with two Exchange organizations that had @domain.com email addresses in them and you set the first server that received email for that domain as authoritative for the accepted domain, no email would ever reach the second Exchange organization. In this case, Authoritative should be seen as the Exchange server saying, “The buck stops here!” for all email in that domain.

Internal Relay

Internal Relay domains are used in situations where more than one Exchange Organization contains users with the same email domain. When an organization is set up to use an Internal Relay domain, it will look for a mailbox that matches the email address in its own organization first, but if it doesn’t find that mailbox, it will send the message off to another organization. This is very important to remember, because you have to decide where the email will go next using a Send Connector (explained later). If you use Internal Relay domains, note that Email routing between organizations *must* stop at an authoritative domain. If it doesn’t, email will get NDRs referencing Loop Detection, which is a pain.

External Relay

An External Relay domain only exists in on-prem Exchange. External Relay works similarly to an Internal Relay domain, except that Exchange will *not* check its own recipient list to see if the email address matches. Email addresses that match an External Relay domain will be immediately forwarded without any real processing. This type of accepted domain has very little functional use these days, except to allow for a hub and spoke architectural design, when a single entity acts as a central point for mail. With an External Relay domain, that central point can relay messages to as many other entities as it wishes without wasting CPU cycles checking the recipient lists before forwarding the messages to their ultimate destination. This type of accepted domain is not available in Exchange Online, simply because Microsoft wants you to check the recipient list before forwarding messages, and to reduce complexity (Since Office 365 is heavily marketed toward smaller businesses whose technical staff may be lacking experience or knowledge, and having this option available might confuse people).

Send Connectors

Accepted domains aren’t enough to properly route mail through Exchange, since they just tell the system what to do with email once received. After messages are processed against the accepted domain list, Exchange has to know what to do with them. This is where send connectors come in.

Each send connector is configured to apply to a specific list of domains (Or all domains, if the connector uses the * address scope). If the transport service sees an email it needs to send outside the Exchange organization, it will process the email domain against the list of send connectors to determine how to send the message. Each send connector has an address “scope” or address space that determines when it’s supposed to be used. If the email domain of the recipient on the email that is being sent outside the organization matches the send connector, that connector’s rules will be used.

One important thing about send connectors that you need to remember is that there should always be a connector with an address space that has just an asterisk (The star symbol) as shown here:address space

This will always be processed *last* and ensures that emails that don’t match any other send connector get routed properly (Unless you only want the Exchange server to route to domains you specify, in which case, leave off the * connector from the list…also, you’re crazy if you want to do this).

If you want to get really, unnecessarily complicated, you can also configure Scoped Send Connectors (to ensure only Exchange servers in a specific AD site can use the connector) or implement secondary connectors for the same domain (If you want to allow Exchange to have a second location to send mail to if the first location fails). I don’t recommend doing either of these things. If you find that you do need to,  you may need to re-examine your Exchange architecture (Up to you).

The Delivery tab of the send connector properties is where the work of a send connector is defined. By default, the connector will follow the MX record settings that the server sees when determining where mail is sent. It’s important to note here that you have to pay attention to what the *server* sees, not what is available to everyone on the Internet. If your Exchange server is set up to use a Domain Controller for DNS, you can create your own MX records for any domain in the world by creating a Forward Lookup Zone for that domain, then creating MX records to route mail for that domain. Again, I don’t recommend doing this, just note that it’s a possibility, so make sure you are taking that into account when troubleshooting.

More common, however, is what is called “Smarthost” delivery. A Smarthost is basically any SMTP server that is capable of determining how to properly handle the message. Almost every mail server in the world can be used as a smarthost, but you should have a specific purpose in mind when using this setting. For instance, if you want to send all email to a spam filter for processing and relay, you would set up an address space of * and set the smarthost to the spam filter’s IP/DNS address. Your Exchange deployment is probably configured to do this already (Even Exchange Online has a hidden send connector that points outgoing email to Exchange Online Protection). If, however, you want to send email for a specific domain to a specific server, you would set the address space to equal the domain, then set the smarthost to be the IP/DNS address of that server.

Summing Up

If you understand the relationship between send connectors and accepted domains, you can do a lot of really cool stuff. For instance, you could have half your users in Office 365 and half in Google Apps (Probably not the best idea, but it’s possible). Exchange Hybrid configurations make heavy use of accepted domains and send connectors to properly route email between cloud and on-prem users. And there are plenty of other use cases. If you’re feeling brave or working in a test lab, tinker around with these settings a bit and see what nifty tricks you can pull, but take care to remember the rules as I’ve explained them. If you don’t, you may spend hours troubleshooting just to find yourself feeling really dumb when you discover that the accepted domain isn’t set right or the send connector sends to the wrong server.

Advertisements

Adam’s O365 Tips and Tricks Part 1: Exchange Online Email Recovery and Retention

With most people moving to Exchange Online or other cloud-based solutions for email, I’ve decided to write up some tips and tricks that might not be well known, but will give you some useful tools for managing Office 365 (Well, I guess they’re calling it Microsoft 365 now), which is the cloud service I am most familiar with. I’ll be expanding and adding articles on the subject as I come up with ideas and remember things I’ve done through the years, so be sure to check back periodically to see what’s new. For this edition, I’ll be covering Exchange Online Backups

Exchange Online Backups Aren’t Necessary!

One thing that drives me bonkers about the third party tools market for Office 365 are the number of companies selling Office 365 Backup Services. Some of that may be helpful for things like OneDrive and SharePoint (Unless you have an E3 license), but Exchange Online provides numerous tools for recovering email and handling retention for all license levels, as long as it’s configured correctly.

Recovering Deleted Emails

The most important thing you can do with Exchange Online is to make sure that a feature called “Single Item Recovery” is enabled. What this feature will do is allow admins to recover any deleted item in any mailbox, even if the user has purged it from the Deleted Items folder (Available by Right-Clicking Inbox and selecting “Recover Deleted Items”). Single Item Recovery will allow items to be deleted, but will retain them for a period of time that you can configure in Exchange Online Powershell (Default is…*Forever*). Recovering emails usually requires the InPlace eDiscovery feature in the compliance tools (Those controls have moved around a lot, so just look for any compliance search features in EoL’s admin portal or the O365 Portal). For a more in-depth look at the feature, visit this Technet Blog.

Fun With Shared Mailboxes

One of the more entertaining features of Exchange Online is Shared Mailboxes. A Shared Mailbox is a limited functionality mailbox that (currently) has a 50GB limit, does not have a password (and so can’t be logged into directly), and is FREE. Yes, you read that correctly. You can have as many shared mailboxes in your EoL tenant as you want and don’t have to pay a license for them. This opens up a world of possibilities for creative admins. Just realize that you have to grant users permission to open these mailboxes before they can be accessed. By default, once you grant permission to a shared mailbox, it will auto-mount in Outlook after about an hour (you can keep it from mounting automatically by using PowerShell to grant the permission with the -automapping switch set to $false).

Shared Mailboxes feel very much like a legal gray area in Exchange Online, because even the entry level subscriptions for EoL allow them and they can be used to mimic many of the higher cost subscription features. If you feel icky about these tips, feel free to ignore them, as the legality of these uses really isn’t documented anywhere. Microsoft’s licensing tactics are notorious for being extremely complicated and confusing (I like to joke that understanding Microsoft’s licensing requires a chicken, a sacred altar, and an obsidian dagger crafted under the light of a blood moon), so take all this under advisement.

Terminated User Retention

If you are off-boarding an employee that is leaving the company for any reason, it is always a good idea to retain a copy of that user’s email for legal or transitional purposes. Most of the time, admins will access the user’s mailbox and export it to a PST for safe-keeping. This is absolutely still a possibility in EoL, but why use your own on-prem data storage to keep the email when you can convert the mailbox to a shared mailbox and have that users’ email available in the cloud for as long as you want without having to pay for it? It’s a great trick for handling data retention following an employee leaving. The EoL admin portal even makes it easy for you. Just click on the recipient and click the “Convert to shared mailbox” button. The process may take a while to finalize, since Shared Mailboxes are stored on different databases with cheaper storage than live mailboxes. Once the process is complete, however, you can either leave the mailbox as is or grant access to people who need it.

Mailbox Extension

This one is more legally questionable than terminated user retention (which seems to be perfectly acceptable, given the ease of implementation), and is entirely theoretical from a licensing standpoint, so if someone knows whether this is allowed or not, feel free to comment and I’ll remove this section. That said, it’s possible to use shared mailboxes to give a user more storage space for their mailbox.

The current limits for Exchange mailboxes are extremely generous, with 50GB for Business and E1 subscriptions, 100GB for E3 and up. Most organizations won’t use up a portion of that storage for email (especially considering the attachment limit of 50MB), but some executives and administrative staff members may break those limits, particularly in larger environments.

To add a shared mailbox as an extended storage space for a user’s mailbox, you need only create the shared mailbox in Exchange Admin > Recipients > Shared and add the necessary user as a “Delegate” with full access permissions. Instruct the user to move or copy emails to the new mailbox once it populates in Outlook, and voila. More mailbox. You can do this as many times as you feel necessary, just understand that adding mailboxes to Outlook can cause significant slowdowns once there are more than 3-4 additional mailboxes mounted.

Additional “FROM:” Addresses

One of the inherent limitations of Exchange that MS has either not been able to solve or has chosen not to solve is that each mailbox can only have a single email address assigned as the “From:” address. If you want to send email using multiple email addresses, you have to have an additional mailbox. The solution for this conundrum in Office 365 is to create a shared mailbox that has the additional email address set as the Primary SMTP address, then grant the user’s regular mailbox Send As permission on the mailbox. You can then choose whether to set up email forwarding on the shared mailbox to redirect messages to the primary mailbox (Preferred) or grant full access to the shared mailbox and mount it as a secondary.

End of Part 1

Hopefully one of these tips proves useful for you (The list is short right now, but I expect it to expand in time), and if you happen to know of a good trick, tool, or tip for other admins, let me know and I’ll add it to the list.

Office 365, ADFS, and SQL

He’s an issue I’ve just run into that there doesn’t seem to be a good answer to on the Internet. When you are building a highly available ADFS farm to enable Single Sign On for Office 365, should you use the Windows Integrated Database (WID) that comes with Windows Server or store the ADFS Configuration on a SQL server? Put simply, if you are using ADFS *only* for Office 365, there is no need to use SQL server for storing the configuration database. Here’s why.

What Does ADFS with SQL Get You?

I spent a good day or so trying to answer this question because I came in after the initial configuration of a Hybrid Exchange migration to Office 365 that was set up using SQL to store ADFS. When we went to test failover, we ran into a *mess* of problems with this configuration. So I’m going to try to explain things in a way that makes sense, rather than just telling you what Technet says in their article on the subject.

Storing your ADFS configuration in a SQL database gives you 4 things:

  1. The ability to detect and block SAML and WS-Federation Token Replay attempts.
  2. The ability to use SAML artifact resolution
  3. The ability to use SQL fail-over to increase availability of the Configuration database
  4. Scalability!!!

Using WID with ADFS prevents these features from being available to you. Here’s what each of those lines of text mean (Why no, I’m not going to just give you that without definitions like Technet does. Can you tell I’m a little grumpy about Technet’s coverage of this subject?)

SAML and WS-Federation Token Replay Attempts

This is actually a potential security vulnerability that comes about because of how Token authentication mechanisms work. In some cases, an application that accepts a federated token identity for access may allow users to “replay” a previously issued authentication Token to bypass the authentication mechanisms used to issue that Token. You can do this by ending a session with a federated application, then returning to the application with a cached version of the interaction. The easiest way to do this is to just push the Back button in a web browser. If the federated application isn’t properly designed, it will accept the cached version of the token used in the previous session and continue operating just like someone never logged out.

It should be noted that this is a serious security issue. If you have an environment where there are systems exposed to public use by multiple users (Kiosks, for example), you will want to make sure that Token Replay attempts are dropped. But here’s the thing, Token Replay is a vulnerability that affects the application you are using federated login to access. It is not a vulnerability of ADFS itself. Basically, if the applications that you are using ADFS to federate with are designed properly, there is no need to have your ADFS environment provide this kind of protection. Office 365 was designed to prevent Token Replay attempts from succeeding. So if you are only federating with Office 365, you don’t need to have this functionality in your ADFS environment. So, no need to have a SQL based Configuration Database for this reason.

SAML Artifact Resolution

I have tried for a while to find a good definition for what SAML Artifact Resolution actually is. I’m still not 100% on it, because the technical documentation for the SAML 2.0 standard is about as informative as a block of graphite. However, it seems like this is essentially a method through which both sides of a SAML token exchange can check on and authenticate one another using a single session, rather than multiple sessions. This comes in handy for situations where load balancing is in use and the application requests verification of the token provider.

The good news is that you don’t have to understand SAML Artifact Resolution. Office 365 doesn’t use it, so you don’t have to worry about having it. No SQL required for this purpose.

SQL Failover Capabilities

At first glance, having SQL server failover capabilities for your ADFS configuration database sounds like a great idea. I mean, SQL server has great failover capabilities built in. The problem is, so does ADFS. When you use WID for your ADFS Configuration database, the first server in the Farm will store a read/write enabled copy of the configuration database. Each additional ADFS server added to the farm will pull a copy of this database and store it in a read only format. So in a WID based ADFS farm, every ADFS server in the farm has a copy of the configuration database by default. ADFS Proxy servers don’t store or need access to the configuration database, so only the backend ADFS servers will benefit at all from SQL integration.

But here’s the thing, for SQL integration to work properly in a way that allows full SQL failover capabilities, you need to have a valid SQL cluster. With SQL 2008R2 and earlier, this means that you have to have at minimum 2 SQL servers installed on a version of Windows that has Cluster Services available. Cluster services requires that you have shared storage before it will create a failover cluster for SQL versions prior to 2012. If you have 2012, you can manage much simpler failover, but if you’re limited to SQL 2008R2 and earlier, you’re going to end up with a significantly higher infrastructure requirement to get SQL failover working. It you can use SQL 2012 to store the ADFS database, there is some advantage to using SQL clustering for ADFS, but the advantage doesn’t actually justify the costs of doing so.

“What about SQL Mirroring?” You ask. Well, that’s a good idea in theory. You can have your ADFS config database mirrored on two SQL servers, but you will run into some major headaches using this technique, especially if you ever need to run a failover. First off, ADFS configuration with SQL server requires that you use the SFConfig command line tool to create the config database and add servers to the farm. This is a huge pain to do, but it’s necessary. Second, if you ever have to fail over to the other SQL server for any reason, you have to reconfigure every one of the servers in the farm. SQL mirroring doesn’t utilize a clustered VIP, so the configuration with SFConfig requires you to point the server to the active SQL server. When you activate the SQL database on the other server, things may or may not work without reconfiguration. If they do work, there’s a chance that they might *stop* working at any time in the future without notice until you run SFConfig on each member of the farm and use the now active mirror’s address in the command. Then you may have to log in to each of your Proxy servers and run the Proxy Config wizard to reinitialize the proxy trust between it and the ADFS farm. That means that if you want to fail over with SQL mirroring, you have to log in to 4 servers at minimum and reconfigure each one every time the active SQL mirror changes. This is a truly unwieldy and excessive failover process. It isn’t worth the loss of expense from not using a full SQL cluster to do this for ADFS’s configuration database.

The failover abilities of SQL allow you to have constantly available access to a writable copy of the ADFS configuration database. If you use WID and the first server in the farm goes down, you cannot make changes to your ADFS configuration. No adding new servers, no modifications to the ADFS functions, etc. The good news is that the only time you need access to a writable copy of the database is when you are adding new servers to the farm or making modifications to ADFS functions. If you’re using Office 365, you almost never have to make changes to either of these once its set up and running. ADFS will continue operating as long as even one server in the farm is accessible without a writable copy of the database. So realistically, you don’t need SQL failover capabilities for ADFS’s configuration database.

Scalability!!!

Scalability is one of my favorite buzz words. Scalability means “I can add more servers to handle this workload without any problems.” And SQL allows you to do that. However, WID integrated ADFS allows you to do so as well. The limitation, though, is that you can only have up to 5 servers in a WID based ADFS farm (NOTE: Proxy servers don’t count against this maximum number). “Only five servers!?” you may ask in heated exasperation. “Yes!” I answer. “Well, I can’t stand limitations! I will use SQL!” you respond. “You will never need more than 4 ADFS servers in ever!” I retort. ADFS is one of the most light-weight roles available for Windows server. I have deployed ADFS on numerous environments with user counts well past the 10,000 mark and the ADFS servers almost never measure a blip on the performance monitors. This is because ADFS does three things, it provides a website for users to log in to, it queries Active Directory to authenticate users, and it generates a small token that contains less than 1KB of data that it then sends to a remote service. This entire process requires about 10 seconds per user per session, and that includes about 9 seconds of someone typing in their credentials. So there isn’t really even much need to have load balancing capabilities on ADFS. One server can handle monstrous loads. If you have an environment with 100,000 users accessing numerous federated applications, you might need to add 1 or 2 extra servers to your farm to handle the load, but I doubt it. You’re better off just adding RAM and processing power to your ADFS servers to improve performance than adding extra servers to the farm.

If you want multi-site high availability, you may need 5 servers, but probably not more. Let’s say you want to have site resiliency in your ADFS configuration as well as failover capability in your primary and secondary site. This configuration requires a minimum of 4 ADFS servers. Two for the primary site, two for the failover site, and 4 proxy servers that don’t count against the maximum of 5 servers. There you have it, full site-resiliency and in-site failover capability using less than 5 servers. You can even add a third site to the farm without having any issues if you want using the remaining server. But let’s be honest here, if there is an emergency that takes down two geographically dispersed datacenters at the same time, you’re going to be looking for the nearest shotgun to fight off zombies/anarchist bandits, not checking to make sure your ADFS farm is still working, so triple hot-site resiliency for ADFS is probably more than anyone needs. So you don’t need SQL for ADFS if you want scalability. It’s scalable enough without SQL.

EXPLOSIONS! I mean Conclusions

If you are planning to build an ADFS farm for Single Sign On with Office 365, you will ask the question, “Do I need SQL?” The answer to that question, in pretty much every possible case is “Not if you’re just going to use it for Office 365.” SQL integrated ADFS configurations add some features, sure, but there is almost no situation where any of those features must exist for ADFS to work with Office 365. So don’t bother even trying it out. It adds an unnecessary level of complexity, cost, and management difficulties and give no advantages whatsoever.

Office 365 Hybrid Configuration Failures

This is just a quick post that is meant to help people out who are having some issues with creating a Hybrid Configuration with Office 365 and Exchange 2010 SP3. There are some serious bumps in the road that you can come across when setting this up that may cause you to spend countless hours troubleshooting without any real success. I’ll elaborate on a couple of the problems that I’ve run into here, and follow up with the solution that worked for me with these issues at the end of the post.

AutoDiscover Failures and Free/Busy Issues

One of the things that you may run into after completing is AutoDiscover failures. You’ll know you have this problem when a cloud (or on-prem) user can log into OWA, but cannot set up their mailbox in Outlook or through Activesync. This can also present in an unusual fashion when you attempt to look up cross-premises Calendar information. Cross-premises calendar sharing utilizes the Exchange Federated Sharing features of Exchange, and this in turn utilizes Autodiscover to work properly. If you can’t view calendars in either direction (from On-Prem to Cloud or Cloud to On-Prem), and you get an error that the Free/Busy information couldn’t be read, look into Autodiscover first.

Generally, there isn’t a whole lot you can do to resolve Autodiscover errors, since Autodiscover is something that you have some pretty limited control over. Microsoft Recommends that the Autodiscover.company.com record that you publish in your Public DNS, so you shouldn’t have to change your Autodiscover record when introducing a Hybrid configuration. Unfortunately, there isn’t much more you can actually do once the Records are configured.

There is, however, a tool you can use to help you troubleshoot some issues with Autodiscover and Office 365 in a hybrid environment. Since Autodiscover is required for Free/Busy exchange to function, it may actually be possible to resolve your error by using Microsoft’s Free/Busy error troubleshooting tool. It’s available here: http://support.microsoft.com/kb/2555008

If you aren’t experiencing Free/Busy errors, the tool may not be as handy, but I suggest trying to go through it a bit anyway, since it can give you some tips for resolving Autodiscover errors. If you have on-prem users that are having trouble configuring clients with autodiscover, tell the tool you have on-prem users that can’t see free-busy for Cloud users. If you have cloud users that are having trouble, do the opposite. If neither are working, use the other option available in the tool.

What Solved My Problem

Interestingly, it took me about 2 or 3 days of digging before I finally found the solution to my autodiscover and free/busy issues. It turned out that my problems were caused by some information that Microsoft failed to let anyone know about.

When you run the Hybrid Configuration tool, it will make some major changes to each of the CAS and HUB servers that you add as Hybrid Endpoints. However, because the hybrid configuration wizard actually makes these changes remotely and on demand, it does not actually complete the setup for you. Once you complete the Hybrid Configuration Wizard and add *any* CAS or HUB servers as hybrid endpoints (All your CAS and HUB servers should be hybrid endpoints for optimum functionality), *make sure to reboot those servers*. The changes that are made by the Hybrid Configuration wizard *will not* apply fully until the World Wide Web Publishing Services and IIS services are restarted. You can achieve the same goal by running IISRESET on your CAS/HUB servers like I did if you are in a situation where rebooting will create unnecessary downtime, but a full reset is a good idea.