Hardening Microsoft Solutions from Attacks

Take a minute to go over this post from Dirk-jan Mollema. Go ahead and read it. I’ll wait…

Did you realize how scary that kind of attack is? As an IT guy who specializes in Exchange server and loves studying security, that article scared the snot out of me. Based on my experience with organizations of all sizes I can say with a good bit of authority that almost every Exchange organization out there is probably vulnerable to this attack. Why? Because Exchange is scary to a lot of people and they don’t really know how to harden it effectively. But I also want to use the above attack as a way to illustrate what I feel is the best strategy for hardening a Windows environment (and, really, any environment).

Take this opportunity to look at your Exchange deployment (if you haven’t already moved to Exchange Online) and think about what you can do to protect your environment from this type of thing. In this post, though, I want to focus on Exchange Server and Windows Server hardening techniques in general, rather than this particular vulnerability because with any hardening effort, you want to examine the network as a whole and work downward without focusing on specific vulnerabilities. If you do the opposite, you will invariably end up playing a never ending game of whack-a-mole, trying to stay ahead of a world full of malicious attackers and never really being successful.

The techniques recommended in the Center for Internet Security’s (CIS) Critical Security Controls follow the top-down approach and represent one of the best guides for approaching information security at a technical level.

IT Hardening, a Quick Intro

Hardening is essentially all actions that you take to make an environment more secure. There are many different types of hardening; server hardening, network hardening, physical hardening, procedural hardening, etc. But these all seek to do the same thing, just in different ways.

If you take a close look at the actions the CIS controls recommend, you’ll (hopefully) notice that they seek to secure as much of the environment as possible when you start at control number 1. As you go through the controls, each subsequent control has a more narrow focus. Once you get to control number 5, you will probably have an environment that will stand up against all but the most determined attacks, but you don’t necessarily want to stop there.

The most important best practice in Information Security is the idea of “Defense in Depth”. This technique involves building layers of protection instead of relying on a single security measure to protect your environment. Having a firewall in place is only one “layer” of defense, and is regarded as the broadest level of protection you can have. Anti-virus tools, Intrusion Detection/Prevention tools, and hardening techniques represent additional layers of defense. You want as many layers as you can justify when measuring cost against risk (a much more difficult topic to cover).

Focusing on Windows

One thing that you hear regularly in the IT industry is the argument about what OS people choose to handle their IT. The common argument is that Linux is a more secure OS than Windows, and this is true, up to a point. The reality is that they are simply different approaches to crafting an OS.

Linux tends to be more modular in its approach. If you implement a Linux environment, you would start with the core OS and add features as needed. This approach is good for limiting the attack surface from the start, but it also has a number of drawbacks.

The biggest drawback for Linux is that there is no centralization for support and maintenance. There are lots of different solutions to the same problem, and there isn’t really a single source of support for all solutions, so you have to either have very capable Linux support specialists or handle lots of different vendors. This usually increases the cost of ongoing maintenance and support of the infrastructure. It’s also not uncommon for different Linux-based open source projects to be abandoned for whatever reason, leaving organizations that implemented that solution without support, and once the guy who knows how to use it effectively leaves, you’re left with a very serious problem.

Windows, on the other hand, is a fairly complete package of capabilities for most situations. Windows server has built in solutions that can do most of the work you will want in an IT environment, within some limits. For instance, Windows server doesn’t handle EMail well right out of the box. You have to also implement Exchange server to have a truly effective method of handling email, but with that solution you also gain a very powerful collaboration tool that handles calendaring, contact management, task management, and other features that you can pick and choose from. Microsoft also invests a lot of time and effort in developing training tools and educational resources to ensure that there is a large pool of talent to support their OS and other software solutions. You don’t often have to worry about finding someone who knows how to manage a Windows environment. There are boatloads of MCSAs and MCSEs looking for work almost all the time.

The major drawback with Windows is, of course, security. With all of the features built in, Windows has a very large attack surface compared to Linux. However, with careful planning and implementation, the attack surface of Windows can be decreased very effectively, such that there is virtually no difference between a standard Linux deployment and a hardened Windows environment.

Hardening Windows

Going back to the vulnerability outlined in the link from the start of this article, a single change to a Windows Active Directory environment will eliminate vulnerability: LDAP signing and channel binding. LDAP signing and channel binding are techniques that are used to prevent Man In the Middle attacks from succeeding. I explain the theory behind LDAP signing in more depth in my article on Understanding Digital Certificates. LDAP channel binding is a technique that prevents clients from using portions of authentication attempt against one DC when communicating with a different DC or client. Put simply, it “binds” a client to the entire authentication attempt by requiring clients to present proof that the authentication traffic it’s sending to the server isn’t forged or copied from a different authentication attempt.

Essentially, LDAP signing configures all Active Directory Domain Controllers to that they are verifying that they are actually talking to the server they are supposed to before doing anything. Implementing this is a little difficult, though, as it requires the use of a Certificate Authority to generate and deploy digital certificates, but once digital certificates are installed on Domain Controllers and Member Servers in a Windows Domain, LDAP signing is available (once systems are configured to require it) and becomes a very effective form of security that prevents a wide swatch of attacks that can be performed to gain unauthorized access.

LDAP signing alone won’t prevent all possible attacks in a Windows environment, though, which is why it’s essential to disable features and roles that each server is not using, and taking effective care of remote access to servers. Windows Remote Desktop is one of the most frequently used tools to breach security in a Windows environment, so limiting access to it is essential. As a rule of thumb, only allow System Administrators to access critical Windows Servers and never, *never* allow remote desktop ports through your firewall.

Check your firewalls now, if you have port 3389 allowed to the Internet, it’s only a matter of time before you get attacked and suffer severe consequences. Remote Desktop is *not* meant for allowing remote workers access over the Internet. Implement secure VPNs and practice effective password security policies if you want people to access your IT environment remotely.

Once all unnecessary features and roles are removed or effectively controlled in a Windows environment, build and maintain an effective patch management strategy. Microsoft regularly deploys patches to close security holes before attackers are regularly attacking them. Any patch management plan should make allowances for testing, approving, deploying, and installing Security-related patches as soon as possible.

Next, focus on granting only permissions necessary for workers to accomplish their tasks. This is a difficult practice to implement, because it takes a lot of investigation to determine what permissions each user needs. Many environments grant Administrative permission to users on company owned equipment, which is a horrible, lazy practice that will get your environment owned by a hacker very quickly.

Once you have all of the above security practices in place, you will then want to start focusing on more specific vulnerabilities. As an example method for preventing the attack in the link at the start of this post, changing a simple registry setting will block the attack. But it will not prevent future attacks that may attack vulnerabilities that aren’t well known.

How Does the Cloud Play Into This?

One of the major benefits of using cloud solutions like Exchange Online is that most of the work outlined above has been done already. Microsoft’s cloud servers are stored in highly secure datacenters with many protections against unauthorized access (as opposed to the common tactic of putting the server in a closet in your office). Servers in cloud environments are hardened as much as possible before being put into operation. Security vulnerabilities are usually addressed across the entire cloud environment within hours of discovery, and the servers don’t function with an eye to backwards compatibility, so things like NTLM and SMBv1 are disabled on all systems.

That said, the cloud poses its own security challenges. You must accept the level of security put in place by the cloud provider and will have little to no control over systems in a way that will let you increase security. Furthermore, utilizing a Hybrid-cloud solution (which is extremely common and will be for years to come) presents unique problems involving the interface between two separately controlled environments. Poor security practices in the on-prem side of a hybrid deployment will make the cloud side just as insecure.

You must accept public availability of your data and accept the reality that you don’t control where that data is (for the most part…this issue is slowly changing as cloud environments mature). In addition, your do not offload the responsibility of securing access to the data you store in the cloud. I’ll cover this subject in another post, but for now, understand that while cloud environments build a lot of security into their solutions, you still have a responsibility to make security a priority.

Conclusion (I never can think of a good heading here)

Security in any IT environment is a major challenge that takes careful planning and effective management. Failing to consider security challenges when deploying new solutions will almost always come back to bite you. But, with the right strategy and guidance, it *is* possible to build a secure environment that can withstand the vast majority of attacks.




Enabling Message Encryption in Office 365

As I mentioned in an earlier post, email encryption is a sticky thing. In a perfect world, everyone would have Opportunistic TLS enabled and all mail traffic would be automatically encrypted with STARTTLS encryption, which is a fantastic method of ensuring security of messages “in transit”. But some messages need to be encrypted “at rest” due to security policies or regulations. Unfortunately, researchers have recently discovered some key vulnerabilities in the S/MIME and OpenPGP. These encryption systems have been the most common ways of ensuring message encryption for messages while they are sitting in storage. The EFAIL vulnerabilities allow HTTP formatted messages to be exposed in cleartext by attacking a few weaknesses.

Luckily, Office 365 subscribers can improve the confidentiality of their email by implementing a feature that is already available to all E3 and higher subscriptions or by purchasing licenses for Azure Information Protection and assigning them to users that plan to send messages with confidential information in them. The following is a short How-To on enabling the O365 Message Encryption (OME) system and setting up rules to encrypt messages.

The Steps

To enable and configure OME for secure message delivery, the following steps are necessary:

  1. Subscribe to Azure Information Protection
  2. Activate OME
  3. Create Rules to Encrypt Messages

Details are below.

Subscribe to Azure Information Protection

The Azure Information Protection suite is an add-on subscription for Office 365 that will allow end users to perform a number of very useful functions with their email. It also integrates with SharePoint and OneDrive to act as a Data Loss Prevention tool. With AIP, users can flag messages or files so that they cannot be copied, forwarded, deleted, or a range of other common actions. For email, all messages that have specific classification flags or that meet specific requirements are encrypted and packaged into a locked HTML file that is sent to the recipient as an attachment. When the recipient receives the message, they have to register with Azure to be assigned a key to open the email. The key is tied to their email address and once registered the user can then open the HTML attachment and any future attachments without having to log in to anything.

Again, if you have E3 or higher subscriptions assigned to your users, they don’t need to also have AIP as well. However, each user that will be sending messages with confidential information in them will need either an AIP license or an E3/E5 license to do so. To subscribe to AIP, perform these steps:

  1. Open the Admin portal for Office 365
  2. Go to the Subscriptions list
  3. Click on “Add a Subscription” in the upper right corner
  4. Scroll down to find the Azure Information Protection
  5. Click the Buy Now option and follow the prompts or select the “Start Free Trial” option to get 25 licenses for 30 days to try it out before purchasing
  6. Wait about an hour for the service to be provisioned on your O365 tenant

Once provisioned, you can then move on to the next step in the process.

Activate OME

This part has changed very recently. Prior to early 2018, Activating OME took a lot of Powershell work and waiting for it to function properly. MS changed the method for activating OME to streamline the process and make it easier to work with. Here’s what you have to do:

  1. Open the Settings option in the Admin Portal
  2. Select Services & Add-ins
  3. Find Azure Information Protection in the list of services and click on it
  4. Click the link that says, “Manage Microsoft Azure Information Protection settings” to open a new window
  5. Click on the Activate button under “Rights Management is not activated”
  6. Click Activate in the Window that pops up

Once this is done, you will be able to use AIP’s Client application to tag messages for right’s management in Outlook. There will also be new buttons and options in Outlook Web App that will allow you to encrypt messages. However, the simplest method for encrypting messages is to use an Exchange Online Transport Rule to automatically encrypt messages.

Create Rules to Encrypt Messages

Once OME is activated, you’ll be able to encrypt messages using just the built in, default Rights Management tools, but as I mentioned, it’s much easier to use specific criteria to do the encryption automatically. Follow these stpes:

  1. Open the Exchange Online Admin Portal
  2. Go to Mail Flow
  3. Select Rules
  4. Click on the + and select “Add a New Rule”
  5. In the window that appears, click “More Options” to switch to the advanced rule system
  6. The rule you use can be anything from Encrypting messages flagged as Confidential to using a tag in the subject line. My personal preference is to use subject/body tags. Make your rule look like the below image to use this technique:Encrypt Rule

When set up properly, the end user will receive a message telling them that they have received a secure message. The email will have an HTML file attached that they can open up. They’ll need to register, but once registered they’ll be able to read the email without any other steps required and it will be protected from outside view.



Do I need Anonymous Relay?


If you have managed an Exchange server in the past, you’ve probably been required to set things up to allow printers, applications, and other devices the ability to send email through the Exchange server. Most often, the solution to this request is to configure an Anonymous Open Relay connector. The first article I ever wrote on this blog was on that very subject: http://wp.me/pUCB5-b .  If you need to know what a Relay is, go read that blog.

What people don’t always do, though, is consider the question of whether or not they need an anonymous relay in Exchange. I didn’t really cover that subject in my first article, so I’ll cover it here.

When you Need an Open Relay

There are three factors that determine whether an organization needs an Open Relay. Anonymous relay is only required if you meet all three of the factors. Any other combination can be worked around without using anonymous relaying. I’ll explain how later, but for now, here are the three factors you need to meet:

  1. Printers, Scanners, and Applications don’t support changes to the SMTP port used.
  2. Printers, Scanners, and Applications don’t support SMTP Authentication.
  3. Your system needs to send mail to email addresses that don’t exist in your mail environment (That is to say, your system sends mail to email addresses that you don’t manage with your own mail server).

At this point, I feel it important to point out that Anonymous relays are inherently insecure. You can make them more secure by limiting access, but using an anonymous relay will always place a technical solution in the environment that is designed specifically to circumvent normal security measures. In other words, do so at your own informed risk, and only when it’s absolutely required.

The First Factor

If the system you want to send SMTP messages doesn’t allow you to send email over a port other than 25, you will need to have an open relay if the messages the system sends are addressed to email addresses outside your environment. The bold stuff there is an important distinction. The SMTP protocol defines port 25 as the “default” port for mail exchange, and that’s the port that every email server uses to receive email from all other systems, which means that, based on modern security concerns, sending mail to port 25 is only allowed if the recipient of the email you send exists on the mail server. So if you are using the abc.com mail server to send messages to bob@xyz.com, you will need to use a relay server to do it, or the mail will be rejected because relay is (hopefully) not allowed.

The Second Factor

If your system doesn’t allow you to specify a username and password in the SMTP configuration it has, then you will have to send messages Anonymously. For our purposes, an “anonymous” user is a user that hasn’t logged in with a username and password. SMTP servers usually talk to one another Anonymously, so it’s actually common for anonymous SMTP access to be valid and is actually necessary for mail exchange to function, but SMTP servers will, by default, only accept messages that are destined for email addresses that they manage. So if abc.com receives a message destined for bob@abc.com, it will accept it. However, abc.com will reject messages to jim@xyz.com, *unless* the SMTP session is Authenticated. In other words, if bob@abc.com wants to send jim @xyz.com a message, he can open an SMTP session with the abc.com mail server, enter his username and password, and send the message. If he does that, the SMTP server will accept the message, then contact the xyz.com mail server and deliver it. The abc.com mail server doesn’t need to have a username and password to do this, because the xyz.com mail server knows who jim@xyz.com is, so it just accepts the message and delivers it to the correct mailbox. So if you are able to set a username and password with the system you need to send mail with, you don’t need anonymous relay.

The Third Factor

Most of the time, applications and devices will only need to send messages to people who have mailboxes in your environment, but there are plenty of occasions where applications or devices that send email out need to be able to send mail to people *outside* the environment. If you don’t need to send to “external recipients” as these users are called, you can use the Direct Send method outlined in the solutions below.


As promised, here are the solutions you can use *other* than anonymous relay to meet the needs of your application if it doesn’t meet *all three* of the deciding factors.

Authenticated Relay (Factor #3 applies)

In Exchange server, there is a default “Receive Connector” that accepts all messages sent by Authenticated users on port 587, so if your system allows you to set a username and password and change the port, you don’t need anonymous relaying. Just configure the system to use your Exchange Hub Transport server (or CAS in 2013) on port 587, and it should work fine, even if your requirements meet the last deciding factor of sending mail to external recipients.

Direct Send (Factor #2 applies and/or #3 doesn’t apply)

If your system needs to send messages to abc.com users using the abc.com mail server, you don’t need to relay or authenticate. Just configure your system to send mail directly to the mail server. The “direct send” method uses SMTP as if it were a mail server talking to another mail server, so it works without additional work. Just note that if you have a spam filter that enforces SPF or blocks messages from addresses in your environment to addresses in your environment, it’s likely these messages will get blocked, so make allowances as needed.

Authenticated Mail on Port 25 (Only factor #1 applies)

If the system doesn’t allow you to change the port number your system uses, but does allow you to authenticate, you can make a small change to Exchange to allow the system to work. This is done by opening the Default Receive connector (AKA – the Default Front End receive connector on Exchange 2013 and later) and adding Exchange Users to the Permission settings on the Security tab as shown with the red X below:


Once this setting is changed, restart the Transport service on the server and you can then perform authenticated relaying on port 25.


If you do find you need to use an anonymous relay, by all means, do so with careful consideration, but always be conscious of the fact that it isn’t always necessary. As always, comments questions on this article and others are always welcome and I’ll do my best to answer as soon as possible.

How Will the Cloud Affect My Career as an IT Professional?

Well, after a year’s hiatus due to some particularly difficult personal trials, I’ve decided to come back to the block and weigh in on one of the big hot-button subjects in the IT industry – How the cloud will affect the job market.

The Push to Cloud

In the modern world, as the Internet has developed and increased in prominence in our lives, the increased infrastructure, security technology, and bandwidth is beginning to allow businesses and individuals to forgo the traditional need to pay big bucks for things like processing power and storage. Companies have been moving their critical systems into third party data-centers for years, but with the development of entirely cloud based solutions like Office 365, Azure, Google Apps, and AWS we’re starting to see a large industry push to reduce infrastructure costs by moving away from self-managed IT solutions. So now we’re starting to see another paradigm shift in the IT industry.

Now, this is not to say that IT hasn’t seen any kind of paradigm shift before, quite the contrary. It seems like every year we’re having to face some new technology that is permeating the industry. From the introduction of Ethernet, to wireless networking, to virtualization and VDI, most of us have dealt with the changes as they’ve come, learning new techniques and adjusting the way we work. But the push to cloud has a lot of IT personnel worried about their jobs.

What About my Job!?

Cost savings has been the primary driver behind the recent push to adopt cloud services. Executives around the world are salivating at the possibility of reducing their costs by shifting the responsibility of IT infrastructure management onto third parties. This shifting of responsibility has a lot of IT people on edge, knowing that if the stuff they do every day is outsourced to a third party service provider, what will happen to their job? If we have no servers or networks to maintain, am I going to lose my job?

The answer here is actually pretty simple. Unless you’re a part of some specific niche industry jobs, you’re pretty likely to keep your job.

Working IT in the Age of the Cloud

While moving to the cloud does reduce the need for critical infrastructure and complex solutions, it doesn’t really reduce the need for administration, problem solving, end-user support, and technical know-how. After performing numerous migrations from various email systems to Office 365, the one thing I’ve discovered is that moving to the cloud doesn’t really make the IT guy’s job any easier. In many ways, it actually makes things more difficult and complex, which means that if you’re a competent IT professional your job is pretty safe.

How will the Cloud Change My Job?

Now, that isn’t to say that your job isn’t going to change. Moving to the cloud requires a good deal of adaptation and adjustment to new ways of thinking and managing resources. For instance, if you want to have any kind of Active Directory integration with Office 365, you’re going to have to use Dirsync, and using Dirsync means you can’t modify things like distribution groups, user accounts, and passwords in Office 365. These things have to be managed in Active Directory, and that means that all of those really menial tasks you’ve been handing off to department heads, like adding distribution group members, are going to land right back into your wheelhouse if you’re as nervous as I am about giving people with no IT experience access to the Active Directory Users and Computers snap-in. For things like password changes, be prepared to face a massive influx of support calls asking for help resetting passwords as well.

In addition to the technical limitations involved with moving to the cloud, you also have to deal with the fact that you lose a lot of direct control over the IT infrastructure. Since your resources are now located on a system owned and operated by someone else, if things break you have to go to the vendor to get it fixed, and that brings up any number of frustrations, depending on who you work with. If you’ve ever spent any time on the phone with Microsoft Support, you’re likely to dread any interaction with them from that point on. You won’t have to do much of the work involved in fixing the problem, but you will have to sit around twiddling your thumbs while someone else does, and that can be a little maddening at times. This does, of course, depend on how competent the person on the other end of the phone is. Sometimes you get lucky and find someone who knows their stuff. Sadly, that’s more of an exception than a rule, so you may need to brush up on your people skills a bit and learn how to light a fire under the support technician on the phone with you.

One Step Forward, Two Steps Back

As more services start moving to the cloud, we’ll probably see a reduction of administrative overhead, but for now cloud solutions will feel like a major step backward to a lot of people, and it really is a step backward. You see, in the old days, most businesses that made investments in IT infrastructure would utilize systems called Mainframes. The mainframe system would perform all of the actual processing work (and was as big as a house in some cases), and people who used computers would interface with the mainframe from a system that was directly connected to it. Sound a lot like a Virtual Desktop Environment? It’s a lot like how cloud services work, too, except that with the Cloud, we replace the centralized Mainframe system with a vast, globe spanning collection of servers. It’d be like if one company owned a single mainframe that was rented out for numerous companies to use at once. As a result, we have to rethink the way we work. Luckily, this does mean fewer trips to the data-center to reboot servers or move cables around, which is a major plus for some people (not me, though. Data-centers relax me for some reason. I’m not sure if it’s the steady humming sound or 50 degree temperatures).

Niche Workers Beware!

With all of this said, there are a couple jobs that are going to start disappearing in the next few years. If you happen to work entirely in one of these areas, you should seriously consider branching out or you may soon find yourself without a reason to work.

1. Backup Operations – This is one niche where the writing has been on the wall for a while now. Companies have been moving toward high-availability solutions for some time now, which means that the need to spend copious funds on backup solutions and storage has been falling. High availability solutions generally rely on having multiple copies of critical data on multiple servers, so the loss of a single server no longer puts people into panic mode. With the cloud, data is placed in systems with so much redundancy, with such a high level of integrity, that data loss is extremely uncommon, and unrecoverable data loss is nearly impossible for some cloud solutions. So if you’re a backup operator and that’s all you’ve done for years, you might find your job under the cross-hairs. It’s time to start expanding your repertoire.

2. Hardware maintenance – To me, it should be obvious that computer hardware specialists are going to see less work with the move to cloud, since there is a definite drop in the amount of server and network hardware required when your company is running there, but I figured I should at least mention it.

3. Internal Network Administration – This particular job won’t ever go away, but with cloud solutions we may see a definite drop in the complexity and overhead required for running a LAN, and the concept of the WAN may begin to disappear as satellite offices will only require an Internet connection to access company resources located in the Cloud.

4. IT Infrastructure Design Specialists – Since the cloud consists almost entirely of prepackaged solutions, the demand for complex architectural designs will start to disappear, meaning that people who make their living designing and implementing IT infrastructure solutions are going to have a lot less work to do. This is the one that makes me sad, since I really enjoy Infrastructure Design. As the cloud push progresses, Infrastructure design will change from designing solutions to developing solutions for managing and interfacing with cloud-based services (which is not nearly as much fun).

5. SAN management – The concept of the Storage Area Network isn’t really even into adolescence yet and here we’re moving away from it? Well, yes, pretty much. As the cloud sees greater levels of adoption, the need for people who focus on the management, provisioning, and optimization of centralized data storage are probably going to see less work.

Now, I’m not saying that these niche jobs will disappear entirely. Nor is this a comprehensive list of jobs that the cloud will be making less important. There will always be companies who avoid the Cloud like the plague, and they’re going to need people who know their stuff. But what I will say is that if you focus in these areas alone, be prepared to branch out or you’re probably going to start spending your days working for a cloud service provider like Microsoft, which might not be as much fun as working for a (much) smaller company.

Adapt or Die

In the end, the cloud will simply force IT professionals to do what most of us do best, adapt to changes in our surroundings. We’ll need to change the way we think and interact in our jobs to be successful in our careers, but we should still have careers despite the changes to the IT landscape.

Should I Switch to Office 365? A Frank Examination

The Cloud – An Explanation

As time moves on, technology moves with it, and times they are a’changin’! There have been many drastic changes in the world of IT over the years, but the most recent change, the move toward cloud computing, is probably one of the most drastic and industry redefining change to occur since the release of HTTP in the early 1990s.

Cloud computing is, put simply, placing your IT infrastructure into the hands of a third party, and it’s becoming big news for companies like Microsoft, Amazon, and Apple, who are working hard to push the IT world into the Cloud so they can take advantage of the recurring income model that their Cloud systems are built around.

A more complex explanation of Cloud Computing almost always requires a metaphor or analogy of some kind, so here’s mine; Cake. I love cake. Everyone loves cake. If you don’t love cake, you’re crazy and you should give your cake to me. But there is a problem with cake. If you live alone, you can never really have cake, because in order to have cake you usually have to buy or make a whole cake, and if you have a whole cake to yourself, you will very quickly regret having purchased a whole cake for yourself as you roll yourself out of bed and out the door each morning on your gigantic rolls of fat. Then you’ll have a heart attack. Cake is meant to be shared. One cake is enough to allow 12 people (or more, or less, depending on how much they love cake) to have a comfortable amount of cake. This is good for everyone because they all get cake and aren’t fat because of it. So, how is the Cloud like cake? Well, now that computing technology has advanced to the point where almost every computer system provides more power than is necessary for most tasks in the corporate environment, buying and building a complex IT infrastructure that meets all the needs of a specific company can be extremely expensive, and it’s highly likely that much of that infrastructure is going to be wasted because it is more than is needed.

Virtualization was the first step to addressing the concerns of excessive computing power. It allowed IT departments to combine multiple server roles, securely segregated, on a single physical server. Prior to the advent of Virtualization, secure segregation of server roles required numerous physical servers, which in turn required a lot of space, power, and resources to maintain. Virtualization started shrinking the corporate Datacenters of the world, and the concept of Cloud computing seeks primarily to not only shrink the corporate datacenter, but to centralize it

Cloud computing, like a giant cake, allows multiple corporate environments to share a single, gigantic infrastructure system. Rather than each company having their own segregated and wholly owned infrastructure that is managed, configured, and maintained separately, Cloud solutions like Office 365 seek to provide all the services of a highly integrated and functional environment without requiring a fully dedicated infrastructure for each company that needs or wants that type of functionality. This is accomplished through the use of highly customized versions of products that are normally available to individual corporations for a one time investment in a packaged, much smaller recurring fee system. The cloud provider builds, maintains, and supports the Infrastructure and the cloud user makes use of the system just like it was owned by them. In the case of Office 365, every individual or company that uses the solution is effectively lumped into the same infrastructure system and uses the portion of that system that they need, rather than using a portion of their own system and wasting the portion they don’t use. This is handy because one of the worst things in the world is stale cake. I mean unused IT resources.

Getting to it – What Does Office 365 Offer

Office 365, being Microsoft’s Cloud solutions environment, provides most of the IT services that companies depend on Microsoft for. Specifically, Email and Calendaring (Through Exchange), Collaboration (Through Sharepoint and Exchange), Instant Messaging (Through Lync), and centralized file management and storage through OneDrive (Formally SkyDrive, formally something else. Microsoft changes the names of stuff every week it seems, so we’ll just call this cloud storage). If you wanted to compare the Infrastructure requirements you would need to meet what Office 365 provides for its monthly per-user fee (or a la cart if you don’t want all the services together) you would need the following things in your environment:

Exchange Online:

  • At minimum 3 Exchange 2013 servers configured with DAG
  • A hardware load balancer
  • A high-speed Fiber-channel SAN with about 55GB of storage per user (For normal Mailboxes) and an additional 5GB for each resource mailbox (Rooms, equipment, shared calendars, etc.).
  • An infinitely expanding low speed SAN (For archive mailboxes, for E3 licenses and above)
  • A secure email delivery solution to provide email stubbing (for E3 licenses and above)
  • Spam filtering services or a spam appliance

Sharepoint Online:

  • At minimum 2 Sharepoint 2013 Servers
  • Additional high-speed Fiber-Channel space, up to 50GB per user (again…this is space for OneDrive/SkyDrive/whatever they call it next)
  • More Load Balancing

Lync Online:

  • At minimum 3 Lync 2013 servers

Generic Software Infrastructure

  • Several Windows 2008/2012 Licenses
  • Active Directory (2 DCs minimum)
  • Multi-Tiered SAN infrastructure (with multi-site geo-replication capabilities)
  • A Load Balancer
  • Highly secured Firewalls


  • Office Professional Plus license for each user

Physical Requirements

  • Multiple Physical locations spread across numerous geographical regions.
  • Each Physical Location should have Concrete security walls, entry barriers, full time security staff, man-traps, and multi-factor authentication before admittance

And that’s just for basic service. You would also need several employees dedicated to maintaining the infrastructure and supporting the environment, since it is very difficult to find individual IT personnel who have the skill set or mental constitution necessary to manage such an environment.

In other words, if you were to build the infrastructure necessary to provide the same functionality and level of service available with Office 365, you would need to spend several hundred thousand dollars in hardware, software, and manpower. This also does not take into account Architecting and development costs for setting up the environment, which is typically done by third party companies or contractors.

Normally, individual companies would need to spend this kind of money every time they upgrade their infrastructure to keep up with new features and changes in technology. But with Office 365, upgrades, patching and new services/features are released regularly, with no need to manage a patching system. All in all, using Office 365 can represent a significant cost savings for companies that need high availability, accessibility, scalability, and security in their IT infrastructure.

Why Wouldn’t I Use Office 365

If it costs a whole lot less and provides a lot of great features, why would you not want to use Office 365? The answer here is that cloud solutions are designed to meet the needs of the average environment, not every environment. As the cloud begins to mature (it’s really just an infant right now, so don’t be surprised if it tosses Cheerios across the room every now and then) it will become much more customizable, but for now, there is very little customization that can happen with Office 365. For instance, any Line of Business application you have in your environment that must me installed on an Exchange Server is not usable. Many software providers that require this type of functionality are moving their focus to cloud based solutions now, but things like Blackberry Enterprise Server will not work with Office 365. As I mentioned, most companies are building systems that integrate with Office 365, for instance Research in Motion has teamed with Microsoft to build in support for BES features into Office 365. This support has to be activated, though. But there are several things that simply can’t be done with Office 365. I’ll try to provide some of the limitations here. For a more detailed explanation of what Office 365 *can* do, check out the official service descriptions available from Microsoft.

  1. Microsoft limits email restorations to 14 days. If an email is deleted from a Mailbox, the Deleted Items folder, and then purged from the hidden recoverable items folder, you will only have the ability to recover that email for 14 days after being fully purged. Within the 14 day window, a support request to Microsoft is required to recover the email. Outside that Window, there is no possible way to recover it. I should mention that this is a technical limitation of the Exchange Online service. Exchange Online utilizes a 14 day lagged, 3+ copy DAG configuration. This configuration allows Microsoft to use Circular Logging on their databases to reduce resource usage, and allows an extremely high level of availability. However, the maximum amount of time that any DAG member can be lagged is 14 days. Once a full email purge is fully committed to the lagged database copy, there is no way to recover it.
  2. Office 365 does not allow unauthenticated email relaying. In order to send email to Office 365, you *must* authenticate with a licensed user account. If you have Line of Business applications or equipment that doesn’t support unauthenticated email, consider upgrading your version to one that supports authenticated relaying. If this is not possible, it is necessary to utilize an SMTP server in your network that supports unauthenticated relay, such as PostFix or the IIS based SMTP server that is included with Windows Server.
  3. Exchange Online has strict limits on the amount of email that each mailbox can send. This is to prevent spamming from Office 365’s mail servers and reduce resource overhead. Each mailbox is allowed to send to at most 10,000 recipients per day (a recipient is considered to be a single email address listed in the To:, CC:, or BCC: fields of an email, so a distribution list can be used to count as a single recipient). In addition, each email can be sent to up to 500 recipients and each mailbox can only send up to 30 messages per minute. Microsoft will not increase these limits even if you ask. If you have a business need to exceed these limitations, consider using a cloud based mass-emailing service like MailChimp.
  4. Many of the administrative capabilities and controls that are available with an On-Premise Exchange, SharePoint, or Lync environment are not exposed to Office 365 tenant administrators. If there is a control that you would like to enable or use and you can’t find it in the Admin Portal, you may be able to do it in the Remote Powershell sessions provided by Microsoft. Even with Powershell, there is only so much you can do. There are 4 different Modules for Managing Office 365 in Powershell, Exchange Online (with some additional things), Lync Online, Windows Azure AD, and SharePoint Online. As a general rule, though, administrative settings that control server level or organizational level functions will not be available to you. If you have a business need to change a high level configuration, you must prepare and submit a Support Request to Microsoft through the Office 365 Admin Portal. This is a fairly simple process, but support requests can take a significant amount of time to complete, so be prepared to wait for your changes to apply.
  5. If it breaks, there’s nothing you can do to fix it in most situations. Unless you are using ADFS and Dirsync in your environment, bringing Office 365 services back online if something fails is completely out of your wheelhouse. If you do use ADFS or Dirsync, the only thing you can really troubleshoot and fix is Active Directory Object syncing or Login issues. Everything else false under Microsoft’s SLAs and is their responsibility to fix. Microsoft guarantees a minimum service up-time of 99.9% (or 43 minutes acceptable downtime per month). The SLA documents provide exact details on service credits granted for falling below that level, but if your IT management has decided that a higher level of up-time is required, Office 365 may not be a good solution for your environment.

There are a number of other limitations to Office 365’s service, so many, in fact, that it would be difficult to outline them all with a small book. Because of that, Microsoft (and I) typically recommend a short Proof of Concept (PoC) period before migrating to the service. A PoC will highlight errors in your system configuration that will negatively impact interoperability with Office 365 and make sure that your environment and business needs can be completely met with Office 365.

I Work in IT – How Will This Impact My Job

One of the greatest causes of push-back from moving to the Cloud comes from IT staff. Most IT people are justifiably twitchy when it comes to keeping their jobs. There are a lot of people competing for IT jobs and one of the major selling points of Cloud Services is decreased employment costs. Add that to the fact that the first group to get targeted for layoffs during a recession is the IT department and questions about how this will impact IT workers becomes a greatly valid question. The truth is that while movements to the cloud will decrease the need for dedicated IT staff in most companies, it also increases the need for IT staff at datacenters and consulting firms. Skilled IT people are in short supply, so keeping up with the technology trends and times is very important. Learning about the cloud and understanding it will keep you from being unemployed for extended periods (I speak from experience on this, I promise).

That said, large environments that maintain IT staff will still need to keep a significant portion of their IT workers even if they move to the cloud. Microsoft and other cloud solutions do not provide End User Support so there will always be a need for Help Desk and on-site support staff. In addition, companies that migrate to the cloud will continue to need IT staff that can interface with Cloud Services Support Staff. A single support request with Microsoft should convince most of the difficulty that exists in managing support requests and maintaining lines of communication during outages and system failures. There will also still be a need for Systems and Network Administrators (In fact, with Cloud services Network Administrators may be in even higher demand as Internet Connection Up-time becomes more important). In all reality, on-site IT staff will still be very necessary, but the nature of the job will begin to change as more cloud services become available. Instead of fighting fires and panicking about system failures and inefficiencies, IT staff can focus on developing processes and making non-cloud based services work better. Cloud services make the typical IT employee’s job easier, not less necessary.

What is Available in the Cloud Besides Office 365

The primary principal of Cloud Computing is that the equipment that runs your infrastructure no longer exists in your physical locations. There are a lot of ways this is accomplished. Terms like Private Cloud are bandied about by Marketing teams with wild abandon without a really concrete definition. Ask three different salesmen what a Private Cloud is and you’ll probably get three completely different answers (Or some really blank stares. Salesmen. *eyeroll* Am I right?). But essentially, you’re in the cloud if you have to have a Public Internet connection to reach your resources. If you have a dedicated MPLS-like connection to a datacenter and someone else manages, maintains, and updates those systems for you, this is not operating in the Cloud. Some people will call this a Private Cloud, but the real term used to define this type of relationship is Managed Services (since that’s what that kind of relationship was called well before the term Private Cloud was coined).

At any rate, cloud services can range from Solutions like Office 365 and Google Apps for Business to things like Drop-Box and Imgur. For IT purposes, some additional services that may be useful include Microsoft’s Infrastructure as a Service (IaaS) Azure solution. Azure allows you to create entirely cloud based Virtual Machines in Microsoft’s datacenters that can be acessed from anywhere. Amazon’s AWS provides a number of services for Web Based businesses to perform necessary functions. Google’s Apps for Business provides similar functionality to Office 365 but with (in my opinion) less polish.

Other Considerations

Migrating to the cloud is a difficult decision to make. There are pros and cons just like any other business decision. Take some time to understand what is involved in moving to the cloud and make sure to plan for life after moving if you decide to do so. One of the best recommendations I can make for people considering a move like this is to contact a company that specializes in cloud services (Like Business and Decision North America, the company I work for. How’s that for shameless plugs?). Aside from being able to explain things in greater detail and help plan for moving to the cloud, using a Microsoft Partner to assist in your move will open up options for quick escalations and better communication with Microsoft’s Support teams in the event of problems. Companies that do not have a partner to assist them must deal with Microsoft’s Office 365 support teams themselves, and this can take up to 2 days to receive the first response , depending on their workload and day of the week (never make a support case at 4PM on Friday. Just a friendly tip). If you work with a Microsoft Partner, this response time can be decreased to the amount of time it takes for someone in Redmond to get their butt kicked (metaphorically speaking). All in all, the world is just now beginning to move into the cloud, so now is the time to begin preparing for the inevitable.

Office 365 Hybrid Configuration Failures

This is just a quick post that is meant to help people out who are having some issues with creating a Hybrid Configuration with Office 365 and Exchange 2010 SP3. There are some serious bumps in the road that you can come across when setting this up that may cause you to spend countless hours troubleshooting without any real success. I’ll elaborate on a couple of the problems that I’ve run into here, and follow up with the solution that worked for me with these issues at the end of the post.

AutoDiscover Failures and Free/Busy Issues

One of the things that you may run into after completing is AutoDiscover failures. You’ll know you have this problem when a cloud (or on-prem) user can log into OWA, but cannot set up their mailbox in Outlook or through Activesync. This can also present in an unusual fashion when you attempt to look up cross-premises Calendar information. Cross-premises calendar sharing utilizes the Exchange Federated Sharing features of Exchange, and this in turn utilizes Autodiscover to work properly. If you can’t view calendars in either direction (from On-Prem to Cloud or Cloud to On-Prem), and you get an error that the Free/Busy information couldn’t be read, look into Autodiscover first.

Generally, there isn’t a whole lot you can do to resolve Autodiscover errors, since Autodiscover is something that you have some pretty limited control over. Microsoft Recommends that the Autodiscover.company.com record that you publish in your Public DNS, so you shouldn’t have to change your Autodiscover record when introducing a Hybrid configuration. Unfortunately, there isn’t much more you can actually do once the Records are configured.

There is, however, a tool you can use to help you troubleshoot some issues with Autodiscover and Office 365 in a hybrid environment. Since Autodiscover is required for Free/Busy exchange to function, it may actually be possible to resolve your error by using Microsoft’s Free/Busy error troubleshooting tool. It’s available here: http://support.microsoft.com/kb/2555008

If you aren’t experiencing Free/Busy errors, the tool may not be as handy, but I suggest trying to go through it a bit anyway, since it can give you some tips for resolving Autodiscover errors. If you have on-prem users that are having trouble configuring clients with autodiscover, tell the tool you have on-prem users that can’t see free-busy for Cloud users. If you have cloud users that are having trouble, do the opposite. If neither are working, use the other option available in the tool.

What Solved My Problem

Interestingly, it took me about 2 or 3 days of digging before I finally found the solution to my autodiscover and free/busy issues. It turned out that my problems were caused by some information that Microsoft failed to let anyone know about.

When you run the Hybrid Configuration tool, it will make some major changes to each of the CAS and HUB servers that you add as Hybrid Endpoints. However, because the hybrid configuration wizard actually makes these changes remotely and on demand, it does not actually complete the setup for you. Once you complete the Hybrid Configuration Wizard and add *any* CAS or HUB servers as hybrid endpoints (All your CAS and HUB servers should be hybrid endpoints for optimum functionality), *make sure to reboot those servers*. The changes that are made by the Hybrid Configuration wizard *will not* apply fully until the World Wide Web Publishing Services and IIS services are restarted. You can achieve the same goal by running IISRESET on your CAS/HUB servers like I did if you are in a situation where rebooting will create unnecessary downtime, but a full reset is a good idea.