One of the things in IT Security that took me a while to figure out was the subject of Password management. There were a few pieces of it that confused me for a while. I knew how to create “secure” passwords, but I didn’t really understand what made them secure or how someone could crack a password. It took a while, but I finally figured it out, and I thought I’d pass on the knowledge to keep others from having to bang their head against this subject.
Important Concepts
First, there are a few things you need to learn about before you can really understand passwords and how they work. I’ll outline them here.
Hashing
Probably the most important concept to understand with passwords is that of “Hashing.” This does not mean cooking diced potatoes. In IT Security, Hashing refers to the process of passing data through a specific mathematical algorithm that cannot be reversed to hide or obscure the data. Data that is “hashed” cannot be discovered easily without knowledge of what was originally passed through the algorithm. If you pass the word “hashbrowns” through a hashing algorithm (there are a number of these, and I’ll go over a couple later), you’d get a string of letters and numbers that represent that word, also known as a fingerprint.
Hashing is pretty important in IT, because we use it for a number of purposes. The most common purpose for hashing is Password Validation, but it is also used in forensic investigations to provide legal documentation and proof that the contents of a seized or otherwise legally obtained data source are not changed during the investigative process. This is done by generating a hash prior to and following the investigation. If the hashes match up, then the investigator made no changes to the data and it can be verified as authentic and valid evidence that has not been tampered with.
In perfectland, that string of numbers and letters generated by the hashing example earlier would represent *only* the word “hashbrowns.” Sadly, we don’t live in perfectland, where everything works the way it should, and some hashing algorithms are not capable of producing enough unique strings of numbers and letters to make certain that each string represents only one set of data. When you have two sets of data that result in the same output, we refer to that as a “Collision.” Collisions are an important weakness in Hashing algorithms, and have caused the death of some algorithms. Particularly, the MD5 hashing algorithm was discovered to have a weakness in it that caused it to produce identical hashes. I’ll explain why this is bad later on in this post. Aside from collisions, there is only one real way to break a hashing algorithm; Brute Force.
Brute Force Attacks
A brute force attack is a little different in IT than it would be in a real world. If you came across a locked door in the real world, you could just use brute force to break down the door. In IT, you can’t really break down the door, but you can certainly try every possible key to the door and hope you get lucky. In a brute force password cracking attack, you attempt every possible password until you get the right one. Most scripts that do this will attempt several “well known” common passwords, and then move on to trying every possible other combination.
Modern security techniques have made brute force password cracking attempts realistically impossible. The most effective security techniques include account lockout and password attempt windows. Account lockouts allow administrators to block user accounts from accessing anything either until a set amount of time passes or until the user directly contacts an administrator to have their account unlocked. Both techniques effectively combat normal brute force attacks by significantly increasing the amount of time it takes for a brute force attack to be successful. Since there can be so many different combinations of letters in a password (depending on the length of the password, there can be billions of different possible passwords to attempt), forcing systems to only allow 10 attempts every 30 minutes longer makes it technically impossible to crack a password using brute force.
Just as an example, if a system requires 8 character passwords using only lowercase letters, there are over 200 *billion* possible passwords that can be used. If you can only try 10 passwords every 30 minutes, it could take as long as 132,437 *years* to get a valid password for a single user account using a brute force attack. Now, it’s possible to get lucky and very quickly get a valid password using a brute force attack (particularly if you use a well known password or something like aaaaaaaa as your password), but the likelihood is very low, and that likelihood decreases even more if you include a greater number of valid characters for your password. Using all possible characters available on a US English keyboard, there are 6 quadrillion possible passwords that are 8 characters long. Adding one more character, for a 9 character password, increases that number to 600 quadrillion. This is why it’s important to allow and use “complex passwords” that utilize as many characters and as much length as is feasible.
You Are the Weakest Link
The problem, though, with using complex passwords is that the more complex they are, the more difficult they are to *remember*. This is a problem, because the first thing a person does when they can’t remember something is to write it down and put it somewhere where it’s easy to get to. Like on a post it note on their monitor. This makes it so much easier to get someone’s password because all you have to do is walk by their computer and it’s sitting there in plain view. Which brings up the most important rule of IT security: The weakest link in any security system is people. People do all kinds of things to make things easier or quicker. Unfortunately, things that are easy and quick are also very much not secure. For instance, removing the password attempt lockout mechanism from a system will keep you from having to spend time unlocking user accounts, but will absolutely open you up to a brute force attempt. When you go from being able to enter only 10 passwords every 30 minutes to being able to enter upwards of 20 billion per second (and beyond if the hacker has a lot of computing power at hand), it becomes a lot easier to break into the system with a brute force attack (so don’t turn off your password lockout policy!).
Since modern security policies allow us to make a brute force attack technically impossible, hackers have come up with other methods to obtain passwords and even bypass password mechanisms altogether. The most effective of these techniques is called Social Engineering.
Social Engineering
The most effective method for getting passwords is called social engineering. Social Engineering techniques rely on a hacker’s ability to take advantage of the weak link in any security system; the human being. For instance, even if you aren’t an IT professional, you’ve probably heard of the term “Phishing” before. This is a technique that uses weaknesses in human psychology to trick people into exposing private details including social security numbers, physical addresses, credit card numbers, and passwords. On very common example of a Phishing attempt is a mass email that purports to be from a supposedly credible source like a national banking company. The email will inform the user of some major problem with their bank account and prompt them to click a link to access their bank account. The link will take them to a web-site that very closely or exactly mimics the official bank web site, but the actual site you are sent to has a slightly different URL. Many people don’t pay very close attention to the URL they are being sent to, so they will unwittingly enter their actual bank access information into the login form on this fake website, resulting in their username and password being written to a database set up by the individual who sent out the email.
Another type of social engineering is something my step-kids love to pull off; shoulder surfing. This involves walking around looking for people who are in the process of entering a password for something (in my step-kids’ case, they can get my wife’s cell phone lock password within 10 minutes of her changing it by doing this). Most people aren’t as aware of their surroundings as they could be when they are entering a password, and shoulder surfing takes advantage of this fact. This is why many credit card machines in stores now come with “privacy shields” that help obscure which buttons are being pushed during a transaction. It’s a little more difficult to accurately enter your password on these devices, but it’s also more difficult for someone to purposefully snag your PIN.
Usually, though, social engineering is a little too involved and specialized to pull off. For a good example of a highly involved social engineering attack on a high security environment, watch the movie “Sneakers” staring Robert Redford (and Dan Akroyd…and Ben Kingsley…and some other people who aren’t as cool). Or go old school and just watch “The Sting”. In fact, just about every technique used by con men will typically fall under the social engineering category. But I digress.
A less involved technique for getting passwords involves the use of “Rainbow Tables.”
The Rainbow Connection
The best way to describe Rainbow Tables is to refer to them as a Distributed Brute Force attack that attacks the biggest weakness in password authentication mechanisms, password validation. As I mentioned earlier, when you enter a password, that password isn’t usually sent to the computer you’re trying to access. The password you enter is instead run through a hashing algorithm and the resulting hash string is sent and compared with the string that is stored on the computer. The weakness in this technique is that, despite the actual password being obscured, the hash is usually sent as is. And, in order for the system you are accessing to verify that you have the correct password, the hash is also stored somewhere in the system. This means that it is possible to discover what your password is by running a brute force attack against the hashing algorithm in use.
One of the biggest security problems with hashing algorithms is that the algorithms themselves are publicly available. This means that anyone can generate hash strings using them. Unfortunately, hashing algorithms *must* be published publicly available to ensure that they can’t be easily broken and are verifiable one-way mathematical algorithms. At any rate, a Rainbow Table is a database or other collection of hash values and the original data used to generate that hash value.
Rainbow Tables are built over a long period of time by a lot of individuals running a string through the hashing algorithm, then recording the resulting hash value. The original string and the hash value are entered into the Rainbow Table and stored permanently. Hackers can then use the rainbow table by comparing hash values that they uncover through various means with the ones stored in the Rainbow Table. The way a Rainbow Table is created, through the use of numerous computers and individuals to generate the information, means that it is a “distributed” effort. The fact that Rainbow Tables attempt to uncover every possible result generated by the hashing algorithm puts it into the Brute Force Attack classification. Thus, a Rainbow Table is the result of a Distributed Brute Force technique. Rainbow Tables are the preferred method for cracking passwords on systems that use hash exchange techniques because hashing algorithms are designed to generate *huge* numbers of values. For example, MD5 can generate 1.20892582e24 values. For those that aren’t aware of Math notation, that’s 1.20892582 with the decimal moved to the right 24 times. Put even more simply, it’s 120,892,582 with 16 more zeroes on the end. That’s a lot of possible passwords. Even assuming you could generate 30 billion hashes per second, it would still take you over a million years to find a string matching every single hash value the algorithm can put out. But, distribute that workload among a million people who are all able to generate 30 billion hashes per second and it will take a little over a year to accomplish the task (In the real world, the number of people generating hashes for Rainbow Tables is probably lower than 1 million and the rate they generate them is significantly lower than 30 billion per second). The Rainbow Table serves as a central repository for all the computers that are running hashes to record their progress and reduce the instances of generating the same hash numerous times. Eventually (and usually in an entirely feasible amount of time), any hashing algorithm can be made useless as a security mechanism for password validation because of the way Rainbow Tables work.
Creating Secure Passwords
Right now, almost every single hashing algorithm currently in wide use has hash values tied to known passwords that are stored in rainbow tables reaching as high as 8 characters with over 90% of all possible combinations. This means that every possible password combination for passwords of less than 8 characters long, using every possible combination of characters available using all characters available on a US keyboard, is recorded in a Rainbow table. What that means to you, dear reader, is that if you are using a password that is 8 characters long or less, there’s a 90% (or greater) chance that your password is listed on a rainbow table, and you should stop using it.
Now, it’s very likely you’ve been supremely frustrated when creating a new password at some point in your history of creating passwords because you were told that your password wasn’t “complex” or “strong” enough. A prevailing attitude in IT is that passwords with more characters available for each character in length makes the password more secure, and that attitude is partially correct. However, passwords that are longer than 8 characters that also have high numbers of characters used for each character in length are *much* more difficult to remember, and very few people ever make a password longer than 8 or 9 characters. As I mentioned, passwords of that length are not nearly as secure as they used to be because they are stored in Rainbow Tables, so to keep your passwords secure, you *have* to make them *longer*.
Here’s a fun fact for you…A password that is 14 characters long and only uses lower-case characters and the space bar is *much* more secure than a password that is 8 characters long and uses every character available. The number of available passwords for the former is a 21 digit number. The latter is 15 digits. This means that it will take significantly longer to generate a Rainbow Table for a 14 character password than for an 8 character password.
Now, I understand that a 14 character password might seem difficult to come up with, but consider this…if all you have to use in that password is lowercase letters and spaces, you can use your favorite movie quote. “may the force be with you” is 25 characters long. It will take 624 billion years for 1 million computers generating 30 billion hashes per second to complete a rainbow table that holds all the passwords possible using 25 characters using *just* lowercase letters and spaces. For reference, scientists estimate that the entire universe has only been around for 14 billion years.
However, there’s a problem with that last paragraph…The hashing algorithm. Some algorithms don’t generate enough hash values to cover the entire list of passwords you can use. For instance, the example above, using 25 character length passwords, means the number of possible passwords is a 36 digit long number. If you were to run that password through a hashing algorithm that generates a short value, CRC16, for example, you would likely end up generating a hash that also matches a much *shorter* password. In fact, the entire CRC16 algorithm will only support passwords of up to 10 characters in length, using the entirety of the US keyboard character set (you can actually generate five times as many passwords than CRC16 hash values using that length and complexity). Luckily, CRC16 isn’t heavily utilized in password validation these days. That said, it is important to point out that there *is* an upper limit to how long a password can be and remain “secure”. For the most part (as of 2015), you should consider a password that is more than 32 characters long to be increasingly likely to generate a hash value that could collide with a shorter password’s hash value.
So the longer your password is, the more secure it is (up to a point). If the system you are creating a password for *requires* you to use more than just lower case letters and spaces, put 1! at the start or end and then capitalize the first letter of your sentence.
And there you have it. That’s a whole lot of explanation on the subject of passwords, and there’s still a lot I can cover. In part 2 of this subject, I’ll cover the fine art of building a secure (and not annoying) password policy. I’ll also touch a little bit on Active Directory passwords, and why it generally *isn’t* possible to crack them with a Rainbow Table (when you only have access to a client machine, at least).