Rijndael 86
Serpent 59
Twofish 31
RC6 23
MARS 13
NIST Analysis
MARS - Too Slow and Complicated
MARS is the most complicated of the candidates. While other algorithms use a single function for all rounds, MARS uses four different functions. This gives it a high security margin, as estimated by attacks on its reduced-round variants. However, the approach's complexity casts doubt on this estimate, and some reviewers argued that MARS required more analysis than was possible within the evaluation schedule.
MARS relies on multiplication, variable rotations, and large data tables. These make it hard to protect MARS against implementation attacks. MARS suffers from serious performance degradation if it's modified to defend against those attacks.
Regarding performance, MARS yielded moderate to poor results. Overall, its software performance landed in the middle range, though results varied significantly depending on the processor and the compiler used. Hardware implementations performed below average, regardless of key size. MARS wasn't well suited for smart-card implementation because it required excessive amounts of ROM. MARS missed the mark for performance and also for an excessively complicated algorithm.
RC6 - Too Much RAM
RC6 is a simple algorithm with an adequate security margin. It's based on RC5, an earlier algorithm developed at RSA Security. This lineage worked to its advantage, since RC5 has been analyzed without uncovering serious problems. Like MARS, RC6 relies on multiplication and variable rotations, making RC6 hard to defend against implementation attacks, though not as difficult as MARS.
RC6 is also fast. It outperformed Rijndael in some cases, particularly in software implementations on 32-bit processors. Hardware implementations, however, only yielded average performance. RC6 also requires a lot of RAM, so it's not well suited for implementation in restricted-space environments such as smart cards. RC6 lost the race for its poor hardware performance.
Serpent - Conservative but Slow
Serpent resembles Rijndael, but instead of executing a small number of more complicated rounds, Serpent executes a larger number of simpler rounds. With its simple, conservative design, Serpent recycled a few features of DES, and overall relies on well-known operations. This simplicity and familiarity made it easier to assess Serpent's security, with studies of reduced-round versions revealing that it has a high security margin. Serpent is among the easiest candidates to defend against implementation attacks.
Unfortunately, software implementations of Serpent were among the slowest of the finalists. In contrast, in some cases testers were able to pipeline hardware implementations, yielding the highest encryption rates of any of the implementations. Execution speed wasn't affected by increasing the key size. Serpent's low storage requirements also made it suitable for smart cards.
Although Serpent offered a better blend of simplicity and security margin than Rijndael, its poor software performance eliminated it.
Twofish - Slow and Subtle
Twofish uses an innovative approach that uses half of its encryption key to modify how the encryption algorithm operates, and this subalgorithm uses the other half of the key as its own encryption key. This feature yields the key separation property, which some observers feared might make a divide and conquer attack possible. Such an attack would try to determine which subalgorithm the key chose, yielding half of the key's value. Testers found no such attack during analysis.
Studies of reduced-round variants of Twofish found that it has a high security margin. As with MARS, however, its unusual structure casts some doubt on the validity of those studies. Some reviewers noted that its complexity made it more difficult to evaluate, given the deadlines. Twofish is vulnerable to implementation attacks but can be modified to defend efficiently against some attacks.
Twofish yielded average overall performance. Software performance was in the lower range, with key setup time particularly slow. Hardware performance was average. Its limited storage requirements make it suitable for smart-card implementation. NIST didn't select Twofish because of its lower performance and algorithm complexity.
Rijndael - The Winner
Rijndael is fast and compact, with a simple mathematical structure. This structure made it easier to analyze its security during evaluation, and NIST found no problems. In contrast, its simplicity gives attackers a smaller mathematical realm to study-if a hidden problem lurks somewhere in Rijndael, someone will eventually find it. The reduced-round attacks showed that Rijndael doesn't have as much of a security margin as other candidates, but adding rounds would slow it down.
Rijndael also did well at resisting implementation attacks, where an attacker tries to crack the encryption by externally measuring its behavior, including power consumption and execution time. NIST examined the candidates' vulnerability to such attacks and their ability to defend against them, usually by special coding that balances the algorithms' power usage. Rijndael proved easy to defend against such attacks because it relies primarily on Boolean operations.
Rijndael's overall software performance was the best. It performed well in tests with smart cards and in hardware implementation. The algorithm has a lot of inherent parallelism, making it easy to use processor resources efficiently. Larger key sizes slowed it down somewhat, since it executes additional rounds to handle the larger keys.
NIST chose Rijndael because it combines simplicity and high performance. While Rijndael does have a slimmer security margin than other finalists, this doesn't pose a practical risk. If you think of the different finalists as hardened steel cages, then Rijndael uses enough steel to keep the monster safely caged, while using less steel than other finalists. (See table for a summary of NIST's assessment of each finalist.)