ja_mageia

Both Mac OS X Leopard and Windows Vista have added significant security features from their predecessors. In this post, we compare the new executable space protection and address space layout randomization features in Leopard and Vista.

Executable Space Protection

Both Mac OS X Leopard and Windows Vista provide some form of executable space protection.

Microsoft Windows' implementation of executable space protection, dubbed DEP, protects both the stack and the heap through either software or hardware mechanisms. Even processors which lack the NX bit have limited protection through software. Modern CPUs have full executable space protection, covering both the heap and stack within both system and application executables.

According to official Apple documentation, Mac OS X Leopard provides executable space protection in stack space on 32-bit Intel processors. Older PPC-based systems are not protected, either by hardware or software. Additionally, heap execute protection is only provided for 64-bit executables:

"In Leopard, stack execute disable is available for both 32- and 64-bit applications. For 64-bit processes, Leopard provides protection from code execution in both heap and stack data areas. "

Unfortunately, most common apps that ship with OS X Leopard are 32-bit applications (note that none are Mach-O 64-bit executable x86_64):

bash-3.2$ file /Applications/Safari.app/Contents/MacOS/Safari
/Applications/Safari.app/Contents/MacOS/Safari: Mach-O universal binary with 2 architectures
/Applications/Safari.app/Contents/MacOS/Safari (for architecture i386): Mach-O executable i386
/Applications/Safari.app/Contents/MacOS/Safari (for architecture ppc7400): Mach-O executable ppc

bash-3.2$ file /Applications/QuickTime\ Player.app/Contents/MacOS/QuickTime\ Player
/Applications/QuickTime Player.app/Contents/MacOS/QuickTime Player: Mach-O universal binary with 2 architectures
/Applications/QuickTime Player.app/Contents/MacOS/QuickTime Player (for architecture ppc7400): Mach-O executable ppc
/Applications/QuickTime Player.app/Contents/MacOS/QuickTime Player (for architecture i386): Mach-O executable i386

bash-3.2$ file /Applications/iChat.app/Contents/MacOS/iChat
/Applications/iChat.app/Contents/MacOS/iChat: Mach-O universal binary with 2 architectures
/Applications/iChat.app/Contents/MacOS/iChat (for architecture i386): Mach-O executable i386
/Applications/iChat.app/Contents/MacOS/iChat (for architecture ppc7400): Mach-O executable ppc

Therefore, this added executable space protection for the heap doesn't apply to most common applications in Leopard, leaving them vulnerable to heap overflow attacks.

Address Space Layout Randomization

Address space layout randomization is a technique to randomize memory addresses of the base of the code, stack, heap, and libraries. First used by PaX and OpenBSD, some forms of ASLR are now used in Mac OS X Leopard and Windows Vista. However, the implementation of ASLR is Leopard and Vista differ significantly.

ASLR in Vista, as documented by Microsoft, randomizes the addresses of the code, stack, heap, and DLLs. In other words, Windows Vista offers a complete implementation of ASLR. However, it is not without issues. Weaknesses have been identified in Vista's ASLR implementation by Ollie Whitehouse of Symantec.

As documented in Apple's official documentation, by David Maynor and others, ASLR in Mac OS X Leopard is limited to library randomization. According to Apple:

"In Leopard, libraries are loaded into random addresses when the system is installed and at any time that library prebinding is updated on the system... For any given Mac, the address of a particular library function will be fixed in one of thousands of random locations between system updates, but across all Mac systems, the address is different."

While this may prevent some basic return-to-libc attacks, it won't raise the bar on other attacks. By refusing to randomize the location of the code, stack, and heap, Apple has introduced an incomplete ASLR implementation in Mac OS X Leopard.

Although the effectiveness of address-space randomization has been challenged in the paper On the Effectiveness of Address-Space Randomization, ASLR remains a useful technique to provide additional system security when combined with executable space protection and other technologies.

Overall verdict: Windows Vista's ASLR coupled with executable space protection provides a superior security solution to that of Mac OS X Leopard. As Leopard increases in market penetration, we should expect to see further attacks exploiting the weaknesses of these security features in Leopard.

Note: This post was composed on an iMac running Mac OS X Leopard, my primary home workstation.