Murdoch University Research Repository

Welcome to the Murdoch University Research Repository

The Murdoch University Research Repository is an open access digital collection of research
created by Murdoch University staff, researchers and postgraduate students.

Learn more

A cryptographically-based operating system security model that protects against privileged attackers

Payne, Christian (2009) A cryptographically-based operating system security model that protects against privileged attackers. PhD thesis, Murdoch University.

PDF - Front Pages
Download (188kB)
PDF - Whole Thesis
Download (5MB)


It has long been recognized that widely used contemporary systems have relatively weak security and stronger operating system security models are required. In particular, the design of widely-used security models is such that the highest level of privileges available on the system are often highly exposed. If an attack is successful and the attacker attains a high level of privilege, all of the security mechanisms on the system may typically be bypassed.

Despite such limitations, weak models remain ubiquitous as more secure alternatives are complex and therefore harder to configure and audit for correctness. This is especially problematic when the user and administrator are the same person, as is often the case in widespread workstation environments. To be used effectively, security models must be simple enough to be easily conceptualised by users and consistent with their requirements.

Careful application of cryptography can often improve security. However, in the domain of operating system security to date, the use of cryptography has largely involved the creation of ad hoc, standalone mechanisms. Cryptographic file systems exist to protect the confidentiality of data, but have little or no connection with existing access control theory. For example, some allow sharing of data between users but none provide all of the expected properties available in conventional, fully-fledged access control mechanisms such as secure and convenient revocation and prevention of authorisation transference. Filesystem integrity checkers use cryptographic hashes or digital signatures to ensure objects have not been modified, thereby protecting against substitution of malicious code. However, existing schemes lack the supporting infrastructure of an underlying security model to properly discriminate between verified and unverified objects. Furthermore, key management-related interdependencies between these different mechanisms have not been recognized and, as a result, work in this area has so far progressed in a somewhat disjointed and piecemeal manner.

This research describes a new security model, known as Vaults, that utilises cryptography to provide improved security. In particular, the new model aims to be secure against an attacker who has achieved a high level of privilege on the system. Vaults provides a cryptographically-enhanced access control model that protects files from unauthorised read and write access. It also facilitates secure, authenticated sharing of data between users using semantics consistent with traditional non-cryptographic access control models.

The Vaults access control mechanism is supported by a flexible and convenient key management architecture that can be used for both file access keys and generic application secrets. Access to these values is controlled by a mechanism for cryptographically verifying the integrity of programs and the data objects with which they interact. However, unlike previous schemes, Vaults not only prevents execution of illicitly modified trusted code, but also assigns different privilege levels to verified and unverified processes. Furthermore, partially-trusted processes can be confined to specifically defined objects if required. This approach provides a mechanism for authenticated user interaction with security-critical system components and therefore represents a new interpretation of the traditional notion of a physical trusted path that can be extended to any appropriate object on the system. Finally, all of these mechanisms apply on both a global and local level, allowing administrators to create system-wide policies, and users to extend and refine these to suit their own security needs.

However this flexibility does not come at the cost of great complexity and the basics of using the scheme can be easily explained to users as they can be expressed using conceptually simple abstractions such as "locking files" and "sharing keys". The use of cryptography in this way also serves to weaken the traditional association between privilege and identity, as access is permitted or denied based upon possession of the required token rather than the identity of the requesting process. Such a design has the dual effects of constraining the powers of privileged users and lowering their exposure to attack by reducing privileges to a token, which is generally easier to protect than an identity.

After developing the model, a series of large-scale attack trees were constructed to analyse its security. The attack trees were used to both refine the design of the security model and also evaluate the assertion that the model retains its security properties when under attack by a user who has gained the ability to bypass the security kernel and directly access the secondary storage device. The results of this analysis demonstrate the advantages of applying cryptography to the problem of operating system security and show that the Vaults model is able to maintain its security properties in the face of attacks that are normally excluded 'by assumption' under existing computer security models. Vaults is therefore a novel and comprehensive model for integrating cryptography into the operating system in a manner that improves security, while remaining both flexible and usable.

Item Type: Thesis (PhD)
Murdoch Affiliation(s): School of Information Technology
Supervisor(s): Sudweeks, Fay
Item Control Page Item Control Page


Downloads per month over past year