Capabilities Compromised: Uncovering Security Risks in CHERI Systems

by

Author: Denis Avetisyan

A new analysis reveals that despite the inherent security advantages of the CHERI architecture, vulnerabilities in core system components can undermine its compartmentalization features.

Exploits targeting dynamic linking, heap management, and stack walking can allow malicious code to bypass CHERI’s capabilities-based security model.

While the CHERI architecture offers promising defenses against memory corruption, its compartmentalization mechanisms are not impenetrable. This paper, ‘A Security Analysis of CheriBSD and Morello Linux’, details critical vulnerabilities in implementations on both BSD and Linux systems, demonstrating how malicious code can bypass intended security boundaries. Our analysis reveals that weaknesses in dynamic linking, heap management, and stack handling allow compromised libraries to escape compartmentalization and access sensitive data. Can these foundational issues be effectively addressed to realize the full security potential of CHERI and build truly isolated, resilient systems?

Beyond Runtime Defenses: A Paradigm Shift in Memory Safety

Conventional memory safety mechanisms, historically centered around runtime checks, are increasingly challenged by the evolving landscape of cyberattacks. These systems operate by verifying memory access permissions during program execution, a process that introduces overhead and creates opportunities for exploitation through vulnerabilities like buffer overflows and use-after-free errors. While offering a degree of protection, the complexity of these checks-and the inherent difficulty in anticipating every potential attack vector-renders them susceptible to sophisticated exploits that bypass or subvert these safeguards. Modern attacks, often leveraging techniques like return-oriented programming and just-in-time code spraying, demonstrate the limitations of relying solely on detecting errors at runtime, highlighting the need for a more robust and preventative approach to memory safety.

The CHERI architecture departs from conventional memory safety methods by embracing capabilities, which function as unforgeable tokens representing legitimate access rights to specific memory regions. Unlike traditional approaches that rely on tagging memory as readable, writable, or executable – and then performing runtime checks to enforce these permissions – CHERI embeds access control directly within the capabilities themselves. Each capability doesn’t just indicate permission; it is the permission, making it incredibly difficult for malicious code to forge access. This design offers a more precise and scalable solution because capabilities can be delegated – allowing controlled access to resources without revealing the underlying data – and can be finely-grained, defining access to individual objects rather than broad memory blocks. The result is a system where programs operate within strictly defined boundaries, dramatically reducing the attack surface and bolstering overall system security.

The CHERI architecture’s capability-based approach fosters a significantly enhanced security model through fine-grained compartmentalisation. Instead of broadly permitting or denying access based on process or memory region, it assigns explicit, unforgeable capabilities – essentially, keys – to specific memory locations. This means code can only access data for which it possesses a valid capability, effectively creating strong isolation boundaries. Consequently, even if malicious code breaches one compartment, its propagation is severely limited; lacking the necessary capabilities, it cannot access data or execute code in other, protected areas. This dramatically reduces the attack surface and contains potential breaches, offering a robust defense against exploitation techniques that bypass traditional memory safety measures by restricting access at a much more granular level.

The Dynamic Linker: A Necessary Component, and a Potential Weakness

The dynamic linker is a core component of most contemporary operating systems, facilitating code reuse and modularity by loading shared libraries – collections of precompiled code – at program runtime. This process is typically initiated by functions such as `dlopen`, which locates and maps the shared library into the program’s address space. Rather than statically linking all required code into a single executable, dynamic linking reduces executable size and allows for updates to shared libraries without requiring recompilation of dependent programs. The dynamic linker manages symbol resolution, determining the memory addresses of functions and data defined within these shared libraries, and establishing the necessary connections for the program to execute the library’s code. This runtime loading and linking significantly enhances system flexibility and resource utilization.

The resolution of function addresses in dynamically linked programs utilizes the Global Offset Table (GOT). The GOT acts as a lookup table where the actual memory addresses of dynamically linked functions are stored, allowing for position-independent code. However, this mechanism introduces a security vulnerability: if an attacker can overwrite entries within the GOT, they can redirect function calls to malicious code. Exploitation typically involves overwriting a GOT entry with the address of attacker-controlled code, effectively hijacking the program’s control flow. Proper security measures, such as GOT relocation randomization and write protection mechanisms (e.g., using the `relro` flag and NX bit), are crucial to mitigate these risks by making it more difficult for attackers to predict or modify GOT entries.

CHERI architecture implements capability-based access control extending to the dynamic linker, intending to restrict loaded shared libraries to explicitly authorized memory regions and enhance system robustness. However, recent research has identified vulnerabilities in this implementation. Specifically, analysis demonstrates that the dlopen infoleak attack vector successfully bypassed compartmentalization mechanisms in both Morello Linux and CheriBSD environments. This bypass allows an attacker to potentially leak information about loaded libraries and their memory layout, despite the presence of CHERI’s capability-based security features, indicating a need for further refinement of dynamic linker security protocols.

Assessing the Boundaries: Attack Vectors and Capability-Based Defenses

Heap storing and heap scavenging represent classes of memory corruption vulnerabilities that can be leveraged to compromise capability-based security. Heap storing involves an attacker overwriting metadata associated with freed memory blocks, potentially allowing them to control the contents of those blocks and, crucially, any capabilities stored within them. Heap scavenging occurs when an attacker can access and reuse memory that has been freed, potentially retrieving capabilities that were not properly cleared or invalidated. Successful exploitation of either technique can provide an attacker with unauthorized capabilities, enabling them to bypass compartmentalization and gain access to protected resources despite the presence of capability-based access control.

Heap revocation is a security mitigation employed by the CHERI architecture to address memory reuse attacks. When a memory block is freed, any capabilities – which represent authorized access to that memory – are simultaneously invalidated. This prevents an attacker from retaining a capability to a previously allocated and freed memory region, thereby disrupting attempts to exploit dangling pointers or use-after-free vulnerabilities. By invalidating capabilities upon memory deallocation, heap revocation effectively limits the lifespan of access rights, reducing the window of opportunity for malicious actors to leverage stale capabilities and compromise system security. This differs from traditional memory management approaches that may allow capabilities to persist even after the underlying memory is no longer valid.

While capability-based architectures are intended to limit information leakage through techniques like stack walking by enforcing precise access control, recent research indicates that practical bypasses of compartmentalization are still achievable. Testing on Morello Linux demonstrated successful exploitation using four attack vectors: stack walking, dynamic library loading (dlopen) infoleaks, heap scavenging, and heap storing. Further analysis on CheriBSD, which incorporates c18n and heap revocation mitigations, revealed that two of these – dlopen infoleaks and heap scavenging – continued to successfully bypass security measures, suggesting that despite architectural defenses, information leakage remains a viable attack path.

Towards a More Secure Foundation: CHERI in Practice

Several operating systems are undergoing adaptation to leverage the security benefits of the CHERI architecture. `CheriBSD`, a direct port of FreeBSD, serves as a research platform and demonstrates the feasibility of CHERI capabilities within a traditional Unix-like environment. Simultaneously, `Morello Linux`, developed by Google and others, is building CHERI support directly into the Linux kernel and user space. These adaptations involve modifying the kernel, compilers, and standard libraries to utilize CHERI’s capability-based access control, aiming to create a system where memory access is explicitly authorized and significantly reduces the attack surface available to malicious code. Both projects contribute to the practical implementation and validation of CHERI’s security model in real-world operating system contexts.

The `c18n` feature within CHERI-based systems enhances compartmentalisation by allocating a dedicated stack space to each compartment. This isolation prevents stack-based attacks, such as buffer overflows and return-oriented programming (ROP), from compromising other compartments. By confining the impact of a potential exploit to the compartment’s allocated stack, `c18n` significantly reduces the attack surface and improves overall system resilience. This approach contrasts with traditional systems where a compromised stack can lead to arbitrary code execution with system-level privileges.

Sealing, a security feature within the CHERI architecture, provides data integrity through cryptographic means. This process utilizes digital signatures generated with a private key to ‘seal’ data, enabling verification of its authenticity and preventing unauthorized modification. Implementation relies on the OpenSSL library for cryptographic operations, allowing for standard key management practices and interoperability. Sealing ensures that data, when unsealed, has not been tampered with since it was originally sealed, offering a robust defense against data corruption and malicious attacks by verifying the data’s origin and ensuring its consistency.

Recursive Capability Scanning is a verification technique employed within CHERI systems to validate the integrity of the capability infrastructure. This process involves recursively checking the validity and authenticity of capabilities – the pointers used to access memory and resources – against a root of trust. By traversing the chain of capabilities and verifying each link, the system can detect any unauthorized modification or compromise of the capability system itself. This scanning isn’t a one-time operation; it can be implemented as a periodic or event-triggered process to ensure ongoing effectiveness and resilience against attacks that attempt to subvert the capability-based security model. The technique aims to prevent malicious actors from gaining control by manipulating the fundamental access control mechanisms of the system.

The research meticulously details how even fundamentally secure architectures like CHERI are susceptible to systemic weaknesses. This echoes Carl Friedrich Gauss’s observation: “If other sciences were as well defined as mathematics, the progress would be greater.” The study reveals that compartmentalization, a core tenet of CHERI’s design, isn’t absolute; vulnerabilities in components like dynamic linking and heap scavenging create pathways for circumvention. Just as a flawed axiom undermines an entire mathematical proof, weaknesses within these foundational system components compromise the architecture’s intended security. The analysis emphasizes that security isn’t solely about novel hardware; it’s about the integrity of the entire software stack and the interactions between its parts.

The Road Ahead

The findings presented here underscore a fundamental truth: architectural innovation alone does not deliver security. The CHERI architecture offers a compelling vision of capability-based compartmentalization, yet the persistent vulnerabilities in established system components-dynamic linking, heap management, stack traversal-reveal the limits of localized defenses. Each new dependency, each convenience offered by these systems, carries a hidden cost – a widening of the attack surface, a potential bypass for even the most rigorously designed memory safety mechanisms.

Future work must move beyond isolated improvements. The focus should shift toward holistic system design, where security is not bolted on, but woven into the very fabric of the operating system. This demands a re-evaluation of long-held assumptions about code loading, library management, and the implicit trust relationships that permeate modern systems. Heap scavenging and stack walking, seemingly benign optimizations, require intense scrutiny – they represent points of structural weakness, opportunities for subversion.

Ultimately, the pursuit of robust security demands a deeper understanding of systemic behavior. A system is an organism; its strength lies not in isolated fortifications, but in the resilience of its interconnected parts. The challenge is not simply to prevent attacks, but to build systems that can gracefully absorb and recover from them – systems where compromise in one area does not necessarily lead to catastrophic failure elsewhere.

Original article: https://arxiv.org/pdf/2601.19074.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-01-28 08:56