Taming Undefined Behavior: Lessons from Ruby on Capability Hardware

Author: Denis Avetisyan

Porting the CRuby virtual machine to the CHERI architecture revealed significant challenges related to undefined behavior in C, necessitating novel approaches to achieve memory safety.

This paper details the pitfalls encountered while porting CRuby to CHERI and proposes workarounds leveraging capability hardware to mitigate memory safety issues stemming from C’s undefined behavior and conservative garbage collection.

While modern software increasingly relies on virtual machines for security, porting these systems to capability-based architectures like CHERI introduces unexpected challenges. This paper, ‘Pitfalls in VM Implementation on CHERI: Lessons from Porting CRuby’, details the issues encountered while porting the CRuby VM, revealing that common assumptions about undefined behavior in C code often conflict with CHERI’s stricter memory safety model. We categorize these pitfalls-stemming from pointer arithmetic and conservative garbage collection-and present practical workarounds validated through our CRuby port and analysis of existing CHERI VM implementations. Ultimately, can these lessons guide the development of safer and more robust virtual machines for a future demanding enhanced memory safety?

The Inevitable Crisis of Mutable State

The foundation of modern computing relies on managing system memory, yet traditional techniques are showing increasing susceptibility to exploitation. Historically, operating systems have employed methods like dynamic memory allocation and pointer arithmetic to efficiently utilize resources. However, these approaches introduce vulnerabilities – specifically, opportunities for attackers to access or manipulate data outside of authorized boundaries. Buffer overflows, use-after-free errors, and dangling pointers – all stemming from flawed memory management – are consistently found in software, enabling malicious code execution and data breaches. The proliferation of interconnected devices and increasingly sophisticated attack vectors exacerbate these risks, demanding a shift towards more robust and inherently safe memory management paradigms to safeguard digital infrastructure and user data.

The enduring prevalence of languages like C in systems programming stems from their performance and low-level control, but this power comes at a cost: inherent permission for undefined behavior. Unlike more modern languages with strict rules, C allows operations that are not clearly defined by the language standard, such as dereferencing null pointers or accessing memory outside allocated bounds. While seemingly innocuous, these actions don’t necessarily cause a program to crash; instead, the compiler is free to do anything – and attackers skillfully exploit this ambiguity. Malicious code can be crafted to subtly manipulate program execution based on these undefined outcomes, creating vulnerabilities ranging from denial-of-service attacks to complete system compromise. The lack of defined behavior transforms potential programming errors into security loopholes, demanding meticulous code reviews and increasingly sophisticated runtime defenses to mitigate the risk.

The implementation of memory safety protections frequently introduces a performance overhead that significantly impacts application efficiency. While techniques like address space layout randomization (ASLR) and data execution prevention (DEP) bolster security, they demand additional computational cycles for tasks such as address translation and permission checks. This trade-off between security and speed presents a considerable challenge, particularly for performance-critical applications like databases, game engines, and high-frequency trading platforms. Developers often face a difficult choice: prioritize security and accept reduced performance, or optimize for speed and potentially increase vulnerability. Research continues to explore methods for minimizing this performance penalty, including hardware-assisted memory safety and more efficient software-based mitigations, but a universally optimal solution remains elusive, forcing ongoing compromise in many systems.

Capability-Based Security: A Shift in Perspective

The CHERI (Capability Hardware Enhanced RISC Instructions) architecture fundamentally alters memory access control by introducing capabilities. These capabilities are not simply pointers, but rather opaque tokens that encapsulate both a memory address and the specific access rights permitted for that address – including read, write, and execute permissions. This differs from traditional pointer-based systems where pointers implicitly allow unrestricted access to the pointed-to memory. Every memory access in CHERI requires a valid capability; attempting an operation without a valid, authorizing capability results in an immediate hardware exception. This explicit permission model forms the foundation of CHERI’s enhanced security, as it shifts the focus from preventing exploitation of memory errors to preventing the errors themselves by enforcing strict access control at the hardware level.

The CHERI architecture fundamentally alters memory access control by associating access rights directly with pointer values, effectively creating what are known as capabilities. Traditional pointer-based systems allow arbitrary memory access based solely on address, leaving them vulnerable to attacks like buffer overflows and use-after-free errors. CHERI, however, encodes permissions – such as read, write, and execute – within the pointer itself. This means any memory access initiated through a CHERI pointer is automatically validated against the permissions embedded within that pointer. Consequently, attempts to access memory outside of the authorized bounds, or in a prohibited manner, are immediately detected and prevented by hardware, thus eliminating entire classes of memory safety vulnerabilities that plague conventional architectures. This approach moves the enforcement of memory safety from software-based runtime checks to a hardware-enforced mechanism, providing a significant performance and security improvement.

Capability Tags are metadata associated with capabilities in the CHERI architecture, extending beyond simple permissions like read, write, or execute. These tags, typically 32-64 bits in size, provide additional context regarding the capability’s validity and intended usage. They allow for the encoding of information such as the scope of the capability – limiting its use to a specific address range or process – and the delegation policy, controlling whether the capability can be transferred or copied. Critically, tags are cryptographically sealed to prevent tampering; any modification to the tag invalidates the capability, preventing unauthorized access or privilege escalation. This granular control enables precise authorization and prevents the misuse of valid capabilities by restricting their application to specific contexts and conditions.

Porting Languages to CHERI: Observing the Constraints

The successful porting of languages including CRuby, MicroPython, and Rust to the CHERI architecture provides practical evidence that capability-based security can be implemented in complex, real-world software systems. While demonstrating feasibility, these ports also highlight significant implementation challenges. These challenges stem from the need to adapt existing language runtimes and compilers to operate within a capability-based security model, requiring modifications to memory management, data alignment, and the handling of potentially unsafe operations. The effort reveals that capability-based systems are not simply a drop-in replacement for traditional pointer-based systems and necessitate careful consideration of existing code behaviors and potential compatibility issues.

Porting languages to CHERI architectures introduces challenges related to data alignment and undefined behavior. Traditional languages often permit flexible data layouts and rely on implicit assumptions about memory access, which are incompatible with CHERI’s strict capability-based memory model. Adapting data alignment to satisfy CHERI’s requirements necessitates modifications to compilers and runtime systems. Furthermore, existing code frequently contains instances of undefined behavior – operations whose results are not specified by the language standard – that may inadvertently bypass CHERI’s security mechanisms. Addressing this requires either rewriting such code or implementing runtime checks to ensure safe operation, adding complexity to the porting process and potentially impacting performance.

Conservative garbage collection is essential when porting languages to CHERI due to the architecture’s fine-grained capabilities and the need to prevent accidental capability leaks through dangling pointers. This approach necessitates treating any memory location potentially reachable from multiple pointers as live, which can increase memory usage and introduce performance overhead. The successful port of CRuby to CHERI leveraged conservative garbage collection and was validated through a rigorous testing suite comprising 894 micro-benchmarks, with 3 benchmarks excluded due to identified issues; this comprehensive testing ensured functional correctness and identified areas for optimization in the garbage collection implementation.

The Illusion of Optimization: Compromises Inevitable

In the pursuit of optimized performance, developers sometimes employ techniques like in-place reallocation – modifying data structures directly within their existing memory locations to avoid the overhead of copying. However, this practice inherently challenges core memory safety principles, potentially introducing vulnerabilities if not meticulously managed. The fundamental issue lies in the risk of overwriting data before it’s fully utilized, leading to unpredictable program behavior or exploitable security flaws. While offering speed gains, in-place reallocation demands rigorous validation of boundaries and dependencies, ensuring that modifications don’t inadvertently corrupt adjacent data or violate expected program logic; failing to do so can swiftly negate any performance benefits with the introduction of difficult-to-detect errors and security risks.

Performance-critical code often relies on pointer arithmetic and bit manipulation to achieve speed, yet these very techniques introduce significant security risks. Direct memory manipulation bypasses many standard safety checks, potentially leading to buffer overflows, use-after-free errors, or arbitrary code execution if not meticulously reviewed. Developers must carefully validate all pointer offsets and bitwise operations to ensure they remain within the bounds of allocated memory and adhere to expected data structures. This scrutiny requires a deep understanding of memory layout and potential attack vectors, making thorough code analysis and robust testing essential to mitigate vulnerabilities arising from these powerful, but potentially dangerous, programming practices.

The CHERI (Capability Hardware Enhanced RISC Instructions) architecture, while fundamentally bolstering memory safety through fine-grained capabilities, doesn’t operate in isolation; it strategically integrates with existing operating system mechanisms like the mprotect system call. This call allows a program to alter the permissions of memory regions – marking them as read-only, executable, or writable – and when used in conjunction with CHERI, provides a complementary layer of defense. While CHERI prevents unauthorized access based on capability mismatches, mprotect can enforce broader policies, such as preventing code injection by marking data regions as non-executable. This synergistic approach addresses vulnerabilities that might bypass CHERI’s inherent protections, or conversely, provides a more coarse-grained control mechanism when fine-grained capability management is impractical, ultimately strengthening the overall security posture of the system by layering different protection strategies.

Toward a Resilient Future: The Promise of Capabilities

The Capability Hardware Enhanced RISC Instructions (CHERI) architecture presents a fundamentally different approach to computer security, moving beyond traditional methods that often rely on preventing attacks after they’ve begun. Instead of simply attempting to contain breaches, CHERI focuses on proactive prevention through the concept of capabilities. These capabilities act as unforgeable tokens granting access to specific memory regions, effectively limiting the damage an attacker can inflict even if they compromise a portion of the system. By redefining how memory access is managed at the hardware level, CHERI drastically reduces the attack surface and mitigates entire classes of vulnerabilities, including buffer overflows and return-oriented programming. This capability-based approach isn’t merely a refinement of existing techniques; it represents a paradigm shift towards building systems where secure-by-design principles are inherent, promising a future where software is demonstrably more resilient against increasingly sophisticated threats.

The promise of CHERI-capable systems hinges not merely on architectural innovation, but on a meticulous approach to software development practices. Traditional programming paradigms often rely on assumptions about memory access that, when violated, lead to exploitable vulnerabilities; CHERI, while mitigating many of these, demands explicit attention to memory safety. Developers must proactively address potential undefined behavior – situations where a program’s outcome is unpredictable due to uninitialized variables, out-of-bounds access, or other similar issues – as these can still undermine the benefits of capability-based security. Effective memory management, including careful allocation, deallocation, and bounds checking, becomes paramount, necessitating both refined compiler technology and a shift in programmer mindset to fully leverage CHERI’s potential for building demonstrably more resilient software.

Widespread adoption of the Capability Hardware Enhanced RISC Instructions (CHERI) architecture hinges on seamless integration with existing programming ecosystems. Recent work focusing on this integration showcased a port of the CRuby virtual machine, demonstrating promising performance results. Benchmarks revealed an average throughput of 0.982 relative to the original Ruby VM, indicating minimal overhead from the capability-based security mechanisms. Importantly, the low standard deviation of 0.038 across these benchmarks suggests consistent performance gains and predictable behavior, bolstering confidence in CHERI’s practicality for mainstream applications and paving the way for more secure software development practices.

The endeavor to port CRuby to the CHERI architecture illuminates a fundamental truth about complex systems: their fragility isn’t a bug, but an inherent property. The paper meticulously documents instances where undefined behavior in C, seemingly innocuous, triggered failures within the capability-based system. This echoes the notion that a system that never breaks is, effectively, dead – a static, unadaptable construct. The researchers didn’t solve memory safety, but rather navigated a landscape of potential failures, revealing how conservative garbage collection and careful handling of pointer arithmetic became necessary compromises. As Ada Lovelace observed, “The Analytical Engine has no pretensions whatever to originate anything. It can do whatever we know how to order it to perform.” The CRuby port wasn’t about building a perfect system, but about defining the boundaries of predictable behavior within a complex, evolving ecosystem.

The Horizon Beckons

The exercise of transplanting a mature virtual machine like CRuby onto CHERI soil reveals a truth often obscured by architectural enthusiasm: capability systems do not prevent undefined behavior, they merely expose it. Each workaround implemented, each conservative garbage collection strategy employed, represents not a victory over complexity, but an acknowledgement of its inherent presence. The system doesn’t become safer, it becomes more visibly dangerous, demanding a constant accounting of what remains unseen. Every new architecture promises freedom until it demands DevOps sacrifices.

Future endeavors must shift focus from attempting to eliminate undefined behavior – a Sisyphean task given the prevalence of C – to embracing its inevitability. Research should concentrate on runtime monitoring and containment strategies, accepting that memory errors will occur, and building systems resilient enough to withstand them. The goal isn’t a perfect machine, but one that fails gracefully, and predictably.

Ultimately, this work reinforces a familiar pattern: systems aren’t built, they grow. The boundaries of safety aren’t drawn with code, but negotiated with chaos. Order is just a temporary cache between failures, and each attempt to formalize it merely reveals the shape of the next, inevitable collapse. The true measure of progress isn’t the absence of bugs, but the speed with which a system can recover from them.

Original article: https://arxiv.org/pdf/2603.05645.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/