Metacarp

For the past few months, the author has been developing a new compiler for the Carp programming language. This compiler, named Metacarp, is written in Carp itself and compiles Carp programs. The reference implementation of Carp, written in Haskell, has been used as a baseline for the development of Metacarp. Since then, the language and compiler have evolved, with many libraries available for various needs.

Metacarp aims to be a second implementation of the reference language. It compiles the reference suite, compiles itself, and finally compiles itself again into byte-identical C code. Metacarp is composed of independent libraries for different compiler phases, and it has a second LLVM backend. The compiler can keep sessions alive between inputs and supports incremental compilation using a Just-In-Time (JIT) compiler.

However, it is not yet a drop-in replacement for the original compiler, as its diagnostics are different, some parts are unstable, and there are still bugs. The author provides an example of how to use Metacarp to compile a simple Carp program, which produces a C translation unit. The compiler understands the annoying aspects of implementing Carp, such as compile-time language and macros, Hindley-Milner type inference, interfaces, monomorphization, pattern matching, closures, and ownership and borrow rules.

It derives copy and delete functions for managed values and inserts calls to them before producing C code. The compiler can also load existing Core and compile existing programs, and it works with the Carpentry libraries. While it is not 100% compatible with the original compiler yet, the differences are mostly in edge cases. Metacarp is written in around 42,000 lines of Carp code, divided into libraries that implement individual compiler phases.

The fun part is that these libraries have their own data models, entry points, tests, and errors, and can be used independently. The author introduces a carp-session library that keeps a compiler alive and allows for a notebook-style coding experience with Metacarp. A session loads, expands, resolves, derives, and infers Core once and keeps a set of definitions provided by the client.

Cells in a notebook can be checked transiently against the session without becoming part of it. The client can commit, replace, or remove definitions, and when a definition changes, the session rebuilds the affected views. This stateful compilation system helps with notebook-style code behavior. Implementing this stateful compilation system proved to be challenging, but once fixed, compiling one cell against Metacarp itself took about thirty seconds, and caching the immutable half reduced compilation time to about two hundred milliseconds on the same machine.