What I Learned After Building a Programming Language From Scratch in Python

After building an entire programming language from scratch using Python, the author gained valuable insights into the inner workings of language implementation. The journey involved creating a pipeline that transforms NexPro source code (.pa files) into executable output, passing through multiple stages such as lexer, parser, abstract syntax tree (AST), interpreter, runtime, and finally the output.

Firstly, the project structure was analyzed, revealing a clear separation between different components responsible for specific tasks. The lexer converts source text into tokens, the parser builds the program structure, AST represents syntax as nodes, interpreter executes the AST, runtime handles runtime behavior/state, and errors manage language-level error handling.

This separation is essential for maintaining language implementations, as mixing lexing, parsing, execution, and runtime logic can lead to difficulties in maintaining the code.

Next, the author demonstrated that NexPro actually executes .pa programs by providing a CLI command. They showcased how simple programs like "say Hello NexPro!" result in the output "Hello NexPro!", and variables can be assigned and used within the program, producing output based on the assigned values. This evidence confirmed that the language pipeline functions as intended, transforming input code into executable output.

The author also delved into the importance of tokens before execution. They explained that a language implementation must first break the source code into meaningful pieces (IDENTIFIER, NUMBER, STRING, etc.) before proceeding with parsing. This step is crucial for understanding the program's structure and enabling proper interpretation.

The AST emerged as a fundamental concept, illustrating how the parser transforms relationships and precedence into a structured representation. For instance, the expression "a = 10 + 20" is not treated as a single string but rather as a tree-like structure with nodes representing the assignment and binary operation. This representation allows the interpreter to execute the expression more effectively by understanding the relationships between nodes and evaluating them correctly.

Finally, the interpreter's role in executing the language became clear. The author emphasized that the interpreter must understand various language elements such as numbers, strings, variables, assignments, binary expressions, and even specific commands like "say". By resolving expressions and evaluating them based on the AST, the interpreter brings the programmer's intent to life, producing the expected output.

Through this hands-on experience, the author gained a deep understanding of the complexities involved in implementing a programming language. They learned that tokenization, parsing, and interpreting are just the beginning, and the real magic happens in the interpreter, where the language becomes executable. This project not only provided practical knowledge but also highlighted the importance of separating concerns and structuring components to create maintainable and efficient language implementations.