I tested my sandbox against Deno and plain Python on 63 AI-written scripts
I've been building a language where a function's signature declares which effects it may perform, and a runtime refuses anything outside a budget you grant. The obvious question is whether that catches anything real, so I built a benchmark against the alternatives. The setup 63 programs — 56 dangerous, 7 harmless controls — each written three times in Velaris, Python and JavaScript, doing the…
A programming language named Velaris, which enforces function signatures declaring potential effects, was compared to Deno and plain Python using 63 AI-generated scripts. Of these, 56 were dangerous programs and 7 were harmless controls, each appearing three times in Velaris, Python, and JavaScript. The purpose was to test whether the language's approach could catch real issues.
The 63 scripts were divided into 11 categories, including file writes hidden in helpers, network calls within helpers, division by user input, off-by-one reads, integer overflows, ignored failures, infinite loops, runaway memory usage, dangerous module reaches, scoped-budget escapes, and control programs. All tools ran under the minimal budget needed for their tasks. Velaris was configured with specific permissions, Deno with matching allowances, and Python without any budget restrictions.
Velaris demonstrated effectiveness in 6 out of 11 categories, catching integer overflow errors, where whole numbers are 64-bit and arithmetic operations trigger a stop if they exceed the limit. Python's and JavaScript's handling of integers differed, with JavaScript rounding to doubles, resulting in no errors being printed. Conversely, Deno and Python failed to detect any issues, despite having stricter permission models.
Four programs were only caught during runtime, while the remaining seven were flagged before execution by a termination rule that identifies loops without a counter approaching a fixed limit. However, this was merely an observation, not a proof.
One scenario where Velaris fell short involved programs computing incorrect answers without providing any contract or verification. Another case involved the program printing a dangerous command like "rm -rf /," but the language did not execute the command. Instead, the tool would flag any program that prints text resembling a command, making it challenging to flag specific instances.
Additionally, seven control programs were included to penalize tools that incorrectly identify harmless code patterns. The experiments were conducted using the Velaris language, and the results showed consistent output across ten runs, with minor normalizations made to account for ephemeral ports, V8 crash messages, and Python's MemoryError display.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
