When Python is Too Slow

Python is a favored language for Agile development due to its flexibility and adaptability to changing requirements. However, its interpreted nature may not always deliver the performance needed in certain situations. To address this, developers often employ python-to-binary bindings, utilizing programming languages like Rust, C++, or Go to build binary code that can be integrated with Python. This article focuses on bindings to Rust-based code.

The binding process involves creating a module with domain-specific functions in Rust, then building it as a C-compatible dynamic library. A Python wrapper is also constructed as a Python package, installed alongside the dynamic library, enabling the import and use of functions that call the corresponding Rust functions. Complex parameters may require serialization as JSON strings or individual primitive parameters during this process.

To demonstrate the effectiveness of Rust bindings, an experiment was conducted that read a large CSV file and generated a new file with duplicate records removed based on column indexes. The test CSV file contained data about European NGOs and was 3 MB in size. Using Rust bindings, the processing time was 4.3 times faster compared to running the same task directly in Python. The source code for this experiment is available in a public repository.

Rust is a modern programming language that offers several advantages over Python. Packages are built with Cargo, the all-in-one build tool for Rust, akin to pip, virtualenv, and setuptools combined. Rust packages are published to crates.io, similar to PyPI. To create a Python-to-Rust binding, the PyO3 library and maturin build tool are used.

While Rust's syntax may be less developer-friendly than Python or Go, AI agents can help in converting Python-native code to Rust relatively easily. Additionally, Rust and Go provide memory management benefits compared to C++, eliminating the need for manual memory management and pointer handling.

In a Django web application context, the main bottlenecks typically lie in database, file system, object storage, and API connections. Rust can prove beneficial when processing large amounts of data or performing heavy computations. Libraries like orjson and Django ORM renderers like django-orjson or drf-orjson-renderer showcase the performance boost Rust can offer in JSON building and parsing, which is roughly 2-10 times faster than a Python-native implementation.

Potential use cases for Rust replacements in Django include background tasks, management commands, template parsing, and occasionally views or middleware.