The Journey Continues: Creating My Own Image Processing Library

Introduction

Throughout my studies in Machine Learning, I studied the concept of Data Augmentation, a technique designed to enhance the generalization ability of Neural Networks when dealing with image-based datasets. At the time, I believed that a Julia library existed specifically for DA, and later discovered Augmentor .jl. However, I wanted to determine if I could independently implement at least one image processing operation without relying on external assistance, namely image rotation.

One afternoon during my work at a manufacturing job, I began visualizing matrices representing images in my mind, and after approximately 2.5 hours of contemplation, I felt that I had successfully figured out how to digitally rotate an image by any arbitrary degree. I returned home, coded the solution in Julia, and was pleasantly surprised to discover that it worked.

Although the initial performance was quite slow (around 1000 ms to rotate a 552x736 grayscale image) and memory usage was substantial, I eventually managed to reduce the time to around 80 ms for the same operation. This performance disparity led me to question why Augmentor's image rotation function was significantly faster than my own method.

After some research, I stumbled upon a writeup about backwards warping, also known as inverse mapping, which explained why interpolation is often used in image processing operations. Understanding this concept finally made the connection between interpolation and its application in image rotation. Consequently, I abandoned my initial approach and adopted the inverse mapping + bilinear interpolation method, which proved to be far more efficient and memory-friendly.

In order to fully emulate Augmentor's capabilities, I decided to implement additional image processing operations in my own library, including distortion filtering, flipping, rotation, scaling, shearing, and zooming. After implementing these functions both on the CPU and GPU, I had created my very own image processing library tailored for use in Machine Learning and other applications. Comparison with Augmentor

While Augmentor is a well-designed and well-organized library, I have identified several areas where my library outperforms it and where there is room for improvement. Firstly, Augmentor hasn't been updated since 2022, which presents compatibility issues when adding it to a Julia work environment due to external library dependencies that often get removed and downgraded.

Additionally, using Augmentor causes errors in the Plots package on my system, rendering it unusable. My library, on the other hand, has been actively developed and does not suffer from these compatibility problems. Another notable difference between the two libraries is the requirement for pre-allocation of output image sizes in Augmentor's batch functions.

My library, however, dynamically determines the size of the output images within the functions, similar to my single-image functions. Furthermore, while Augmentor offers only one Gaussian blur filtering option, my library allows users to input any convolution kernel, enabling a wider range of filtering operations. In terms of multithreading, Augmentor only parallelizes its Gaussian blur function, whereas all of my single-image functions are designed to be multithreaded.

Lastly, Augmentor lacks the ability to chain various image manipulation operations together, which is a feature currently under development in my library. Notwithstanding these strengths, my library does have some limitations compared to Augmentor. For instance, it provides less flexibility in cropping options, and Augmentor includes more advanced features like a greater variety of random variations in rotation and other relevant functions.

Nevertheless, these shortcomings can be addressed before publishing my library to ensure it meets the expectations of potential users.