Predicting the Financial Health of Small Businesses with Python

Small businesses play a significant role in many African economies, yet entrepreneurs often face challenges when seeking loans, insurance and other financial services. To address this issue, a project was initiated to predict the financial health of small businesses based on their survey responses. The data used in this project came from the Financial Health Prediction Challenge on Zindi, and the objective was to determine whether a business had low, medium or high financial health.

The dataset consisted of 9,618 small business owners from Eswatini, Lesotho, Malawi and Zimbabwe, with 39 columns each. These questions covered various aspects of the business and its owner, including age, gender, turnover, age, financial products used, money handling practices, and future outlook. Each business was classified into one of three groups using a Financial Health Index, which takes into account savings and assets, debt and repayment ability, resilience to shocks, and access to credit.

Before diving into the data, it was crucial to understand the information at hand. The dataset included a variable definitions sheet that clarified the purpose of each column. After reviewing this documentation, basic data checks were performed. These checks included verifying the dataset's dimensions, identifying duplicate rows, counting missing values, and assessing the distribution of the data.

The dataset was divided into categorical (text) columns and numerical columns, as different data types often require distinct solutions. Categorical columns were addressed first, with extra spaces and capital letters being trimmed and all text converted to lowercase. However, it was discovered that some categorical columns contained unique apostrophe characters and variations in answering "don't know."

Additionally, there were hidden invisible characters and unexpected 0 values. These inconsistencies were addressed by merging similar entries into a single spelling.

Moving on to numerical columns, it was observed that several columns had extreme outliers that significantly deviated from the rest of the data. Some columns also contained missing values. To tackle these issues, the data was split into categorical and numerical groups.

In the case of categorical columns, missing values were handled based on specific criteria. Columns with missing data exceeding 35% were treated differently than the rest. A total of twelve columns exceeded this threshold, with missing answers ranging from 37% to 47% across the countries. Upon further investigation, it was found that Lesotho did not have any responses for eleven of the categorical columns, such as gender, credit card ownership, and insurance. This was likely due to the fact that these questions were not asked in that particular country.

For training data, columns with more than 35% missing values were categorized as 'unknown', given that the absence of an answer could also provide valuable information. Six other categorical columns had minimal missing data, so they were filled with the mode (the most frequently occurring value). The decision to apply the 35% cut-off was a personal judgment and not a predetermined rule. This approach was applied within each country instead of uniformly across all countries to account for regional variations.

The most challenging aspect of the project was handling monetary data, as income, expenses, and turnover were recorded in each country's local currency. To standardize currency, a function was created to look up each row's country, retrieve the corresponding exchange rate, and convert the monetary values to US dollars.