Big Data in Peer-to-peer lending

The small balance, consumer credit orientation of peer-to-peer lending leads to the issuance of a large volume of loans.
The abundance of relatively homogenized loan data lends itself very well to statistical/quantitative analysis.
We use state-of-the-art techniques to clean, analyze and visualize the data before getting it into machine learning-ready shape.
Perhaps unsurprisingly, as with all machine learning tasks, this stage makes up more than 50% of the total effort.

Since the beginning of the decade, the P2P platform we analyze has issued more than 1.6 million loans.
Between application data from the borrower and data pulled from credit bureaus and other third parties, we have access to more than 150 fields of data per loan.
We also have information on the platform's own risk assessment of the loan.

We have nearly 30 million rows of time series data detailing month-by-month payment history of every loan.
Memory (RAM) management is essential when dealing with such large datasets.
This extensive dataset can be used to compute return generated (total return, IRR etc.) on a per loan basis or after applying different aggregations (by risk grade, vintage etc.).
This dataset is key to mapping machine learning output to optimal portfolio construction.

As with all real world data, the loan data is messy and requires significant effort to clean.
To achieve this goal, we use sophisticated Python libraries that implement fast, vectorized operations in lower level languages (C, C++).
We combine multiple datasets together to create a varied set of features per loan to feed our algorithms. This requires extensive clean-up and mapping to make the datasets "talk" to each other.

Once the data has been cleaned, it needs to be pre-processed to bring it to a machine learning-ready state.
We use the Scikit Learn and Pandas libraries in Python for this task.
Among others, this step includes the creation of ordinal scales and pre-processing pipelines along with feature scaling, etc.
Data needs to be split into training, cross-validation and test sets. Learning is performed on the training data while hyperparameters are tuned using the cross-validation set. The (out of sample) test set is used for the evaluation of model efficacy.

This section is only viewable on larger screen sizes. Please use a tablet or bigger. For best experience, please view on a desktop computer and use Google Chrome browser.