Published in 2017 and preserved substantially as written; no longer maintained.

P2P Lending Case Study

Read the summary

The full case study is laid out for larger screens. On a desktop or large tablet you will find every chart and the complete commentary.

Big Data in Peer-to-peer lending

1.6 million loans, 150 data fields 30 million rows of time series data The Gap Can We Do Better?
  • The small balance, consumer credit orientation of peer-to-peer lending leads to the issuance of a large volume of loans.
  • The abundance of relatively homogenized loan data lends itself very well to statistical/quantitative analysis.
  • We use state-of-the-art techniques to clean, analyze and visualize the data before getting it into machine learning-ready shape.
  • Perhaps unsurprisingly, as with all machine learning tasks, this stage makes up more than 50% of the total effort.
  • Since the beginning of the decade, the P2P platform we analyze has issued more than 1.6 million loans.
  • Between application data from the borrower and data pulled from credit bureaus and other third parties, we have access to more than 150 fields of data per loan.
  • We also have information on the platform's own risk assessment of the loan.
  • We have nearly 30 million rows of time series data detailing month-by-month payment history of every loan.
  • Memory (RAM) management is essential when dealing with such large datasets.
  • This extensive dataset can be used to compute return generated (total return, IRR etc.) on a per loan basis or after applying different aggregations (by risk grade, vintage etc.).
  • This dataset is key to mapping machine learning output to optimal portfolio construction.
  • As with all real world data, the loan data is messy and requires significant effort to clean.
  • To achieve this goal, we use sophisticated Python libraries that implement fast, vectorized operations in lower level languages (C, C++).
  • We combine multiple datasets together to create a varied set of features per loan to feed our algorithms. This requires extensive clean-up and mapping to make the datasets "talk" to each other.
  • Once the data has been cleaned, it needs to be pre-processed to bring it to a machine learning-ready state.
  • We use the Scikit Learn and Pandas libraries in Python for this task.
  • Among others, this step includes the creation of ordinal scales and pre-processing pipelines along with feature scaling, etc.
  • Data needs to be split into training, cross-validation and test sets. Learning is performed on the training data while hyperparameters are tuned using the cross-validation set. The (out of sample) test set is used for the evaluation of model efficacy.