Local processing of massive databases with R: a national analysis of a Brazilian social programme

Paz, Hellen, Maia, Mateus, Moraes, Fernando, Lustosa, Ricardo, Costa, Lilia, Macêdo, Samuel, Barreto, Marcos E. ORCID: 0000-0002-7818-1855 and Ara, Anderson (2020) Local processing of massive databases with R: a national analysis of a Brazilian social programme. Stats, 3 (4). 444 - 464. ISSN 2571-905X

Text (Barreto_local-processing-of-massive-databases--published) - Published Version Available under License Creative Commons Attribution. Download (910kB)

• Author

Abstract

The analysis of massive databases is a key issue for most applications today and the use of parallel computing techniques is one of the suitable approaches for that. Apache Spark is a widely employed tool within this context, aiming at processing large amounts of data in a distributed way. For the Statistics community, R is one of the preferred tools. Despite its growth in the last years, it still has limitations for processing large volumes of data in single local machines. In general, the data analysis community has difficulty to handle a massive amount of data on local machines, often requiring high-performance computing servers. One way to perform statistical analyzes over massive databases is combining both tools (Spark and R) via the sparklyr package, which allows for an R application to use Spark. This paper presents an analysis of Brazilian public data from the Bolsa Família Programme (BFP—conditional cash transfer), comprising a large data set with 1.26 billion observations. Our goal was to understand how this social program acts in different cities, as well as to identify potentially important variables reflecting its utilization rate. Statistical modeling was performed using random forest to predict the utilization rated of BFP. Variable selection was performed through a recent method based on the importance and interpretation of variables in the random forest model. Among the 89 variables initially considered, the final model presented a high predictive performance capacity with 17 selected variables, as well as indicated high importance of some variables for the observed utilization rate in income, education, job informality, and inactive youth, namely: family income, education, occupation and density of people in the homes. In this work, using a local machine, we highlighted the potential of aggregating Spark and R for analysis of a large database of 111.6 GB. This can serve as proof of concept or reference for other similar works within the Statistics community, as well as our case study can provide important evidence for further analysis of this important social support programme.

Item Type:	Article
Official URL:	https://www.mdpi.com/journal/stats
Additional Information:	© 2020 The Authors
Divisions:	Statistics
Subjects:	Q Science > QA Mathematics > QA76 Computer software H Social Sciences > HA Statistics
Date Deposited:	03 Aug 2022 07:51
Last Modified:	01 Apr 2024 08:36
URI:	http://eprints.lse.ac.uk/id/eprint/115770

Actions (login required)

View Item