SlimML: Removing Non- critical Input Data in Large-scale Iterative Machine Learning

Rui Han, Chi Harold Liu, Shilin Li, Lydia Chen, Guoren Wang, Jian Tang, Jieping Ye

Research output: Contribution to journalArticleScientificpeer-review

6 Downloads (Pure)

Abstract

The core of many large-scale machine learning (ML) applications, such as neural networks (NN), support vector machine (SVM), and convolutional neural network (CNN), is the training algorithm that iteratively updates model parameters by processing massive datasets. From a plethora of studies aiming at accelerating ML, being data parallelization and parameter server, the prevalent assumption is that all data points are equivalently relevant to model parameter updating. In this paper, we challenge this assumption by proposing a criterion to measure a data point’s effect on model parameter updating, and experimentally demonstrate that the majority of data points are non-critical in the training process. We develop a slim learning framework, termed SlimML, which trains the ML models only on the critical data and thus significantly improves training performance. To such an end, SlimML efficiently leverages a small number of aggregated data points per iteration to approximate the criticalness of original input data instances. The proposed approach can be used by changing a few lines of code in a standard stochastic gradient descent (SGD) procedure, and we demonstrate experimentally, on NN regression, SVM classification, and CNN training, that for large datasets, it accelerates model training process by an average of 3.61 times while only incurring accuracy losses of 0.37%.
Original languageEnglish
Number of pages15
JournalIEEE Transactions on Knowledge and Data Engineering
DOIs
Publication statusE-pub ahead of print - Apr 2019

Keywords

  • Iterative machine learning
  • large input datasets
  • model parameter updating
  • MapReduce

Fingerprint Dive into the research topics of 'SlimML: Removing Non- critical Input Data in Large-scale Iterative Machine Learning'. Together they form a unique fingerprint.

Cite this