Large-scale Data Classification based on K-means Clustering and Deep Learning

Abstract

Common problems in classifying large data are revealed as long processing time and a lot of training data in order to maintain high accuracy. To solve these problems, researchers study methods for classifying large data to reduce the use of large amounts of training data without sacrificing high classification efficiency. The proposed method reduces the size of the training data by combining K-means and deep learning. To study the effectiveness of the proposed method, the accuracy and AUC values were determined. In addition, it was compared with the original deep learning method using 80% and 90% training data out of the total data and was compared with the original deep learning using the same amount of training data. The results show that the proposed method can significantly reduce the size of the training data. Less than 1% of the total data size was used as training data, but the method yielded the high average percent of accuracy and the high average AUC of the classification. In the case of normal distribution and the size is 1,000,000 × 5 (N × Feature), the proposed method exhibits the average percent of accuracy as high as 97.4878% and the average AUC as 0.9735. When the proposed method was compared with the deep learning method using training data about 80% and 90% of the total data size, classification efficiency was relatively as high as that of the deep learning, but the classification time was 2–4 times less than the processing time of the deep learning method.