Abstract To address the issue of overly heavy computational load of extreme learning machine(ELM) in the big data environment, parallel optimization for ELM is studied. A multi-partition relaxed alternating direction method of multipliers(ADMM) for regularized ELM along with two scalarwise implementations in the N- and N/2-equipartition cases is proposed. By the multi-partition, the proposed algorithm has a highly parallel structure and the combination with relaxation technique improves the convergence rate of the proposed algorithm. Through analysis, a necessary and sufficient convergence condition is established, and optimal convergence ratio and optimal parameters are obtained. Through simulations on bench-mark datasets, the relationship between the convergence ratio and the number of partitioned blocks is calculated, and convergence rates and GPU acceleration ratios of different algorithms are compared. Experimental results demonstrate that the proposed algorithm has low computational complexity and high parallelism.
Fund:Supported by National Natural Science Foundation of China(No.61573123,61503104,U1909209)
Corresponding Authors:
LAI Xiaoping, Ph.D., professor. His research interests include optimization method, machine learning and digital filter design.
About author:: ZHANG Lijia, master student. His resear-ch interests include machine learning.Cao Jiuwen, Ph.D., professor. His research interests include machine learning, neural networks and intelligent information processing.
[1] HUANG G B, ZHU Q Y, SIEW C K. Extreme Learning Machine: Theory and Applications. Neurocomputing, 2006, 70(1/2/3): 489-501. [2] HUANG G B, ZHOU H M, DING X J, et al. Extreme Learning Machine for Regression and Multiclass Classification. IEEE Transactions on Systems, Man, and Cybernetics(Cybernetics), 2012, 42(2): 513-529. [3] CHYZHYK D, SAVIO A, GRAÑA M. Computer Aided Diagnosis of Schizophrenia on Resting State fMRI Data by Ensembles of ELM. Neural Networks, 2015, 68: 23-33. [4] CAO J W, ZHANG K, LUO M X, et al. Extreme Learning Machine and Adaptive Sparse Representation for Image Classification. Neural Networks, 2016, 81: 91-102. [5] HUANG Z Y, YU Y L, GU J, et al. An Efficient Method for Traffic Sign Recognition Based on Extreme Learning Machine. IEEE Transactions on Cybernetics, 2017, 47(4): 920-933. [6] 孙 锐,张东东,高 隽.基于分层极限学习机和局部稀疏模型的视觉跟踪算法.模式识别与人工智能, 2017, 30(4): 302-313. (SUN R, ZHANG D D, GAO J. Visual Tracking via Hierarchical Extreme Learning Machine and Local Sparse Model. Pattern Recognition and Artificial Intelligence, 2017, 30(4): 302-313.) [7] 刘阳阳,张 骏,高欣健,等.基于卷积递归神经网络和核超限学习机的3D目标识别.模式识别与人工智能, 2017, 30(12): 1091-1099. (LIU Y Y, ZHANG J, GAO X J, et al. 3D Object Recognition via Convolutional-Recursive Neural Network and Kernel Extreme Lear-ning Machine. Pattern Recognition and Artificial Intelligence, 2017, 30(12): 1091-1099.) [8] 刘天亮,陈克虎,戴修斌,等.融合RGB-D 多模特征和半径边缘约束超限学习的动作识别.模式识别与人工智能, 2018, 31(10): 958-964. (LIU T L, CHEN K H, DAI X B, et al. Action Recognition Fused with RGB-D Multi-modal Features and Radius-Margin Bounded Extreme Learning. Pattern Recognition and Artificial Intelligence, 2018, 31(10): 958-964.) [9] TANG J X, DENG C W, HUANG G B. Extreme Learning Machine for Multilayer Perceptron. IEEE Transactions on Neural Networks and Learning Systems, 2016, 27(4): 809-821. [10] CAO J W, ZHANG K, YONG H W, et al. Extreme Learning Machine with Affine Transformation Inputs in an Activation Function. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(7): 2093-2107. [11] HE Q, SHANG T F, ZHUANG F Z, et al. Parallel Extreme Lear-ning Machine for Regression Based on MapReduce. Neurocomputing, 2013, 102: 52-58. [12] BI X, ZHAO X G, WANG G R, et al. Distributed Extreme Lear-ning Machine with Kernels Based on MapReduce. Neurocompu-ting, 2015, 149: 456-463. [13] WANG Y Q, DOU Y, LIU X W, et al. PR-ELM: Parallel Regularized Extreme Learning Machine Based on Cluster. Neurocompu-ting, 2016, 173: 1073-1081. [14] 刘 鹏,王学奎,黄宜华,等.基于Spark的极限学习机算法并行化研究.计算机科学, 2017, 44(12): 33-37. (LIU P, WANG X K, HUANG Y H, et al. Study of ELM Algorithm Parallelization Based on Spark. Computer Science, 2017, 44(12): 33-37.) [15] DUAN M X, LI K L, LIAO X K, et al. A Parallel Multiclassification Algorithm for Big Data Using an Extreme Learning Machine. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(6): 2337-2351. [16] CEVHER V, BECKER S, SCHMIDT M. Convex Optimization for Big Data: Scalable, Randomized, and Parallel Algorithms for Big Data Analytics. IEEE Signal Processing Magazine, 2014, 31(5): 32-43. [17] SLAVAKIS K, GIANNAKIS G B, MATEOS G. Modeling and Optimization for Big Data Analytics: (Statistical) Learning Tools for Our Era of Data Deluge. IEEE Signal Processing Magazine, 2014, 31(5): 18-31. [18] 何 清,李 宁,罗文娟,等.大数据下的机器学习算法综述.模式识别与人工智能, 2014, 27(4): 327-336. (HE Q, LI N, LUO W J, et al. A Survey of Machine Learning Algorithms for Big Data. Pattern Recognition and Artificial Intelligence, 2014, 27(4): 327-336.) [19] BERTSEKAS D P. Convex Optimization Algorithms. Belmont, USA: Athena Scientific, 2015. [20] BOYD S, PARIKH N, CHU E, et al. Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. Foundations and Trends in Machine Learning, 2011, 3(1): 1-122. [21] LUO M N, ZHANG L L, LIU J, et al. Distributed Extreme Lear-ning Machine with Alternating Direction Method of Multiplier. Neurocomputing, 2017, 261: 164-170. [22] WANG H, FENG R B, HAN Z F, et al. ADMM-Based Algorithm for Training Fault Tolerant RBF Networks and Selecting Centers. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(8): 3870-3878. [23] LAI X P, CAO J W, HUANG X F, et al. A Maximally Split and Relaxed ADMM for Regularized Extreme Learning Machines [J/OL]. [2019-06-20]. http://ieeexplore.ieee.org/document/8789676. [24] ANTSAKLIS P J, MICHEL A N. A Linear Systems Primer. Boston, USA: Birkhuser, 2007.
2019, Vol. 32 
