Faster Distributed Machine Learning for Free

If one computer can process a set of data in 10 hours, it would be reasonable to assume that 10 computers could process that data in one hour. The research of Xiaorui Liu, an MSU graduate student in the Department of Computer Science and Engineering, reveals that this is not the case. The bottleneck for achieving this speedup comes from the slow process of transmitting information between the devices.

In the big data era, the availability of huge volumes of data significantly improves the learning capability of machine learning models, while also bringing tremendous challenges in designing efficient and effective learning algorithms. Xiaorui works to reconcile this dilemma in the Data Science and Engineering (DSE) Lab led by Dr. Jiliang Tang and the Optimization Lab led by Dr. Ming Yan in the Department of Computational Mathematics, Science and Engineering. His research interests focus on distributed machine learning to reduce the time required to transmit information between machines. The distribution of tasks across machines can utilize the computational power of massive computing devices to speed up processes.

To reduce the bottleneck of communication between machines, Xiaorui developed two strategies: communication compression and decentralization. Communication compression reduces the data communication bits by over 95% during transmission. Decentralization enables distributed learning algorithms with only local communication between directly connected machines. This makes the learning algorithm more flexible and can reduce the cost of communication between devices, in comparison to classic centralized learning algorithms where all machines need to exchange information with the central server. Furthermore, Xiaorui shows that with refined algorithm design, these two strategies can coexist, which brings remarkable speedup. His work enables faster distributed machine learning for free, without reducing the effectiveness of machine learning tasks.

Xiaorui notes that the computational resources and services provided by ICER help him to quickly prototype his research ideas, flexibly explore different algorithm designs, and finally test his algorithms in various environment configurations in a reliable way. With these resources, he has prominently advanced the frontiers of distributed machine learning and published numerous articles at top-tier machine learning conferences.

This research is non-specific and can therefore be applied to a large class of problems in scientific computing. Communication compression and decentralization can reduce energy consumption for supercomputing systems and accelerate the development of new research ideas. The positive impact this work can have on society is the reason Xiaorui is so passionate about his research. He wants to ultimately apply it in industrial practice, and he continues to pioneer creative research directions in academia. This work will contribute to the development of the science and technology communities.