Heyang Qin

SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement

Large scale training requires massive parallelism to finish the training within a reasonable amount of time. To support massive parallelism, large batch training is the key enabler but often at the cost of generalization performance. We propose a fully automated and lightweight adaptive batching methodology to enable fine-grained batch size adaption (e.g., at a mini-batch level) that can achieve state-of-the-art performance with record breaking batch sizes. The core component of our method is a lightweight yet efficient representation of the critical gradient noise information. Extensive evaluations on popular benchmarks (e.g., CIFAR10, ImageNet, and BERT-Large) demonstrate that the proposed methodology outperforms state-of-the-art methodologies using adaptive batching approaches or hand-tuned static strategies in both performance and batch size. Particularly, we achieve a new state-of-the-art batch size of 78k in BERT-Large pretraining with SQuAD score 90.69 compared to 90.58 reported in previous state-of-the-art with 59k batch size.

Region Based Reinforcement Learning Scheduling Framework for MLaaS

The parallelism settings in Machine Learning as a Service (MLaaS) have critical impact on the system performance. It is a challenge to tune the parallel config because of the complex dependency and large search space. We propose a region based reinforcement learning (RRL) approach that can converge to near-optimal config magnitude faster than the traditional reinforcement learning. The proposed RRL is prototyped and evaluated using several real-world machine learning workloads. Both theoretical analysis and experiment evaluation show that RRL outperforms state-of-the-art tuning algorithms for MLaaS.

RRL Plus: Adaptive Region Based Reinforcement Learning for Machine Learning

RRL is sensitive to region size. Excessive region size leads to large performance gap between RRL solution and optimal one whereas inadequate region size leads to longer learning process. We further expand the Region Based Reinforcement Learning algorithm by Bayesian optimization and heuristic algorithm and enable it to automatically adjust the region size to achieve fast converge and near-optimal solution.

Semi-supervised Learning for Large Scale Noisy Data

A large amount of data is available for scientific use. Unfortunately, accurate labels of training data are usually manually-labeled and expensive, leading to insufficient labeled data to train machine learning model. We solve this problem by using generative model to resemble multiple state-of-the-art models to achieve better detection from very noisy training data.