Shaohuai Shi

Publication Activity (10 Years)

Years Active: 2010-2024
Publications (10 Years): 76

Top Topics

Disparity Estimation

Top Venues

Publications

Jing Peng, Zihan Li, Shaohuai Shi, Bo Li
Sparse Gradient Communication with AlltoAll for Accelerating Distributed Deep Learning. ICPP (2024)
Xinglin Pan, Wenxiang Lin, Shaohuai Shi, Xiaowen Chu, Weinong Sun, Bo Li
Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules. INFOCOM (2024)
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xinmei Tian, Tongliang Liu, Bo Han, Xiaowen Chu
FedImpro: Measuring and Improving Client Update in Federated Learning. CoRR (2024)
Xinglin Pan, Wenxiang Lin, Shaohuai Shi, Xiaowen Chu, Weinong Sun, Bo Li
Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules. CoRR (2024)
Zichen Tang, Junlin Huang, Rudan Yan, Yuxin Wang, Zhenheng Tang, Shaohuai Shi, Amelie Chi Zhou, Xiaowen Chu
Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning. ICPP (2024)
Yizhou Luo, Qiang Wang, Shaohuai Shi, Jiaxin Lai, Shuhan Qi, Jiajia Zhang, Xuan Wang
Scheduling Deep Learning Jobs in Multi-Tenant GPU Clusters via Wise Resource Sharing. CoRR (2024)
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xinmei Tian, Tongliang Liu, Bo Han, Xiaowen Chu
FedImpro: Measuring and Improving Client Update in Federated Learning. ICLR (2024)
Hucheng Liu, Shaohuai Shi, Xuan Wang, Zoe L. Jiang, Qian Chen
Performance Analysis and Optimizations of Matrix Multiplications on ARMv8 Processors. DATE (2024)
Shaohuai Shi, Xinglin Pan, Qiang Wang, Chengjian Liu, Xiaozhe Ren, Zhongzhe Hu, Yu Yang, Bo Li, Xiaowen Chu
ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling. EuroSys (2024)
Longteng Zhang, Xiang Liu, Zeyu Li, Xinglin Pan, Peijie Dong, Ruibo Fan, Rui Guo, Xin Wang, Qiong Luo, Shaohuai Shi, Xiaowen Chu
Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models. CoRR (2023)
Lin Zhang, Shaohuai Shi, Bo Li
Eva: A General Vectorized Approximation Framework for Second-order Optimization. CoRR (2023)
Lin Zhang, Shaohuai Shi, Xiaowen Chu, Wei Wang, Bo Li, Chengjian Liu
DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining. ICDCS (2023)
Chen Qiu, Yulin Wu, Weixin Huang, Botao Liu, Shaohuai Shi, Xuan Wang
A Generic Multi-Player Transformation Algorithm for Solving Large-Scale Zero-Sum Extensive-Form Adversarial Team Games. CoRR (2023)
Lin Zhang, Shaohuai Shi, Bo Li
Eva: Practical Second-order Optimization with Kronecker-vectorized Approximation. ICLR (2023)
Zhenheng Tang, Shaohuai Shi, Bo Li, Xiaowen Chu
GossipFL: A Decentralized Federated Learning Framework With Sparsified and Adaptive Communication. IEEE Trans. Parallel Distributed Syst. 34 (3) (2023)
Lin Zhang, Shaohuai Shi, Xiaowen Chu, Wei Wang, Bo Li, Chengjian Liu
Decoupling the All-Reduce Primitive for Accelerating Distributed Deep Learning. CoRR (2023)
Lin Zhang, Shaohuai Shi, Bo Li
Accelerating Distributed K-FAC with Efficient Collective Communication and Scheduling. INFOCOM (2023)
Lin Zhang, Longteng Zhang, Shaohuai Shi, Xiaowen Chu, Bo Li
Evaluation and Optimization of Gradient Compression for Distributed Deep Learning. ICDCS (2023)
Yuxin Wang, Shaohuai Shi, Xin He, Zhenheng Tang, Xinglin Pan, Yang Zheng, Xiaoyu Wu, Amelie Chi Zhou, Bingsheng He, Xiaowen Chu
Reliable and Efficient In-Memory Fault Tolerance of Large Language Model Pretraining. CoRR (2023)
Lin Zhang, Longteng Zhang, Shaohuai Shi, Xiaowen Chu, Bo Li
Evaluation and Optimization of Gradient Compression for Distributed Deep Learning. CoRR (2023)
Zhenheng Tang, Yuxin Wang, Xin He, Longteng Zhang, Xinglin Pan, Qiang Wang, Rongfei Zeng, Kaiyong Zhao, Shaohuai Shi, Bingsheng He, Xiaowen Chu
FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs. CoRR (2023)
Zhenheng Tang, Xiaowen Chu, Ryan Yide Ran, Sunwoo Lee, Shaohuai Shi, Yonggang Zhang, Yuxin Wang, Alex Qiaozhong Liang, Salman Avestimehr, Chaoyang He
FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training. CoRR (2023)
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, Bo Li
LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning. CoRR (2023)
Shaohuai Shi, Xinglin Pan, Xiaowen Chu, Bo Li
PipeMoE: Accelerating Mixture-of-Experts through Adaptive Pipelining. INFOCOM (2023)
Lin Zhang, Shaohuai Shi, Wei Wang, Bo Li
Scalable K-FAC Training for Deep Neural Networks With Distributed Preconditioning. IEEE Trans. Cloud Comput. 11 (3) (2023)
Qiang Wang, Shaohuai Shi, Kaiyong Zhao, Xiaowen Chu
EASNet: Searching Elastic and Accurate Network Architecture for Stereo Matching. ECCV (32) (2022)
Lin Zhang, Shaohuai Shi, Wei Wang, Bo Li
Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning. CoRR (2022)
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, Xiaowen Chu
Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning. CoRR (2022)
Yang Xiang, Zhihua Wu, Weibao Gong, Siyu Ding, Xianjie Mo, Yuang Liu, Shuohuan Wang, Peng Liu, Yongshuai Hou, Long Li, Bin Wang, Shaohuai Shi, Yaqian Han, Yue Yu, Ge Li, Yu Sun, Yanjun Ma, Dianhai Yu
Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters. CoRR (2022)
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, Xiaowen Chu
Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning. ICML (2022)
Shaohuai Shi, Qing Yang, Yang Xiang, Shuhan Qi, Xuan Wang
An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learning. CoRR (2022)
Qiang Wang, Shaohuai Shi, Kaiyong Zhao, Xiaowen Chu
EASNet: Searching Elastic and Accurate Network Architecture for Stereo Matching. CoRR (2022)
Qiang Wang, Shaohuai Shi, Shizhen Zheng, Kaiyong Zhao, Xiaowen Chu
FADNet++: Real-Time and Accurate Disparity Estimation with Configurable Networks. CoRR (2021)
Shaohuai Shi, Xiaowen Chu, Bo Li
MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning. IEEE Trans. Parallel Distributed Syst. 32 (8) (2021)
Xin He, Shihao Wang, Xiaowen Chu, Shaohuai Shi, Jiangping Tang, Xin Liu, Chenggang Yan, Jiyong Zhang, Guiguang Ding
Automated Model Design and Benchmarking of Deep Learning Models for COVID-19 Detection with Chest CT Scans. AAAI (2021)
Shaohuai Shi, Lin Zhang, Bo Li
Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks. CoRR (2021)
Shaohuai Shi, Xianhao Zhou, Shutao Song, Xingyao Wang, Zilin Zhu, Xue Huang, Xinan Jiang, Feihu Zhou, Zhenyu Guo, Liqiang Xie, Rui Lan, Xianbin Ouyang, Yan Zhang, Jieqian Wei, Jing Gong, Weiliang Lin, Ping Gao, Peng Meng, Xiaomin Xu, Chenyang Guo, Bo Yang, Zhibo Chen, Yongjian Wu, Xiaowen Chu
Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters. MLSys (2021)
Shaohuai Shi, Lin Zhang, Bo Li
Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks. ICDCS (2021)
Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, Chengjian Liu, Wei Wang, Bo Li
A Quantitative Survey of Communication Optimizations in Distributed Deep Learning. IEEE Netw. 35 (3) (2021)
Xin He, Shihao Wang, Xiaowen Chu, Shaohuai Shi, Jiangping Tang, Xin Liu, Chenggang Yan, Jiyong Zhang, Guiguang Ding
Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scans. CoRR (2021)
Shaohuai Shi, Xiaowen Chu, Bo Li
Exploiting Simultaneous Communications to Accelerate Data Parallel Distributed Deep Learning. INFOCOM (2021)
Qiang Wang, Shaohuai Shi, Shizhen Zheng, Kaiyong Zhao, Xiaowen Chu
FADNet: A Fast and Accurate Network for Disparity Estimation. ICRA (2020)
Shaohuai Shi, Qiang Wang, Xiaowen Chu
Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format. ICPADS (2020)
Shaohuai Shi, Zhenheng Tang, Qiang Wang, Kaiyong Zhao, Xiaowen Chu
Layer-Wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees. ECAI (2020)
Shaohuai Shi, Xianhao Zhou, Shutao Song, Xingyao Wang, Zilin Zhu, Xue Huang, Xinan Jiang, Feihu Zhou, Zhenyu Guo, Liqiang Xie, Rui Lan, Xianbin Ouyang, Yan Zhang, Jieqian Wei, Jing Gong, Weiliang Lin, Ping Gao, Peng Meng, Xiaomin Xu, Chenyang Guo, Bo Yang, Zhibo Chen, Yongjian Wu, Xiaowen Chu
Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters. CoRR (2020)
Zhenheng Tang, Shaohuai Shi, Xiaowen Chu, Wei Wang, Bo Li
Communication-Efficient Distributed Deep Learning: A Comprehensive Survey. CoRR (2020)
Qiang Wang, Shaohuai Shi, Canhui Wang, Xiaowen Chu
Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs. CoRR (2020)
Shaohuai Shi, Qiang Wang, Xiaowen Chu, Bo Li, Yang Qin, Ruihao Liu, Xinxiao Zhao
Communication-Efficient Distributed Deep Learning with Merged Gradient Sparsification on GPUs. INFOCOM (2020)
Zhenheng Tang, Shaohuai Shi, Xiaowen Chu
Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection. CoRR (2020)
Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, Chengjian Liu, Wei Wang, Bo Li
Communication-Efficient Distributed Deep Learning: Survey, Evaluation, and Challenges. CoRR (2020)
Yuxin Wang, Qiang Wang, Shaohuai Shi, Xin He, Zhenheng Tang, Kaiyong Zhao, Xiaowen Chu
Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training. CCGRID (2020)
Shaohuai Shi, Qiang Wang, Xiaowen Chu
Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format. CoRR (2020)
Zhenheng Tang, Shaohuai Shi, Xiaowen Chu
Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection. ICDCS (2020)
Qiang Wang, Shaohuai Shi, Shizhen Zheng, Kaiyong Zhao, Xiaowen Chu
FADNet: A Fast and Accurate Network for Disparity Estimation. CoRR (2020)
Shaohuai Shi, Qiang Wang, Kaiyong Zhao, Zhenheng Tang, Yuxin Wang, Xiang Huang, Xiaowen Chu
A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networks. ICDCS (2019)
Xin He, Xiaoming Liu, Zhili Wu, Wu Yu, Xiaowen Chu, Shihao Wang, Shaohuai Shi, Zhenheng Tang, Yuxin Wang, Zhihao Zhao, Jing Dai, Ronghao Ni, Xiaofeng Zhang
Computer-Aided Clinical Skin Disease Diagnosis Using CNN and Object Detection Models. IEEE BigData (2019)
Shaohuai Shi, Zhenheng Tang, Qiang Wang, Kaiyong Zhao, Xiaowen Chu
Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees. CoRR (2019)
Xin He, Shihao Wang, Shaohuai Shi, Zhenheng Tang, Yuxin Wang, Zhihao Zhao, Jing Dai, Ronghao Ni, Xiaofeng Zhang, Xiaoming Liu, Zhili Wu, Wu Yu, Xiaowen Chu
Computer-Aided Clinical Skin Disease Diagnosis Using CNN and Object Detection Models. CoRR (2019)
Shaohuai Shi, Xiaowen Chu, Bo Li
MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning. CoRR (2019)
Shaohuai Shi, Kaiyong Zhao, Qiang Wang, Zhenheng Tang, Xiaowen Chu
A Convergence Analysis of Distributed SGD with Communication-Efficient Gradient Sparsification. IJCAI (2019)
Shaohuai Shi, Qiang Wang, Kaiyong Zhao, Zhenheng Tang, Yuxin Wang, Xiang Huang, Xiaowen Chu
A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networks. CoRR (2019)
Shaohuai Shi, Xiaowen Chu, Bo Li
MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms. INFOCOM (2019)
Shaohuai Shi, Xiaowen Chu, Ka Chun Cheung, Simon See
Understanding Top-k Sparsification in Distributed Deep Learning. CoRR (2019)
Shaohuai Shi, Qiang Wang, Xiaowen Chu, Bo Li
A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning. ICPADS (2018)
Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, Tiegang Chen, Guangxiao Hu, Shaohuai Shi, Xiaowen Chu
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes. CoRR (2018)
Shaohuai Shi, Xiaowen Chu
MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms. CoRR (2018)
Shaohuai Shi, Qiang Wang, Xiaowen Chu, Bo Li
Modeling and Evaluation of Synchronous Stochastic Gradient Descent in Distributed Deep Learning on Multiple GPUs. CoRR (2018)
Shaohuai Shi, Qiang Wang, Xiaowen Chu
Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs. DASC/PiCom/DataCom/CyberSciTech (2018)
Shaohuai Shi, Pengfei Xu, Xiaowen Chu
Improving the Performance of Fully Connected Neural Networks by Out-of-Place Matrix Transpose. CoRR (2017)
Pengfei Xu, Shaohuai Shi, Xiaowen Chu
Performance Evaluation of Deep Learning Tools in Docker Containers. CoRR (2017)
Shaohuai Shi, Xiaowen Chu
Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units. CoRR (2017)
Pengfei Xu, Shaohuai Shi, Xiaowen Chu
Performance Evaluation of Deep Learning Tools in Docker Containers. BigCom (2017)
Shaohuai Shi, Pengfei Xu, Xiaowen Chu
Supervised Learning Based Algorithm Selection for Deep Neural Networks. ICPADS (2017)
Shaohuai Shi, Xiaowen Chu
Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs. CoRR (2017)
Shaohuai Shi, Qiang Wang, Pengfei Xu, Xiaowen Chu
Benchmarking State-of-the-Art Deep Learning Software Tools. CCBD (2016)
Shaohuai Shi, Qiang Wang, Pengfei Xu, Xiaowen Chu
Benchmarking State-of-the-Art Deep Learning Software Tools. CoRR (2016)
Shuhan Qi, Xuan Wang, Shaohuai Shi
Mixed Precision Method for GPU-based FFT. CSE (2011)
Jiang-Feng Peng, Hu Chen, Shaohuai Shi
The GPU-based String Matching System in Advanced AC Algorithm. CIT (2010)