Cookies?
Library Header Image
LSE Research Online LSE Library Services

QSGD: communication-efficient SGD via gradient quantization and encoding

Alistarh, Dan, Grubic, Demjan, Li, Jerry Z., Tomioka, Ryota and Vojnovic, Milan (2017) QSGD: communication-efficient SGD via gradient quantization and encoding. . arXiv.

[img] Text (Vojnovic_qsgd-communication-efficient-sgd) - Published Version
Download (918kB)

Abstract

Parallel implementations of stochastic gradient descent (SGD) have received significant research attention, thanks to its excellent scalability properties. A fundamental barrier when parallelizing SGD is the high bandwidth cost of communicating gradient updates between nodes; consequently, several lossy compresion heuristics have been proposed, by which nodes only communicate quantized gradients. Although effective in practice, these heuristics do not always converge. In this paper, we propose Quantized SGD (QSGD), a family of compression schemes with convergence guarantees and good practical performance. QSGD allows the user to smoothly trade off communication bandwidth and convergence time: nodes can adjust the number of bits sent per iteration, at the cost of possibly higher variance. We show that this trade-off is inherent, in the sense that improving it past some threshold would violate information-theoretic lower bounds. QSGD guarantees convergence for convex and non-convex objectives, under asynchrony, and can be extended to stochastic variance-reduced techniques. When applied to training deep neural networks for image classification and automated speech recognition, QSGD leads to significant reductions in end-to-end training time. For instance, on 16GPUs, we can train the ResNet-152 network to full accuracy on ImageNet 1.8× faster than the full-precision variant.

Item Type: Monograph (Working Paper)
Official URL: https://arxiv.org/
Additional Information: © 2017 The Authors
Divisions: Statistics
Subjects: Q Science > Q Science (General)
Q Science > QA Mathematics
H Social Sciences > HA Statistics
Date Deposited: 02 Feb 2021 09:51
Last Modified: 03 Feb 2021 00:25
URI: http://eprints.lse.ac.uk/id/eprint/108587

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics