synchronous-pipeline-parallel

Chimera

Understanding Chimera: A Pipeline Model Parallelism Scheme Chimera is a model parallelism scheme designed to train large-scale models efficiently. Its unique feature is the combination of bidirectional pipelines, namely down and up pipelines, to accomplish the task. The aim is to execute a large number of micro-batches by each worker within a training iteration with the minimum of four pipeline stages. How Chimera Pipeline Works? Chimera pipeline, as shown in the figure, consists of four pip

GPipe

GPipe is a distributed model parallel method for neural networks that allows for faster and more efficient training of deep learning models. What is GPipe? GPipe is a distributed model parallel method for neural networks that was developed by Google to improve the efficiency and speed of training deep learning models. It works by dividing the layers of a model into cells, which can then be distributed across multiple accelerators. By doing this, GPipe allows for batch splitting, which divides

1 / 1