Speed up Mixtral 8x7B pre-training with professional parallelism on Amazon SageMaker

Combination of Consultants (MoE) architectures for big language fashions (LLMs) have just lately gained recognition attributable to their means to extend mannequin capability and computational effectivity in comparison with absolutely dense fashions. By using sparse professional subnetworks that course of completely different subsets of tokens, MoE fashions can successfully enhance the variety of parameters whereas requiring much less computation per token throughout coaching and inference. This permits less expensive coaching of bigger fashions inside mounted compute budgets in comparison with dense architectures.

Regardless of their computational advantages, coaching and fine-tuning massive MoE fashions effectively presents some challenges. MoE fashions can wrestle with load balancing if the tokens aren’t evenly distributed throughout consultants throughout coaching, and a few consultants could change into overloaded whereas others are under-utilized. MoE fashions have excessive reminiscence necessities, as a result of all professional parameters have to be loaded into reminiscence though solely a subset is used for every enter.

On this submit, we spotlight new options of the Amazon SageMaker mannequin parallelism library that allow environment friendly coaching of MoE fashions utilizing professional parallelism. Professional parallelism is a sort of parallelism that handles splitting consultants of an MoE mannequin throughout separate staff or units, much like how tensor parallelism can partition dense mannequin layers. We display learn how to use these new options of SMP by pre-training the 47 billion parameter Mixtral 8x7B MoE mannequin utilizing professional parallelism. To be taught extra, discuss with our GitHub repo and Professional parallelism.

Professional parallelism

The Mixtral 8x7B mannequin has a sparse MoE structure, containing eight professional subnetworks with round 7 billion parameters every. A trainable gate community referred to as a router determines which enter tokens are despatched to which professional. With this structure, the consultants concentrate on processing completely different features of the enter knowledge. The whole Mixtral 8x7B mannequin has a complete of 47 billion parameters, however solely round 12.9 billion (two consultants, for this mannequin structure) are activated for any given enter token; this ends in improved computational effectivity relative to a dense mannequin of the identical whole dimension. To be taught extra in regards to the MoE structure on the whole, discuss with Making use of Combination of Consultants in LLM Architectures.

SMP provides help for professional parallelism

SMP now helps professional parallelism, which is important to performant MoE mannequin coaching. With professional parallelism, completely different professional subnetworks that comprise the MoE layers are positioned on separate units. Throughout coaching, completely different knowledge is routed to the completely different units, with every machine dealing with the computation for the consultants it comprises. By distributing consultants throughout staff, professional parallelism addresses the excessive reminiscence necessities of loading all consultants on a single machine and permits MoE coaching on a bigger cluster. The next determine gives a simplified have a look at how professional parallelism works on a multi-GPU cluster.

The SMP library makes use of NVIDIA Megatron to implement professional parallelism and help coaching MoE fashions, and runs on high of PyTorch Totally Sharded Knowledge Parallel (FSDP) APIs. You possibly can maintain utilizing your PyTorch FSDP coaching code as is and activate SMP professional parallelism for coaching MoE fashions. SMP gives a simplified workflow the place it’s essential specify the expert_parallel_degree parameter, which can evenly divide consultants throughout the variety of GPUs in your cluster. For instance, to shard your mannequin whereas utilizing an occasion with 8 GPUs, you’ll be able to set the expert_parallel_degree to 2, 4, or 8. We suggest that you just begin with a small quantity and steadily enhance it till the mannequin suits within the GPU reminiscence.

SMP’s professional parallelism is suitable with sharded knowledge parallelism

SMP’s professional parallel implementation is suitable with sharded knowledge parallelism, enabling extra memory-efficient and quicker coaching. To grasp how this works, take into account an MoE mannequin within the following instance with eight consultants (N=8) coaching on a easy cluster with one node containing 4 GPUs.

SMP’s professional parallelism splits the MoE consultants throughout GPUs. You management what number of consultants are instantiated on every machine through the use of the expert_parallel_degree parameter. For instance, for those who set the diploma to 2, SMP will assign half of the eight consultants to every knowledge parallel group. The diploma worth should be an element of the variety of GPUs in your cluster and the variety of consultants in your mannequin. Knowledge is dynamically routed to and from the GPU or GPUs internet hosting the chosen professional utilizing all-to-all GPU communication.

Subsequent, sharded knowledge parallelism partitions and distributes the consultants in addition to the non-MoE layers of the mannequin, like consideration or routers, throughout your cluster to cut back the reminiscence footprint of the mannequin. The hybrid_shard_degree parameter controls this. For instance, a hybrid_shard_degree of two will shard the mannequin states (together with consultants and non-MoE layers) throughout half of the GPUs in our cluster. The product of expert_parallel_degree and hybrid_shard_degree shouldn’t exceed the world dimension of the cluster. Within the following instance, hybrid_shard_degree * expert_parallel_degree = 4 is a sound configuration.

Resolution overview

With the background out of the best way, let’s dig into the parts of our distributed coaching structure. The next diagram illustrates the answer structure.

On this instance, we use SageMaker coaching jobs. With SageMaker coaching jobs, you’ll be able to launch and handle clusters of high-performance situations with easy API calls. For instance, you should utilize the SageMaker Estimator to specify the sort and amount of situations to make use of in your distributed techniques with only a few traces of code. Later on this submit, we use a cluster of two ml.p4d.24xlarge situations to coach our mannequin by specifying these parameters in our Estimator. To study SageMaker coaching jobs, see Prepare a Mannequin with Amazon SageMaker.

On this submit, we use the SMP library to effectively distribute the workload throughout the cluster utilizing hybrid sharded knowledge parallelism and professional parallelism. Along with these implementations, SMP gives many different performance-improving and memory-saving strategies, corresponding to:

• Combined precision coaching and fp8 help for dense Llama fashions (which accelerates distributed coaching and takes benefit of the efficiency enhancements on P5 situations)
• Tensor parallelism composable with sharded knowledge parallelism
• Delayed parameter initialization
• Activation checkpointing (a way to cut back reminiscence utilization by clearing activations of sure layers and recomputing them in the course of the backward cross)

For the newest updates, discuss with SageMaker mannequin parallelism library v2.

Together with SMP, this instance additionally makes use of the SageMaker distributed knowledge parallel library (SMDDP). As you scale your workload and add situations to your cluster, the overhead of communication between situations additionally will increase, which may result in a drop in total computational efficiency and coaching effectivity. That is the place SMDDP helps. SMDDP consists of optimized communication collectives corresponding to AllGather which are designed for AWS community infrastructure. Due to this, SMDDP can outperform different extra common communications libraries corresponding to NCCL when coaching on SageMaker.

Collectively, the SMP and SMDDP libraries can speed up massive distributed coaching workloads by as much as 20%. Moreover, these libraries are suitable with normal PyTorch APIs and capabilities, which makes it handy to adapt any present PyTorch FSDP coaching script to the SageMaker coaching platform and make the most of the efficiency enhancements that SMP and SMDDP present. To be taught extra, see SageMaker mannequin parallelism library v2 and Run distributed coaching with the SageMaker distributed knowledge parallelism library.

Within the following sections, we showcase how one can speed up distributed coaching of the Hugging Face Transformers Mixtral 8*7B mannequin on P4 situations utilizing SMP and SMDDP.

Stipulations

It is advisable full some stipulations earlier than you’ll be able to run the Mixtral pocket book.

First, be sure you have created a Hugging Face entry token so you’ll be able to obtain the Hugging Face tokenizer for use later. After you may have the entry token, it’s essential make just a few quota enhance requests for SageMaker. It is advisable request a minimal of two P4d situations ranging to a most of 8 P4d situations (relying on time-to-train and cost-to-train trade-offs to your use case).

On the Service Quotas console, request the next SageMaker quotas:

• P4 situations (ml.p4d.24xlarge) for coaching job utilization: 2–8

It could take as much as 24 hours for the quota enhance to get authorised.

Now that you just’re prepared to start the method to pre-train the Mixtral mannequin, we begin with dataset preparation within the subsequent step.

Put together the dataset

We start our tutorial with getting ready the dataset. This may cowl loading the GLUE/SST2 dataset, tokenizing and chunking the dataset, and configuring the information channels for SageMaker coaching on Amazon Easy Storage Service (Amazon S3). Full the next steps:

1. You first must load the GLUE/SST2 dataset and cut up it into coaching and validation datasets:

   hyperparameters = {
       "cache_dir": "tmp",
       "dataset_config_name": "sst2",
       "dataset_name": "glue",
       "do_train": True,
       "do_eval": True,
   }
   
   raw_datasets = load_dataset(
       hyperparameters["dataset_name"],
       hyperparameters["dataset_config_name"],
   )
   
   del raw_datasets["validation"]
   
   if "validation" not in raw_datasets.keys():
       validation_percentage = "10%"
   
       raw_datasets["validation"] = load_dataset(
           hyperparameters["dataset_name"],
           hyperparameters["dataset_config_name"],
           cut up=f"practice[:{validation_percentage}]",
           cache_dir=hyperparameters["cache_dir"],
       )
   
       raw_datasets["train"] = load_dataset(
           hyperparameters["dataset_name"],
           hyperparameters["dataset_config_name"],
           cut up=f"practice[{validation_percentage}:]",
           cache_dir=hyperparameters["cache_dir"],
       )

2. Load the Mixtral-8x7B tokenizer from the Hugging Face Transformers library:

   tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-v0.1", **tokenizer_kwargs)

Subsequent, you outline two utility capabilities: tokenize_function() and group_texts(). The tokenize_function() runs the tokenizer on the textual content knowledge. The group_texts() perform concatenates all texts from the dataset and generates chunks of a block dimension that corresponds to the mannequin’s enter size (2048) for this instance. By chunking the textual content knowledge into smaller items, you make sure that the mannequin can course of the complete dataset throughout coaching, even when some textual content examples are longer than the enter size (2048).

3. Outline the capabilities with the next code:

   def tokenize_function(examples):
       ...
       
       output = tokenizer(examples[text_column_name])
       return output
   def group_texts(examples):
       # Concatenate all texts.
       concatenated_examples = {ok: sum(examples[k], []) for ok in examples.keys()}
       total_length = len(concatenated_examples[list(examples.keys())[0]])
       
       if total_length >= block_size:
           total_length = (total_length // block_size) * block_size
           # Cut up by chunks of max_len.
           consequence = {
               ok: [t[i : i + block_size] for i in vary(0, total_length, block_size)]
               for ok, t in concatenated_examples.objects()
           }
       consequence["labels"] = consequence["input_ids"].copy()
       return consequence

4. Name the previous utility capabilities in your dataset to tokenize and generate chunks appropriate for the mannequin:

   tokenized_datasets = raw_datasets.map(tokenize_function, batched=True,num_proc=1,remove_columns=column_names)
   lm_datasets = tokenized_datasets.map(group_texts, batched=True)

5. Put together the coaching and validation datasets for SageMaker coaching by saving them as JSON information and setting up the S3 paths the place these information might be uploaded:

   train_dataset = lm_datasets["train"]
   train_dataset.to_json("./coaching.json")
   training_dataset_location = f"s3://{default_bucket}/dataset/practice/"
   
    
   eval_dataset = lm_datasets["validation"]
   eval_dataset.to_json("./validation.json")
   validation_dataset_location = f"s3://{default_bucket}/dataset/validation/"

6. Lastly, arrange the information channels for SageMaker coaching by creating TrainingInput objects from the offered S3 bucket paths for the coaching and take a look at/validation datasets:

   practice = sagemaker.inputs.TrainingInput(
               s3_train_bucket, distribution="FullyReplicated", 
               s3_data_type="S3Prefix"
           )
   data_channels = {"practice": practice}
   
   take a look at = sagemaker.inputs.TrainingInput(
               s3_test_bucket, distribution="FullyReplicated", 
               s3_data_type="S3Prefix"
           )
   data_channels["test"] = take a look at

You’re now able to run pre-training or fine-tuning on the dataset.

Pre-train Mixtral 8x7B with professional parallelism on SMP

To pre-train the Mixtral 8x7B mannequin, full the next steps:

1. Initialize the script with torch.sagemaker.init() to activate the SMP library:

   import torch.sagemaker as tsm
   tsm.init()

2. Import the MoEConfig class from the torch.sagemaker.remodel API. We use the MoEConfig class to allow the mannequin to make use of the SMP implementation of MoE:

   from torch.sagemaker.moe.moe_config import MoEConfig

3. Create a mannequin configuration for Mixtral 8x7B mannequin. This might be handed to AutoModelForCausalLM.from_config(model_config, attn_implementation="flash_attention_2") from the Hugging Face Transformers library to initialize the mannequin with random weights. If you wish to fine-tune, you’ll be able to present the trail to the pre-trained weights as a substitute of the mannequin configuration.

   model_config = MixtralConfig(
               vocab_size=args.vocab_size, # 32000,
               hidden_size=args.hidden_width, # 4096,
               intermediate_size=args.intermediate_size, # 14336,
               num_hidden_layers=args.num_layers, # 32,
               num_attention_heads=args.num_heads, # 32,
               num_key_value_heads=args.num_key_value_heads, # 8,
               hidden_act="silu",
               max_position_embeddings=args.max_context_width, # 4096 * 32,
               initializer_range=args.initializer_range, # 0.02,
               rms_norm_eps=1e-5,
               use_cache=False,
               pad_token_id=None,
               bos_token_id=1,
               eos_token_id=2,
               tie_word_embeddings=False,
               rope_theta=1e6,
               sliding_window=args.sliding_window, # None,
               attention_dropout=0.0,
               num_experts_per_tok=args.num_experts_per_tok, # 2,
               num_local_experts=args.num_local_experts, # 8,
               output_router_logits=False,
               router_aux_loss_coef=0.001,
           )
          
   mannequin = AutoModelForCausalLM.from_config(model_config, dtype=dtype, attn_implementation="flash_attention_2" )

Within the instance Jupyter Pocket book, you utilize a create_model() perform that invokes the AutoModelForCausalLM.from_config() perform.

4. Create the SMP MoE configuration class. Within the following code, you specify parameters within the coaching estimator within the subsequent steps. To be taught extra in regards to the SMP MoEConfig class, see torch.sagemaker.moe.moe_config.MoEConfig.

   moe_config = MoEConfig(
                       smp_moe=args.use_smp_implementation > 0, #Whether or not to make use of the SMP-implementation of MoE. The default worth is True.
                       random_seed=args.seed, # A seed quantity for the random operations in expert-parallel distributed modules. This seed might be added to the professional parallel rank to set the precise seed for every rank. It's distinctive for every professional parallel rank. The default worth is 12345.
                       moe_load_balancing=args.moe_load_balancing, #Specify the load balancing kind of the MoE router. Legitimate choices are aux_loss, sinkhorn, balanced, and none. The default worth is sinkhorn.
                       global_token_shuffle=args.global_token_shuffle > 0,  #Whether or not to shuffle tokens throughout EP ranks inside the identical professional parallel group. The default worth is False
                       moe_all_to_all_dispatcher=args.moe_all_to_all_dispatcher > 0, #Whether or not to make use of all-to-all dispatcher for the communications in MoE. The default worth is True.
                   )

5. With the mannequin and MoE configuration prepared, you wrap the mannequin with the SMP remodel API and cross the MoE configuration. Right here, the tsm.remodel methodology adapts the mannequin from Hugging Face format to SMP format. For extra info, discuss with torch.sagemaker.remodel.

   mannequin = tsm.remodel(
           mannequin, 
           config=moe_config,
       )

6. Outline the coaching hyperparameters, together with the MoE configuration and different settings particular to the mannequin and coaching setup:

   hyperparameters = {
       # MoE config
       "moe": 1,
       "moe_load_balancing": "sinkhorn",
       "moe_all_to_all_dispatcher": 1,
       "seed": 12345,
       #remainder of hyperparameters
       ...
       "model_type": "mixtral",
       "sharding_strategy": "hybrid_shard",
       "delayed_param": 1, 
       "epochs": 100,
       "activation_checkpointing": 1,
       "beta1": 0.9,
       "bf16": 1,
       "fp8": 0,
       "checkpoint_dir": "/choose/ml/checkpoints",
       ...
       ...
       
   }

We allow delayed parameter initialization in SMP, which permits initializing massive fashions on a meta machine with out attaching knowledge. This could resolve restricted GPU reminiscence points whenever you first load the mannequin. This strategy is especially helpful for coaching LLMs with tens of billions of parameters, the place even CPU reminiscence may not be enough for initialization.

SMP helps varied routing methods, together with sinkhorn, balanced, and aux_loss. Every offers distinct load balancing approaches to attain equitable token task amongst consultants, thereby sustaining balanced workload distribution.

7. Specify the parameters for expert_parallel_degree and hybrid_shard_degree:

   expert_parallel_degree = 2  # An integer in [1, world_size]
   hybrid_shard_degree = (
       8  # An integer in [0, world_size // expert_parallel_degree] and its default worth is 0.
   )

Hybrid sharding is a reminiscence saving approach between `FULL_SHARD` and `NO_SHARD`, with `FULL_SHARD` saving essentially the most reminiscence and `NO_SHARD` not saving any. This system shards parameters inside the hybrid shard diploma (HSD) group and replicates parameters throughout teams. The HSD controls sharding throughout GPUs and could be set to an integer from 0 to `world_size`.

An HSD of 8 applies `FULL_SHARD` inside a node after which replicates parameters throughout nodes as a result of there are 8 GPUs within the nodes we’re utilizing. This ends in diminished communication quantity as a result of costly all-gathers and reduce-scatters are solely finished inside a node, which could be extra performant for medium-sized fashions. Usually, you need to use the smallest HSD that doesn’t trigger out of reminiscence (OOM) errors. If you happen to’re experiencing OOM, strive growing the hybrid shard diploma to cut back reminiscence utilization on every node.

8. With all the mandatory configurations in place, you now create the PyTorch estimator perform to encapsulate the coaching setup and launch the coaching job. We run the pre-training on the two ml.p4d.24xlarge situations, the place every occasion comprises 8 A100 Nvidia GPUs:

   smp_estimator = PyTorch(
       entry_point="practice.py",
       hyperparameters=hyperparameters,
       position=position,
       checkpoint_s3_uri=checkpoint_s3_uri,
       checkpoint_local_path=hyperparameters["checkpoint_dir"] 
       instance_type="ml.p4d.24xlarge",
       volume_size=400,
       instance_count=2,
       sagemaker_session=sagemaker_session,
       ...
       distribution={
           "torch_distributed": {
               "enabled": True,
           },
           "smdistributed": {
               "modelparallel": {
                   "enabled": True,
                   "parameters": {
                       "activation_loading_horizon": activation_loading_horizon,
                       "hybrid_shard_degree": hybrid_shard_degree,
                       "sm_activation_offloading": offload_activations,
                       "expert_parallel_degree": expert_parallel_degree,
                   },
               }
           },
       },
       py_version="py310",
       framework_version="2.2.0",
       output_path=s3_output_bucket,
   )

9. Lastly, launch the pre-training workload:

   smp_estimator.match(inputs=data_channels)

Clear up

As a part of cleanup, you’ll be able to delete the SageMaker default bucket created to host the GLUE/SST2 dataset.

Conclusion

Coaching massive MoE language fashions just like the 47 billion parameter Mistral 8x7B could be difficult attributable to excessive computational and reminiscence necessities. By utilizing professional parallelism and sharded knowledge parallelism from the SageMaker mannequin parallelism library, you’ll be able to successfully scale these MoE architectures throughout a number of GPUs and staff.

SMP’s professional parallelism implementation seamlessly integrates with PyTorch and the Hugging Face Transformers library, permitting you to allow MoE coaching utilizing easy configuration flags with out altering your present mannequin code. Moreover, SMP offers efficiency optimizations like hybrid sharding, delayed parameter initialization, and activation offloading and recomputation to additional enhance coaching effectivity.

For the entire pattern to pre-train and fine-tune Mixtral 8x7B, see the GitHub repo.

Particular thanks

Particular due to Rahul Huilgol, Gautam Kumar, and Luis Quintela for his or her steering and engineering management in growing this new functionality.

In regards to the Authors

Roy Allela is a Senior AI/ML Specialist Options Architect at AWS primarily based in Munich, Germany. Roy helps AWS prospects—from small startups to massive enterprises—practice and deploy massive language fashions effectively on AWS. Roy is captivated with computational optimization issues and bettering the efficiency of AI workloads.

Kanwaljit Khurmi is a Principal Options Architect at Amazon Internet Companies. He works with AWS prospects to supply steering and technical help, serving to them enhance the worth of their options when utilizing AWS. Kanwaljit focuses on serving to prospects with containerized and machine studying functions.

Robert Van Dusen is a Senior Product Supervisor with Amazon SageMaker. He leads frameworks, compilers, and optimization strategies for deep studying coaching.

Teng Xu is a Software program Improvement Engineer within the Distributed Coaching group in AWS AI. He enjoys studying.

Suhit Kodgule is a Software program Improvement Engineer with the AWS Synthetic Intelligence group engaged on deep studying frameworks. In his spare time, he enjoys climbing, touring, and cooking.