Accelerate documentation

๐Ÿค— accelerate์˜ Context Parallel

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

๐Ÿค— accelerate์˜ Context Parallel

์ด ๊ฐ€์ด๋“œ์—์„œ๋Š” ๐Ÿค—accelerate์—์„œ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์„ ์‚ฌ์šฉํ•˜๋Š” ๊ธฐ๋ณธ ์‚ฌํ•ญ์„ ๋‹ค๋ฃน๋‹ˆ๋‹ค. ๋” ์ž์„ธํ•œ ๋‚ด์šฉ์„ ์•Œ๊ณ  ์‹ถ์€ ๋…์ž๋ฅผ ์œ„ํ•ด ํ›„๋ฐ˜๋ถ€ ์„น์…˜์—์„œ๋Š” ์ผ๋ถ€ ๊ธฐ์ˆ ์ ์ธ ์‚ฌํ•ญ๋„ ๋‹ค๋ฃน๋‹ˆ๋‹ค.

๋งค์šฐ ๊ด€๋ จ์ด ์žˆ๋Š” Sequence Parallelism ๊ฐ€์ด๋“œ๋„ ์ฐธ๊ณ ํ•˜์„ธ์š”.

์™œ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์„ ์‚ฌ์šฉํ•ด์•ผ ํ•˜๋‚˜์š”?

๋Œ€ํ˜• ์–ธ์–ด ๋ชจ๋ธ, ๊ทธ๋ฆฌ๊ณ  ์ตœ๊ทผ์—๋Š” ์ถ”๋ก  ๋ชจ๋ธ์˜ ๋“ฑ์žฅ์œผ๋กœ ์‹œํ€€์Šค ๊ธธ์ด๊ฐ€ ๋น ๋ฅด๊ฒŒ ์ฆ๊ฐ€ํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์— attention์˜ 2์ฐจ ๋ฉ”๋ชจ๋ฆฌ ๋ณต์žก๋„๊ฐ€ ๋”ํ•ด์ง€๋ฉด์„œ, ๊ธด ์‹œํ€€์Šค๋กœ ๋ชจ๋ธ์„ ํ•™์Šตํ•˜๋Š” ๋” ํšจ์œจ์ ์ธ ๋ฐฉ์‹์— ๋Œ€ํ•œ ํ•„์š”์„ฑ์ด ์ปค์กŒ์Šต๋‹ˆ๋‹ค. ์‹œํ€€์Šค ๊ธธ์ด๊ฐ€ 128k์ผ ๋•Œ, vanilla attention ๊ตฌํ˜„์„ ๊ฐ€์ •ํ•˜๋ฉด bf16 ์ •๋ฐ€๋„์—์„œ attention matrix์˜ ๋ฉ”๋ชจ๋ฆฌ ์š”๊ตฌ๋Ÿ‰์€ 128k * 128k * 2 bytes * num_heads = ~32 GB * num_heads์ž…๋‹ˆ๋‹ค. ๋ฌผ๋ก , ์ด๋Ÿฌํ•œ attention ๊ฐ€์ค‘์น˜๋ฅผ ๋ฌผ๋ฆฌ์ ์œผ๋กœ ์ƒ์„ฑํ•˜์ง€ ์•Š๋Š” flash attention๋‚˜ SDPA๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰์ด ํฌ๊ฒŒ ์ค„์ง€๋งŒ, ๋ฉ”๋ชจ๋ฆฌ ์š”๊ตฌ๋Ÿ‰์˜ ์ฆ๊ฐ€ ํญ์€ ์—ฌ์ „ํžˆ ์ƒ๋‹นํ•ฉ๋‹ˆ๋‹ค.

Context parallelism์€ attention ๊ณ„์‚ฐ์˜ ์ž…๋ ฅ์„ ์‹œํ€€์Šค ์ฐจ์›๋ณ„๋กœ ์ƒค๋”ฉํ•˜์—ฌ ์—ฌ๋Ÿฌ GPU์—์„œ attention์„ ๋ณ‘๋ ฌ๋กœ ๊ณ„์‚ฐํ•  ์ˆ˜ ์žˆ๊ฒŒ ํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฅผ ํ†ตํ•ด ๊ธด ์‹œํ€€์Šค๋ฅผ ๊ฐ€์ง„ ๋ชจ๋ธ์„ ํ•™์Šตํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ, ์ž ์žฌ์ ์œผ๋กœ 1M ์ด์ƒ์˜ ์‹œํ€€์Šค ๊ธธ์ด๊นŒ์ง€ ํ™•์žฅํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ(context parallelism)์„ ์‚ฌ์šฉํ•˜๋Š” ๋ฐฉ๋ฒ•

from accelerate.utils import ParallelismConfig, TorchContextParallelConfig

+ cp_config = TorchContextParallelConfig(
+       cp_comm_strategy="alltoall", # no need to use cp_config at all, if you want to use the default "allgather"
+ )

+ parallelism_config = ParallelismConfig(
+     cp_size=8,
+     cp_handler=cp_config,  # or just cp_size=8, if you want to use the default "allgather"
+ )

accelerator = Accelerator(
    ...,
    parallelism_config=parallelism_config,
)

๐Ÿค—accelerate์˜ ๋‹ค๋ฅธ ๋ชจ๋“  ๊ธฐ๋Šฅ๊ณผ ๋งˆ์ฐฌ๊ฐ€์ง€๋กœ, accelerate launch์— ํ•ด๋‹น ํ”Œ๋ž˜๊ทธ๋ฅผ ์ „๋‹ฌํ•˜์—ฌ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ๋„ ํ™œ์„ฑํ™”ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ๊ฒฝ์šฐ์—๋„ ๋‹ค๋ฅด์ง€ ์•Š์Šต๋‹ˆ๋‹ค:

accelerate launch --parallelism-config-cp-size 8 --parallelism-config-cp-comm-strategy [allgather|alltoall] ...

accelerate config ๋ช…๋ น์—์„œ cp_size์™€ cp_comm_strategy๋ฅผ ์„ค์ •ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ๊ฐ’๋“ค์€ accelerate ๊ตฌ์„ฑ ํŒŒ์ผ์— ์ €์žฅ๋˜๋ฏ€๋กœ, ์Šคํฌ๋ฆฝํŠธ๋ฅผ ์‹คํ–‰ํ•  ๋•Œ๋งˆ๋‹ค ๋งค๋ฒˆ ์ „๋‹ฌํ•  ํ•„์š”๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค.

Context parallelism์€ data parallelism, tensor parallelism, FSDP2์™€ ๊ฐ™์€ ๋‹ค๋ฅธ ๋ณ‘๋ ฌํ™” ์ „๋žต๊ณผ ํ˜ธํ™˜๋ฉ๋‹ˆ๋‹ค. --parallelism-config-dp-size 8 --parallelism-config-tp-size 2 --parallelism-config-cp-size 8์ฒ˜๋Ÿผ ๋ณ‘๋ ฌํ™” ํฌ๊ธฐ๋ฅผ ์›ํ•˜๋Š” ๊ฐ’์œผ๋กœ ์„ค์ •ํ•˜์—ฌ ๊ฐ„๋‹จํžˆ ์กฐํ•ฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋˜๋Š” ParallelismConfig ํด๋ž˜์Šค๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ํ”„๋กœ๊ทธ๋ž˜๋ฐ ๋ฐฉ์‹์œผ๋กœ ์„ค์ •ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

Context parallelism์€ FSDP2์™€ ๊ธด๋ฐ€ํ•˜๊ฒŒ ๊ฒฐํ•ฉ๋˜์–ด ์žˆ์œผ๋ฉฐ, FSDP2 ์†Œ๊ฐœ์—์„œ ์ž์„ธํžˆ ์•Œ์•„๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ฆ‰, context parallelism์€ ํ”„๋กœ๊ทธ๋žจ์— FullyShardedDataParallelPlugin ๋˜๋Š” ๋ฒ„์ „์ด 2๋กœ ์„ค์ •๋œ --use-fsdp๋ฅผ ์‚ฌ์šฉํ•  ๋•Œ๋งŒ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค. FSDP2๋ฅผ ์‚ฌ์šฉํ•˜์ง€ ์•Š์œผ๋ฉด ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

Context parallelism์€ SDPA์—์„œ๋งŒ ์ž‘๋™ํ•˜๋ฉฐ, mask๋‚˜ causal mask๊ฐ€ ์—†๋Š” ๊ฒฝ์šฐ์—๋งŒ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด๋ฅผ ์ž๋™์œผ๋กœ ์ ์ ˆํ•˜๊ฒŒ ๊ฐ์ง€ํ•  ์ˆ˜ ์—†์œผ๋ฏ€๋กœ, mask๋‚˜ causal mask ์—†์ด SDPA๋ฅผ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ๋Š”์ง€ ํ™•์ธํ•˜๋Š” ๊ฒƒ์€ ์‚ฌ์šฉ์ž์˜ ์ฑ…์ž„์ž…๋‹ˆ๋‹ค. ๋‹ค๋ฅธ attention ๊ตฌํ˜„์„ ์‚ฌ์šฉํ•˜๋ฉด ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

์œ„์—์„œ ์–ธ๊ธ‰ํ•œ ๋ฐฉ๋ฒ•์œผ๋กœ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์„ ํ™œ์„ฑํ™”ํ•œ ํ›„, ์ด๋ฅผ ํ•™์Šต ๋ฃจํ”„์— ์ ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ํ•™์Šต ๋ฃจํ”„์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” torch.distributed.tensor.experimental.context_parallel์˜ ์–‡์€ ๋ž˜ํผ๋ฅผ ์ œ๊ณตํ•˜๋ฉฐ, ์ด๋Š” ํ•ด๋‹น ๊ธฐ๋Šฅ ์‚ฌ์šฉ์˜ ์ผ๋ถ€ ๋ณต์žก์„ฑ์„ ์ถ”์ƒํ™”ํ•ฉ๋‹ˆ๋‹ค(์ดํ›„ ์ž์„ธํžˆ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค). ํ•™์Šต ๋ฃจํ”„์—์„œ ์ˆ˜ํ–‰ํ•ด์•ผ ํ•˜๋Š” ๋ณ€๊ฒฝ ์‚ฌํ•ญ์„ ์ตœ์†Œํ™”ํ•˜๊ธฐ ์œ„ํ•ด, ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์ด ๋น„ํ™œ์„ฑํ™”๋˜์–ด ์žˆ์œผ๋ฉด noop์œผ๋กœ ๋™์ž‘ํ•˜๊ณ  ํ™œ์„ฑํ™”๋˜์–ด ์žˆ์œผ๋ฉด ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์„ ์ ์šฉํ•˜๋Š” ์ปจํ…์ŠคํŠธ ๋งค๋‹ˆ์ €๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฅผ ํ†ตํ•ด ๋ณ‘๋ ฌ ๊ตฌ์„ฑ์— ๋”ฐ๋ผ ์ฝ”๋“œ๋ฅผ ์ˆ˜์ •ํ•˜์ง€ ์•Š๊ณ ๋„ ํ•™์Šต ๋ฃจํ”„์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

for batch in dataloader:
    with accelerator.maybe_context_parallel(
        buffers=[batch["input_ids"], batch["attention_mask"]],
        buffer_seq_dims=[1, 1],
        no_restore_buffers={batch["input_ids"], batch["labels"]},
    ):
        outputs = model(**batch)
        ...

์œ„ ์˜ˆ์‹œ์™€ ๊ฐ™์ด ์ด ์ปจํ…์ŠคํŠธ ๋งค๋‹ˆ์ €๋Š” ๊ฐ ํ•™์Šต ๋‹จ๊ณ„๋งˆ๋‹ค ๋‹ค์‹œ ์ƒ์„ฑ๋˜์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฅผ ์ˆ˜ํ–‰ํ•˜๋Š” ๊ฒƒ์ด ๋งค์šฐ ์ค‘์š”ํ•ฉ๋‹ˆ๋‹ค.

์ด๋ฅผ ํ†ตํ•ด ์ปจํ…์ŠคํŠธ ํฌ๊ธฐ๋ฅผ 1M+ ์‹œํ€€์Šค ๊ธธ์ด๊นŒ์ง€ ํ™•์žฅํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์•„๋ž˜์—์„œ๋Š” ์ตœ๋Œ€ 256k ์ปจํ…์ŠคํŠธ ํฌ๊ธฐ์— ๋Œ€ํ•œ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์˜ ์†๋„์™€ ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค. ์ปจํ…์ŠคํŠธ ํฌ๊ธฐ์™€ GPU ์ˆ˜๋ฅผ 2๋ฐฐ๋กœ ๋Š˜๋ฆฌ๋ฉด ์ผ๊ด€๋œ ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰์„ ์œ ์ง€ํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ, ์ด๋ฅผ ํ†ตํ•ด ๋ฌดํ•œํ•œ ์ปจํ…์ŠคํŠธ ๊ธธ์ด ํ™•์žฅ์ด ๊ฐ€๋Šฅํ•ด์งˆ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

context parallelism memory usage
๊ทธ๋ฆผ 1: ์ตœ๋Œ€ 256k ์ปจํ…์ŠคํŠธ ํฌ๊ธฐ์—์„œ์˜ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰ ๋ฐ ์†๋„.

์ด ์˜ˆ์ œ๋“ค์€ examples ํด๋”์—์„œ ์ฐพ์„ ์ˆ˜ ์žˆ๋Š” ์Šคํฌ๋ฆฝํŠธ๋กœ ์ž‘์„ฑ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. 8๊ฐœ์˜ H100 GPU(128k ์‹œํ€€์Šค ๊ธธ์ด)์—์„œ ์˜ˆ์ œ๋ฅผ ์‹คํ–‰ํ•˜๋ ค๋ฉด ๋‹ค์Œ ๋ช…๋ น์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

accelerate launch --use-fsdp --fsdp-activation-checkpointing=TRUE examples/torch_native_parallelism/nd_parallel.py --cp-size=8 --sequence-length=128000

Accelerate์˜ ์ธํ„ฐํŽ˜์ด์Šค

์ปจํ…์ŠคํŠธ ๋งค๋‹ˆ์ €๋Š” ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์„ ๊ตฌ์„ฑํ•˜๋Š” ๋ฐ ์‚ฌ์šฉ๋˜๋Š” ๋ช‡ ๊ฐ€์ง€ ์ธ์ˆ˜๋ฅผ ๋ฐ›์Šต๋‹ˆ๋‹ค.

  • buffers: ์‹œํ€€์Šค ์ฐจ์›์— ๊ฑธ์ณ ์ƒค๋”ฉ๋  ํ…์„œ ๋ชฉ๋ก์ž…๋‹ˆ๋‹ค. ์ด๋Ÿฌํ•œ ํ…์„œ๋Š” ์ผ๋ฐ˜์ ์œผ๋กœ input ids, labels, attention mask์ž…๋‹ˆ๋‹ค.
  • buffer_seq_dims: buffers ๋ฆฌ์ŠคํŠธ์˜ ์ˆœ์„œ๋Œ€๋กœ ๊ฐ ๋ฒ„ํผ์˜ ์‹œํ€€์Šค ์ฐจ์›์„ ์ง€์ •ํ•˜๋Š” ์ •์ˆ˜ ๋ฆฌ์ŠคํŠธ์ž…๋‹ˆ๋‹ค. buffers=[input_ids, shift_labels]๋ฅผ ์ „๋‹ฌํ•˜๋˜, ๋‘ ํ…์„œ์˜ shape๊ฐ€ ๋ชจ๋‘ [batch_size, sequence_length]์ธ ๊ฒฝ์šฐ, ์‹œํ€€์Šค ์ฐจ์›์ด ํ…์„œ์˜ ๋‘ ๋ฒˆ์งธ ์ฐจ์›์ด๋ฏ€๋กœ buffer_seq_dims=[1, 1]์„ ์ „๋‹ฌํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ด๋Š” ๋ชจ๋ธ ์ถœ๋ ฅ์„ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊ณ„์‚ฐํ•˜๋Š” ๋ฐ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  • no_restore_buffers: ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ ๊ตฌํ˜„์€ ๋ฒ„ํผ๋ฅผ ์ธํ”Œ๋ ˆ์ด์Šค๋กœ ์ˆ˜์ •ํ•˜์—ฌ torch.distributed.tensor.Dtensor๋กœ ๋ณ€ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ์ปจํ…์ŠคํŠธ ๋งค๋‹ˆ์ €๋ฅผ ๋ฒ—์–ด๋‚˜๋ฉด ๋ฒ„ํผ๋ฅผ ์›๋ž˜ ์ƒํƒœ๋กœ ๋ณต์›ํ•˜๊ธฐ ์œ„ํ•ด ํ†ต์‹  ์ปค๋„(๋ณดํ†ต all-gather)์„ ์‹คํ–‰ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ด ๊ณผ์ •์— ์‹œ๊ฐ„์ด ์†Œ์š”๋˜๋ฏ€๋กœ, ์ปจํ…์ŠคํŠธ ๋งค๋‹ˆ์ €๋ฅผ ๋ฒ—์–ด๋‚œ ํ›„ ๋ฒ„ํผ๋ฅผ ์‚ฌ์šฉํ•ด์•ผ ํ•œ๋‹ค๋Š” ํ™•์‹ ์ด ์—†๋Š” ํ•œ ๋ถˆํ•„์š”ํ•œ ํ†ต์‹ ์„ ํ”ผํ•˜๊ธฐ ์œ„ํ•ด buffers ์ธ์ž์™€ ๋™์ผํ•œ ํ…์„œ๋ฅผ ์ „๋‹ฌํ•˜๋Š” ๊ฒƒ์ด ๊ถŒ์žฅ๋ฉ๋‹ˆ๋‹ค.

Context parallelism์€ ๐Ÿค— transformers์˜ ๋ชจ๋ธ์ด ์ž์ฒด์ ์œผ๋กœ ์‹œํ”„ํŠธํ•˜์—ฌ causal language modeling์„ ํ™œ์„ฑํ™”ํ•  ์ˆ˜ ์žˆ๋Š” input_ids์˜ ๋ณต์‚ฌ๋ณธ์ธ labels์™€ ํ˜ธํ™˜๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๊ฒฝ์šฐ๋ฅผ ์ƒ๊ฐํ•ด ๋ณด์„ธ์š”: labels = [l1, l2, l3, l4, โ€ฆ li] context parallelism์„ ์ ์šฉํ•˜๋ฉด ๊ฐ rank๊ฐ€ labels์˜ ์ผ๋ถ€, ์˜ˆ๋ฅผ ๋“ค์–ด ๋‹ค์Œ๊ณผ ๊ฐ™์€ ํ˜•ํƒœ๋ฅผ ๊ฐ–๊ฒŒ ๋ฉ๋‹ˆ๋‹ค: labels_rank_0 = [l1, l2], labels_rank_1 = [l3, l4], โ€ฆ transformers ๋ชจ๋ธ๋ง ์ฝ”๋“œ๊ฐ€ labels๋ฅผ ์‹œํ”„ํŠธํ•œ ํ›„์—๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๊ฒฐ๊ณผ๊ฐ€ ๋ฉ๋‹ˆ๋‹ค: labels_rank_0 = [l2, PAD], labels_rank_1 = [l3, PAD], โ€ฆ ์—ฌ๊ธฐ์„œ PAD๋Š” padding token์ž…๋‹ˆ๋‹ค. ์ด๋กœ ์ธํ•ด labels๊ฐ€ ๋” ์ด์ƒ ์ž…๋ ฅ๊ณผ ์ •๋ ฌ๋˜์ง€ ์•Š์•„ ์ž˜๋ชป๋œ loss ๊ณ„์‚ฐ์ด ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค. ์ด ๋•Œ๋ฌธ์— ๋ชจ๋ธ์— ์ „๋‹ฌํ•˜๊ธฐ ์ „์— labels๋ฅผ ์ˆ˜๋™์œผ๋กœ ์‹œํ”„ํŠธํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

๊ตฌ์„ฑ ๊ฐ€๋Šฅํ•œ ์˜ต์…˜

Accelerate๋Š” cp_size๋ฅผ ์ œ์™ธํ•˜๊ณ  ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์„ ๊ตฌ์„ฑํ•˜๊ธฐ ์œ„ํ•œ ๋‹จ์ผ ์˜ต์…˜๋งŒ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค

  • cp_comm_strategy: ์ƒค๋“œ์—์„œ ์‚ฌ์šฉํ•  ๋กœํ…Œ์ด์…˜ ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค. ๋Œ€๋ถ€๋ถ„์˜ ๊ฒฝ์šฐ "alltoall"๋ณด๋‹ค "allgather"๊ฐ€ ๋” ๋‚˜์€ ์„ฑ๋Šฅ์„ ๋‚ผ ๊ฐ€๋Šฅ์„ฑ์ด ๋งค์šฐ ๋†’์œผ๋ฏ€๋กœ, ์ด ๊ฐ’์„ "allgather"๋กœ ์œ ์ง€ํ•˜๋Š” ๊ฒƒ์„ ๊ฐ•๋ ฅํžˆ ๊ถŒ์žฅํ•ฉ๋‹ˆ๋‹ค.

Context parallel size์€ ์ด๋ฆ„ ๊ทธ๋Œ€๋กœ, ์ž…๋ ฅ์ด ์ƒค๋”ฉ๋  rank์˜ ์ˆ˜๋ฅผ ์˜๋ฏธํ•ฉ๋‹ˆ๋‹ค. Context parallel shard rotation์€ ์ž…๋ ฅ์˜ ์ƒค๋“œ๊ฐ€ rank ๊ฐ„์— ์–ด๋–ป๊ฒŒ ํšŒ์ „๋˜๋Š”์ง€๋ฅผ ์ •์˜ํ•ฉ๋‹ˆ๋‹ค. ๋‹ค์Œ ์„น์…˜์—์„œ ๋‘ ๊ฐ€์ง€ ์˜ต์…˜์„ ์ž์„ธํžˆ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

ND parallel example ํŒŒ์ผ์—์„œ 8xH100 ๋…ธ๋“œ 1๋Œ€์—์„œ ์ตœ๋Œ€ 128k ์ปจํ…์ŠคํŠธ ๊ธธ์ด๋ฅผ ๊ฐ€์ง„ 8B ๋ชจ๋ธ์„ ํ•™์Šตํ•˜๋Š” ๋์—์„œ ๋(end-to-end) ์˜ˆ์‹œ๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ฉ€ํ‹ฐ ๋…ธ๋“œ ํ•™์Šต์„ ์‚ฌ์šฉํ•˜๋ฉด ์—ฌ๋Ÿฌ GPU์—์„œ ์‹œํ€€์Šค ๊ธธ์ด๋ฅผ 1M ์ด์ƒ์œผ๋กœ ํ™•์žฅํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋˜ํ•œ ํ•„์š”์— ๋”ฐ๋ผ ๋‹ค๋ฅธ ๋ณ‘๋ ฌํ™” ์ „๋žต๊ณผ๋„ ์›ํ™œํ•˜๊ฒŒ ๊ฒฐํ•ฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๊ธฐ์ˆ ์  ์„ธ๋ถ€ ์‚ฌํ•ญ

์ด ์„น์…˜์€ ์ƒ๋‹นํžˆ ๊ธฐ์ˆ ์ ์ธ ๋‚ด์šฉ์ž…๋‹ˆ๋‹ค. ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์˜ ๋‚ด๋ถ€ ๊ตฌ์กฐ๋ฅผ ํ•™์Šตํ•  ํ•„์š”๊ฐ€ ์—†๋‹ค๋ฉด ๊ฑด๋„ˆ๋›ฐ๊ณ  ๋ฐ”๋กœ ๊ตฌ์ถ•์„ ์‹œ์ž‘ํ•˜์„ธ์š” ๐Ÿš€

๋‹ค์Œ ์„น์…˜์—์„œ shard๋ผ๋Š” ์šฉ์–ด๋ฅผ ๊ด‘๋ฒ”์œ„ํ•˜๊ฒŒ ์‚ฌ์šฉํ•  ์˜ˆ์ •์ด๋ฏ€๋กœ, ๋จผ์ € ์ •์˜ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค. ํ…์„œ๋ฅผ Dth ์ฐจ์›์„ ๋”ฐ๋ผ N๊ฐœ์˜ rank์— ๊ฑธ์ณ sharded๋ผ๊ณ  ๋ถ€๋ฅผ ๋•Œ, ์ด๋Š” ํ•ด๋‹น ํ…์„œ๊ฐ€ N๊ฐœ์˜ ๋ถ€๋ถ„์œผ๋กœ ๋‚˜๋‰˜์–ด ์žˆ์œผ๋ฉฐ ๊ฐ ๋ถ€๋ถ„์˜ shape์ด [..., D//N, ...]์ž„์„ ์˜๋ฏธํ•ฉ๋‹ˆ๋‹ค.

๊ทธ๋ ‡๋‹ค๋ฉด ์–ด๋–ป๊ฒŒ ์ž‘๋™ํ•˜๋‚˜์š”?

Context parallelism์€ Q, K and V ํ–‰๋ ฌ์„ ์‹œํ€€์Šค ์ฐจ์›๋ณ„๋กœ ์ƒค๋”ฉํ•˜๋Š” ๋ฐฉ์‹์œผ๋กœ ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค. ๊ฐ ๋žญํฌ์—๋Š” ํ• ๋‹น๋œ Q์˜ ์ƒค๋“œ๊ฐ€ ์žˆ์œผ๋ฉฐ, ์ด๋ฅผ Q_i๋ผ๊ณ  ๋ถ€๋ฅด๊ฒ ์Šต๋‹ˆ๋‹ค. ์ด ํ–‰๋ ฌ์€ ์ „์ฒด ๊ณ„์‚ฐ ๊ณผ์ • ๋™์•ˆ ํ•ด๋‹น ๋žญํฌ์—๋งŒ ์œ ์ง€๋ฉ๋‹ˆ๋‹ค. ๋งˆ์ฐฌ๊ฐ€์ง€๋กœ, ๊ฐ ๋žญํฌ์—๋Š” K์™€ V์˜ ์ž์ฒด ์ƒค๋“œ๊ฐ€ ์žˆ์œผ๋ฉฐ, ์ด๋ฅผ ๊ฐ๊ฐ K_i์™€ V_i๋ผ๊ณ  ๋ถ€๋ฅด๊ฒ ์Šต๋‹ˆ๋‹ค. ๊ทธ๋Ÿฐ ๋‹ค์Œ, ๊ฐ ๋žญํฌ๋Š” ์ž์‹ ์˜ Q_i, K_i, V_i ์ƒค๋“œ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ attention์„ ๊ณ„์‚ฐํ•˜๋ฉฐ, ์ด๋ฅผ attn_i๋ผ๊ณ  ๋ถ€๋ฅด๊ฒ ์Šต๋‹ˆ๋‹ค. ์ด ๊ณ„์‚ฐ ์ค‘์— ๋‹ค๋ฅธ ๋ชจ๋“  ๋žญํฌ์—์„œ Ks์™€ Vs๋ฅผ ์ˆ˜์ง‘ํ•˜๊ธฐ ์œ„ํ•œ ํ†ต์‹  ์ปค๋„์ด ์‹คํ–‰๋ฉ๋‹ˆ๋‹ค. ์‚ฌ์šฉ๋˜๋Š” ํ†ต์‹  ์›์‹œ(primitive)๋Š” context_parallel_shard_rotation ์˜ต์…˜์— ๋”ฐ๋ผ ๋‹ฌ๋ผ์ง‘๋‹ˆ๋‹ค. ์ด ๋ฐฉ์‹์„ ํ†ตํ•ด ๊ฐ ๋žญํฌ๋Š” ๋จผ์ € Q_i, K_i, V_i๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ, ๊ทธ ๋‹ค์Œ์—๋Š” ๋‹ค๋ฅธ ๋ชจ๋“  ๋žญํฌ์˜ K_j์™€ V_j๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋กœ์ปฌ attention์„ ๊ณ„์‚ฐํ•˜๊ฒŒ ๋ฉ๋‹ˆ๋‹ค. ๊ฐ ๋žญํฌ๊ฐ€ ์‹œํ€€์Šค ์ฐจ์›๋ณ„๋กœ ์ƒค๋”ฉ๋œ Q, K and V ํ–‰๋ ฌ์„ ๋ณด์œ ํ•˜๋ฏ€๋กœ, ๊ฒฐ๊ณผ ํ–‰๋ ฌ์˜ ํฌ๊ธฐ๊ฐ€ ์ž‘์•„์ ธ ๋‹จ์ผ GPU์— ์ ์žฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋‹ค์Œ ์˜์‚ฌ ์ฝ”๋“œ๋กœ ์ด๋ฅผ ๊ณต์‹ํ™”ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

comm_kernel = {"allgather": allgather, "alltoall": alltoall}[context_parallel_shard_rotation]
Qi, Ki, Vi = shard(Q, K, V, seq_dim)
attn[i] = attn(Qi, Ki, Vi)
for j in range(context_parallel_size):
    Kj, Vj = comm_kernel()
    attn[j] = attn(Qi, Kj, Vj) # [batch, num_heads, seq_len // context_parallel_size, head_dim]

final_attn = combine(attn)

all-to-all๊ณผ all-gather

all-gather

all-to-all๊ณผ all-gather์˜ ์ฐจ์ด๋Š” ๋ฌด์—‡์ผ๊นŒ์š”? all-gather์˜ ๊ฒฝ์šฐ ํ†ต์‹ ์ด ๋งค์šฐ ๋‹จ์ˆœํ•ฉ๋‹ˆ๋‹ค. (๋ณดํ†ต ๋” ์˜ค๋ž˜ ๊ฑธ๋ฆฌ๋ฏ€๋กœ) ์‹ค์ œ๋กœ๋Š” ๊ณ„์‚ฐ ์ „์—, ๋กœ์ปฌ ์–ดํ…์…˜ attn_i์„ ๊ณ„์‚ฐํ•œ ํ›„ all-gather๋ฅผ ์‹œ์ž‘ํ•˜์—ฌ ๋‹ค๋ฅธ ๋ชจ๋“  rank์˜ Ks์™€ Vs๋ฅผ ์ˆ˜์ง‘ํ•ฉ๋‹ˆ๋‹ค. ์ด ํ†ต์‹ ์ด ์™„๋ฃŒ๋˜๋ฉด ๊ฐ rank๋Š” ๋‹ค๋ฅธ ๋ชจ๋“  rank์˜ Ks์™€ Vs๋ฅผ ๋ชจ๋‘ ๋ณด์œ ํ•˜๊ฒŒ ๋˜๋ฉฐ, ์ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ์ˆœ์ฐจ์ ์œผ๋กœ ์–ดํ…์…˜์„ ๊ณ„์‚ฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด์ƒ์ ์ธ ์‹œ๋‚˜๋ฆฌ์˜ค์—์„œ๋Š” attn_i์˜ ๊ณ„์‚ฐ์ด ์™„๋ฃŒ๋˜๋Š” ์ •ํ™•ํ•œ ์ˆœ๊ฐ„์— all-gather๊ฐ€ ๋๋‚ฉ๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋‚˜ ์‹ค์ œ๋กœ๋Š” ์ด๋Ÿฐ ์ผ์ด ์ผ์–ด๋‚˜์ง€ ์•Š์œผ๋ฏ€๋กœ, attn_i ์ „์ฒด๊ฐ€ ํ†ต์‹ ์˜ ์ผ๋ถ€์™€ ์˜ค๋ฒ„๋žฉ๋˜๊ณ , K_j์™€ V_j๋ฅผ ์‚ฌ์šฉํ•œ ๊ณ„์‚ฐ์„ ์‹œ์ž‘ํ•˜๊ธฐ ์œ„ํ•ด all-gather๊ฐ€ ์™„๋ฃŒ๋  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฌ๋Š” ๊ฒƒ์ด ์ด์ƒ์ ์ธ ์‹ค์ œ ์˜ค๋ฒ„๋žฉ์ž…๋‹ˆ๋‹ค.

all-to-all

All-to-all, or sometimes called ring-rotation utilizes a ring-like communication pattern. After concluding attn_i computation, an all-to-all is launched to send K_i and V_i to the neighbouring ranks. We then repeat this context_parallel_size-1 times, so that each rank sees all the shards of K and V from all other ranks once. In ideal scenario, we prefetch shards K_i+1 and V_i+1 from the neighbouring rank and this communication is exactly overlapped with computation of our current attn_i. Again, realistically, this perfect overlap doesnโ€™t ever happen. Given the nature of this approach, if we donโ€™t achieve perfect overlap, the penalty is way larger than with all-gather.

์–ด๋–ค ํšŒ์ „ ๋ฐฉ๋ฒ•์„ ์„ ํƒํ•ด์•ผ ํ• ๊นŒ์š”?

In theory, all-to-all should be the better choice. Though in practice, it rarely is. Therefore, we default to all-gather, as itโ€™s more likely to achieve better performance. Extensive benchmarks from the torchtitan team also show that all-to-all rarely outperforms all-gather. Though, we still provide both options, as you might find one to be better for your use case.

์•„๋ž˜ ์ด๋ฏธ์ง€์˜ ํ”„๋กœํŒŒ์ผ๋Ÿฌ ์ถœ๋ ฅ์—์„œ ์ด ๋ฌธ์ œ๋ฅผ ์ง์ ‘ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

all-to-all profiler output
Figure 1: In red you can see the idle time, while we wait for the all-to-all kernel to finish. Highlighted in the first blue bar, you can see that it takes ~250us to finish, which is repeated N-1 times for each attention call, where N is the context parallel size.

์™œ FSDP2๋งŒ ์‚ฌ์šฉํ• ๊นŒ์š”?

FSDP2์™€ ํ•จ๊ป˜ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ๋งŒ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. context_parallel_size์™€ dp_shard_size์˜ ๊ฒฐํ•ฉ๋œ ๋ฉ”์‹œ๋ฅผ ์ƒ์„ฑํ•˜์—ฌ ๊ทธ ์ž ์žฌ๋ ฅ์„ ์ตœ๋Œ€ํ•œ ํ™œ์šฉํ•˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค. ๋™์ž‘ ๋ฐฉ์‹์€ ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค. cp_size*dp_shard_size ํฌ๊ธฐ์˜ ๊ฒฐํ•ฉ๋œ ๋ฉ”์‹œ์— ๊ฑธ์ณ ๋ชจ๋ธ์„ ์ƒค๋”ฉํ•˜์—ฌ ๋ฉ”๋ชจ๋ฆฌ ์ ˆ๊ฐ ํšจ๊ณผ๋ฅผ ๊ทน๋Œ€ํ™”ํ•ฉ๋‹ˆ๋‹ค. ์•„๋ž˜ ์ด๋ฏธ์ง€์— ํ‘œ์‹œ๋œ ๋ฐ”์™€ ๊ฐ™์ด FSDP ํ†ต์‹ ์ด ์–ดํ…์…˜ ๊ณ„์‚ฐ๊ณผ ์™„์ „ํžˆ ์˜ค๋ฒ„๋žฉ๋˜๋ฏ€๋กœ, ์ผ์ข…์˜ '๋ฌด์ƒ ํ˜œํƒ'์ด๋ผ๊ณ  ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

why FSDP2+CP
๊ทธ๋ฆผ 2: ํŒŒ๋ž€์ƒ‰ ์‚ฌ๊ฐํ˜•(Stream 23)์—์„œ `FSDP ์ƒค๋“œ์˜ ํ”„๋ฆฌํŽ˜์น˜๊ฐ€ ์–ดํ…์…˜ ๊ณ„์‚ฐ(Stream 7)๊ณผ ์™„์ „ํžˆ ๊ฒน์ณ ์žˆ๋Š” ๊ฒƒ์„ ๋ณผ ์ˆ˜ ์žˆ์œผ๋ฉฐ, ๋นจ๊ฐ„์ƒ‰ ์‚ฌ๊ฐํ˜•(Stream 24)์—์„œ๋Š” ์˜ฌ๊ฐ€๋” ์ปค๋„๋กœ ์ธํ•ด ์œ ํœด ์‹œ๊ฐ„ ๋ฒ„๋ธ”์ด ๋ฐœ์ƒํ•˜์—ฌ ๊ณ„์‚ฐ ์ŠคํŠธ๋ฆผ(7)์ด ์œ ํœด ์ƒํƒœ๊ฐ€ ๋˜๋Š” ๊ฒƒ์„ ๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

In the figure above, you can also note the difference between all-to-all and all-gather. While in all-to-all (Figure 1), we launch a communication kernel N-1 times for each attention call, in all-gather (Figure 2), we launch a communication kernel only once. This results in a bigger bubble, but it only happens once per attention call, while in all-to-all, it happens N-1 times.

์กฐ์ธํŠธ ๋ฉ”์‹œ์—์„œ์˜ ๋ฐ์ดํ„ฐ ๋””์ŠคํŒจ์นญ

cp ์„œ๋ธŒ๊ทธ๋ฃน ์ „์ฒด์— ๋™์ผํ•œ ๋ฐ์ดํ„ฐ ๋ฐฐ์น˜๋กœ ๋””์ŠคํŒจ์น˜ํ•˜์—ฌ ๊ฒฐ๊ณผ๊ฐ€ ์ •ํ™•ํ•˜๋„๋ก ๋ณด์žฅํ•ฉ๋‹ˆ๋‹ค. (์ฆ‰, cp ์„œ๋ธŒ๊ทธ๋ฃน์˜ ๊ฐ ๋žญํฌ๋Š” ๋™์ผํ•œ ๋ฐ์ดํ„ฐ ๋ฐฐ์น˜๋ฅผ ๋ฐ›์Šต๋‹ˆ๋‹ค.) ํ•˜์ง€๋งŒ dp_shard ๊ทธ๋ฃน์˜ ๊ฐ ๋žญํฌ์—๋Š” ์„œ๋กœ ๋‹ค๋ฅธ ๋ฐฐ์น˜๋ฅผ ๋””์ŠคํŒจ์น˜ํ•ฉ๋‹ˆ๋‹ค. ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ƒ๊ฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

# 8 GPUS, --dp_shard_size 4, --cp_size 2
# mesh = [[0, 1], [2, 3], [4, 5], [6, 7]]
# model is sharded across the whole mesh (each GPU holds 1/8 of the model)
# GPUs 0,1 = batch 0
# GPUs 2,3 = batch 1
... and so on.
Update on GitHub