You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
我这样改成多卡的:
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
model = torch.nn.DataParallel(model, device_ids=[0, 1, 2, 3, 4, 5, 6, 7])
model = model.half().cuda()
会报错:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument index in method wrapper__index_select)
Activity
thaumstrial commentedon Mar 20, 2023
目前正在添加trainer和deepspeed的支持,就可以支持多卡了,或者你换成trainer应该也能支持多卡
thaumstrial commentedon Mar 20, 2023
可以参考peft库中,使用accelerate分布式训练大模型
https://github.com/huggingface/peft/blob/main/examples/causal_language_modeling/peft_lora_clm_accelerate_big_model_inference.ipynb
cs-liangchen-work commentedon Mar 26, 2023
成功啦,多谢大佬