I encountered a RuntimeError while running the training

[XGraphKhipu]

Hello,

when I run: python train.py name=humanml_trans_enc_512_4gpu_600k dataset=humanml training.eval_during_training=0 model.cond_mask_prob=0.1 guidance_param=2.5 training.overwrite=1 training.log_interval=1000 batch_size=128 training.num_steps=600000 num_workers=8 input_text=./assets/example_text_prompts.txt is_debug=0

I get this error message:

return module_to_run(*inputs[0], **kwargs[0])
  File "/home/user/anaconda3/envs/motionfm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/motionfm/model/mdm_flow.py", line 323, in forward
    x = self.input_process(x)
  File "/home/user/anaconda3/envs/motionfm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/motionfm/model/mdm_flow.py", line 423, in forward
    x = self.poseEmbedding(x)  # [seqlen, bs, d]
  File "/home/user/anaconda3/envs/motionfm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/motionfm/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (25088x251 and 263x512)

What could be happening?