Sequence packing with ChatDataset

[LiweiPE]

Hello, may I know how to use gpt_sft_chat_dataset with sequence packing? I couldnt find documentation related to it. I only found to convert from input-output format.

[qijiaxing]

Sequence packing steps can found here

To make it work for chat dataset, you probably need to add the chat format option, like this:

python /workspace/sequence_packing/tokenize_dataset.py \
  model.data.train_ds.file_names=[/path/to/training.jsonl] \
  model.data.train_ds.max_seq_length=4096 \
  +model.data.chat=True \
  +model.data.chat_prompt_tokens.system_turn_start='<extra_id_0>' \
      +model.data.chat_prompt_tokens.turn_start='<extra_id_1>' \
      +model.data.chat_prompt_tokens.label_start='<extra_id_2>' \
      +model.data.chat_prompt_tokens.end_of_turn="\x0A" \
      +model.data.chat_prompt_tokens.end_of_name="\x0A" \
  model.restore_from_path=/path/to/starcoder2.nemo \ # any starcoder2 .nemo models works here
  +output_path=/path/to/my_dataset.npy