Add chatbot example using Qwen2 #282
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Zero is a value that is "safe" for more inputs with an `_ids` suffix, such as `position_ids`.
In HuggingFace models this input's sequence axis is expected to have the same size as the sequence length, not the length of the input IDs that are being provided at the current step. The length was correct for the initial prompt but wrong for subsequent generation steps. However when only one new token was added during iterative decoding, the `attention_mask` worked despite being the wrong size because 1-sized inputs are broadcast by various operators. When appending multiple tokens after the initial generation however, such as when adding a tokenized chat message from a user, this broadcasting failed.
- Fix a design mistake that using `stop_on_token` would cause generation to silently stop without propagating the error to the caller. - Support multiple end-of-turn token IDs. Qwen2 for example can emit either `<|endoftext|>` or `<|im_end|>` tokens.
This is useful in chat applications for example where generation alternates between iterative decoding and feeding in tokenized user input.
This allows loading `tokenizer.json` files which specify NFC normalization, but doesn't actually implement the normalization yet. This is OK as long as the input text doesn't require it.
Qwen2 was chosen as an initial chatbot example because its tokenization is very similar to the already-supported GPT-2 and it is one of the best very small instruction-tuned models.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a chatbot demo. This uses the Qwen2 model since it provides 0.5b and 1.5b sizes that are best-in-class, and the tokenization is conveniently a derivative of the GPT-2 tokenization which rten-text supports. Larger models would produce better results, but since RTen only supports fp32 precision at present, such models become slow due to the memory bandwidth requirements. 1.5b is about the largest size that produces "usable" speed on my Intel i5.
In the process it was necessary to add some capabilities to rten-generate and rten-text to better support chat-like applications:
Generator::append_promptattention_maskinput. It is supposed to have the length of the input sequence not just the new input IDs. Previously it was incorrect after the initial step, but the code worked because on subsequent steps it had size 1 and that gets broadcast to whatever size is required.<|imend|>and<|endoftext|>TODO
append_promptReplace dummy NFC normalization with proper implementation(Deferred for later)