Use accurate token counts #1024
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `total` token count is now based upon an identical format conversation sent to the LLM during response generation. This results in counts that should be accurate, and prevent token limit errors entirely.
|
this seems to be even more confusing now. From user perspective I don't understand why empty code studio shows 129 tokens used and when I add a file with 6.7k tokens the total token counter goes to 8.2k. I think this needs some further thought 
|
|
it is much closer now, but I still see some weird discrepancy |
anastasiya1155
force-pushed
the
fix-token-counts
branch
from
30ba9ca
to
ab1b017
Compare
ggordonhall
approved these changes
Oct 10, 2023
# Conflicts: # client/src/locales/en.json # client/src/locales/es.json # client/src/locales/ja.json # client/src/locales/zh-CN.json
Dead4W
added a commit
to Dead4W/bloop
that referenced
this pull request
Oct 15, 2023
* Use accurate token counts (BloopAI#1024) * Use accurate token counts The `total` token count is now based upon an identical format conversation sent to the LLM during response generation. This results in counts that should be accurate, and prevent token limit errors entirely. * Match token counts precisely, and add baseline count * Calculate `total` as summation of other section token counts * add message to response if it ended when token limit exceeded, no translations --------- Co-authored-by: anastasiia <anastasiya1155@gmail.com> * Send full redirect target to cognito (BloopAI#1034) * Update Helm chart (BloopAI#1033) * update helm chart * remove secret.yaml * put back secret.yaml * mandatory Semantic in /search (BloopAI#1039) * Upgrade qdrant version to 1.6 (BloopAI#1037) * Upgrade qdrant version * Update qdrant binaries * Fix race in credentials polling (BloopAI#1042) Previously the assumption here was that this path is locked & safe when there is a furnished github cred in the system. However, when the user logs out, the `unwrap()` call can blow up the task, and this may cause issues. * Enable paid features for desktop users (BloopAI#1038) * Add pro features to default builds * Check user's status through `User` object * Adapt webserver layer for paid feature gate * Just enforce schema right out of the gate * Fix date parsing logic * allow paid users sync branches * Fix clippy features * Disable branch switching for local repos * cloud implies pro * Update dockerfile --------- Co-authored-by: anastasiia <anastasiya1155@gmail.com> * Update flake (BloopAI#1044) * Debug logs for initialisation (BloopAI#1036) * debug logs for initialisation * Scope logging * DB too * Clippy error --------- Co-authored-by: rsdy <p@symmetree.dev> * Allow indexing of large files (BloopAI#1040) * WIP * Support indexing large files, but lazy load them from local file system * bump tantivy to v0.21 (BloopAI#1043) * bump tantivy to v0.22 * address clippy * fix broken tests * bump version to 0.5.6 (BloopAI#1047) --------- Co-authored-by: calyptobai <111788964+calyptobai@users.noreply.github.com> Co-authored-by: anastasiia <anastasiya1155@gmail.com> Co-authored-by: rsdy <rsdy@users.noreply.github.com> Co-authored-by: Gabriel Gordon-Hall <ggordonhall@gmail.com> Co-authored-by: rsdy <p@symmetree.dev> Co-authored-by: akshay <nerdy@peppe.rs> Co-authored-by: Anastasiia Solop <35258279+anastasiya1155@users.noreply.github.com> Co-authored-by: Ilya Zedgenizov <izedgenizov@saber.games>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
totaltoken count is now based upon an identical format conversation sent to the LLM during response generation.This results in counts that should be accurate, and prevent token limit errors entirely.
Closes BLO-1724