sgl-project sglang · Discussions · GitHub

[Bug] Why does max_new_tokens affect token usage？
Q&A CSEEduanyu

Sort by: Latest activity

Categories

Discussions

You must be logged in to vote

Concurrent Request Processing

rangehow asked Sep 19, 2024 in Q&A · Unanswered

1
You must be logged in to vote

Mistral Nemo -> CUDA out of memory on 2 x 80GB H100

draqos asked Sep 16, 2024 in Q&A · Unanswered

1
You must be logged in to vote

Issues with throughput for 405B model

dmakhervaks started Aug 27, 2024 in General · Closed

1
You must be logged in to vote

dp and tp size for 8 x A100

Desmond819 asked Aug 16, 2024 in Q&A · Closed · Answered

8
You must be logged in to vote

question about model pipeline across nodes

db24 started Aug 19, 2024 in General · Closed

2
You must be logged in to vote

[Feature] Question about kvcache offloading to disk or CPU memory

MazarineGlacier asked Aug 13, 2024 in Q&A · Closed · Answered

1
You must be logged in to vote

Why is there only one gpu being used even when I set --dp 2?

catalpa-bungei asked Aug 25, 2024 in Q&A · Closed · Answered

1
You must be logged in to vote

Stuck in creating runtime

Ricardokevins asked Aug 21, 2024 in Q&A · Closed · Answered

1
You must be logged in to vote

Token usage always shows 0.00

Desmond819 asked Aug 16, 2024 in Q&A · Closed · Answered

4
You must be logged in to vote

The accuracy of using the batch processing function of the client is low

guleng asked Sep 9, 2024 in Q&A · Closed · Answered

3
You must be logged in to vote

the function of triton_start_loc

zhaoofengyang asked Sep 5, 2024 in Q&A · Closed · Unanswered

1
You must be logged in to vote

How to send many messages to open AI endpoint in one request?

Stealthwriter started Sep 4, 2024 in General · Closed

1
You must be logged in to vote

Llava-OV video inference

ehayeshaiper asked Sep 4, 2024 in Q&A · Closed · Answered

1
You must be logged in to vote

Mistral model supported by sglang

mikelfried asked Sep 6, 2024 in Q&A · Closed · Answered

1
You must be logged in to vote

Sglang combined with ray distributed inference

guleng asked Sep 9, 2024 in Q&A · Closed · Answered

7
You must be logged in to vote

json decode：What exactly is the meaning of "token fusion between input and output"?

CSEEduanyu started Sep 8, 2024 in General · Closed

1
You must be logged in to vote

What are the advantages of flashinfer.sampling over the torch version?

CSEEduanyu started Aug 30, 2024 in General · Closed

1
You must be logged in to vote

[Feature] In Sglang ，Is chunked-prefill use fused(prefill+decode) batch?

CSEEduanyu asked Aug 20, 2024 in Q&A · Closed · Answered

6
You must be logged in to vote

Support for returning Logits and Calculating Perplexity During Model Evaluation?

hxer7963 asked Sep 3, 2024 in Q&A · Closed · Answered

1
You must be logged in to vote

CohereForAI/aya model support

SSeanin asked Aug 19, 2024 in Q&A · Closed · Answered

1
You must be logged in to vote

Global config finetuning

Desmond819 started Aug 14, 2024 in General · Closed

2
You must be logged in to vote

Do you support SGLang video inference for Llava-OneVision ?

ehayeshaiper asked Sep 2, 2024 in Q&A · Closed · Answered

1
You must be logged in to vote

More Quantization Options?

Abdulhanan535 asked Sep 4, 2024 in Q&A · Closed · Unanswered

2
You must be logged in to vote

[Bug] Why does max_new_tokens affect token usage？

CSEEduanyu asked Aug 7, 2024 in Q&A · Unanswered

13
You must be logged in to vote

Does sglang Support Inference for Multiple Models at the Compilation Stage?

DopeorNope-Lee asked Aug 30, 2024 in Q&A · Unanswered

0