[Serve][Example] Authentication example #3251
Conversation
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| uvicorn.run(app, host="0.0.0.0", port=8087) |
There was a problem hiding this comment.
maybe can give an example of client program that accesses the server with the verify token
There was a problem hiding this comment.
We've already has one in the Task YAML's comment?
There was a problem hiding this comment.
Implement in python is rather challenging as I didn't find any default support for this; ref https://stackoverflow.com/questions/34359529/python-requests-equivalent-for-curls-location-trusted
There was a problem hiding this comment.
Hmm, not being able to programmatically interacting with the service seems like a key limitation in this mode.
There was a problem hiding this comment.
Yeah... I guess that is another reason we might want to use a proxy. Will investigate more
There was a problem hiding this comment.
Can we test it with OpenAI API? Maybe their library handles the authentication key automatically with redirection.
There was a problem hiding this comment.
I suppose the problem is our controller does not forward the request with previous auth headers..? Correct me if I'm wrong, but why does replica-side Openai API have a relation to this problem..?
There was a problem hiding this comment.
Oh, I mean the client side OpenAI python API, to see if the client querying the endpoint programmatically will have the similar behavior as curl --trust-location.
If this is an issue, we should start migrating our load balancer to proxy based solution.
There was a problem hiding this comment.
Just tried, the OpenAI Client does not support forwarding auth headers as well. It could access the replica endpoint but not the load balancer endpoint.
|
Can we add a README for the example folder to explain how this will work and how to access the service? Also, can we make the example a real LLM serving, e.g. using vllm? |
|
TODO:
|
|
Is this outdated? Should we close this @cblmemo? |
Yes. Closing now |
A simple example with authentication.
Tested (run the relevant ones):
bash format.shpytest tests/test_smoke.pypytest tests/test_smoke.py::test_fill_in_the_namebash tests/backward_comaptibility_tests.sh