Python: #6761 Onnx Connector #8106

nmoeller · 2024-08-14T07:39:51Z

Motivation and Context

Why is this changed required ?
To enable Onnx Models with Semantic Kernel, there was the issue Python: Add support for local models via ONNX #6761 in the Backlog to add a Onnx Connector
What problem does it solve ?
It solves the problem, that semantic kernel is not yet integrated with Onnx Gen Ai Runtime
What scenario does it contribute to?
The scenario is to use different connector than HF,OpenAI or AzureOpenAI. When User's want to use Onnx they can easliy integrate it now
If it fixes an open issue, please link to the issue here.
Python: Add support for local models via ONNX #6761

Description

The changes made are designed by my own based on other connectors, i tried to stay as close as possible to the structure.
For the integration i installed the mistral python package in the repository.

I added the following Classes :

OnnxCompletionBase --> Responsible to control the inference
OnnxTextCompletion --> Inherits from OnnxCompletionBase
- Support for Text Completion with and without Images
- Ready for Multimodal Inference
- Ready for Text Only Inference
- Supports all Models on onnxruntime-genai
OnnxChatCompletion -->Inherits from OnnxCompletionBase
- Support for Text Completion with and without Images
- The user needs to provide the corresponding chat template to use this class
- Ready for Multimodal Inference
- Ready for Text Only Inference
- Supports all Models on onnxruntime-genai

What is integrated yet :

OnnxCompletionBase Class
OnnxChatCompletionBase Class with Dynamic Template Support
OnnxTextCompletionBase Class
Sample Multimodal Inference with Phi3-Vision
Sample of OnnxChatCompletions with Phi3
Sample of OnnxTextCompletions with Phi3
Integration Tests
Unit Tests

Some Notes

Contribution Checklist

The code builds clean without any errors or warnings
The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
All unit tests pass, and I have added new tests where possible
I didn't break anyone 😄

python/pyproject.toml

python/semantic_kernel/connectors/ai/onnx/services/onnx_text_completion.py

…/nmoeller/semantic-kernel into issue-6761-ONNX-gen-ai-Connector

…i-Connector

…i-Connector # Conflicts: # python/tests/integration/completions/chat_completion_test_base.py # python/uv.lock

python/semantic_kernel/connectors/ai/onnx/onnx_gen_ai_prompt_execution_settings.py

python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai_completion_base.py

python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai_chat_completion.py

python/tests/conftest.py

TaoChenOSU · 2024-09-18T22:51:10Z

python/tests/integration/completions/test_text_completion.py

-    if os.environ["OLLAMA_MODEL"]:
-        ollama_setup = True
-except KeyError:
+    OllamaTextCompletion()


Hmm, if we do this, it's very likely that we will never know if our test pipeline is setup correctly since all of the ollama tests will be skipped instead of failing.

Same for the others in chat & text completion

python/semantic_kernel/connectors/ai/onnx/onnx_gen_ai_settings.py

python/tests/integration/completions/test_utils.py

TaoChenOSU · 2024-09-18T23:05:24Z

Regarding our offline conversation on the prompt template, is using a prompt template to parse the chat history to some format an overkill? Prompt template can do much more that substituting arguments. Is it possible to override the _prepare_chat_history_for_request method to get what Onnx wants?

…_completion_base.py Co-authored-by: Tao Chen <taochen@microsoft.com>

…_chat_completion.py Co-authored-by: Tao Chen <taochen@microsoft.com>

…/nmoeller/semantic-kernel into issue-6761-ONNX-gen-ai-Connector

eavanvalkenburg

We need to rethink the prompt based setup but the rest looks good!

python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai_chat_completion.py

…i-Connector

nmoeller · 2024-09-20T14:36:51Z

As disscussed with @eavanvalkenburg i introduced hardcoded Templates in the onnx_utils.py file and integrated applying the Template into the _prepare_chat_history_for_request.

If people want to introduce custom templates they can overwrite the _prepare_chat_history_for_request.
Also the output from string to the SK Objects can now be overwritten, since they are seperated into _create_chat_message_content and _create_streaming_chat_message_content.

eavanvalkenburg

Couple of small improvements, but looking good!

python/semantic_kernel/connectors/ai/onnx/onnx_utils.py

Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>

setup for onnx connector

ff979ba

markwallace-microsoft added the python Pull requests for the Python Semantic Kernel label Aug 14, 2024

nmoeller changed the title ~~Python : Issue-6761-Onnx-Connector~~ Python: Issue-6761-Onnx-Connector Aug 14, 2024

nmoeller changed the title ~~Python: Issue-6761-Onnx-Connector~~ Python: #6761 Onnx Connector Aug 14, 2024

eavanvalkenburg reviewed Aug 14, 2024

View reviewed changes

python/pyproject.toml Outdated Show resolved Hide resolved

TaoChenOSU reviewed Aug 15, 2024

View reviewed changes

python/semantic_kernel/connectors/ai/onnx/services/onnx_text_completion.py Outdated Show resolved Hide resolved

nmoeller added 11 commits August 16, 2024 11:08

initial implementation commit

49a2a72

Merge branch 'main' into issue-6761-ONNX-gen-ai-Connector

b2c5a70

initial unit tests for onnx text completion

342db1d

Merge branch 'issue-6761-ONNX-gen-ai-Connector' of https://github.com…

9c371de

…/nmoeller/semantic-kernel into issue-6761-ONNX-gen-ai-Connector

added chat completion support

adf262b

added small comment regarding Image Opening

a40a6cb

Merge remote-tracking branch 'origin/main' into issue-6761-ONNX-gen-a…

fd6d9b4

…i-Connector

migrated to uv

b118396

Merge remote-tracking branch 'origin/main' into issue-6761-ONNX-gen-a…

0da7615

…i-Connector

Merge remote-tracking branch 'origin/main' into issue-6761-ONNX-gen-a…

0b6df05

…i-Connector

added unit tests and integration tests

3c41141

nmoeller marked this pull request as ready for review September 17, 2024 14:32

nmoeller requested a review from a team as a code owner September 17, 2024 14:32

nmoeller requested a review from TaoChenOSU September 17, 2024 14:32

nmoeller added 2 commits September 18, 2024 17:39

added unit tests and integration tests

81bf663

Merge remote-tracking branch 'origin/main' into issue-6761-ONNX-gen-a…

bd21157

…i-Connector # Conflicts: # python/tests/integration/completions/chat_completion_test_base.py # python/uv.lock

TaoChenOSU reviewed Sep 18, 2024

View reviewed changes

nmoeller and others added 6 commits September 19, 2024 08:44

Update python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai…

21f585f

…_completion_base.py Co-authored-by: Tao Chen <taochen@microsoft.com>

Update python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai…

58702f3

…_completion_base.py Co-authored-by: Tao Chen <taochen@microsoft.com>

Update python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai…

8908cb9

…_completion_base.py Co-authored-by: Tao Chen <taochen@microsoft.com>

Update python/semantic_kernel/connectors/ai/onnx/services/onnx_gen_ai…

af15de6

…_chat_completion.py Co-authored-by: Tao Chen <taochen@microsoft.com>

integrated pr feedback

14128e2

Merge branch 'issue-6761-ONNX-gen-ai-Connector' of https://github.com…

352dede

…/nmoeller/semantic-kernel into issue-6761-ONNX-gen-ai-Connector

integrated pr feedback

9471cbb

eavanvalkenburg requested changes Sep 19, 2024

View reviewed changes

nmoeller added 6 commits September 20, 2024 09:37

Merge remote-tracking branch 'origin/main' into issue-6761-ONNX-gen-a…

49514d6

…i-Connector

integrated new template architecture

322afd8

adjusted default max length for int test with image

7c31645

fixed documentation in samples

477db50

fixed docstring for chat completion

030254a

Merge branch 'main' into issue-6761-ONNX-gen-ai-Connector

627f0b7

nmoeller requested review from TaoChenOSU and eavanvalkenburg September 20, 2024 14:34

eavanvalkenburg reviewed Sep 20, 2024

View reviewed changes

python/semantic_kernel/connectors/ai/onnx/onnx_utils.py Outdated Show resolved Hide resolved

python/semantic_kernel/connectors/ai/onnx/onnx_utils.py Outdated Show resolved Hide resolved

python/semantic_kernel/connectors/ai/onnx/onnx_utils.py Outdated Show resolved Hide resolved

nmoeller and others added 2 commits September 21, 2024 11:45

Update python/semantic_kernel/connectors/ai/onnx/onnx_utils.py

e86db81

Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>

implemented pr feedback

cdb18ba

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python: #6761 Onnx Connector #8106

Python: #6761 Onnx Connector #8106

nmoeller commented Aug 14, 2024 •

edited

Loading

TaoChenOSU Sep 18, 2024 •

edited

Loading

TaoChenOSU Sep 18, 2024 •

edited

Loading

TaoChenOSU commented Sep 18, 2024

eavanvalkenburg left a comment

nmoeller commented Sep 20, 2024 •

edited

Loading

eavanvalkenburg left a comment

Python: #6761 Onnx Connector #8106

Are you sure you want to change the base?

Python: #6761 Onnx Connector #8106

Conversation

nmoeller commented Aug 14, 2024 • edited Loading

Motivation and Context

Description

Some Notes

Contribution Checklist

TaoChenOSU Sep 18, 2024 • edited Loading

Choose a reason for hiding this comment

TaoChenOSU Sep 18, 2024 • edited Loading

Choose a reason for hiding this comment

TaoChenOSU commented Sep 18, 2024

eavanvalkenburg left a comment

Choose a reason for hiding this comment

nmoeller commented Sep 20, 2024 • edited Loading

eavanvalkenburg left a comment

Choose a reason for hiding this comment

nmoeller commented Aug 14, 2024 •

edited

Loading

TaoChenOSU Sep 18, 2024 •

edited

Loading

TaoChenOSU Sep 18, 2024 •

edited

Loading

nmoeller commented Sep 20, 2024 •

edited

Loading