Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: Add metrics instrumentation #8317

Conversation

TaoChenOSU
Copy link
Contributor

@TaoChenOSU TaoChenOSU commented Aug 21, 2024

Motivation and Context

We very recently added instrumentation to kernel functions: #8280. That PR added logs and traces for observing a kernel function. This PR adds metrics and modifies the telemetry sample app to show how to export metric instruments to Application Insights.

Description

  1. Kernel function now produces metrics data for observability.
  2. Add code to export metric instruments to the telemetry sample.
  3. Add Google AI as a second service to the telemetry sample to show case nested spans.
  4. Mics optimizations.

image

image

image

Contribution Checklist

@TaoChenOSU TaoChenOSU added the python Pull requests for the Python Semantic Kernel label Aug 21, 2024
@TaoChenOSU TaoChenOSU self-assigned this Aug 21, 2024
@TaoChenOSU TaoChenOSU requested review from a team as code owners August 21, 2024 21:56
@github-actions github-actions bot changed the title Add metrics instrumentation Python: Add metrics instrumentation Aug 21, 2024
@markwallace-microsoft markwallace-microsoft added .NET Issue or Pull requests regarding .NET code kernel.core labels Aug 21, 2024
@github-actions github-actions bot changed the title Python: Add metrics instrumentation .Net: Python: Add metrics instrumentation Aug 21, 2024
@TaoChenOSU TaoChenOSU changed the title .Net: Python: Add metrics instrumentation Python: Add metrics instrumentation Aug 21, 2024
@TaoChenOSU TaoChenOSU linked an issue Aug 21, 2024 that may be closed by this pull request
Copy link
Member

@eavanvalkenburg eavanvalkenburg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couple of notes, but looks good otherwise!

@markwallace-microsoft
Copy link
Member

Python Test Coverage

Python Test Coverage Report
FileStmtsMissCoverMissing
semantic_kernel
   kernel.py1994776%150, 161, 165, 315–318, 425, 439–482
semantic_kernel/agents/group_chat
   agent_chat.py116199%91
   agent_group_chat.py67297%113, 135
   broadcast_queue.py72199%35
semantic_kernel/agents/open_ai
   open_ai_assistant_base.py361399%243, 321–322
semantic_kernel/connectors/ai/anthropic/services
   anthropic_chat_completion.py942573%171–181, 196–205, 222–236
semantic_kernel/connectors/ai/azure_ai_inference/services
   azure_ai_inference_chat_completion.py150497%174, 324–325, 441
semantic_kernel/connectors/ai/embeddings
   embedding_generator_base.py8188%50
semantic_kernel/connectors/ai/google
   shared_utils.py39295%97, 119
semantic_kernel/connectors/ai/google/google_ai/services
   google_ai_chat_completion.py150497%153, 265, 299–300
   utils.py63395%139, 157–162
semantic_kernel/connectors/ai/google/vertex_ai/services
   utils.py65395%141, 159–164
   vertex_ai_chat_completion.py150497%147, 258, 292–293
semantic_kernel/connectors/ai/hugging_face/services
   hf_text_completion.py54394%97, 105, 136
   hf_text_embedding.py32584%79–83
semantic_kernel/connectors/memory/azure_ai_search
   azure_ai_search_collection.py87298%150, 152
semantic_kernel/connectors/memory/redis
   redis_collection.py160299%146, 316
   utils.py451176%145–146, 164, 166, 173–188
semantic_kernel/connectors/openapi_plugin
   openapi_manager.py58297%110–111
   openapi_parser.py88199%127
   openapi_runner.py105298%181–182
semantic_kernel/connectors/openapi_plugin/models
   rest_api_operation.py129199%242
semantic_kernel/contents
   function_call_content.py97199%201
semantic_kernel/data
   vector_store_record_collection.py2491992%410, 470–474, 482–486, 526–529, 536–539
   vector_store_record_utils.py26292%50, 52
semantic_kernel/functions
   kernel_function_decorator.py98199%102
   kernel_function_from_method.py96199%153
   kernel_function_from_prompt.py154795%165–166, 180, 201, 219, 239, 322
   kernel_function_log_messages.py431272%20, 26, 32, 40–46, 58, 64, 70
   kernel_plugin.py187299%472, 475
semantic_kernel/planners
   plan.py2344581%54, 163–165, 197, 214–227, 264, 269, 277–278, 288–291, 308, 313, 329, 332–337, 355, 360, 363, 365, 372, 386–388, 393–397
semantic_kernel/planners/function_calling_stepwise_planner
   function_calling_stepwise_planner.py116497%145, 189–190, 198
semantic_kernel/planners/sequential_planner
   sequential_planner.py64691%71, 75, 109, 125, 134–135
   sequential_planner_extensions.py50982%31–32, 56, 110–124
   sequential_planner_parser.py771284%66–74, 93, 117–120
semantic_kernel/schema
   kernel_json_schema_builder.py119596%50, 84, 206, 210–211
semantic_kernel/template_engine/blocks
   code_block.py77199%119
   named_arg_block.py43198%98
semantic_kernel/utils/telemetry
   decorators.py113496%232, 240, 243, 252
   user_agent.py16288%18–19
TOTAL1078226398% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
2370 1 💤 0 ❌ 0 🔥 51.218s ⏱️

@TaoChenOSU TaoChenOSU added this pull request to the merge queue Aug 22, 2024
Merged via the queue into microsoft:main with commit 6680b2e Aug 22, 2024
28 checks passed
@TaoChenOSU TaoChenOSU deleted the taochen/python-instrument-kernel-function-metrics branch August 22, 2024 15:45
LudoCorporateShark pushed a commit to LudoCorporateShark/semantic-kernel that referenced this pull request Aug 25, 2024
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
We very recently added instrumentation to kernel functions:
microsoft#8280. That PR added
logs and traces for observing a kernel function. This PR adds metrics
and modifies the telemetry sample app to show how to export metric
instruments to Application Insights.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
1. Kernel function now produces metrics data for observability.
2. Add code to export metric instruments to the telemetry sample.
3. Add Google AI as a second service to the telemetry sample to show
case nested spans.
4. Mics optimizations.


![image](https://github.com/user-attachments/assets/141e9bd2-06b2-4399-a913-f8aeb12aa8af)


![image](https://github.com/user-attachments/assets/fd35ad84-c59f-418c-b9e6-84beb4b22eb1)


![image](https://github.com/user-attachments/assets/1f40fac0-23fa-4b3e-b6c9-644e654c0cbd)


### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation kernel.core .NET Issue or Pull requests regarding .NET code python Pull requests for the Python Semantic Kernel
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Python: Instrument kernel functions: traces, logs and metrics
4 participants