Python: Bug: VertexAIChatCompletion can't handle classes or unions in kernel functions.

[q-stefanmuscalu]

Describe the bug
Kernel functions can't handle classes or unions when using VertexAIChatCompletion

To Reproduce
Steps to reproduce the behavior:

1. Enable auto function calling
   VertexAIChatPromptExecutionSettings(function_choice_behavior=FunctionChoiceBehavior.Auto())
2. Define a plugin that returns a class:

class PersonDetails(KernelBaseModel):
    age: int
    address: str

class SearchPlugin:

    @kernel_function(
        name="SearchPerson",
        description="Search details of a person given their id."
    )
    def search_person(
        self,
        person_id: Annotated[str, "The person ID to search"]
    ) -> Annotated[PersonDetails, "The details of the person"]:
        print(f"\n\n called with person id {person_id} \n\n")
        return PersonDetails(age=42, address="123 Main Street London UK")

3. Add the plugin to the kernel and ask it to search for something:

    kernel = Kernel()

    chat_function = kernel.add_function(
        prompt="{{$chat_history}}{{$user_input}}",
        plugin_name="ChatBot",
        function_name="Chat",
    )

    kernel.add_plugin(SearchPlugin(), plugin_name="SearchPlugin")
    
        chat_completion = VertexAIChatCompletion(
        project_id="???",
        region="???",
        gemini_model_id="gemini-1.5-pro"
    )

    kernel.add_service(chat_completion)

    kernel_args = KernelArguments(
        settings=VertexAIChatPromptExecutionSettings(
            function_choice_behavior=FunctionChoiceBehavior.Auto(
                filters={"excluded_plugins": ["ChatBot"]}
            )
        )
    )

    kernel_args["user_input"] = "Search for person id 9b3f6e40"
    kernel_args["chat_history"] = ChatHistory()

    result = await kernel.invoke(chat_function, arguments=kernel_args)

4. See error:

.venv/lib/python3.10/site-packages/semantic_kernel/functions/kernel_function_from_prompt.py", line 177, in _invoke_internal
    raise FunctionExecutionException(f"Error occurred while invoking function {self.name}: {exc}") from exc
semantic_kernel.exceptions.function_exceptions.FunctionExecutionException: Error occurred while invoking function Chat: Unable to coerce value: PersonDetails(age=42, address='123 Main Street London UK')

Setting the person_id to a class instead of a string results in a different error:

    def search_person(
        self,
        person_id: Annotated[PersonId, "The person ID to search"]
    ) -> Annotated[str, "The details of the person"]:

Error:

.venv/lib/python3.10/site-packages/pydantic/main.py", line 568, in model_validate
    return cls.__pydantic_validator__.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for PersonId
  Input should be a valid dictionary or instance of PersonId [type=model_type, input_value='9b3f6e40', input_type=str]

A different issue, but similar to the above:
I get a similar error if I use a union in the function argument:

    def search_person(
        self,
        person_id: Annotated[str | int, "The person ID to search"]
    ) -> Annotated[str, "The details of the person"]

Error:

.venv/lib/python3.10/site-packages/semantic_kernel/functions/kernel_function_from_prompt.py", line 177, in _invoke_internal
    raise FunctionExecutionException(f"Error occurred while invoking function {self.name}: {exc}") from exc
semantic_kernel.exceptions.function_exceptions.FunctionExecutionException: Error occurred while invoking function Chat: Protocol message Schema has no "anyOf" field.

Expected behavior
I would expect the functions to handle classes and unions.

Screenshots
If applicable, add screenshots to help explain your problem.

Platform

• OS: Mac
• IDE: PyCharm
• Language: Python
• Source: SK version 1.5.0

Additional context
Gemini with VertexAI, Python 3.10, Gemini model gemini-1.5-pro

[TaoChenOSU]

Thanks for reporting! I was able to reproduce the issues and working on fixing them.

There are 3 issues reported:

1. The return parameter of a function is a complex type: This is a bug and will be fixed.
2. The input parameter of a function is a complex type: This is not reproducible.
3. The input of the parameter of a function is a union type: Unfortunately, Google function calling currently only supports the following types: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/function-calling#schema

[q-stefanmuscalu]

@TaoChenOSU thank you for looking into this! I will try the next release and let you know if the complex input parameter is still an issue.