Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for data with schema with non-struct top level #78

Open
ikbalkaya opened this issue May 31, 2022 · 0 comments
Open

Add support for data with schema with non-struct top level #78

ikbalkaya opened this issue May 31, 2022 · 0 comments
Labels
enhancement New feature or improved functionality.

Comments

@ikbalkaya
Copy link
Contributor

ikbalkaya commented May 31, 2022

We need to consider converting data with schema where schema.type != STRUCT at top level. Currently messages with top level schemas other than STRUCT won't be JSONified. BYTES and STRINGs will be passed as is as a data to Ably message. Other top level types are not supported and connector will throw an exception. While this seems enough good for most cases, we should consider supporting all top level Kafka connect schemas.

I am not sure how likely it is a top level schema could be a primitive type, but I think we should still address it. For complex types we already support BYTES and STRUCT, We just need to address ARRAY and MAP types.

This issue is created from conversation below

If I understand correctly, this will fail if valueSchema != null and valueSchema != STRUCT, i.e. there is a schema set, but it's not a Struct type, such as Array, String, Integer, etc, as we'll still be passing a Connect data type to ably-java, which it won't know how to process?

Have we made a deliberate decision to only support Struct as the top-level schema? If so, we should probably add a more explicit error, else it'll be raised as a bug should somebody try to send an array or similar. I'm not sure it's really necessary to require this? Are not 1, "foo", 3.145, [1, 2, 3] etc all valid JSON values? The docs suggest we at least support arrays.

If others agree, I'm happy with us just raising a ticket to add support for other top-level types later rather than delaying this PR.

Originally posted by jaley in #73 (comment)

┆Issue is synchronized with this Jira Uncategorised by Unito

@ikbalkaya ikbalkaya self-assigned this May 31, 2022
@ikbalkaya ikbalkaya added the enhancement New feature or improved functionality. label May 31, 2022
@ikbalkaya ikbalkaya removed their assignment May 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or improved functionality.
Development

No branches or pull requests

1 participant