serde: Require T::serde_fields for all serde structs

[oleiman]

With the exception of those that implement BOTH a custom reader and a custom writer. In that case, the fields tuple is not needed, so envelope_to_tuple will not participate in overload resolution.

In effect, this PR modifies serde to require using serde_fields for all structs EXCEPT those implementing both serde_read and serde_write. In these cases envelope_to_tuple won't generally participate in overload resolution

Why do this? #21431 provides a motivating example. Where possible, we try to avoid compat-breaking serde changes by appending new fields to the end of the binary format. The implicit coupling between aggregate field ordering and binary format makes it easy to introduce subtle serialization bugs or unnecessary compat breaks.

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v24.2.x
• v24.1.x
• v23.3.x

Release Notes

• none

[vbotbuildovich]

new failures in https://buildkite.com/redpanda/redpanda/builds/54056#0191c394-ac5e-4c83-afcb-2e446caf341f:

"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.S3"

[vbotbuildovich]

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/54056#0191c394-ac64-43a8-8fc0-ecaacfa9d975

[oleiman]

CI Failure:

• build - Looks like CI Failure (Internal Server Error) in EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures #14898 (known, unrelated)
  • note that this is a non-functional change, so the build passing across the board should be sufficient for acceptance.

[oleiman]

force push to remove has_serde_fields concept.

[oleiman]

bazel build failed initially - looks to have been a transient 502 error pulling something from github

[oleiman]

CI Failure:

• cluster_cloud_metadata_rpfixture timed out build