feat: fallback to build protobuf from source #67

tisonkun · 2023-06-19T16:33:44Z

No description provided.

1. Avoid bundle binaries 2. Leverage C++ portability Signed-off-by: tison <wander4096@gmail.com>

See https://protobuf.dev/support/version-support/ Signed-off-by: tison <wander4096@gmail.com>

src/protobuf_impl.rs

Signed-off-by: tison <wander4096@gmail.com>

BusyJay · 2023-06-20T02:47:31Z

I'm afraid building from source would require extra dependencies like C/C++ compilers.

tisonkun · 2023-06-20T02:58:08Z

I'm afraid building from source would require extra dependencies like C/C++ compilers.

This patch include three changes:

Support new protoc version string.
Prost build respects bundled protoc
Using protobuf-src over binaries to support macos aarch64 fallback

If you'd prefer add macos aarch64 binary to resolve 3 and agree on the changes for 1 and 2, I can find time to restore and update the binaries.

BusyJay

I believe protoc-osx-aarch64 was omitted because it exceeded the size limit (10MiB at that time) of a crate that can be published to crates.io. Better check if the problem still exists.

BusyJay · 2023-06-20T05:52:00Z

src/lib.rs

+            let caps = ver_re.captures(from_utf8(&o.stdout).unwrap()).unwrap();
+            let major = caps.get(1).unwrap().as_str().parse::<i16>().unwrap();
+            let minor = caps.get(2).unwrap().as_str().parse::<i16>().unwrap();
+            if (major == 3 && minor >= 1) || (major >= 20) {


How about (major, minor) >= (3, 1)?

See the link in commit c1462c6.

I tried to follow protobuf's version policy that they change the version pattern only after 20.x. But it should be physically OK to use (major, minor) >= (3, 1).

BusyJay · 2023-06-20T05:59:26Z

Note protobuf-src may not work for Windows.

tisonkun · 2023-06-20T06:02:02Z

Note protobuf-src may not work for Windows.

OK. Then I'll try the restore and update binaries way. Perhaps I can run cargo publish --dry-run locally to check the limitation but I'm not sure if dry-run requires push permission to the crate.

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2023-06-20T18:07:00Z

@BusyJay Updated. I use 23.3 protoc and check it works well on osx-aarch_64 env.

BusyJay · 2023-06-21T01:24:15Z

The resulting package is clearly larger than 10MiB, I'm afraid this change can't be published to crates.io.

tisonkun · 2023-06-21T05:06:32Z

cargo publish --dry-run
    Updating crates.io index
   Packaging protobuf-build v0.14.1 (/Users/tison/Brittani/protobuf-build)
   Verifying protobuf-build v0.14.1 (/Users/tison/Brittani/protobuf-build)
    Updating crates.io index
   Compiling memchr v2.5.0
   Compiling protobuf v2.28.0
   Compiling bytes v1.4.0
   Compiling regex-syntax v0.7.2
   Compiling bitflags v1.3.2
   Compiling aho-corasick v1.0.2
   Compiling regex v1.8.4
   Compiling protobuf-codegen v2.28.0
   Compiling protobuf-build v0.14.1 (/Users/tison/Brittani/protobuf-build/target/package/protobuf-build-0.14.1)
    Finished dev [unoptimized + debuginfo] target(s) in 13.22s
    Packaged 31 files, 53.8MiB (17.7MiB compressed)
   Uploading protobuf-build v0.14.1 (/Users/tison/Brittani/protobuf-build)
warning: aborting upload due to dry run

It seems dry-run doesn't test against it.

cc @nrc could you please take a look and check the cargo limitation?

cc @weihanglo

weihanglo · 2023-06-21T09:41:22Z

crates.io has its own stricter limitations that Cargo doesn't check at this time. Part of the reason Cargo has less checks because Cargo is not only for crates.io. We would like to see a new publish check API but the work is not on priority so far I can tell.

The 10MiB limit is documented here.

See also

tisonkun · 2023-06-21T09:50:56Z

@BusyJay Perhaps we drop the support to linux-ppcle_64 and linux-x86_32 (or perhaps linux-aarch_64, I don't know) as they are least widely used? And if we can add a feature flag to support users to fallback to protobuf-src on their demand.

-rwxr-xr-x@  1 tison  staff   7.8M Jan  1  1980 protoc-linux-aarch_64
-rwxr-xr-x@  1 tison  staff   9.1M Jan  1  1980 protoc-linux-ppcle_64
-rwxr-xr-x@  1 tison  staff   8.5M Jan  1  1980 protoc-linux-x86_32
-rwxr-xr-x@  1 tison  staff   7.8M Jan  1  1980 protoc-linux-x86_64
-rwxr-xr-x@  1 tison  staff   6.1M Jun 14 23:12 protoc-osx-aarch_64
-rwxr-xr-x@  1 tison  staff   5.6M Jun 14 23:12 protoc-osx-x86_64
-rwxr-xr-x@  1 tison  staff   8.8M Jan  1  1980 protoc-win32.exe

-rw-r--r--  1 tison  staff    18M Jun 21 12:56 target/package/protobuf-build-0.14.1.crate

tisonkun · 2023-06-21T09:52:39Z

@weihanglo ... and Thank you!

BusyJay · 2023-06-24T08:50:50Z

I think fallback to build from source may not be a bad idea giving that the community prefers not to use bundled binaries (like prost-build recently did). But we still need to think a way to support Windows.

BusyJay · 2023-06-24T08:51:48Z

linux-ppcle_64 and linux-aarch_64 are mainly used on server side.

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2023-06-26T14:02:37Z

@BusyJay Updated. As we discussed offline -

Bundle win only;
Others compile from source.

tisonkun · 2023-06-26T14:03:13Z

linux-ppcle_64 and linux-aarch_64 are mainly used on server side.

Yeah. I suspect on server side they're used less, while aarch_64 can be a trend :D

BusyJay

Rest LGTM

src/lib.rs

Signed-off-by: tison <wander4096@gmail.com>

sunxiaoguang

LGTM

tisonkun · 2023-07-01T11:36:45Z

Maybe we can release a 0.15.0 now and upgrade downsteams like raft-rs and tikv.

tisonkun added 2 commits June 20, 2023 00:29

feat: fallback to build protobuf from source

1ffed5c

1. Avoid bundle binaries 2. Leverage C++ portability Signed-off-by: tison <wander4096@gmail.com>

feat: support new protobuf versions

c1462c6

See https://protobuf.dev/support/version-support/ Signed-off-by: tison <wander4096@gmail.com>

tisonkun requested review from BusyJay and hicqu June 19, 2023 16:33

tisonkun commented Jun 19, 2023

View reviewed changes

src/protobuf_impl.rs Show resolved Hide resolved

tisonkun mentioned this pull request Jun 19, 2023

keep compatible with newer protoc version #65

Merged

tisonkun added 2 commits June 20, 2023 00:40

try ensure fallback works

0ea1f6c

Signed-off-by: tison <wander4096@gmail.com>

refactor: align protoc lookup

76430ad

Signed-off-by: tison <wander4096@gmail.com>

tisonkun mentioned this pull request Jun 20, 2023

build: force use latest protobuf-build tikv/raft-rs#510

Merged

BusyJay reviewed Jun 20, 2023

View reviewed changes

restore binaries

33bf6d0

Signed-off-by: tison <wander4096@gmail.com>

tisonkun requested a review from BusyJay June 20, 2023 18:07

BusyJay approved these changes Jun 21, 2023

View reviewed changes

bundle for win only

4e2867c

Signed-off-by: tison <wander4096@gmail.com>

hawkingrei approved these changes Jun 26, 2023

View reviewed changes

BusyJay reviewed Jun 27, 2023

View reviewed changes

src/lib.rs Show resolved Hide resolved

sync bundles to v21.5

7faef5e

Signed-off-by: tison <wander4096@gmail.com>

BusyJay approved these changes Jun 28, 2023

View reviewed changes

sunxiaoguang approved these changes Jul 1, 2023

View reviewed changes

sunxiaoguang merged commit d4c4831 into master Jul 1, 2023
3 checks passed

sunxiaoguang deleted the use-protobuf-src branch July 1, 2023 11:11

JmPotato mentioned this pull request Jul 4, 2023

Is it support 22.x.x protoc? #66

Closed

tisonkun mentioned this pull request Jul 18, 2023

feat: Add vendored support tikv/client-rust#411

Merged

lizhanhui mentioned this pull request Oct 10, 2023

Bump prost to v0.12 #70

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: fallback to build protobuf from source #67

feat: fallback to build protobuf from source #67

tisonkun commented Jun 19, 2023

BusyJay commented Jun 20, 2023

tisonkun commented Jun 20, 2023

BusyJay left a comment

BusyJay Jun 20, 2023

tisonkun Jun 20, 2023

BusyJay commented Jun 20, 2023

tisonkun commented Jun 20, 2023 •

edited

Loading

tisonkun commented Jun 20, 2023

BusyJay commented Jun 21, 2023

tisonkun commented Jun 21, 2023

weihanglo commented Jun 21, 2023

tisonkun commented Jun 21, 2023 •

edited

Loading

tisonkun commented Jun 21, 2023

BusyJay commented Jun 24, 2023

BusyJay commented Jun 24, 2023

tisonkun commented Jun 26, 2023

tisonkun commented Jun 26, 2023

BusyJay left a comment

sunxiaoguang left a comment

tisonkun commented Jul 1, 2023

feat: fallback to build protobuf from source #67

feat: fallback to build protobuf from source #67

Conversation

tisonkun commented Jun 19, 2023

BusyJay commented Jun 20, 2023

tisonkun commented Jun 20, 2023

BusyJay left a comment

Choose a reason for hiding this comment

BusyJay Jun 20, 2023

Choose a reason for hiding this comment

tisonkun Jun 20, 2023

Choose a reason for hiding this comment

BusyJay commented Jun 20, 2023

tisonkun commented Jun 20, 2023 • edited Loading

tisonkun commented Jun 20, 2023

BusyJay commented Jun 21, 2023

tisonkun commented Jun 21, 2023

weihanglo commented Jun 21, 2023

tisonkun commented Jun 21, 2023 • edited Loading

tisonkun commented Jun 21, 2023

BusyJay commented Jun 24, 2023

BusyJay commented Jun 24, 2023

tisonkun commented Jun 26, 2023

tisonkun commented Jun 26, 2023

BusyJay left a comment

Choose a reason for hiding this comment

sunxiaoguang left a comment

Choose a reason for hiding this comment

tisonkun commented Jul 1, 2023

tisonkun commented Jun 20, 2023 •

edited

Loading

tisonkun commented Jun 21, 2023 •

edited

Loading