how to compile redpanda?

[unicomp21]

[286/697] Building CXX object src/v/cluster/CMakeFiles/v_cluster.dir/members_table.cc.o
[287/697] Building CXX object src/v/cluster/CMakeFiles/v_cluster.dir/tx_gateway_frontend.cc.o
FAILED: src/v/cluster/CMakeFiles/v_cluster.dir/tx_gateway_frontend.cc.o
/usr/bin/g++ -DFMT_LOCALE -DSEASTAR_API_LEVEL=6 -DXXH_PRIVATE_API -DZSTD_STATIC_LINKING_ONLY -I../src/v -Isrc/v -isystem deps_install/include -isystem deps_install/include/hdr -fPIC -fdiagnostics-color=always -fcoroutines -O3 -DNDEBUG -fPIC -Wall -Wextra -Werror -Wno-missing-field-initializers -std=c++20 -U_FORTIFY_SOURCE -Wno-maybe-uninitialized -DSEASTAR_SSTRING -Werror=unused-result -std=c++2a -MD -MT src/v/cluster/CMakeFiles/v_cluster.dir/tx_gateway_frontend.cc.o -MF src/v/cluster/CMakeFiles/v_cluster.dir/tx_gateway_frontend.cc.o.d -o src/v/cluster/CMakeFiles/v_cluster.dir/tx_gateway_frontend.cc.o -c ../src/v/cluster/tx_gateway_frontend.cc
../src/v/cluster/tx_gateway_frontend.cc: In member function 'seastar::future<boost::outcome_v2::basic_result<cluster::tm_transaction, cluster::tx_errc, boost::outcome_v2::policy::throw_bad_result_access<cluster::tx_errc, void> > > cluster::tx_gateway_frontend::get_ongoing_tx(seastar::shared_ptrcluster::tm_stm, model::producer_identity, kafka::transactional_id, seastar::lowres_clock::duration)':
../src/v/cluster/tx_gateway_frontend.cc:1242:1: error: statement has no effect [-Werror=unused-value]
1242 | }
| ^
cc1plus: all warnings being treated as errors
[288/697] Building CXX object src/v/cluster/CMakeFiles/v_cluster.dir/topic_updates_dispatcher.cc.o

[emaxerrno]

what is the issue, what is the operating system, what is the compiler. this stack trace has no real useful information.

what memory profile for the compiler, how many threads, what build (make|ninja), etc.

try gcc11

[emaxerrno]

note that we compile with coroutines, so gcc11 or clang12 needed.

[unicomp21]

Apologies, failed to mention, I'm using ubuntu 20.10, in docker, w/ the instructions here:

For hackers, here is the short and sweet:
sudo ./install-dependencies.sh && ./build.sh

https://github.com/vectorizedio/redpanda

Are there env vars I need to set before running the above scripts? Am I looking in the wrong place for build instructions?

[BenPope]

I think Alex got it right, you'll need to sudo apt install g++-11

And then invoke like this:

sudo ./install-dependencies.sh && CC=gcc-11 CXX=g++-11 ./build.sh

[emaxerrno]

@unicomp21 please close the tix if it answers your question

[emaxerrno]

I'm turning this issue into a discussion

also updated the readme w/ this. https://github.com/vectorizedio/redpanda/blob/dev/README.md#build-manually

[unicomp21]

#2760

looks like we're getting close to a fix.

[aparavind]

The following is the confugration of my EC2 instance ..... which i had taken for compilation.

# cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

The following are the observations.

1. z1d.xlarge : I first started with this . but the entire compilation could not complete.
2. then i upgraded the instance to z1d.2xlarge .. the compilation completed.

but still the following 2 test cases have been failed.

Label Time Summary:
archival          =  62.22 sec*proc (1 test)
bytes             =   1.46 sec*proc (1 test)
cli_parse         =   0.92 sec*proc (1 test)
cloud_roles       =   3.43 sec*proc (2 tests)
cloud_storage     = 796.01 sec*proc (2 tests)
cluster           = 510.87 sec*proc (34 tests)
compression       =   1.17 sec*proc (1 test)
config            =   1.13 sec*proc (1 test)
coproc            =  90.85 sec*proc (1 test)
disable_on_ci     = 336.75 sec*proc (2 tests)
features          = 224.45 sec*proc (1 test)
hashing           =   0.93 sec*proc (2 tests)
http              =   2.25 sec*proc (1 test)
http_imposter     =   2.89 sec*proc (1 test)
json              =   1.17 sec*proc (1 test)
kafka             =  99.18 sec*proc (9 tests)
kafka_protocol    =   5.76 sec*proc (2 tests)
model             =   2.04 sec*proc (2 tests)
net               =  21.72 sec*proc (2 tests)
pandaproxy        =  75.18 sec*proc (10 tests)
raft              =  91.14 sec*proc (3 tests)
random            =   0.78 sec*proc (1 test)
reflection        =   1.93 sec*proc (2 tests)
rpc               =  20.13 sec*proc (3 tests)
s3                =   2.55 sec*proc (1 test)
serde             =   1.62 sec*proc (1 test)
ssx               =   1.15 sec*proc (1 test)
storage           = 222.19 sec*proc (6 tests)
utils             =  18.04 sec*proc (3 tests)

Total Test time (real) = 3881.67 sec

The following tests FAILED:
         24 - coproc_fixture_unstable_rpunit (Failed)
         73 - test_features_rpunit (Failed) 

so do i need to recompile considering the failed cases. ??

[BenPope]

Can you describe what you did?

I'd like to know which version you checked out, which compiler you used, and what instructions/scripts you followed.

[aparavind]

1. Did a git clone of the the redpanda repository. so it was the latest dev branch
2. Selected a z1d.2xlarge (62GB) (100GB) (8 core) the standard ubuntu AMI (Ubuntu version 22.4)
3. executed the command with their status

Attempt 1

sudo apt install g++-11 
./install-dependencies.sh && CC=gcc-11 CXX=g++-11 ./build.sh

THe above resulted in error.

[91/93] : && /usr/bin/cmake -E rm -f libseastar_testing.a && /usr/bin/ar qc libseastar_testing.a  CMakeFiles/seastar_testing.dir/src/testing/entry_point.cc.o CMakeFiles/seastar_testing.dir/src/testing/random.cc.o CMakeFiles/seastar_testing.dir/src/testing/seastar_test.cc.o CMakeFiles/seastar_testing.dir/src/testing/test_runner.cc.o && /usr/bin/ranlib libseastar_testing.a && :
[92/93] /usr/bin/ccache /usr/bin/g++-11 -DFMT_LOCALE -DSEASTAR_API_LEVEL=6 -DSEASTAR_DEFERRED_ACTION_REQUIRE_NOEXCEPT -DSEASTAR_HAS_MEMBARRIER -DSEASTAR_HAVE_ASAN_FIBER_SUPPORT -DSEASTAR_HAVE_HWLOC -DSEASTAR_HAVE_LZ4_COMPRESS_DEFAULT -DSEASTAR_HAVE_NUMA -DSEASTAR_SCHEDULING_GROUPS_COUNT=16 -I/opt/redpanda/build/deps_build/seastar-prefix/src/seastar/include -I/opt/redpanda/build/deps_build/seastar-prefix/src/seastar-build/gen/include -isystem /opt/redpanda/build/deps_install/include -fPIC -fdiagnostics-color=always -std=c++20 -O3 -DNDEBUG -fPIC -fvisibility=hidden -UNDEBUG -Wall -Werror -Wno-array-bounds -Wno-error=deprecated-declarations -gz -Wno-error -fcoroutines -U_FORTIFY_SOURCE -Wno-maybe-uninitialized -DSEASTAR_SSTRING -Werror=unused-result -MD -MT CMakeFiles/seastar_perf_testing.dir/tests/perf/perf_tests.cc.o -MF CMakeFiles/seastar_perf_testing.dir/tests/perf/perf_tests.cc.o.d -o CMakeFiles/seastar_perf_testing.dir/tests/perf/perf_tests.cc.o -c /opt/redpanda/build/deps_build/seastar-prefix/src/seastar/tests/perf/perf_tests.cc
[93/93] : && /usr/bin/cmake -E rm -f libseastar_perf_testing.a && /usr/bin/ar qc libseastar_perf_testing.a  CMakeFiles/seastar_perf_testing.dir/src/testing/random.cc.o CMakeFiles/seastar_perf_testing.dir/tests/perf/perf_tests.cc.o CMakeFiles/seastar_perf_testing.dir/tests/perf/linux_perf_event.cc.o && /usr/bin/ranlib libseastar_perf_testing.a && :
ninja: build stopped: subcommand failed.
CMake Error at cmake/main.cmake:106 (message):
  Build step for v::build failed: 1
Call Stack (most recent call first):
  CMakeLists.txt:5 (include)


-- Configuring incomplete, errors occurred!
See also "/opt/redpanda/build/CMakeFiles/CMakeOutput.log".

clearly the system was not liking CC=gcc-11 CXX=g++-11 as indicated in the comments above .. so reverted back to clang

Attempt 2
executed the following shell script

#!/bin/bash
set -e

bash -x install-dependencies.sh
CC=clang CXX=clang++ ./build.sh

this built completely .. but 2 test cases failed.

Total Test time (real) = 1797.41 sec

The following tests FAILED:
         64 - test_health_monitor_test_rpunit (Failed)
         74 - test_features_rpunit (Failed)
Errors while running CTest

the logs of attempt 2 are completely stored in this file

https://drive.google.com/file/d/14uRafPbMRGWexRtKsPpQ-U8CPtdyW3ch/view?usp=sharing