aws s3 static build fails on the aarch64 GNU/Linux running on the Nvidia Jetson-AGX

[user959]

Hello! We need to link against s3 statically, so we don't need to install it on all of our servers. So, I configured aws sdk with a command cmake ../aws-sdk-cpp -DCMAKE_BUILD_TYPE=Debug -DCMAKE_PREFIX_PATH=/usr/local/ -DCMAKE_INSTALL_PREFIX=/usr/local/ -DBUILD_ONLY="s3" -DBUILD_SHARED_LIBS=OFF, but make command failed with

[ 99%] Building CXX object aws-cpp-sdk-core-tests/CMakeFiles/aws-cpp-sdk-core-tests.dir/utils/xml/XmlSerializerTest.cpp.o
[ 99%] Linking CXX executable aws-cpp-sdk-core-tests
/usr/bin/ld: warning: libcrypto.so.1.1, needed by /usr/lib/gcc/aarch64-linux-gnu/7/../../../aarch64-linux-gnu/libcurl.so, may conflict with libcrypto.so.3
s2n_init() failed: 402653266 (The libcrypto major version number seen at compile-time is different from the major version number seen at run-time)
Fatal error condition occurred in /home/tarcxpoint/kuryakov/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-io/source/s2n/s2n_tls_channel_handler.c:197: 0 && "s2n_init() failed"
Exiting Application
No call stack information available
Aborted (core dumped)
aws-cpp-sdk-core-tests/CMakeFiles/aws-cpp-sdk-core-tests.dir/build.make:907: recipe for target 'aws-cpp-sdk-core-tests/aws-cpp-sdk-core-tests' failed
make[2]: *** [aws-cpp-sdk-core-tests/aws-cpp-sdk-core-tests] Error 134
make[2]: *** Deleting file 'aws-cpp-sdk-core-tests/aws-cpp-sdk-core-tests'
CMakeFiles/Makefile2:1125: recipe for target 'aws-cpp-sdk-core-tests/CMakeFiles/aws-cpp-sdk-core-tests.dir/all' failed
make[1]: *** [aws-cpp-sdk-core-tests/CMakeFiles/aws-cpp-sdk-core-tests.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 99%] Linking CXX static library libaws-cpp-sdk-s3.a
[ 99%] Built target aws-cpp-sdk-s3
Makefile:148: recipe for target 'all' failed
make: *** [all] Error 2

The same happens if I add -DFORCE_SHARED_CRT=OFF.
You even mention this issue in your cmake file: https://github.com/aws/aws-sdk-cpp/blob/main/CMakeLists.txt#L194. Is there a work around? We couldn't find one

[jmklix]

What version of openssl are you using?

It looks like one of the tests is failing, one of the possible workarounds for that is to disable the tests.
You can do this by adding: -DENABLE_TESTING=OFF

[user959]

Hi! Thanks for the response.

tarcxpoint@vovacart:~/kuryakov$ openssl version
OpenSSL 3.2.0-dev  (Library: OpenSSL 3.2.0-dev )

UPD:
With this flag everything builds, though I can see s2n_init() failed: 402653266 (The libcrypto major version number seen at compile-time is different from the major version number seen at run-time) warning during builds. However, our project crashes on startup with

s2n_init() failed: 402653266 (The libcrypto major version number seen at compile-time is different from the major version number seen at run-time)
Signal: SIGTRAP (Trace/breakpoint trap)

whereas simple minimal project runs ok.
It might be that our project uses grpc, I found such an issue: #1533 (comment)

[user959]

Adding options.cryptoOptions.initAndCleanupOpenSSL = false; as suggested in the issue does not help

[user959]

Yes, the problem is in grpc. Here is the minimal project which crashes on startup:

CMakeLists.txt

cmake_minimum_required(VERSION 3.10)

find_package(Protobuf REQUIRED)
find_package(gRPC CONFIG REQUIRED)
find_package(AWSSDK REQUIRED COMPONENTS s3)

add_executable(prjct main.cpp)

target_link_libraries(prjct
        gRPC::grpc++
        ${AWSSDK_LINK_LIBRARIES}
        ${AWSSDK_PLATFORM_DEPS}
)

main.cpp

#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>

#include <grpcpp/client_context.h>

#include <iostream>

void not_even_called() {
    grpc::ClientContext ctx;
}

int main() {
    Aws::SDKOptions options;
    Aws::InitAPI(options);

    std::cout << "Oplya\n";
}

If I build and run it on the Nvidia Jetson-AGX with a statically built aws s3 I get

s2n_init() failed: 402653266 (The libcrypto major version number seen at compile-time is different from the major version number seen at run-time)
Fatal error condition occurred in /home/tarcxpoint/kuryakov/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-io/source/s2n/s2n_tls_channel_handler.c:197: 0 && "s2n_init() failed"
Exiting Application
No call stack information available
Aborted (core dumped)

The mind-blowing thing is, if I comment grpc::ClientContext ctx it doesn't crash, though it is not even called. Without Aws::InitAPI(options) it doesn't crash too.

Do you have any idea how to fix it, please?

[JonathanHenson]

just a hunch, is grpc calling dlload() and dlsym()? This is assuming the build closure is the same, and the only difference is commenting out the grpc init code.

[JonathanHenson]

my suspicion is, grpc is calling dlsym with the global flag and it's preferring those symbols over the ones referenced by the linker. One possibility would be to isolate the symbols that get loaded.

[user959]

Thanks for the reply! Yes, build closure is exactly the same. No, I believe grpc does not call it. I ran grep -rn "dlsym" /usr and grep -rn "dlload" /usr and there was nothing about grpc