SDK crashing with s2n_init() failed (Embedded Linux, buildroot)

[omri-s-electreon]

Hello every,

We are currently migrating our IoT solution to AWS and we are stuck at the last phase - adjusting our devices code to connect with IoT Core using the aws-iot-device-sdk-cpp-v2 library.

Our devices are based on Microchp's SAMA5D27-SOM1 (ARM Cortex-V5 CPU, 128MB RAM) running an Embedded Linux OS. Our user space application (C++ daemon) is the software communicating with IoT Core (MQTT over Websocket). As our cross compilation system, we use buildroot, which cross compiles the Linux Kernel, libraries, application and packages everything into a bootable image and a cross compiled custom SDK for application development (all the headers, libraries etc. used to build applications for the target platform). The AWS SDK is built as part of buildroot (AKA. a "package") and we can verify that our application is linking successfully with the shared objects created (for ARM). We are using SDK v1.20.0 (the tag in the github repo).

The issue we are facing is at the initial usage of the SDK, as seen in the samples code:

ApiHandle apiHandle;

which causes the application to terminate with the following error printed to the terminal:

s2n_init() failed: 402653194 (a safety check failed)
Fatal error condition occurred in ../../../../../crt/aws-crt-cpp/crt/aws-c-io/source/s2n/s2n_tls_channel_handler.c:197: 0 && "s2n_init() failed"
Exiting Application
No call stack information available

The AWS C++ SDK is defined as the following buildroot package and cross-builds all shared libraries successfully (which are also linked successfully with our application's executable and installed correctly to the target's root file system / sysroot):

AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_VERSION = v1.20.0
AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_SITE = https://github.com/aws/aws-iot-device-sdk-cpp-v2
AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_SITE_METHOD = git
AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_GIT_SUBMODULES = YES
AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_INSTALL_STAGING = YES
AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_INSTALL_TARGET = YES
AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_CONF_OPTS = -DPERFORM_HEADER_CHECK=OFF
AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_DEPENDENCIES = openssl libcurl util-linux host-cmake
AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_LICENSE = GPLv2
AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_LICENSE_FILES = LICENSE
AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_SUPPORTS_IN_SOURCE_BUILD = NO

AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_PRE_CONFIGURE_HOOKS += AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_CMAKE_MOVE_HOOK

define AWS_IOT_DEVICE_SDK_CPP_V2_ELECTREON_CMAKE_MOVE_HOOK
    mkdir $(@D)/build
endef

$(eval $(cmake-package))

Looking into #343 which seems somewhat similar we tried cross-building the library with the following cmake flags, with no effect:

-DS2N_NO_PQ_ASM=ON <---- Had no effect
-DS2N_NO_PQ=ON <----- Had no effect
-DLIBCRYPTO_SUPPORTS_EVP_RC4=0 <---- Had no effect
-DUSE_OPENSSL=ON <---- Had no effect but there might be a clue here.

As the issue above (#343) suggested, we also tried tweaking the CMakeLists.txt files which also had no effect.

We did notice differences between the compiler output when compiling with the USE_OPENSSL flag and when compiling without it.

When building with -DUSE_OPENSSL the cmake script finds the correct libcrypto.so file but declares a strange version - The version found is 1.1.1g while the actual version is 1.1.1p - This is in spite of the libcrypto.so file being of the correct version (1.1.1p).

-- Packaging currently only supported on Fedora.
-- Detected CMAKE_SYSTEM_PROCESSOR as armv7l
-- Detected 32-Bit system - disabling PQ crypto assembly optimizations
-- S2N_NO_PQ_ASM flag was detected - disabling PQ crypto assembly code
-- Disabling stacktrace functionality
-- madvise() support detected
-- clone() support detected
-- FOUND AWS-LC CRYPTO cmake config - shared
-- Using libcrypto from the cmake path
-- Found OpenSSL: /home/omri/electreon/repos/electreon-microchip-sama5d27-som-linux/electreon-som-ext-tree/host/arm-buildroot-linux-uclibcgnueabihf/sysroot/usr/lib/libcrypto.so (found version "1.1.1g")  
-- Using libcrypto from system: /home/omri/electreon/repos/electreon-microchip-sama5d27-som-linux/electreon-som-ext-tree/host/arm-buildroot-linux-uclibcgnueabihf/sysroot/usr/lib/libcrypto.so
-- Performing Test AWS_ARM32_CRC
-- Performing Test AWS_ARM32_CRC - Success
-- Configuring done
-- Generating done

When building without the flag the build seems to include the OpenSSL headers from the correct location (our custom Linux SDK). I assume it is linking with the libcrypto.so library although not mentioned (?).

-- Detected CMAKE_SYSTEM_PROCESSOR as armv7l
-- Detected 32-Bit system - disabling PQ crypto assembly optimizations
-- S2N_NO_PQ_ASM flag was detected - disabling PQ crypto assembly code
-- Disabling stacktrace functionality
-- madvise() support detected
-- clone() support detected
-- S2N found target: crypto
-- Using libcrypto from AWS-LC
-- Performing Test AWS_ARM32_CRC
-- Performing Test AWS_ARM32_CRC - Success
-- Configuring done
-- Generating done

In both builds we can see that the OpenSSL include files are coming from the correct path.

Looking into this file: https://github.com/aws/s2n-tls/blob/main/CMakeLists.txt , we have also tried the INTERNING flag, but we were unable to build it, even with a static library (libcrypto.a).

So we are a bit confused if and what we're missing and how to approach and solve this. It seems like something wrong is being picked up by cmake

If any more info is required - we will be happy to work on getting it.

Your assistance is highly appreciated,

Omri

[rpcme]

@omri-s-electreon which embedded linux build system are you using? I know Microchip supports both yocto and buildroot - if you are using yocto, are you using the aws bitbake layer? https://github.com/aws4embeddedlinux/meta-aws

EDIT: Nevermind, I see you're using buildroot. We have a contributor who maintains buildroot recipes. https://github.com/aduskett/buildroot-aws-iot - recipes might be outdated and require version adjustment.

[omri-s-electreon]

Thanks Richard,
Just to clarify before starting a time consuming task: You recommend not using the recursive build (that is - the code that comes from the git submodules), but to build each library as a package (like a yocto recipe) separately. Correct?
How should we know which versions (git tags) to select for the libraries? Is there some documentation on version compatibility between the libraries?
I'm referring to all of the small libraries (18 in total, are there more?):

aws-c-auth
aws-c-cal
aws-c-common
aws-c-compression
aws-c-event-stream
aws-c-http
aws-c-io
aws-c-iot
aws-c-mqtt
aws-c-s3
aws-c-sdkutils
aws-checksums
aws-crt-cpp
aws-iot-device-sdk-cpp-v2
aws-lambda-cpp
aws-lc
aws-sdk-cpp
s2n

[rpcme]

We have found success in managing them separately (and it is more philosophically inline with YP). True the configuration management is a challenge but we've done this through scripting. note aws-crt-cpp is the parent for most of the modules.

What I can say is the development team is also looking at this issue so please try to hold off until they reproduce and respond. There might be a tactical fix to get your immediate issue sorted but yes... long term I think separate package management is more inline with embedded linux philosophies and will also give you the opportunity to tweak per package build / stage options if needed as you get further along.

[omri-s-electreon]

Hi Richard,
I found some time to check your suggestion (and recommendation) about building all libraries as separate packages and not relying on the code from the submodules. As you suggested I took the packages file from this excellent repo and tweaked them for our needs. We mainly changed the versions of the libraries to the latest tag for each library repo and some naming changes to fit our buildroot code conventions.
This seems to work with no issues and we are able to execute our application passed the ApiHandle apiHandle; line.

Please let me know if you require more information and whether I should mark this issue as resolved (and how?).
BTW - We will be happy to share our buildroot configurations.

Thanks again. We are now able to make progress ;-)

Omri

Oh.... almost forgot - Happy holidays and an ever happier new year ;-)

[jmklix]

This should no longer be a problem as s2n is no longer a dependency. Please let us know if you run into any other problems with this sdk.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.