Invalid socket options on older kernels

[jkalez]

Problem:

The syscall to sendmmsg in s2n-quic-platform/src/syscall/mmsg.rs uses the msghdr.msg_control to set socket options for each call to sendmmsg. It seems that at least one of these options (level SOL_UDP, option UDP_SEGMENT) is not present on older kernels. Specifically, I have found that this option does not exist on Linux 4.14 and earlier.

The result is that s2n-quic believes GSO is supported, but the kernel does not support it. Running the io::tokio::tests::ipv4_test in s2n-quic-platform on 4.15 results in the kernel combining multiple UDP messages, forcing the test to execute until the timeout, then failing. This can be replicated by running cargo test in an Ubuntu 18.04 machine with a 4.15 kernel.

In all fairness, support for this option started appearing in 4.19, so I do think it's very fair to say that 4.15 is too old and isn't supported. The more general problem though is that there is no runtime detection for the socket options being used, which could cause pretty bad, silent bugs like this down the road. For example, it appears that GRO support didn't land until ~5.0. However, the UDP_GRO socket option may be used regardless of kernel support.

Solution:

Unfortunately, the sendmmsg and sendmsg (and their recv counterpart) syscalls do not appear to return any errors if passed an invalid socket option. An alternative may be to identify which socket options we'd like to use early on and attempt to set them with setsockopt. If we fail, then ensure we don't try to set those options, and turn off features which may rely on those options (e.g. GSO and/or GRO).

• Does this change what s2n-quic sends over the wire? Yes, though specifically how may vary by platform.
• Does this change any public APIs? --> No

Requirements / Acceptance Criteria:

A handful of tests currently timeout on 4.15 kernels:

• io::tokio::tests::ipv4_test
• io::tokio::tests::ipv4_two_socket_test
• io::tokio::tests::ipv6_test
• io::tokio::tests::ipv6_two_socket_test

When these tests pass on older kernels, its likely that this issue is resolved.

As mentioned above, I think it's pretty fair to say you only support specific kernel versions & up. If that's the case though, that should probably be documented. It would be even better if when attempting to run on those kernels, we detect that and report an error or panic.

Out of scope:

• Compile-time detection of libc-wrappers for relevant syscalls is also broken, and closely related, but addressed separately in Platform detection of sendmsg/sendmmsg does not work as intended #2260.

[mxinden]

Potentially related: quic-go/quic-go#4446

[WesleyRosenblum]

Thanks for the issue. We'll either document that we don't support older kernel versions or disable GSO for kernel versions older than 5.