rdma: add separate bounce buffer freelist for data (eager) messages

[rauteric]

Separate out bounce buffer freelists into a smaller-sized freelist for
control messages and a larger size for data (eager) messages

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[rauteric]

The CI failure(s) are real:

scatter_perf: nccl_ofi_rdma.c:2100: alloc_bounce_req: Assertion `NCCL_OFI_IS_PTR_ALIGNED(&bounce_fl_item->bounce_msg, BOUNCE_BUFFER_ALIGNMENT)' failed.

Need to fix up some asserts.

[rauteric]

Force-push to 3494556 fixes the CI failure by having alignment on both (eager) bounce buffers and ctrl recv buffers.

This requirement will be relaxed when we refactor the data structure in the related #600.

[rauteric]

Rebased on master

[AmedeoSapio]

bot:aws:retest

[AvivBenchorin]

bot:aws:retest

[rauteric]

Rebase on master

[rajachan]

Shouldn't this be a function out max outstanding requests, which today we have at 128?

[rauteric]

That's a bit tricky, because the outstanding requests max (NCCL_OFI_MAX_REQUESTS) is

1. A per-communicator value, whereas this parameter is a per-endpoint per-rail value, so we'd have to multiply this parameter by the number of communicators, which we don't know in advance
2. Much higher than NCCL actually keeps in flight typically, so setting this param to NCCL_OFI_MAX_REQUESTS x num_comms will be overkill.

That said, we may need to tune this value, but the current value in this PR is already significantly higher than what is in master today (which is 16-32 per rail, and shared between eager and ctrl recv buffers).

[rauteric]

Rebased on master

[bwbarrett]

I don't think this is the right approach. Now would be a great time to clean up some code, but this only makes the code more confusing.

Bounce buffers for eager messages and control messages are different things. Have different structures with different names so that there's no reason to figure out what's what. I'm also not sure I understand the desire to move the freelists from the endpoint into the per-rail structures, given that most of the data is really operations that will be per-endpoint.

[bwbarrett]

per what rail? device, endpoint, etc?

[rauteric]

Bounce buffers for eager messages and control messages are different things. Have different structures with different names so that there's no reason to figure out what's what.

They are different things, but there is a lot of code that is shared between them (e.g., all of the buffer pool management code, posting receives). We should rename these structures to something more generic (like receive buffers instead of bounce buffers [which I wanted to do in a followup patch]), but I think duplicating all the buffer management code is overkill.

I'm also not sure I understand the desire to move the freelists from the endpoint into the per-rail structures, given that most of the data is really operations that will be per-endpoint.

I'm confused... isn't having the freelists per-rail exactly what you were advocating in #614 (comment)?

I think it makes sense to have the freelists per-rail because posted recv buffer counts are maintained per-rail.

[bwbarrett]

I think the problem with your patch is that it's way too much cognitive load to figrue out what a bounce buffer is. And the rest is just too confusing to review. Fix that.