Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP webhook integration #82

Open
1 task done
tomtom215 opened this issue Aug 15, 2023 · 5 comments
Open
1 task done

HTTP webhook integration #82

tomtom215 opened this issue Aug 15, 2023 · 5 comments
Labels
A-Other Area: Other enhancement New feature or request

Comments

@tomtom215
Copy link

tomtom215 commented Aug 15, 2023

Is there an existing proposal for this?

  • I have searched the existing proposals

Is your feature request related to a problem?

The ability to send JSON via HTTP posts to a wehbook endpoint which then takes that JSON and adds it to the message queue.

Describe the solution you'd like

I know MQ's + HTTP is kind of an antipatern and usually the advice is to write your own middleware. So far I have not seen a message queue that offers this feature, but it seems like a common enough problem that people should not have to write their own customer middleware every time.

An example of this is can be found trying to search for ways to integrate HTTP messages with RabbitMQ and finding this project https://github.com/dougwatson/http2amqp

The ideal scenario would be if you could create a webhook endpoint on BlazingMQ which would then be able to receive JSON messages from HTTP (push/pub) sources which would convert the JSON into a message and place it into a queue.

Alternatives you considered

Custom middleware

@tomtom215 tomtom215 added the enhancement New feature or request label Aug 15, 2023
@quarter-note
Copy link
Contributor

Hi @tomtom215, thanks for your interest in BlazingMQ! Internally, we have an adapter which provides HTTP interface which a producer can use to post messages to BlazingMQ. This adapter is closely tied to enterprise layer and cannot be open sourced.

One solution here would be to implement a similar adapter for OSS. A simple application which listens over HTTP to the application on one side, and talks to BlazingMQ backend over BlazingMQ protocol on the other side (by using BlazingMQ SDK). This is not on our roadmap, but will keep in mind.

Out of curiosity, what interested you in BlazingMQ, and why are you looking for HTTP hook?

Thanks

@tomtom215
Copy link
Author

Hi @quarter-note thanks for your response!

This seems to validate my initial assumption that if you want to have an HTTP interface for a MQ that you need to roll your own middleware and it is important enough that you would make it a licenses feature of your product.

So far one of the only instances I've found of this being out of the box by a OSS tool has been Apache SeaTunnel

I came across BlazingMQ because I am always keeping an eye of event/streaming platforms and components in the industry and when I saw BlazingMQ had been battle tested at Bloomberg and offered some architectural advantages over RabbitMQ, I decided to dig deeper.

Our use case for an HTTP webhook is that we interact with some systems and applications that have interfaces that push DB updates via JSON messages. It's close to CDC, but since we don't have influence over the source systems to setup something like Debezium, we considered a webhook -> MQ -> DB so that we didn't have to manage message lifecycle or delivery guarantees with our own custom middleware if it was available from BlazingMQ.

@quarter-note
Copy link
Contributor

@tomtom215 If we provide a HTTP/HTTP2 proxy which will enable you to talk to BlazingMQ, would you consider using BlazingMQ in your setup?

@tomtom215
Copy link
Author

@tomtom215 If we provide a HTTP/HTTP2 proxy which will enable you to talk to BlazingMQ, would you consider using BlazingMQ in your setup?

Honestly, we would. As mentioned it seems to be a big feature gap in the available message queue space and the main reason I've been looking around is to avoid either rolling our own middleware or by adding more complexity to our stack by introducing something as heavy as SeaTunnel or Kafka connect just to handle this use case.

If marketed correctly, I think this feature in an OSS offering will be a big advantage for BlazingMQ over other message queues and could help to simplify some architectural decisions.

Ps. Thanks for hearing me out and the good discussion!

@quarter-note quarter-note added the A-Other Area: Other label Sep 5, 2023
@Montana
Copy link

Montana commented Aug 6, 2024

If we provide a HTTP/HTTP2 proxy which will enable you to talk to BlazingMQ, would you consider using BlazingMQ in your setup?

It definitely would make it significantly more convenient to integrate BlazingMQ into setups requiring HTTP-based message ingestion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Other Area: Other enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants