Replace SQS with an open-source alternative.

[prshant70]

Let's make NotifyOne a cloud agnostic tool. And to achieve this, we first need to replace the AWS specific tools being used (s3, sqs) with other open-source counterparts.

This issue is about replacing SQS with and open-source counterpart.

In my opinion -

• Replace SQS with Kafka

[riki-nitdgp]

Hi @prshant70
I checked the code. Before moving to Kafka few points are needed:

• We can't move CRITICAL events to Kafka as it is API Based.
• What will be the partition key at each level (Gateway, core, handler) It is one of the major factors in deciding scalability. My suggestion will be partitioned by the receiver's identifier (Phone, Email, etc). This will maintain the Sequence of communication also (in Ideal Case).
• Exponential Backoff is not supported in Kafka, What's your opinion on handling failed cases?
• Can the user change/configure the partition key?
• Code is tightly coupled with 1mg Commons (Torpido) Do you prefer Kafka Common Code need to be written in (Torpido) or in the same repo?
• APM is one of the components that is tightly coupled, Use OpenTelemetry for Application traces (Telemetry Data). Most of the vendors support OpenTelemetry.
• Tracing is one more challenge in Kafka.

[prshant70]

Hi @riki-nitdgp

Thanks for your time. These are important questions that must be answered before we move forward.

Please find my opinion inline to the pointers raised by you -

We can't move CRITICAL events to Kafka as it is API Based.

Yes, CRITICAL priority events are expected to be handled with synchronous processing. This is important to prioritise
these events correctly in the system. Ideally, we would never want to use a queue to process CRITICAL events.

What will be the partition key at each level (Gateway, core, handler) It is one of the major factors in deciding scalability.
My suggestion will be partitioned by the receiver's identifier (Phone, Email, etc). This will maintain the Sequence of
communication also (in Ideal Case)

In my opinion, we must start with default partitioning logic where messages are routed to partitions randomly. Sticking partition logic to message attributes like event_id or phone/email, etc might imbalance the number of messages in different partitions and thus result in hampering the priority of processing the events.
However, we must keep the number of partitions count on higher side (must be controlled by NotifyOne) to make sure that when we scale-up the consumers group, consumers don't sit idle.

Exponential Backoff is not supported in Kafka, What's your opinion on handling failed cases?

I think we need to handle this using the retry logic and the max retry attempts. The subscriber code in notifyone-core is already structured in a way that it won't trigger the event if the a messages is duplicate (subscribed more than once).

Can the user change/configure the partition key?

No. These must be system controlled to guarantee the scale and priority processing.

Code is tightly coupled with 1mg Commons (Torpedo) Do you prefer Kafka Common Code need to be written in (Torpedo) or in the same repo?

Ideally, we should add the Kafka wrapper in the commonutils repo. This utility is already being used across all NotifyOne services and is intended to contain common utility code needed across services.

APM is one of the components that is tightly coupled, Use OpenTelemetry for Application traces (Telemetry Data). Most of the vendors support OpenTelemetry.

This is already in plan and will be release as a new torpedo release. We can skip this from the current context.

Tracing is one more challenge in Kafka.

We can look into this separately when we work on replacing the current elastic APM integration with open-telemetry.

[prshant70]

@riki-nitdgp

I know there are certain challenges when we use Kafka as message queue and consumers scale-up and scale-down frequently. We don't necessarily need to use Kafka here, if you have any other alternative in mind or think that other open source alternatives to SQS might be a better fit here, we are open to discuss.