How to create REST API that stores the payload to Kafka and send HTTP response only after successful storage?

General
,
1

Hi

I am new to node-red. I am using node-red to accept HTTP Post from another server. I need to respond to the with the success or failure status (200 or 400 response). If the received message is successfully stored to a kafka queue (or to database) then send http response 200, otherwise send 400.

I tried building a flow using KafkaJS node-red-contrib-kafkajs . I am able to stores the message to kafka queue successfully, but has this node has no output. So I am unable to send the response after this.

How do I achieve this use case?

2

Try pointing a complete node at the node. Attach a debug to that. Does it send a message when the kafka node succeeds / fails?

Alternatively, does the node display any status underneath it after completion that signals success/failure? If so, try a status node looking at the kafka node.

Additionally (and likely the better solution), I would raise an issue on the nodes repo describing your requirement. If the dev is responsive, they might add an output to the node that indicates success or failure.

3

Try pointing a complete node at the node. Attach a debug to that. Does it send a message when the kafka node succeeds / fails?

I tried with "Complete". It does not send success / fail.

does the node display any status underneath it after completion that signals success/failure? If so, try a status node looking at the kafka node.

The kafka node does display the status underneath. However it is not useful. When I connect debug to the node, it keeps sending this connectivity messages every second or so.

I think, I will have raise and issue.

Thanks for your help.

4

You could use a switch node to ignore the status messages - hopefully then you'll see success/fail messages?

Please provide a link to the raised issue here so others with the same issue can keep track.

Thanks, Steve.

5

Issue raised for reference For Producer need to know if the message was successfully so that further actions could be taken. · Issue #5 · emrebekar/node-red-contrib-kafkajs · GitHub .
I must try and alternate approach too.

6

If I understand you correctly, you're trying to make a Kafka client act as if it was synchronous...

If this is the case, bear in mind that:

  • Default Kafka producer send API is asynchronous and nonblocking.
  • HTTP is a synchronous protocol.

I don't think it will be easy for you to convince a node-red-contrib maintainer to develop what you say you need.

7

@lu4t Probably you are right. Yes, it looks like I am trying to make Kafka client synchronous. If I was writing this code in, let's say "express", I would have handled it that way but managed it with async calls.

This begs a different question. (forgive me if I am going back to basics). In my given scenario should I just return the response (http 200) even though I am not sure if the client successfully stored the message in kafka? Otherwise, client just keeps waiting for node-red to respond, it will either timeout eventually and if it makes thousands of requests each of those requests will have to timeout.

What's the ideal way to handle this in node-red?

8

I might be wrong here, but isn't kafka a pubsub system ? Couldn't you just listen on (ie: consume) the topic ?

9

@bakman2 No. You cannot. Because you have to respond to http request with the http response in the same branch of the flow where "Http in" node is. Otherwise you don't get the "response" object.

I have tried this with "Complete" as well as "Status" nodes. They don't work because you do not have http response object when responding. It has to be in the same path where "Http In" is. This is FYI for @Steve-Mcl response to earlier discussion.

10

Got it. You could store the output of the kafka client in a context variable and show that as output to the http response node. Probably wise to keep the context for this only in memory (not write to disk).

11

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.