# How to create REST API that stores the payload to Kafka and send HTTP response only after successful storage?

**URL:** <https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255>\
**Category:** General\
**Tags:** http-request, database\
**Created:** [26 January 2022 12:51 UTC](https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255 "2022-01-26T12:51:19Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![aj\_dev](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@aj\_dev](https://discourse.nodered.org/u/aj_dev)\
**Post date:** [26 January 2022 12:51 UTC](https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255/1 "2022-01-26T12:51:19Z")

</div>

Hi

I am new to node-red. I am using node-red to accept HTTP Post from another server. I need to respond to the with the success or failure status (200 or 400 response). If the received message is successfully stored to a kafka queue (or to database) then send http response 200, otherwise send 400.

I tried building a flow using KafkaJS node-red-contrib-kafkajs . I am able to stores the message to kafka queue successfully, but has this node has no output. So I am unable to send the response after this.

How do I achieve this use case?

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [26 January 2022 13:09 UTC](https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255/2 "2022-01-26T13:09:02Z")

</div>

> [@aj\_dev](#):
>
> I am able to stores the message to kafka queue successfully, but has this node has no output

Try pointing a `complete` node at the node. Attach a debug to that. Does it send a message when the kafka node succeeds / fails?

Alternatively, does the node display any status underneath it after completion that signals success/failure? If so, try a status node looking at the kafka node.

Additionally (and likely the better solution), I would raise an issue on the nodes repo describing your requirement. If the dev is responsive, they might add an output to the node that indicates success or failure.

---

<div class="post-metadata">

**Author:** ![aj\_dev](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@aj\_dev](https://discourse.nodered.org/u/aj_dev)\
**Post date:** [26 January 2022 15:42 UTC](https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255/3 "2022-01-26T15:42:29Z")

</div>

> Try pointing a `complete` node at the node. Attach a debug to that. Does it send a message when the kafka node succeeds / fails?

I tried with "Complete". It does not send success / fail.

> does the node display any status underneath it after completion that signals success/failure? If so, try a status node looking at the kafka node.

The kafka node does display the status underneath. However it is not useful. When I connect debug to the node, it keeps sending this connectivity messages every second or so.

I think, I will have raise and issue.

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [26 January 2022 16:33 UTC](https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255/4 "2022-01-26T16:33:39Z")

</div>

> [@aj\_dev](#):
>
> The kafka node does display the status underneath. However it is not useful. When I connect debug to the node, it keeps sending this connectivity messages every second or so.

You could use a switch node to ignore the status messages - hopefully then you'll see success/fail messages?

> [@aj\_dev](#):
>
> I think, I will have raise and issue.

Please provide a link to the raised issue here so others with the same issue can keep track.

Thanks, Steve.

---

<div class="post-metadata">

**Author:** ![aj\_dev](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@aj\_dev](https://discourse.nodered.org/u/aj_dev)\
**Post date:** [26 January 2022 19:32 UTC](https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255/5 "2022-01-26T19:32:49Z")

</div>

Issue raised for reference [For Producer need to know if the message was successfully so that further actions could be taken. · Issue #5 · emrebekar/node-red-contrib-kafkajs · GitHub](https://github.com/emrebekar/node-red-contrib-kafkajs/issues/5) .  
I must try and alternate approach too.

---

<div class="post-metadata">

**Author:** ![lu4t](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/lu4t/32/32276_2.png) [@lu4t](https://discourse.nodered.org/u/lu4t)\
**Post date:** [27 January 2022 00:54 UTC](https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255/6 "2022-01-27T00:54:32Z")

</div>

If I understand you correctly, you're trying to make a Kafka client act as if it was synchronous...

If this is the case, bear in mind that:

- Default Kafka producer send API is **asynchronous** and nonblocking.
- HTTP is a **synchronous** protocol.

I don't think it will be easy for you to convince a node-red-contrib maintainer to develop what you say you need.

---

<div class="post-metadata">

**Author:** ![aj\_dev](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@aj\_dev](https://discourse.nodered.org/u/aj_dev)\
**Post date:** [27 January 2022 06:19 UTC](https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255/7 "2022-01-27T06:19:00Z")

</div>

@lu4t Probably you are right. Yes, it looks like I am trying to make Kafka client synchronous. If I was writing this code in, let's say "express", I would have handled it that way but managed it with async calls.

This begs a different question. (forgive me if I am going back to basics). In my given scenario should I just return the response (http 200) even though I am not sure if the client successfully stored the message in kafka? Otherwise, client just keeps waiting for node-red to respond, it will either timeout eventually and if it makes thousands of requests each of those requests will have to timeout.

What's the ideal way to handle this in node-red?

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [27 January 2022 06:39 UTC](https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255/8 "2022-01-27T06:39:00Z")

</div>

I might be wrong here, but isn't kafka a pubsub system ? Couldn't you just listen on (ie: consume) the topic ?

---

<div class="post-metadata">

**Author:** ![aj\_dev](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@aj\_dev](https://discourse.nodered.org/u/aj_dev)\
**Post date:** [27 January 2022 06:58 UTC](https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255/9 "2022-01-27T06:58:11Z")

</div>

@bakman2 No. You cannot. Because you have to respond to http request with the http response in the same branch of the flow where "Http in" node is. Otherwise you don't get the "response" object.

I have tried this with "Complete" as well as "Status" nodes. They don't work because you do not have http response object when responding. It has to be in the same path where "Http In" is. This is FYI for @Steve-Mcl response to earlier discussion.

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [27 January 2022 07:28 UTC](https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255/10 "2022-01-27T07:28:32Z")

</div>

Got it. You could store the output of the kafka client in a context variable and show that as output to the http response node. Probably wise to keep the context for this only in memory (not write to disk).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [28 March 2022 07:29 UTC](https://discourse.nodered.org/t/how-to-create-rest-api-that-stores-the-payload-to-kafka-and-send-http-response-only-after-successful-storage/57255/11 "2022-03-28T07:29:02Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.