Reading uploaded CSV and how to do the same with JSON

Hello, I am new to Node-RED and would appreciate any help I can get with this. I made a flow which allows you to upload a file, type in the files name and it will then read that file and use the headers for two drop downs, You can choose two headers and +,-,* and / and it will do that.

My current issues are:
I need it to pass the file name down when a file uploaded rather than have the user type in the file.

Currently it is set so that after 5 messages go through the flow it will carry on, the issue with that is if someone does it in the wrong order or changes something, it will crash. Is it possible to resolve this?

Finally is it possible to do something like this with JSON or Kafka?

I have spent a couple of hours looking, I might just be looking in the wrong places, but I haven't found anything that resolves these issues. Any links or advice would be greatly apricated, thank you.

Flow:

[{"id":"708ff173.7f789","type":"csv","z":"5e2fd65b.332578","name":"","sep":",","hdrin":false,"hdrout":"all","multi":"mult","ret":"\\n","temp":"","skip":"0","strings":true,"include_empty_strings":"","include_null_values":"","x":850,"y":40,"wires":[["69e6a7bc.4054a8"]]},{"id":"53158969.3182a8","type":"file in","z":"5e2fd65b.332578","name":"","filename":"TestingFileInput","format":"utf8","chunk":false,"sendError":false,"encoding":"none","x":660,"y":40,"wires":[["708ff173.7f789"]]},{"id":"99ee054.ff743f8","type":"python-function","z":"5e2fd65b.332578","name":"","func":"call = \"\";\n\ni = 1;\nmsg[\"options\"] = [];\nmsg[\"options\"].append(msg[\"payload\"][0][\"col\" + str(i)]);\ni += 1;\nwhile i <= len(msg[\"payload\"][0]):\n    msg[\"options\"].append(msg[\"payload\"][0][\"col\" + str(i)]);\n    i+= 1;\n\nreturn [msg, msg, msg, msg] ","outputs":4,"x":580,"y":180,"wires":[["cf7a832c.14142"],["719990d.627dc7"],["6744ce88.fc713"],["c78575c4.6a2e48"]]},{"id":"cf7a832c.14142","type":"ui_dropdown","z":"5e2fd65b.332578","name":"","label":"","tooltip":"","place":"Select column","group":"d984ee28.de125","order":3,"width":0,"height":0,"passthru":true,"multiple":false,"options":[],"payload":"","topic":"","x":940,"y":80,"wires":[["17338d8e.9aaae2"]]},{"id":"4e579112.18cac","type":"ui_dropdown","z":"5e2fd65b.332578","name":"","label":"","tooltip":"","place":"Select option","group":"d984ee28.de125","order":4,"width":0,"height":0,"passthru":false,"multiple":false,"options":[{"label":"+","value":"+","type":"str"},{"label":"/","value":"/","type":"str"},{"label":"*","value":"*","type":"str"},{"label":"-","value":"-","type":"str"}],"payload":"","topic":"msg.d4","x":1000,"y":180,"wires":[["17338d8e.9aaae2"]]},{"id":"719990d.627dc7","type":"ui_dropdown","z":"5e2fd65b.332578","name":"","label":"","tooltip":"","place":"Select column","group":"d984ee28.de125","order":5,"width":0,"height":0,"passthru":false,"multiple":false,"options":[],"payload":"","topic":"","x":940,"y":120,"wires":[["17338d8e.9aaae2"]]},{"id":"17338d8e.9aaae2","type":"join","z":"5e2fd65b.332578","name":"","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"5","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"num","reduceFixup":"","x":1350,"y":140,"wires":[["a3006da8.bc73c"]]},{"id":"a3006da8.bc73c","type":"python-function","z":"5e2fd65b.332578","name":"","func":"i = 1 \nmsg[\"second\"] = []\nthird = \"\"\nfirstarray= []\nthirdarray = []\ntes = []\nfirst = \"\"\nstri = \"\"\n\n\nwhile i <= len(msg[\"payload\"][0][0]):\n    stri = \"col\" + str(i)\n#itterating to see what col was chosen by the first choice\n    if msg[\"payload\"][1] == msg[\"payload\"][0][0][stri]:\n        first = stri\n        \n    if msg[\"payload\"][3] == msg[\"payload\"][0][0][stri]:\n        third = stri\n    i += 1\n\ni = 1\n\nwhile i + 1 < len(msg[\"payload\"][0]):\n    firstarray.append(msg[\"payload\"][0][i][first])\n    i += 1\n\ni = 1\nwhile i + 1 < len(msg[\"payload\"][0]):\n    thirdarray.append(msg[\"payload\"][0][i][third])\n    i += 1\n\ni = 1\n\nwhile i + 1 < len(firstarray):\n    if msg[\"payload\"][2] == \"+\":\n        tes.append(int(firstarray[i]) + int(thirdarray[i]))\n    if msg[\"payload\"][2] == \"*\":\n        tes.append(int(firstarray[i]) * int(thirdarray[i]))\n    if msg[\"payload\"][2] == \"/\":\n        tes.append(int(firstarray[i]) / int(thirdarray[i]))\n    if msg[\"payload\"][2] == \"-\":\n        tes.append(int(firstarray[i]) - int(thirdarray[i]))\n    i += 1\ntext = msg[\"payload\"][4]\nmsg[\"payload\"][0] = text\nmsg[\"payload\"][1] = tes\nreturn msg","outputs":1,"x":1520,"y":220,"wires":[["c8aa0120.ae3f2"]]},{"id":"69e6a7bc.4054a8","type":"change","z":"5e2fd65b.332578","name":"","rules":[{"t":"set","p":"all","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":390,"y":120,"wires":[["99ee054.ff743f8"]]},{"id":"641a6e69.d5335","type":"ui_text_input","z":"5e2fd65b.332578","name":"","label":"name of new column","tooltip":"","group":"d984ee28.de125","order":6,"width":0,"height":0,"passthru":false,"mode":"text","delay":"0","topic":"","x":960,"y":240,"wires":[["17338d8e.9aaae2"]]},{"id":"6744ce88.fc713","type":"change","z":"5e2fd65b.332578","name":"","rules":[{"t":"set","p":"options","pt":"msg","to":"[\"+\", \"-\", \"/\" ,\"*\"]","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":840,"y":180,"wires":[["4e579112.18cac"]]},{"id":"b6439579.e0efe8","type":"ui_text_input","z":"5e2fd65b.332578","name":"","label":"File name","tooltip":"","group":"d984ee28.de125","order":2,"width":0,"height":0,"passthru":true,"mode":"text","delay":"0","topic":"","x":280,"y":20,"wires":[["f73070e.2f7739"]]},{"id":"f73070e.2f7739","type":"change","z":"5e2fd65b.332578","name":"","rules":[{"t":"set","p":"filename","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":470,"y":40,"wires":[["53158969.3182a8"]]},{"id":"c78575c4.6a2e48","type":"change","z":"5e2fd65b.332578","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":760,"y":220,"wires":[["641a6e69.d5335"]]},{"id":"dbb49191.58209","type":"ui_upload","z":"5e2fd65b.332578","group":"d984ee28.de125","title":"upload","name":"","order":5,"width":0,"height":5,"chunk":256,"transfer":"text","x":70,"y":20,"wires":[["b6439579.e0efe8"]]},{"id":"c8aa0120.ae3f2","type":"debug","z":"5e2fd65b.332578","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1710,"y":220,"wires":[]},{"id":"d984ee28.de125","type":"ui_group","name":"Two","tab":"1e101955.bd39b7","order":2,"disp":true,"width":"6","collapse":false},{"id":"1e101955.bd39b7","type":"ui_tab","name":"LastDemo","icon":"dashboard","disabled":false,"hidden":false}]

Hi and welcome

To get help you will need to fix your posted flow, it is corrupt due to posting without code tags.

please read this
https://discourse.nodered.org/t/how-to-share-code-or-flow-json/506/4

Ah, thank you, is it working now?

Yes ok now.

if you want to input filename then you have to leave filename blank in the file in node.

I dont have a clue after that as i don't know what your python node is doing. You will need to explain more.

So, the first python node sets msg.options to be what ever is within msg.payload[0]["col" + i], This means that what ever is in the top line of the csv will be set to options, which is then used in the drop downs. There is a join node which combines all of the msg.payloads into an array. It then does this:

while i <= len(msg["payload"][0][0]):
stri = "col" + str(i)
This just means it will keep going till it's used all of the top row, it also sets stri to be col + i
if msg["payload"][1] == msg["payload"][0][0][stri]:
first = stri
It then checks which of the options was chosen, once it finds the correct answer it sets first to be that answer

if msg["payload"][3] == msg["payload"][0][0][stri]:
    third = stri

It does the same with the third input (also a drop down)

while i + 1 < len(msg["payload"][0]):
firstarray.append(msg["payload"][0][i][first])
i += 1

i = 1
while i + 1 < len(msg["payload"][0]):
thirdarray.append(msg["payload"][0][i][third])
i += 1

It goes through and adds everything in the correct columns to arrays.

while i + 1 < len(firstarray):
if msg["payload"][2] == "+":
tes.append(int(firstarray[i]) + int(thirdarray[i]))
if msg["payload"][2] == "*":
tes.append(int(firstarray[i]) * int(thirdarray[i]))
if msg["payload"][2] == "/":
tes.append(int(firstarray[i]) / int(thirdarray[i]))
if msg["payload"][2] == "-":
tes.append(int(firstarray[i]) - int(thirdarray[i]))
i += 1

Finally it checks if the second box was +, - * or / and will do that to the arrays. So if durations, * and products were chose, it would do that. If that makes sense

i have tried to convert to javascript function as i do not have python, or the format of you csv file. I think you would be better making a form using ui-template as you could validate it with javascript.

hope this moves you forward

[{"id":"f3b01072.3cab7","type":"inject","z":"792e0cca.a348cc","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":370,"y":220,"wires":[["1da790ad.73234f"]]},{"id":"1da790ad.73234f","type":"template","z":"792e0cca.a348cc","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"1,2,3,4\n1,2,3,4\n5,6,7,8","output":"str","x":550,"y":220,"wires":[["2617ae4b.5aa23a"]]},{"id":"2617ae4b.5aa23a","type":"csv","z":"792e0cca.a348cc","name":"","sep":",","hdrin":false,"hdrout":"all","multi":"mult","ret":"\\n","temp":"","skip":"0","strings":true,"include_empty_strings":"","include_null_values":"","x":690,"y":220,"wires":[["a9070655.b7db68"]]},{"id":"a9070655.b7db68","type":"change","z":"792e0cca.a348cc","name":"","rules":[{"t":"set","p":"all","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":400,"y":280,"wires":[["2046bfa6.52438"]]},{"id":"2046bfa6.52438","type":"function","z":"792e0cca.a348cc","name":"","func":"let msg1 = {\"options\":Object.values(msg.payload[0])}\nreturn [msg1,msg1,{options:[\"+\", \"-\", \"/\" ,\"*\"]},{payload:\"\"}];","outputs":4,"noerr":0,"initialize":"","finalize":"","x":570,"y":300,"wires":[["e9a5cf20.644458","703616a2.8a512"],["89a07c29.d96d58"],["c370d7cb.12ec08"],["34dd2350.9a81a4"]]},{"id":"e9a5cf20.644458","type":"debug","z":"792e0cca.a348cc","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":960,"y":220,"wires":[]},{"id":"703616a2.8a512","type":"ui_dropdown","z":"792e0cca.a348cc","name":"","label":"","tooltip":"","place":"Select column","group":"f22ae04f.7dcf9","order":3,"width":0,"height":0,"passthru":false,"multiple":false,"options":[],"payload":"","topic":"drop1","x":830,"y":280,"wires":[["ed379b69.6c4a5"]]},{"id":"89a07c29.d96d58","type":"ui_dropdown","z":"792e0cca.a348cc","name":"","label":"","tooltip":"","place":"Select column","group":"f22ae04f.7dcf9","order":5,"width":0,"height":0,"passthru":false,"multiple":false,"options":[],"payload":"","topic":"drop2","x":830,"y":320,"wires":[["ed379b69.6c4a5"]]},{"id":"34dd2350.9a81a4","type":"ui_text_input","z":"792e0cca.a348cc","name":"","label":"name of new column","tooltip":"","group":"f22ae04f.7dcf9","order":6,"width":0,"height":0,"passthru":false,"mode":"text","delay":"0","topic":"drop4","x":880,"y":400,"wires":[["ed379b69.6c4a5"]]},{"id":"c370d7cb.12ec08","type":"ui_dropdown","z":"792e0cca.a348cc","name":"","label":"","tooltip":"","place":"Select option","group":"f22ae04f.7dcf9","order":4,"width":0,"height":0,"passthru":false,"multiple":false,"options":[{"label":"+","value":"+","type":"str"},{"label":"/","value":"/","type":"str"},{"label":"*","value":"*","type":"str"},{"label":"-","value":"-","type":"str"}],"payload":"","topic":"drop3","x":830,"y":360,"wires":[["ed379b69.6c4a5"]]},{"id":"ed379b69.6c4a5","type":"join","z":"792e0cca.a348cc","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"num","reduceFixup":"","x":1070,"y":320,"wires":[["ed8fe5c8.f1749"]]},{"id":"ed8fe5c8.f1749","type":"function","z":"792e0cca.a348cc","name":"","func":"if (Object.keys(msg.payload).length === 4){\n msg.payload = Object.values(msg.payload); //to convert to array\n    return msg;\n}else{\nmsg.payload =\"error\";\nreturn [null,msg];\n}","outputs":2,"noerr":0,"initialize":"","finalize":"","x":1230,"y":340,"wires":[["a6704991.f6c18"],["34dd2350.9a81a4"]]},{"id":"476e3a9.9c08b44","type":"change","z":"792e0cca.a348cc","name":"","rules":[{"t":"set","p":"complete","pt":"msg","to":"true","tot":"bool"},{"t":"delete","p":"payload","pt":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":990,"y":440,"wires":[["ed379b69.6c4a5"]]},{"id":"a6704991.f6c18","type":"debug","z":"792e0cca.a348cc","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1430,"y":340,"wires":[]},{"id":"f138dc46.cab328","type":"ui_button","z":"792e0cca.a348cc","name":"","group":"f22ae04f.7dcf9","order":6,"width":0,"height":0,"passthru":false,"label":"send","tooltip":"","color":"","bgcolor":"","icon":"","payload":"complete","payloadType":"str","topic":"","x":810,"y":440,"wires":[["476e3a9.9c08b44"]]},{"id":"f22ae04f.7dcf9","type":"ui_group","name":"Two","tab":"91b691a5.209438","order":2,"disp":true,"width":"6","collapse":false},{"id":"91b691a5.209438","type":"ui_tab","name":"LastDemo","icon":"dashboard","disabled":false,"hidden":false}]
1 Like

Sorry, I didn't get a notification for this. The program worked for any csv file as long as the top row contained the headers. Thank you for the help, though I am a little confused, where would I put the csv file within the flow that you just sent?

You would add back your you first 4 nodes ui-upload to file in node.

I removed and added a inject and template to simulate the first 4 blocks as i do not have ui builder installed

1 Like

Awesome, thanks that helped massively. I think I can do the rest on my own. Have an amazing day :slight_smile:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.