New file creator from dashboard with directory

i want to build a dashboard to create new files to save the information that i get from my sensors, i was looking for some examples or things that someone did it but i did't found something that really helps

First you have to decide what data and where are you going to save the data, Are you going to save it to a text file? An Excel file, CSV, database (which one)?

basically your flow will be:

               / => send to dashboard
read sensor => 
               \ => send to database (or file etc)

That is about all the help I can give with what little you have provided. If your are using a database, you need to setup the database first.

Take it one step at a time. make sure you can read the sensor

  • send the output to a debug node to see that it is working
  • then send the data to a node to formate it for the file storage then store the data.
  • go read the data to make sure it is there
  • build your dashboard
  • send data to dashboard and make sure it looks good

take it one step at a time (best way to learn)

I would like to make sure that you are clear about the separation between the "Dashboard" (which runs in a users browser) and the Node-RED server (which runs on your server of course)?

I assume that you mean that you wan to save files on the server that record your sensor data? So actually a Dashboard and the saved files are not directly related.

As Paul says, most people would use a database for this. You need to pay particular attending to the amount of data you are getting and how quickly it arrives. Sensor data can really build up over time so saving to files may become problematic unless you keep trimming the data. A database such as InfluxDB is specifically designed for this kind of "timeseries" data and includes commands that will let you set up an auto-trim of the data. For example to just keep the last 7d, 1 month/yr/etc. It also has special features that allows you to auto-summarise data to reduce volume over time.

So for example, I keep detail sensor data for 7 days at about 1 minute intervals. I have InfluxDB auto-trimming that data to 7d but it also auto-summarises the per minute data to hourly (avg, max & min) which is then kept for a couple of years.

@TotallyInformation so basically what i have until now is this. but i want to make my inject nodes into buttoms to create the new file and put the name also.

[{"id":"365c7e15.3dc1c2","type":"ui_template","z":"7d115c90.a35f24","group":"57838259.42276c","name":"UI Table JSON","order":1,"width":"","height":"","format":"<style>\n.table\n{\n    height:400px;\n    width:700px;\n    background:lightblue;\n}\n</style>\n<div class=\"table\">\n<table style=\"width:100%\">\n  <tr>\n    <th>TIMESTAMP</th> \n    <th>SENSOR</th>\n    <th>RAW DATA (JSON)</th>\n  </tr>\n  <tr ng-repeat=\"x in msg.payload | limitTo:10\">\n    <td>{{msg.payload[$index].TIMESTAMP}}</td>\n    <td>{{msg.payload[$index].SENSOR}}</td>\n    <td>{{msg.payload[$index].RAW_DATA}}</td>\n  </tr>\n</table>\n</div>","storeOutMessages":true,"fwdInMessages":true,"templateScope":"local","x":760,"y":60,"wires":[[]]},{"id":"e9bd5e7b.d5efa","type":"inject","z":"7d115c90.a35f24","name":"Test Data","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":300,"wires":[["981a1490.5e9c38"]]},{"id":"981a1490.5e9c38","type":"function","z":"7d115c90.a35f24","name":"test data","func":"var count=context.get('count');\nvar timestamp=context.get('timestamp');\n\nif (count==undefined)\n{\n    count=1;\n    timestamp=1;\n}\nvar data1={\"sensor\":\"sensor1\",\"temperature\":21,\"humidity\":61};\nvar data2={\"sensor\":\"sensor2\",\"temperature\":22,\"humidity\":62};\nvar data3={\"sensor\":\"sensor3\",\"temperature\":23,\"humidity\":63};\nif (count%20==0)\n    count=-count;\ndata1.temperature+=(count*.25);\n//data2.temperature+=(count*.5);\n//data3.temperature+=(count*.75);\ndata1.humidity+=(count*.25);\n//data2.humidity+=(count*.5);\n//data3.humidity+=(count*.75);\nmsg.payload=[data1];\nvar msg1={};\nvar msg2={};\nvar msg3={};\n\ndata1.timestamp=timestamp;\ntimestamp+=1;\ndata2.timestamp=timestamp;\ntimestamp+=1;\ndata3.timestamp=timestamp;\n\nmsg1.payload=JSON.stringify(data1);\nmsg2.payload=JSON.stringify(data2);\nmsg3.payload=JSON.stringify(data3);\ntimestamp+=1;\ncount+=1;\ncontext.set('count',count);\ncontext.set('timestamp',timestamp);\nreturn [[msg1,msg2,msg3]];","outputs":1,"noerr":0,"x":240,"y":500,"wires":[["bfdee763.da2578","d063bd43.b93ac"]]},{"id":"d063bd43.b93ac","type":"function","z":"7d115c90.a35f24","name":"write query","func":"var raw_data=msg.payload;\nvar temp=JSON.parse(msg.payload);\nvar timestamp=temp.timestamp;\nvar sensor=temp.sensor;\nmsg=\"(timestamp,sensor,raw_data) values(\"+timestamp +\",\\'\"+sensor +\"\\'\"+ \",\\'\"+raw_data+\"\\'\"  +\")\";\nvar topic=\"INSERT INTO DATA \" +msg;\nvar msg1={};\nmsg1.topic=topic;\nmsg1.payload=timestamp;\nreturn msg1;","outputs":1,"noerr":0,"x":350,"y":380,"wires":[["de705f2c.35357","c62c694f.3f9788"]]},{"id":"de705f2c.35357","type":"debug","z":"7d115c90.a35f24","name":"sql out","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":530,"y":340,"wires":[]},{"id":"bfdee763.da2578","type":"debug","z":"7d115c90.a35f24","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":490,"y":520,"wires":[]},{"id":"c62c694f.3f9788","type":"delay","z":"7d115c90.a35f24","name":"","pauseType":"delay","timeout":"1","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":560,"y":440,"wires":[["f3b96514.a3f678"]]},{"id":"f3b96514.a3f678","type":"sqlite","z":"7d115c90.a35f24","mydb":"b7f1797e.26cdb8","sqlquery":"msg.topic","sql":"","name":"","x":550,"y":200,"wires":[[]]},{"id":"88b9677a.d02a38","type":"inject","z":"7d115c90.a35f24","name":"show RECORDS","topic":"SELECT * FROM DATA ORDER BY TIMESTAMP DESC LIMIT 100","payload":"","payloadType":"date","repeat":"1","crontab":"","once":true,"onceDelay":"","x":230,"y":80,"wires":[["66be463c.163598"]]},{"id":"66be463c.163598","type":"sqlite","z":"7d115c90.a35f24","mydb":"b7f1797e.26cdb8","sqlquery":"msg.topic","sql":"","name":"","x":470,"y":60,"wires":[["365c7e15.3dc1c2"]]},{"id":"d4238074.7588d","type":"inject","z":"7d115c90.a35f24","name":"create database table","topic":"CREATE TABLE data(TIMESTAMP INT PRIMARY KEY NOT NULL, SENSOR TEXT NOT NULL,RAW_DATA TEXT NOT NULL)","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":"","x":240,"y":140,"wires":[["f3b96514.a3f678"]]},{"id":"5d910dde.bb94c4","type":"inject","z":"7d115c90.a35f24","name":"delete records","topic":"DELETE FROM DATA WHERE TIMESTAMP>=0","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":"","x":240,"y":180,"wires":[["f3b96514.a3f678"]]},{"id":"eb5750f.00c35b","type":"inject","z":"7d115c90.a35f24","name":"Drop Table","topic":"drop table data","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":"","x":240,"y":240,"wires":[["f3b96514.a3f678"]]},{"id":"57838259.42276c","type":"ui_group","z":"","name":"Sensors","tab":"e23e9cd7.e914d","order":1,"disp":true,"width":"14","collapse":false},{"id":"b7f1797e.26cdb8","type":"sqlitedb","z":"","db":"/home/pi/sensordata.db","mode":"RWC"},{"id":"e23e9cd7.e914d","type":"ui_tab","z":"","name":"Home","icon":"dashboard"}]

Currently you are saving the data to a database. Are you sayng you want to save it to a file instead?

no i mean, if there is a way to make something like Excel when you are saving an Excel file you can choose the directory and put the name on the file, i know that i have a database on sqlite but i was looking for this because i want to use multiple databases and i want to put on each one a date or something to have that kind of control con the data i don't know if i explained well.

The idea of using a different database for each day would be difficult at best since you can not create databases dynamically.

The idea of creating a new table every day would also be difficult since you would need to change the name of the table in all the SQL statements you used and/or have duplicate code for each day.

If you want to use a database, I suggest creating a table that includes a column for the date. That way you could retreive all the rows for a particular date.

What are you using the data for? Maybe there is a better way to do what you want to do.

alright i get if i use a txt file it is possible?

1 Like

Before I answer that how much data are we talking about and what will you use the files for?

maybe like 18000 per day and i will use to export to excel

If you have a database, would would you want to save the data to separate excel files ?
You could even connect to the database from excel directly

That is a lot of Excel data.

So if the 18,000 readings come in over 24 hours, you will be taking a reading every 4.8 seconds, If it is over 12 hours it would be a reading every 2.4 seconds.

What are you measuring that needs a reading that frequently? And if you put that in an excel spread sheet will each reading be a row? who would ever scroll thru 18,000 rows?

How much data will be written for each sensor reading? Have you calculated how big each days file will be and hw much storage it will take up in a week, month, year?

Also I hope you are not running on a Pi because I'd be concerned about the life of the SD card.

i mean that will be the top of the maximun i get what you said i already think about it i mean i just want to know if i can what i told just just that and how can i do it

Well you can do what you want but if you are collecting the data all day, why do you need a button to start a new file for the day? What happens if you forget to press the button? Wouldn't that cause todays data to be written to yesterdays file?

I would think that you would want this automatic so that at midnight, any new data would start a new file. You could build the file name using the date of the current day and send each record to one of the file out nodes set to 'append' to the file.

I would use an inject node to emulate the sensor data coming in before connecting it to your sensor input. Once you know it works then attach your sensor node to the flow.