Best practise for a nodered gateway "local>cloud" CSV files and OPC data

General
1

What would be the best way to upload my data to the cloud.
I see many people doing all different things and using different providers etc... It gets me really confusing after days/weeks of experimenting and reading.

Currently;

  • Nodered is installed on NAS and gathering data from MQTT and OPC and converts it into 100 line samples that get emailed and saved to google drive every 10 minutes.

Goals;

  • Free or cheap cloud storage provider (or my own NAS). Who would you recommend?
  • Make a flow that saves the incoming data, sends it to the cloud and deletes the data locally that is send to cloud. (This is my struggle as i cannot get a feedback to work that my data is successfully received.).
  • Make it robust, so when my internet drops it retains the data and resend it when connection is restored. (this i have not figured out).

Could somebody please help me in the right direction.

2

There are dozens of options but it depends on the total size you want and what type of access.

Well, that is the problem with cheap cloud storage, you may not get any feedback from the process. Really, you want to use a dedicate API or at least a process that does provide feedback such as SFTP.

The rest of those steps can be split out. (1) save a chunk of data to file (presumably a filename with the date/time in it), have a slight delay to let the filing system catch up (2) delete the batch of data. At that point, you are independent of the file transfer.

3

Thanks, That's indeed the problem :frowning: . There are to many options to choose from.
Azure, Google, IBM etc...and all have some examples on the internet but nothing i could find with saving the data locally and getting feedback of successfully uploaded files.

I will definitely look into SFTP as from my quick search it could go on my NAS.

4

If you want free, Google Drive or Microsoft OneDrive, Dropbox or Box.net or any number of others. But these are, as you've already seen just remote drives usually with a dedicated client application.

Google Drive certainly has an API. I also seem to remember that there may be a command-line client for dropbox for Linux.

If you can't get a positive confirmation of the upload working, you might have to upload the file, wait a few seconds and then check that the file size is what you expect.

5

Thanks, i am working on the moment with DropBox.
This seems to work flawless. And quick and easy.
Thanks for pointing me in the correct direction.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.