Clarify - Time Series Visualization

[bbergshaven]

Hi, I'm Bernt-Johan, one of the founders at Clarify.io.

I've been part of the Node-Red community for a while now, during which I've been developing the Clarify Node-Red Node. And with our release of a totally free plan, I feel it's time to tell you folks, more about what Clarify is and see how it can be useful for some of you.

Clarify is a time-series visualization platform that helps users quickly turn data into information. We do this by storing and presenting time-series data alongside comments, photos, videos, or more generically, the context of the data. This allows the raw data to live alongside the context that is required to understand it.

I've seen many people asking in the forums and here on slack about ways to visualize their time-series data, and even though some great solutions already exist with combinations of e.g influxDB and Grafana, I think some might appreciate the turnkey SaaS solution that Clarify can offer.

You can create a free account and start streaming your time-series data from Node-Red to Clarify by simply adding the node found in the palette, and doing some minor data transformation. node-red-contrib-clarify (node) - Node-RED

I'm using it for my home automation system and power surveillance and would love to share and discuss ideas on how to integrate and use Clarify.

I'd really appreciate honest feedback and improvement suggestions. If you have questions ask in or shoot me a DM.

[Colin]

Can you summarise what you see as the pros and cons of using Clarify vs Influx/Grafana?

[bbergshaven]

Hello Colin, and thank you for asking.

I'll start off by saying that Clarify and Influx/Grafana are not mutually exclusive, and we have several customers that use Grafana as an add-on to Clarify.

In Clarify we focus a lot on usability and the fact that it should be a tool for the whole workforce. With this, we feel that it is easier to explore data and create timelines that fit your need. We've seen that for many people that need data for decision-making, but working in "non-data jobs" in the industry, the threshold to find, organize, and combine data is too high in existing tools. Clarify is built for sharing and collaboration around data, for the whole industrial ecosystem. With our built-in social element that lets you tag data, comment, and include other people easily. We also have native apps for Android and iOS, so it's super easy to access and comment on your data.

Clarify is a fully hosted solution with an integrated time-series database, so once you create an account there is no additional setup needed. And with the contrib-clarify node, it's super easy to get data into Clarify, enrich it, and put it to use through both the API and the UI. We have good admin tools built into the product for managing integrations and signals, and there is a clear differentiation between signals and items (published signals) so that what an admin/engineer sees in the admin panel can be published as something that the whole workforce understands. If you want to add "data science" to any data signal in Clarify, you can easily connect to i.e. Jupyter Notebooks and even put them in production and visualize the results in Clarify, through our collab with orchest.io.

Hope this clarifies some of the differences, happy to answer any questions!

[Colin]

OK, thanks.

When you say it is fully hosted do you mean that the database is in the cloud? If so, does that mean you loose all visualisation if the internet is down?

[Paul-Reed]

Just tried the 'Maritime - Demo', but zooming into a chart is not intuitive.
I thought it would be similar to Grafana - click & select a period in the timeline, and it would zoom into it.
But that doesn't seem to be the case....

Chrome browser
Windows 10

[bbergshaven]

Hello Paul-Reed and thank you for the feedback. We have tried to make it even more intuitive than click & select, but we might have a way to go to explain that

Here are some short notes and videos explaining how to navigate around and explore your data. But in general, the zooming is done by hovering the area of interest and just scrolling your mouse-wheel or a similar gesture on a trackpad.

We have put a lot of work into our backend to be able to zoom from seconds to years of data with as little lag as possible.

Hoping to get more feedback

[Colin]

@bbergshaven Did you see my question about the hosting?

[bbergshaven]

Yes, you are right Colin. This is a fully SaaS package where everything is hosted online, so without internet no access to the visualizations. This might be a showstopper for some critical systems, but a lot of our customers and users find the trade-off to be reasonable.

If the data that you stream to Clarify is critical I would recommend a local buffer / retry so that if there are connectivity issues your flow would retry.
I'll note this down as a feature to build into the contrib-clarify node

[Colin]

@bbergshaven

without internet no access to the visualizations. This might be a showstopper for some critical systems

Well if you count home automation as critical then yes, it might be a showstopper. My internet was out for a month recently, after a storm took the fibre out, and the idea of being without my charts (currently Grafana) for that length of time would fill me with despair.

Colin

[Paul-Reed]

@bbergshaven

This is a fully SaaS package where everything is hosted online

...a lot of users who are running a Pi on a SD card would welcome this approach, as it's not writing/reading data to the SD card, and thereby prolonging it's life.

@bbergshaven

the zooming is done by hovering the area of interest and just scrolling your mouse-wheel or a similar gesture on a trackpad.

Not working for me I'm afraid - Acer ES_571 laptop with touchpad.

[bbergshaven]

@Paul-Reed

Not working for me I'm afraid - Acer ES_571 laptop with touchpad.

I'll have our product team look into it. I might contact you on the Node-Red slack to ask for some more information, hope that's ok

[Paul-Reed]

@bbergshaven

I'll have our product team look into it

I've updated my touchpad driver & it's now allowing me to zoom in OK, so sorted thanks.

[bbergshaven]

@Paul-Reed

I've updated my touchpad driver & it's now allowing me to zoom in OK, so sorted thanks.

Great, then I'll let the developers know so they can have a normal weekend
Thanks for letting me know!

[dynamicdave]

Having seen this thread, I signed-up for a free account this afternoon and had a go at plotting the readings from the DS18B20 sensor used on my home-grown Room Temperature Display (RTD) units.

Here's part of the experimental Node-RED flow I created to filter and format the readings for Clarify.

This is what the readings look like in Clarify for three of the RTD units. I think you can see when (approx 3:15pm) I left the house this afternoon and it started to cool-down (as I turned the central heating off).

It took a while to find my way around the various menus and to understand the nomenclature used by Clarify. All-in-all probably 15-minutes effort to get things up-and-running.

[Paul-Reed]

@bbergshaven Sorry for so many questions!

• Can the vertical plot area be increased? by default, the y axis is small, and only has 3 vertical scale markers. This also means that that it's hard to visualise small changes in data, when larger data changes are present in the same view.
• Is it possible to add more than one series to the same plot? for example temperature & humidity in the same plot, overlapping, instead of separate plots.
• Are we confined to white lines on a black background?
• How do we set a default view? for example display the last 30 minutes when opening the timeline, instead of scrolling and navigating to get that view?
• I've probably missed it, but I can't find anywhere what your data retention policy is, are there limits to the amount of user data that you will process/store, and api call limits?

In a similar way to Grafana;

[dynamicdave]

Hi @bbergshaven I have to agree with a number of Paul's comments.

I have found the x-axis will auto-scale when I zoom in to sections of the graph. A very nice feature.

Is there a way to embed just the graph part of Clarify (in an iFrame like I've tried to do here) ?

Like Paul asked, is there a way to submit a number of values at the same time?
For example, it would be nice to be able to submit 'voltage', 'current', 'power-factor' and 'power' readings I obtain from my PZEM-004T (Tasmota-based) unit.

I tried 'inviting' another person to join (actually it's me but with a different email address) and get this error.

[bbergshaven]

Hello Paul. Noting to be sorry about! I'll try to answer your questions below.

@Paul-Reed

• Can the vertical plot area be increased? by default, the y axis is small, and only has 3 vertical scale markers. This also means that that it's hard to visualise small changes in data, when larger data changes are present in the same view.

As of now we don't support changing the vertical plot area, but you're not the first person to ask about this so I know the product team has it on the radar. We have many features and upgrades on the roadmap for the months and year ahead and all feedback is welcomed.

• Is it possible to add more than one series to the same plot? for example temperature & humidity in the same plot, overlapping, instead of separate plots.
• Are we confined to white lines on a black background?

That's possible. You just drag and drop the data you want into the same visualization.
Our nomentclature is: One timeline consist of several visualizations that can contain one or more items.
Items are connected to one or more signals(raw data).

Here's a quick screen recording on how to add more items to a single visualization, renaming and changing colors.

• How do we set a default view? for example display the last 30 minutes when opening the timeline, instead of scrolling and navigating to get that view?

No support for this atm, but I'll pass it over to our product team right away.

• I've probably missed it, but I can't find anywhere what your data retention policy is, are there limits to the amount of user data that you will process/store, and api call limits?

This can be a bit easier to find, I'll see to it. For now you can find some info on our pricing page. For the free plan the limit is now 30 signals with 1second resolution. We don't delete any data, so historic data will be kept and accessible through the UI and/or APIs.
Here is a link to our Terms of Service

Thank you so much for the feedback!
Be sure to check out our Android or iOS app as well:

Best regards
Bernt-Johan

[bbergshaven]

@dynamicdave

Hi @bbergshaven I have to agree with a number of Paul's comments.

I have found the x-axis will auto-scale when I zoom in to sections of the graph. A very nice feature.

Hello Dynamic Dave! Thanks for the feedback.

Is there a way to embed just the graph part of Clarify (in an iFrame like I've tried to do here) ?

As of now we don't support any embedding of the Graphs, but is something we have in our feature inbox and I'll let the product team know of your request as well

Like Paul asked, is there a way to submit a number of values at the same time?
For example, it would be nice to be able to submit 'voltage', 'current', 'power-factor' and 'power' readings I obtain from my PZEM-004T (Tasmota-based) unit.

This is a typical, both yes and no answer. When we made the contrib-clarify node we decided to put inputID in msg.topic, metadata in msg.signal and data in msg.payload to make it as easy as possible to use deadband nodes etc. that filters on topic. The clarify-input node will actually combine several signals internally before doing the request to our API.

If you were to use our JSON RPC API directly the request would look something like this:

{
      "jsonrpc": "2.0",
      "method": "integration.insert",
      "id": "1",
      "params": {
        "integration": "<YOUR INTEGRATION ID>",
        "data":{
          "times": ["2021-03-11T21:49:06Z","2021-03-11T21:50:06Z","2021-03-11T21:51:06Z"],
          "series": {
            "<YOUR INPUT ID1>" : [1337,1338,null],
            "<YOUR INPUT ID2>" : [1337,null,1339]
          }
        }
      }
    }

I'll make a note of the request of sending multiple signals in one message. What I have been doing in my flows for now, is just looping through data and doing a node.send(msg) after building a signal message.

I tried 'inviting' another person to join (actually it's me but with a different email address) and get this error.

I'll have a look at our logs to see if I can figure out why this happened to you.

Regards
Bernt-Johan

[Paul-Reed]

Thanks @bbergshaven the video is very helpful, and I was up and running in no time.
We do lose a lot of detail when the 2 data sources are far apart, such as temperature 9 deg and humidity 55% as both are scaled the same, and are visualised as straight lines.

It would be nice if we could have the choice to scale to the left axis or the right axis, so for example the left axis range would auto adjust from 53 to 56 (for humidity), and the right axis auto adjust from 8 to 11 (for temperature).

I know we can't compare products here, as they are very different products, but that's how it's tackled in Grafana.

I don't want to sound negative about Clarify, but you did ask for feedback
I am liking Clarify more though as I am discovering more about it, and it's very kind of you to allow such a generous allowance in the free plan for hobby users, which include quite a few in the node-RED community.

[dynamicdave]

@bbergshaven

What I have been doing in my flows for now, is just looping through data and doing a node.send(msg) after building a signal message.

Using your suggestion I've split up the javascript object I receive from Tasmota as follows...

// Create a date-time-stamp
let date_now = new Date().toISOString();

// Pickup the readings from Tasmota PZEM-004T
let power   = msg.payload.ENERGY.ApparentPower;
let voltage = msg.payload.ENERGY.Voltage;
let current = msg.payload.ENERGY.Current;

// Send readings to Clarify with the same date-time-stamp

msg.topic = 'power';
msg.signal = {name: 'power', type: 'numeric', description: 'Home energy readings'};
msg.payload = {times: [date_now], values: [power]};
node.send(msg);

msg.topic = 'voltage';
msg.signal = {name: 'voltage', type: 'numeric', description: 'Home energy readings'};
msg.payload = {times: [date_now], values: [voltage]};
node.send(msg);

msg.topic = 'current';
msg.signal = {name: 'current', type: 'numeric', description: 'Home energy readings'};
msg.payload = {times: [date_now], values: [current]};
node.send(msg);

node.done();

return null;

This produces the following graph in Clarify, so I'm really pleased with what I've achieved with your help.