Remove duplicates from an array of objects

General
1

Talk about a self fulfilling prophecy!!
I've noticed that duplicate entries have started to appear in my data feed, which are creating errors in a 7 day rolling average calculation.

So I'm getting now something like;

[{
	"x": 1591315200000,
	"y": 8
}, {
	"x": 1591315200000,
	"y": 8
}, {
	"x": 1591315200000,
	"y": 8
}, {
	"x": 1591228800000,
	"y": 30
}]

So 3 of the 4 entries are identical, and I need just a unique entry for each timestamp, like;

[{
	"x": 1591315200000,
	"y": 8
}, {
	"x": 1591228800000,
	"y": 30
}]

I've googled, and found several solutions, but not really found one that works.
Is there an easy way to filter out the duplicates?

2

I would take the lazy approach and use a function node.

var prev = null;

msg.payload = msg.payload.filter((item) => {
    var keep = (item.x !== prev);
    prev = item.x;
    return keep;
});

return msg;

This assumes you only want to dedupe on 'x', and they are all in sequence. You would have to modify it a bit if that's not the case.

1 Like
3

Perfect! thanks Michael.

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.