# Comparing Large Data Set

**URL:** <https://discourse.nodered.org/t/comparing-large-data-set/70611>\
**Category:** General\
**Created:** [15 November 2022 03:03 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611 "2022-11-15T03:03:06Z")\
**Posts on this page:** 16\
**Page:** 2

<div class="post-metadata">

**Author:** ![interscope101](https://avatars.discourse-cdn.com/v4/letter/i/a88e4f/32.png) [@interscope101](https://discourse.nodered.org/u/interscope101)\
**Post date:** [19 November 2022 17:29 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/21 "2022-11-19T17:29:40Z")

</div>

Thanks, after the array is filtered down I expect it will be \<500 records. I'll know for sure as soon as I figure out how to implement the filtering.

---

<div class="post-metadata">

**Author:** ![interscope101](https://avatars.discourse-cdn.com/v4/letter/i/a88e4f/32.png) [@interscope101](https://discourse.nodered.org/u/interscope101)\
**Post date:** [19 November 2022 17:53 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/22 "2022-11-19T17:53:36Z")

</div>

> [@TotallyInformation](#):
>
> Wiser node.js

Could you guide me how to test your library with my data set ?

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [19 November 2022 18:28 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/23 "2022-11-19T18:28:45Z")

</div>

There are a few object diff libraries on npm: [deep-object-diff - npm search (npmjs.com)](https://www.npmjs.com/search?q=deep-object-diff) - the one I use is the exact match on that link but others are available.

To use it in a function node, either require it in settings.js on a global property or turn on the capability to install npm packages in function nodes (settings also in settings.js) and install it in a function node. Instructions on using the library are in its docs and it is very straight-forward.

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [19 November 2022 18:31 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/24 "2022-11-19T18:31:21Z")

</div>

Roughly tested this:

```auto
let prev = context.get("coins") 
const input = msg.payload
// first time set context only
if(prev == undefined) {
    context.set("coins", input)
    return 
}

input.forEach(coin=>{
    const search = prev.find(p => p.symbol === coin.symbol)
    const last = parseFloat(coin.last)
    const prev_last = parseFloat(search.last)
    const calc = (((last - prev_last) / prev_last) * 100).toFixed(2);   
    // coin.gain = calc
    
    node.send({payload:{symbol:coin.symbol, gain: calc}})
})
context.set("coins",msg.payload)
//node.send({payload:input})

```

You could remove the `//` to send the whole array at once instead.

Input payload are arrays:

```auto
[
{
    "symbol":"GALA/USDT",
    "last": "0.35"
},
{
    "symbol":"BTC/USDT",
    "last": "16071.02"
}
]

```

You should be able to process this every second, depending on your cpu, it can handle 1000's of operations per millisecond. Arrays live in memory, which is extremely fast.

Output:

 ![SCR-20221119-raz](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/1/f/1fb42b5ff046536b932c81f4c6b572f30a26e50e.png)

---

<div class="post-metadata">

**Author:** ![interscope101](https://avatars.discourse-cdn.com/v4/letter/i/a88e4f/32.png) [@interscope101](https://discourse.nodered.org/u/interscope101)\
**Post date:** [19 November 2022 18:40 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/25 "2022-11-19T18:40:43Z")

</div>

> [@bakman2](#):
>
> ```auto
> let prev = context.get("coins") 
> const input = msg.payload
> // first time set context only
> if(prev == undefined) {
> context.set("coins", input)
> return 
> }
> 
> input.forEach(coin=>{
> const search = prev.find(p => p.symbol === coin.symbol)
> const last = parseFloat(coin.last)
> const prev_last = parseFloat(search.last)
> const calc = (((last - prev_last) / prev_last) * 100).toFixed(2);   
> // coin.gain = calc
>     
> node.send({payload:{symbol:coin.symbol, gain: calc}})
> })
> context.set("coins",msg.payload)
> //node.send({payload:input})
> 
> ```

Getting ""TypeError: input.forEach is not a function" error ... which of the two data formats did you try?

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [19 November 2022 18:42 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/26 "2022-11-19T18:42:24Z")

</div>

> which of the two data formats did you try?

Arrays (your 2nd screenshot). And note that your payload data is not the same as my function (instead of symbol you will need to use `currency_pair`

---

<div class="post-metadata">

**Author:** ![interscope101](https://avatars.discourse-cdn.com/v4/letter/i/a88e4f/32.png) [@interscope101](https://discourse.nodered.org/u/interscope101)\
**Post date:** [19 November 2022 18:50 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/27 "2022-11-19T18:50:54Z")

</div>

Oh, I see, you're using a modified input - I wasn't able to get that far transforming the whole array, just one record. Could you suggest how to best filter it (at the same time as preparing the input, based on these three variables?

```auto
1) payload.payload["PRQ/USDT"].symbol - only those with "USDT" base
2) payload.payload["PRQ/USDT"].last - only those > .01
3) payload.payload["SOL/USDT"].baseVolume only those > 10000

```

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [19 November 2022 18:56 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/28 "2022-11-19T18:56:52Z")

</div>

Why is the payload called payload.payload ?

First try to make this work on the whole array (i tried to change it so that it "should work"):

```auto
let prev = context.get("coins") 
const input = msg.payload.payload // 
// first time set context only
if(prev == undefined) {
    context.set("coins", input)
    return 
}

input.forEach(coin=>{
    const search = prev.find(p => p.currency_pair === coin.currency_pair)
    const last = parseFloat(coin.last)
    const prev_last = parseFloat(search.last)
    const calc = (((last - prev_last) / prev_last) * 100).toFixed(2);   
    // coin.gain = calc
    
    node.send({payload:{symbol:coin.currency_pair, gain: calc}})
})
context.set("coins",input)

```

If this works, from there we can look into filtering the things that you actually want.

---

<div class="post-metadata">

**Author:** ![interscope101](https://avatars.discourse-cdn.com/v4/letter/i/a88e4f/32.png) [@interscope101](https://discourse.nodered.org/u/interscope101)\
**Post date:** [19 November 2022 19:04 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/29 "2022-11-19T19:04:27Z")

</div>

> [@bakman2](#):
>
> ```auto
> let prev = context.get("coins") 
> const input = msg.payload.payload // 
> // first time set context only
> if(prev == undefined) {
> context.set("coins", input)
> return 
> }
> 
> input.forEach(coin=>{
> const search = prev.find(p => p.currency_pair === coin.currency_pair)
> const last = parseFloat(coin.last)
> const prev_last = parseFloat(search.last)
> const calc = (((last - prev_last) / prev_last) * 100).toFixed(2);   
> // coin.gain = calc
>     
> node.send({payload:{symbol:coin.currency_pair, gain: calc}})
> })
> context.set("coins",msg.payload)
> 
> ```

My function example was dealing with the first data format (hence the different naming). So you're testing on this one? Unfortunately still getting the same "TypeError: prev.find is not a function"

```auto
{"currency_pair":"PRQ_USDT","last":"0.0853","lowest_ask":"0.0853","highest_bid":"0.0852","change_percentage":"-3.06","change_utc0":"-0.81","change_utc8":"1.18","base_volume":"1074494.223282","quote_volume":"91759.208421527","high_24h":"0.0939","low_24h":"0.0811"}

```

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/5/8/5821f189db7a848dc6e9f8e081bed5a1732d3fe2.png)

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [19 November 2022 19:05 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/30 "2022-11-19T19:05:32Z")

</div>

Copy the code again, I modified it because of the payload.payload the context was not set properly. Now it should.

---

<div class="post-metadata">

**Author:** ![interscope101](https://avatars.discourse-cdn.com/v4/letter/i/a88e4f/32.png) [@interscope101](https://discourse.nodered.org/u/interscope101)\
**Post date:** [19 November 2022 19:07 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/31 "2022-11-19T19:07:37Z")

</div>

> [@bakman2](#):
>
> ```auto
> [
> {
> "symbol":"GALA/USDT",
> "last": "0.35"
> },
> {
> "symbol":"BTC/USDT",
> "last": "16071.02"
> }
> ]
> 
> ```

P.S. you indicated that the input format is this? not the actual incoming array?

```auto
[
{
    "symbol":"GALA/USDT",
    "last": "0.35"
},
{
    "symbol":"BTC/USDT",
    "last": "16071.02"
}
]

```

---

<div class="post-metadata">

**Author:** ![interscope101](https://avatars.discourse-cdn.com/v4/letter/i/a88e4f/32.png) [@interscope101](https://discourse.nodered.org/u/interscope101)\
**Post date:** [19 November 2022 19:09 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/32 "2022-11-19T19:09:40Z")

</div>

> [@bakman2](#):
>
> ```auto
> let prev = context.get("coins") 
> const input = msg.payload.payload // 
> // first time set context only
> if(prev == undefined) {
> context.set("coins", input)
> return 
> }
> 
> input.forEach(coin=>{
> const search = prev.find(p => p.currency_pair === coin.currency_pair)
> const last = parseFloat(coin.last)
> const prev_last = parseFloat(search.last)
> const calc = (((last - prev_last) / prev_last) * 100).toFixed(2);   
> // coin.gain = calc
>     
> node.send({payload:{symbol:coin.currency_pair, gain: calc}})
> })
> context.set("coins",msg.payload)
> 
> ```

I did, that was with the update:

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/7/1/7143f8bb9fa06896d0f4c6f93b05990487c9485d.png)

---

<div class="post-metadata">

**Author:** ![interscope101](https://avatars.discourse-cdn.com/v4/letter/i/a88e4f/32.png) [@interscope101](https://discourse.nodered.org/u/interscope101)\
**Post date:** [19 November 2022 19:13 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/33 "2022-11-19T19:13:18Z")

</div>

Here is the full data format (first three records)

```auto
{
    "exchange": "Gate.io",
    "api": "get_tickers",
    "arguments": "",
    "payload": [
        {
            "currency_pair": "SRK_ETH",
            "last": "0.00000018",
            "lowest_ask": "0.000000183",
            "highest_bid": "0.000000178",
            "change_percentage": "1.69",
            "change_utc0": "3.44",
            "change_utc8": "2.27",
            "base_volume": "79985174.98",
            "quote_volume": "14.0636925668",
            "high_24h": "0.000000181",
            "low_24h": "0.000000172"
        },
        {
            "currency_pair": "LBL_USDT",
            "last": "0.004457",
            "lowest_ask": "0.004456",
            "highest_bid": "0.004352",
            "change_percentage": "4.5",
            "change_utc0": "9.21",
            "change_utc8": "-3.56",
            "base_volume": "80688419.381082",
            "quote_volume": "378113.53589448",
            "high_24h": "0.005666",
            "low_24h": "0.003946"
        },
        {
            "currency_pair": "ISP_ETH",
            "last": "0.0000005256",
            "lowest_ask": "0.000000527",
            "highest_bid": "0.0000005225",
            "change_percentage": "-2.21",
            "change_utc0": "-2.08",
            "change_utc8": "-0.52",
            "base_volume": "43767933",
            "quote_volume": "23.5053001431",
            "high_24h": "0.00000055",
            "low_24h": "0.00000052"
        },

```

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [19 November 2022 19:14 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/34 "2022-11-19T19:14:08Z")

</div>

I send you a DM.

---

<div class="post-metadata">

**Author:** ![interscope101](https://avatars.discourse-cdn.com/v4/letter/i/a88e4f/32.png) [@interscope101](https://discourse.nodered.org/u/interscope101)\
**Post date:** [11 December 2022 02:14 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/35 "2022-12-11T02:14:31Z")

</div>

Hello, you mentioned using finhubb ... I would like test it out, just got an API key ... could you show me the configuration to use where I can input the symbol via msg. Thank you.

```auto
const finnhub = require('finnhub');

const api_key = finnhub.ApiClient.instance.authentications['api_key'];
api_key.apiKey = "ceahq1aad3i1o5buvh00ceahq1aad3i1o5buvh0g"
const finnhubClient = new finnhub.DefaultApi()

finnhubClient.supportResistance("AAPL", "D", (error, data, response) => {
  console.log(data)
});

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [9 February 2023 02:15 UTC](https://discourse.nodered.org/t/comparing-large-data-set/70611/36 "2023-02-09T02:15:12Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

[Previous page](https://discourse.nodered.org/t/comparing-large-data-set/70611.md?page=1)