Comparing Large Data Set

Thanks, after the array is filtered down I expect it will be <500 records. I'll know for sure as soon as I figure out how to implement the filtering.

Could you guide me how to test your library with my data set ?

There are a few object diff libraries on npm: deep-object-diff - npm search (npmjs.com) - the one I use is the exact match on that link but others are available.

To use it in a function node, either require it in settings.js on a global property or turn on the capability to install npm packages in function nodes (settings also in settings.js) and install it in a function node. Instructions on using the library are in its docs and it is very straight-forward.

Roughly tested this:

let prev = context.get("coins") 
const input = msg.payload
// first time set context only
if(prev == undefined) {
    context.set("coins", input)
    return 
}

input.forEach(coin=>{
    const search = prev.find(p => p.symbol === coin.symbol)
    const last = parseFloat(coin.last)
    const prev_last = parseFloat(search.last)
    const calc = (((last - prev_last) / prev_last) * 100).toFixed(2);   
    // coin.gain = calc
    
    node.send({payload:{symbol:coin.symbol, gain: calc}})
})
context.set("coins",msg.payload)
//node.send({payload:input})

You could remove the // to send the whole array at once instead.

Input payload are arrays:

[
{
    "symbol":"GALA/USDT",
    "last": "0.35"
},
{
    "symbol":"BTC/USDT",
    "last": "16071.02"
}
]

You should be able to process this every second, depending on your cpu, it can handle 1000's of operations per millisecond. Arrays live in memory, which is extremely fast.

Output:

Getting ""TypeError: input.forEach is not a function" error ... which of the two data formats did you try?

which of the two data formats did you try?

Arrays (your 2nd screenshot). And note that your payload data is not the same as my function (instead of symbol you will need to use currency_pair

Oh, I see, you're using a modified input - I wasn't able to get that far transforming the whole array, just one record. Could you suggest how to best filter it (at the same time as preparing the input, based on these three variables?

1) payload.payload["PRQ/USDT"].symbol  - only those with "USDT" base
2) payload.payload["PRQ/USDT"].last - only those  > .01
3) payload.payload["SOL/USDT"].baseVolume only those > 10000

Why is the payload called payload.payload ?

First try to make this work on the whole array (i tried to change it so that it "should work"):

let prev = context.get("coins") 
const input = msg.payload.payload // 
// first time set context only
if(prev == undefined) {
    context.set("coins", input)
    return 
}

input.forEach(coin=>{
    const search = prev.find(p => p.currency_pair === coin.currency_pair)
    const last = parseFloat(coin.last)
    const prev_last = parseFloat(search.last)
    const calc = (((last - prev_last) / prev_last) * 100).toFixed(2);   
    // coin.gain = calc
    
    node.send({payload:{symbol:coin.currency_pair, gain: calc}})
})
context.set("coins",input)

If this works, from there we can look into filtering the things that you actually want.

My function example was dealing with the first data format (hence the different naming). So you're testing on this one? Unfortunately still getting the same "TypeError: prev.find is not a function"

{"currency_pair":"PRQ_USDT","last":"0.0853","lowest_ask":"0.0853","highest_bid":"0.0852","change_percentage":"-3.06","change_utc0":"-0.81","change_utc8":"1.18","base_volume":"1074494.223282","quote_volume":"91759.208421527","high_24h":"0.0939","low_24h":"0.0811"}

Copy the code again, I modified it because of the payload.payload the context was not set properly. Now it should.

P.S. you indicated that the input format is this? not the actual incoming array?

[
{
    "symbol":"GALA/USDT",
    "last": "0.35"
},
{
    "symbol":"BTC/USDT",
    "last": "16071.02"
}
]

I did, that was with the update:

Here is the full data format (first three records)

{
    "exchange": "Gate.io",
    "api": "get_tickers",
    "arguments": "",
    "payload": [
        {
            "currency_pair": "SRK_ETH",
            "last": "0.00000018",
            "lowest_ask": "0.000000183",
            "highest_bid": "0.000000178",
            "change_percentage": "1.69",
            "change_utc0": "3.44",
            "change_utc8": "2.27",
            "base_volume": "79985174.98",
            "quote_volume": "14.0636925668",
            "high_24h": "0.000000181",
            "low_24h": "0.000000172"
        },
        {
            "currency_pair": "LBL_USDT",
            "last": "0.004457",
            "lowest_ask": "0.004456",
            "highest_bid": "0.004352",
            "change_percentage": "4.5",
            "change_utc0": "9.21",
            "change_utc8": "-3.56",
            "base_volume": "80688419.381082",
            "quote_volume": "378113.53589448",
            "high_24h": "0.005666",
            "low_24h": "0.003946"
        },
        {
            "currency_pair": "ISP_ETH",
            "last": "0.0000005256",
            "lowest_ask": "0.000000527",
            "highest_bid": "0.0000005225",
            "change_percentage": "-2.21",
            "change_utc0": "-2.08",
            "change_utc8": "-0.52",
            "base_volume": "43767933",
            "quote_volume": "23.5053001431",
            "high_24h": "0.00000055",
            "low_24h": "0.00000052"
        },

I send you a DM.

Hello, you mentioned using finhubb ... I would like test it out, just got an API key ... could you show me the configuration to use where I can input the symbol via msg. Thank you.

const finnhub = require('finnhub');

const api_key = finnhub.ApiClient.instance.authentications['api_key'];
api_key.apiKey = "ceahq1aad3i1o5buvh00ceahq1aad3i1o5buvh0g"
const finnhubClient = new finnhub.DefaultApi()

finnhubClient.supportResistance("AAPL", "D", (error, data, response) => {
  console.log(data)
});