UI Tabular Table Sorting

interscope101 · 2 December 2022 07:40

In respect to saving data. Why use external file save/ read nodes vs simply saving to localfilesystem in the function? Isn't this the same thing: flow.set('recordKeeper', check, 'file') ?

Regarding frequency/triggers. The API call is made every minute. There may be hours with no new records and there may be instances when you get a handful created within minute. So my thinking is to grab 10-20 at a time ( to make sure there a large-enough overlap) and at that point they are saved - basically that's what we already have. The table and charts need to be constantly updating with the new/full data set.

I didn't realize that this entire block is commented out, so in fact none of these variables are getting set.

 tempObj.in = obj.note.split("_");

        /*tempObj.frames = obj.in[0];
        tempObj.gain = obj.in[1];
        tempObj.gain_st = obj.in[2];
        tempObj.freq = obj.in[3];
        tempObj.analysis = obj.in[4];
        tempObj.tp_a = obj.in[5];
        tempObj.tp_trl_a = obj.in[6];
        tempObj.sl_a = obj.in[7];
        tempObj.sl_trl_a = obj.in[8];
        tempObj.tm_a = obj.in[9];*/

Let me clarify what this is. To associate/track certain parameters with the transaction I create a string of those parameters with a separator and inject it into the note field (as the transaction gets initiated). Now I am downloading the finished transaction and parsing back that note field to separate those parameters and that's working fine. You see all the parameters separated in the sub-array [via empObj.in = obj.note.split("_")]

Now I need to assigned them to variables. Could there be an issue with a changed path reference? This the path for "S.BUY" ["21628617"].in[4]

madhouse · 2 December 2022 16:25

Not quite. Using something like flow.set to save your file stores it in a designated file built for the node-red environment. You could have a lot of stuff going into that file and there's not a lot of control without writing a lot of different lines of code to handle the manipulation of the file. That's not saying it can't be done. It's just not an easy lift to make it happen the way you need it.

If you want to try some file reading and saving, look up these tutorials for using the fs module in Node.js for reading and writing. I haven't used this yet, but it looks pretty straight forward and should handle well in Node-Red. The one warning is I don't know the default storage location, so you'll likely need to go hunting for your file if want to look at it's contents outside of Node-Red or assign it directly to a fully defined folder on your computer.

This is very possible. When the code you're building is working, you don't need to limit it to 10-20. You could bring in the maximum your API will allow and parse that as needed if you want. The code will handle it. It won't do to much to system resources either. While it looks like a lot of data visually, it's really only a few KB of data to the system. Not much to worry about. If you grab 100 results and there is only one new one, it will pull that one new one and add it to your data.

I also didn't pay attention to that block of code to see what it was doing. My fault. Late night coding is not a really good idea... You should be able to replace the code with this to make it work.

var check = context.get('recordKeeper');

const input = msg.payload.filter(s => s.status.type == "stop_loss_finished" || s.status.type == "finished");

var tempObj ={};
var tempArr = [];
for(let obj of input){

    if(!(check.hasOwnProperty(obj.id))){
        tempObj = {};
        tempObj.pair = (String(obj.pair)).replace("USDT_", "");
        tempObj.trade_id = obj.id;
        tempObj.profit_usd = Number(obj.profit.usd).toFixed(2);
        tempObj.profit_percent = obj.profit.percent;
        tempObj.position = Number(obj.position.total.value).toFixed(0);
        tempObj.volume = obj.data.current_price.quote_volume;
        tempObj.close_price = obj.data.average_close_price;
        tempObj.closed = obj.data.closed_at;

        if (obj.status.type === "stop_loss_finished"){tempObj.status = "SL";}
            else if (obj.status.type === "finished"){tempObj.status = "TP";}
            else if (obj.status.type === "panic_sold"){tempObj.status = "➦";}
            else {tempObj.status = obj.status.type}

        if (obj.stop_loss.enabled) {
            tempObj.to_sl = ((1 - obj.stop_loss.conditional.price.value / obj.data.average_enter_price) * 100).toFixed(2);
        }

        if (obj.status.type == "stop_loss_finished") {
            tempObj.S_L = (((obj.data.average_enter_price - obj.stop_loss.conditional.price.value) / obj.data.average_enter_price) * 100).toFixed(2);
        }

        if (obj.status.type == "finished") {
            tempObj.T_P = (((obj.take_profit.steps[0].price.value - obj.data.average_enter_price) / obj.data.average_enter_price) * 100).toFixed(2);
        }

        tempArr = obj.note.split("_");
        
        tempObj.frames = tempArr[0];
        tempObj.gain = tempArr[1];
        tempObj.gain_st = tempArr[2];
        tempObj.freq = tempArr[3];
        tempObj.analysis = tempArr[4];
        tempObj.tp_a = tempArr[5];
        tempObj.tp_trl_a = tempArr[6];
        tempObj.sl_a = tempArr[7];
        tempObj.sl_trl_a = tempArr[8];
        tempObj.tm_a = tempArr[9];

        node.send({bundle:"Object", tempObj});

        check[obj.id] = tempObj;
    
  }
}

context.set('recordKeeper', check);
msg = {};
msg.payload = check;
msg.bundle = "Final Message";
return msg;

I hadn't been accounting for you assigning the split of that object into the array. Try this code and see if it works any better. If this works, I'll put in the stuff to start sending it to the table.

interscope101 · 2 December 2022 16:50

Hi, unfortunately still breaking ...

This may be the way to separate/define local storage - you're saying that flow.set is not suitable for this purpose regardless?

contextStorage: {
   default: "memoryOnly",
   memoryOnly: { module: 'memory' },
   file: { module: 'localfilesystem', dir: '/home/user/.node-red/context1' },
   file2: { module: 'localfilesystem', dir: '/home/user/.node-red/context2' },
   file3: { module: 'localfilesystem', dir: '/home/user/.node-red/context3' }
},

madhouse · 2 December 2022 17:31

Try this in that block:

        tempArr = obj.note.split("_");
        node.send(tempArr);
        node.send(tempArr[0]);
        /*tempObj.frames = tempArr[0];
        tempObj.gain = tempArr[1];
        tempObj.gain_st = tempArr[2];
        tempObj.freq = tempArr[3];
        tempObj.analysis = tempArr[4];
        tempObj.tp_a = tempArr[5];
        tempObj.tp_trl_a = tempArr[6];
        tempObj.sl_a = tempArr[7];
        tempObj.sl_trl_a = tempArr[8];
        tempObj.tm_a = tempArr[9];*/

Maybe we should see what's happening to tempArr to see if we can even do what we're trying to do.

No, I'm not saying it's not suitable. I'm saying you don't have the level of control you may want. Though defining multiple files like you did counters what I was thinking... It looks like you're on to something with that... If you have those defined, you may now have the level of control you're looking for, though it gives your variable a much wider scope of access than just your function. In any case, you have the ability to store the file. Give it a try.

You should actually be able to replace your context.set with your flow.set you've setup and accomplish both long term variable storage and file storage at the same time, since it will store a variable just like context.set will. You'll also need to change context.get to flow.get to access it since it will now be a flow variable instead of just a context variable. Not a big deal to change.

Let me know what you get out for tempArr. It should be the same as what you got before. It should also tell us if we can access the element of the array we're looking for. That will dictate what we have to do with that.

interscope101 · 2 December 2022 17:45

This version actually just got it done!! May fault; I made some of the filesystem mods last night, so our functions were a bit out of sync - after adjusting back all the vars are coming in full.

madhouse · 2 December 2022 17:59

Sweet! Everything is being processed correctly! Integrating in your flow.set and flow.get changes should be a quick and easy change. Just work with what you've defined as files and you'll be good.

Now on to sending the data to the table...

This is a copy of the last function code I posted last night. Probably the one you just used. The important part will be down at the bottom.

var check = context.get('recordKeeper');
var keyTemp = context.get('recordKeys');

const input = msg.payload.filter(s => s.status.type == "stop_loss_finished" || s.status.type == "finished");

var tempObj ={};
var tempArr = [];
for(let obj of input){

    if(!(check.hasOwnProperty(obj.id))){
        tempObj = {};
        tempObj.pair = (String(obj.pair)).replace("USDT_", "");
        tempObj.trade_id = obj.id;
        tempObj.profit_usd = Number(obj.profit.usd).toFixed(2);
        tempObj.profit_percent = obj.profit.percent;
        tempObj.position = Number(obj.position.total.value).toFixed(0);
        tempObj.volume = obj.data.current_price.quote_volume;
        tempObj.close_price = obj.data.average_close_price;
        tempObj.closed = obj.data.closed_at;

        if (obj.status.type === "stop_loss_finished"){tempObj.status = "SL";}
            else if (obj.status.type === "finished"){tempObj.status = "TP";}
            else if (obj.status.type === "panic_sold"){tempObj.status = "➦";}
            else {tempObj.status = obj.status.type}

        if (obj.stop_loss.enabled) {
            tempObj.to_sl = ((1 - obj.stop_loss.conditional.price.value / obj.data.average_enter_price) * 100).toFixed(2);
        }

        if (obj.status.type == "stop_loss_finished") {
            tempObj.S_L = (((obj.data.average_enter_price - obj.stop_loss.conditional.price.value) / obj.data.average_enter_price) * 100).toFixed(2);
        }

        if (obj.status.type == "finished") {
            tempObj.T_P = (((obj.take_profit.steps[0].price.value - obj.data.average_enter_price) / obj.data.average_enter_price) * 100).toFixed(2);
        }

        tempArr = obj.note.split("_");
        
        tempObj.frames = tempArr[0];
        tempObj.gain = tempArr[1];
        tempObj.gain_st = tempArr[2];
        tempObj.freq = tempArr[3];
        tempObj.analysis = tempArr[4];
        tempObj.tp_a = tempArr[5];
        tempObj.tp_trl_a = tempArr[6];
        tempObj.sl_a = tempArr[7];
        tempObj.sl_trl_a = tempArr[8];
        tempObj.tm_a = tempArr[9];

        check[obj.id] = tempObj;
        keyTemp.push(obj.id);
    
  }
}

context.set('recordKeeper', check);
context.set('recordKeys', keyTemp);
msg = {};
msg.payload = [];
for(let k of keyTemp){
    msg.payload.push(check[k]);
}
return msg;

The part you'll notice is I've now referenced recordKeys at the top and used it at the bottom. We're recording all the individual transaction IDs in that variable and using that variable to loop through all the different transactions we've been storing to create an array. That array will now go out to the debug (afterwards table) and we should get a list of everything we want!

I've also removed the other outputs so now you should only get the final array showing up on the debug output. Give it a try and let me know what happens.

I haven't put anything in about file storage in this code. But if it works, you just need to change the context stuff to make it work.

interscope101 · 2 December 2022 18:00

Ok, so it seems I was able to separate and save data set in a separate system file called "transact". However oddly I am not able to locate the actual file (what that name) in my system.

madhouse · 2 December 2022 18:06

Did you define the file as "transact" in your setup? Whatever is defined in the system will be where it's saved. If you defined "file1" to go to a file called "transact", that will save it to "transact" whenever you reference "file1".

interscope101 · 2 December 2022 18:10

Ok, the msg output is now a single array

And both recordKeeper and recordKeys are getting saved (looking identical)

To get in sync here is the latest version of the function with all the mods


var check = flow.get('recordKeeper', 'transact');
var keyTemp = flow.get('recordKeys', 'transact');


const input = msg.payload.filter(s => s.status.type == "stop_loss_finished" || s.status.type == "finished"  || s.status.type == "panic_sold");   

var tempObj ={};
var tempArr = [];
for(let obj of input){

    if(!(check.hasOwnProperty(obj.id))){
        tempObj = {};
        tempObj.pair = (String(obj.pair)).replace("USDT_", "");
        tempObj.trade_id = obj.id;
        tempObj.profit_usd = Number(obj.profit.usd).toFixed(2);
        tempObj.profit_percent = obj.profit.percent;
        tempObj.position = Number(obj.position.total.value).toFixed(0);
        tempObj.volume = obj.data.current_price.quote_volume;
        tempObj.close_price = obj.data.average_close_price;
        tempObj.closed = obj.data.closed_at;

        if (obj.status.type === "stop_loss_finished"){tempObj.status = "SL";}
            else if (obj.status.type === "finished"){tempObj.status = "TP";}
            else if (obj.status.type === "panic_sold"){tempObj.status = "EX";}
            else {tempObj.status = obj.status.type}

        if (obj.stop_loss.enabled) {
            tempObj.to_sl = ((1 - obj.stop_loss.conditional.price.value / obj.data.average_enter_price) * 100).toFixed(2);
        }

        if (obj.status.type == "stop_loss_finished") {
            tempObj.S_L = (((obj.data.average_enter_price - obj.stop_loss.conditional.price.value) / obj.data.average_enter_price) * 100).toFixed(2);
        }

        if (obj.status.type == "finished") {
            tempObj.T_P = (((obj.take_profit.steps[0].price.value - obj.data.average_enter_price) / obj.data.average_enter_price) * 100).toFixed(2);
        }

        tempArr = obj.note.split("_");
        
        tempObj.frames = tempArr[0];
        tempObj.gain = tempArr[1];
        tempObj.gain_st = tempArr[2];
        tempObj.freq = tempArr[3];
        tempObj.analysis = tempArr[4];
        tempObj.tp_a = tempArr[5];
        tempObj.tp_trl_a = tempArr[6];
        tempObj.sl_a = tempArr[7];
        tempObj.sl_trl_a = tempArr[8];
        tempObj.tm_a = tempArr[9];

        check[obj.id] = tempObj;
        keyTemp.push(obj.id);
    
  }
}

flow.set('recordKeeper', check, 'transact');
flow.set('recordKeys', check, 'transact');

msg = {};
msg.payload = [];
for(let k of keyTemp){
    msg.payload.push(check[k]);
}
return msg;

interscope101 · 2 December 2022 18:15

Yes, defined, and all seeming looks/works correct in the app. The mystery is that there is no "transact" file in the specified directory, nor anywhere else in the drive (I searched) - which gives some concern that the files are not in fact separated.

madhouse · 2 December 2022 19:45

Success! Now you have something to push to your table! That should be all the processing required for the array to put into your table. Now onto storage...

This is a problem, but an easy one to solve. You have this in your code:

flow.set('recordKeeper', check, 'transact');
flow.set('recordKeys', check, 'transact');

The problem is you're assigning "check" to both variables instead of just to the first. It should look like this:

flow.set('recordKeeper', check, 'transact');
flow.set('recordKeys', keyTemp, 'transact');

I'm not sure what will happen since it's using the same file for storage, but I guess we'll find out once it's tried.

Linux can be quirky when using files, and Node-Red just adds to the quirkiness. While you may be defining a file by its absolute path from root, Node-Red may just be taking that path and adding it to its default path. Try taking out the paths and just leave the file name. Then look under ~/.node-red and see if you can find it in one of the directories under that. This is new territory for me as well, so I can't offer much other than suggestions.

Let's see what happens with the change in your flow.set line. That might be the last piece you need for your function. Let me know what happens.

interscope101 · 2 December 2022 19:50

So the table is getting populated - is that data already coming from transact stored file ?

interscope101 · 2 December 2022 19:53

Looking good, except the "profit.percent" (%) data seems to be missing ...

madhouse · 2 December 2022 19:54

That will come in your table code. What's the current definition for that column?

interscope101 · 2 December 2022 20:07

Fixed it! It seems there was a small change in the variable name.

So, could you please clarify the difference between recordKeeper and recordKeys data sets

madhouse · 2 December 2022 20:52

Yep.
recordKeeper: This variable stores all the individual entries you're wanting to display in your table. Each entry is assigned a property name that is the same as the transaction ID. The reason it's done this way is it makes it possible to reference everything based off that unique ID.

recordKeys: This is more of a convenience variable than anything. recordKeeper is doing all the work with storing the data. recordKeys is simply so we have an array ready to use when going through all the data in recordKeeper. The same values could probably be obtained using Object.keys(recordKeeper), but I wasn't sure if it would only give the unique keys at the main level or if it would try to give every key in existence in the object. Feel free to experiment on that if you want. You can just to this with a debug:

node.sent(Object.keys(recordKeeper));

And see what comes out on your debug. If all you get is the unique transaction IDs as the keys, then you don't actually need recordKeys as a variable anymore and we can take all that stuff out. It's not something I've had to work with before though so that's why I put in the convenience variable.

But that's good you got your table working! Let me know if there's anything else you need help with.

Also, since this thread is turning out to be really long, feel free to DM me and we can take care of stuff that way as well. Just click my name badge and hit message (gotta put that in there cause some people don't know how).

interscope101 · 2 December 2022 21:05

Can't thank you enough for your help, time, and patience!

madhouse · 2 December 2022 21:07

Certainly welcome! Even learned a few new things myself. Thanks!

system · 16 December 2022 21:08

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Sort ui_table data by a datetime programatically at the start Dashboard	3	794	28 October 2020
Help to sort by a column the data entered in node ui_tablet General	3	1151	4 February 2022
Simple ui_table to sort "desc" on load Dashboard	5	154	1 May 2024
Ui-table date sort FAQs	6	2497	29 February 2020
UI_TABLE : How to add row at the end of the table Dashboard	4	2492	15 July 2020

UI Tabular Table Sorting

Related topics