Hello All,
Ive been pulling some big data from an API and noticed that im only getting max 50 records
is there a way around this or another xml node ?
I dont know of any built in limitation. Can you show us the data before the XML node?
Failing that, you could always DIY it by adding any XML parsing lib from NPM into a function node.
Hi Steve.. Thanks for replying
Bit hard to show you all the data as its confidential data.. however it is showing 897217 bytes of data coming from the http node and in the state for the xml its showing 1480 records however it only returns 50?
I dont need to see the actual data. Just that there are as many row as you suggest.
Try capturing the data in a file (feed the result of the HTTP Request to a file node), then open that file and verify how many jobs there are in the source. It may turn out there are only 50 because the API pages the data (meaning you need to either make subsequent requests with the next page ID or add a parameter to set the page size larger!)
Once you have made a capture to file, you can try it out in an online XML to JSON convertor (like this one)
EDIT: Actually, its right there in your screen shot. The data is paged. You will need to read the API docs to request more.
Thank you .. wasnt sure if that was related.
I will go to the vendor and ask 
or try the node-red-contrib-fastxml node
(I am the author so would be interested if that fails also)
Yes - maybe it is a "paging" api and only sends that many in a chunk
I really wish the node-red core XML looked so clean.
in what way ? - so clean as what ?
Check the actual text you are getting from the API and confirm that there is only the first page there. Output it to a file and you should easily be able to check by searching for, for example, current_page.
Maybe you can ask the API to send 1480, all, or (say) 99999 entries ? Probably a parameter to can add to the query to set it.
Ah yes - indeed - I did try to make it so it could replace core - but the fastxml lib can't quite be a 100% match so it would break some users so I had to release it separately :-(... As it really is just in the margins of attribute handling it was mostly OK - but... yeah 
if we could swap I would
You can set the xml output options to not use arrays if it is a single object (for example), there is a long list with options available
2 Likes
Just an update.. YES it was the API that's limiting
@dceejay I have change to your XML node... can notice a difference.
1 Like
The issue is that if you then do have any attributes that you can't easily go from XML -> JSON and then back to XML without losing anything. OK - not maybe not a problem for most but is useful if you need to - and was the default behaviour chosen.
1 Like
