HTTP GET request help needed

[grant1]

Over a year ago, I was trying to find a way to acquire values from a compressor that is connected to our LAN. The device (at 192.168.8.134) supplies a simple web page that displays live operating data. I am interested in the Pressure and Temperature, both of which are displayed in the Analog Inputs section.

I am revisiting this topic since I never resolved it, but I think I am getting close. I looked at the Network tab in the dev tools and believe the Request URL shown is what I need to get the data values for the Analog Inputs.

However, when I enter http://192.168.8.134/js/AnalogInputs.js?_=1638627336885 into a tab of the web browser, it downloads a file called AnalogInputs.js that simply contains the code to retrieve the value.

Does it appear that there a way to get the actual temperature of pressure value via some request method?

[hardillb]

I think you are going to need to look lower down the network request tab for where it actually pulls the data

That request is returning a Javascript file that is being executed in the browser and it possibly doing some conversion of what is being returned from the server to what is actually displayed.

[grant1]

Thank you. There is another function called CalculatedAnalogInputs that shows this Javascript file:

and then the table.js file shows this:

which seems to be where the value is actually inserted:

Any pointers would be much appreciated. If it appears that I am chasing a ghost (meaning data values retrieved via JS will never be retrievable using the HTTP GET), then let me know and I will move on to Puppeteer or something like that.

[shrickus]

On lines 237-238, the Javascript is using Jquery to lookup UI elements by id (the $('#' ... ) bits), and then setting the values you are looking for using the .html(...) function -- so the values must be coming from the function inside there, namely language.get("TABLEVALUE",1)

If you put a breakpoint in your JS source at the start of that switch statement on line 232, then you might be able to see more about the language object and trace the data back to the actual ajax request that retrieved it. I'd look closer at that "English.txt" call 3 lines down from the one you have highlighted (since the language you have chosen is English). More likely, that is some UI text strings, but SOMEWHERE in this stack there is data being returned in text format, so it can be shown inside the table... good luck!

[grant1]

Thank you @shrickus.

I did open up the Englist.txt and while it contains hundreds of lines, I was able to find the two UI text strings that I want:

MPL_509$$Compressor Outlet
MPL_1008$$Element Outlet

I am well in over my head on this (have no idea how to do breakpoints and such), but based on what you have seen thus far, does it appear that there will be a way to request this info via HTML?

[shrickus]

I see at the bottom of the init.js script, there is a call to the refresh_data() function... inside there should be the logic for pulling the values and updating the DOM elements.

For me, I would keep digging around -- in the end you just need the raw data, not all of this ui layout + js logic to wade through. They might have obfuscated the data enough to make it hard to find in its raw state. In that case, you may search for documentation online just to see if there are headers you can use to retrieve a json or xml version of the data, instead of the full ui html/js pages.

[grant1]

Looking into the refresh_data() function, and exploring the ANALOGINPUTS section, I see a function location of VM96:56...

and then drilling down a bit further, I see a value of 1E540080 that may be hex or something? (sorry, but this becomes like Chinese at this point)

The value I am expecting should be around 7 or 8.

and drilling a bit further, I was hoping to find something more, but no dice:

Following VM96:56, I find this:

which I presume is where the value is actually retrieved? Is there a way for me to tie this function together with the two UI text strings that I had earlier, i.e. MPL_509$$Compressor Outlet and MPL_1008$$Element Outlet?

Re: documentation, I did search and could not find anything, and also quizzed the technician who works on this, and he was clueless. Said that "all that computer stuff" is info that he does not know how to get.

[shrickus]

And the plot thickens... I can see where the raw DATA value is highlighted, but I can only guess what is implemented within the vDATA.Int16(1) function call. The function signature has one arg called "word" which might be an offset?

I wonder if focusing on those two UI text strings will be less useful than simply returning the AnalogInput array data, using the same functions like Int16(word) to extract the values within node-red.

[grant1]

So as I have spent time looking at all the js functions and what they retrieve, I came to realize that while the temperature and pressure (the two values with the UI text strings that were identified) were useful, it would indeed be better (and likely easier) to get the whole array of data into Node-RED and then sort it out there.

What is the general approach for returning an array that can only be executed via a script on a machine like this, and get the output to display in Node-RED?

[Steve-Mcl]

Can you look through the code and try to find the Questions class? I suspect you will find that within its logic is an async call to refresh data (possibly from an endpoint)

Essentially, if you find out how Questions gets its data, you can maybe replicate that (avoiding the need to run scripts at all)

[grant1]

Below is the text from Questions. Does this line in yellow indicate what is returned to the browser?

function Questions()
{
    //PRIVATE
    var vQUESTIONS=new Array();
    function Post(URL,QUESTION)
    {
		$('#error').hide();
		
		var return_data = null;
		$.ajax({
			url: URL, 
			type: 'POST',
			data: 'QUESTION=' + QUESTION,
			success: function(data, textStatus, jqXHR) {
				return_data = data;
			},
			error: function(jqXHR, textStatus, errorThrown) {
				show_error('Error occured during data retrieval');
				throw new Error();
			}
		});

		return return_data;
    }
    function HexString(VALUE,LENGTH)
    {
        var v=VALUE.toString(16);
        while(v.length<LENGTH)
            v="0"+v;
        return v;
    }
    
    //PUBLIC
    this.Add=function(INDEX,SUBINDEX)
    {
        var vQuestion=new Question();
        vQuestion.INDEX=INDEX;
        vQuestion.SUBINDEX=SUBINDEX;
        vQUESTIONS[vQUESTIONS.length]=vQuestion;
    }
    this.getData=function(INDEX,SUBINDEX)
    {
        for(var i=0;i<vQUESTIONS.length;i++)
        {
            if((vQUESTIONS[i].INDEX==INDEX)&&(vQUESTIONS[i].SUBINDEX==SUBINDEX)) {
                return vQUESTIONS[i].getData();
			}
        }
    }
    this.SendReceive=function()
    {
		for(var idx=0;idx<vQUESTIONS.length;idx+=1000)
		{
			var vQuestionsSlice;
			if((vQUESTIONS.length-idx)<=1000)
				vQuestionsSlice=vQUESTIONS.slice(idx,vQUESTIONS.length);
			else
				vQuestionsSlice=vQUESTIONS.slice(idx,idx+1000);
			
			var vQuestions="";
			for(var iQ=0;iQ<vQuestionsSlice.length;iQ++)
				vQuestions+=HexString(vQuestionsSlice[iQ].INDEX,4)+HexString(vQuestionsSlice[iQ].SUBINDEX,2);    
			
			var vAnswers=Post("http://"+document.location.hostname+"/cgi-bin/mkv.cgi",vQuestions);//

			for(var iQ=0,iA=0;iQ<vQuestionsSlice.length;iQ++)
			{
				if(vAnswers != null && vAnswers.charAt(iA)!="X")
				{
					vQuestionsSlice[iQ].setData(vAnswers.substring(iA,iA+8));
					iA+=8;
				}
				else
				{
					vQuestionsSlice[iQ].setData("X");
					iA++;
				}
			}
		}
    }
}
function Question() {
    //PRIVATE
    var vDATA=null;
    
    //PUBLIC
    this.INDEX=0;
    this.SUBINDEX=0;
    this.setData = function(DATA) {
        var vData = new Data();
        vData.DATA = DATA;
        vDATA = vData;
    }
    this.getData = function(){return vDATA;}
}
function Data() {
    this.DATA=null;

    this.UInt32=function()  {
        if(this.DATA=="X")
            throw new Error();
        return parseInt(this.DATA,16);
    }
    this.Int32=function() {
        if(this.DATA=="X")
            throw new Error();
        var v=parseInt(this.DATA,16);
        if(v>>>31)
            v=-2147483648+(v&0x7FFFFFFF);
        return v;
    }
    this.UInt16=function(WORD) {
        if(this.DATA=="X")
            throw new Error();
        return parseInt(this.DATA.substring((1-WORD)*4,(2-WORD)*4),16);
    }
    this.Int16=function(WORD) {
        if(this.DATA=="X")
            throw new Error();
        var v=parseInt(this.DATA.substring((1-WORD)*4,(2-WORD)*4),16);
        if(v>>>15)
            v=-32768+(v&0x00007FFF);
        return v;
    }
    this.Byte=function(BYTE) {
        if(this.DATA=="X")
            throw new Error();
        return parseInt(this.DATA.substring((3-BYTE)*2,(4-BYTE)*2),16);
    }
}

[Steve-Mcl]

That URL is where the data actually comes from. Use a http request node to get raw data. Once you have that working, you can examine it's content and begin dissecting it.

[grant1]

OK, so if I inspect the mkv.cgi fetch file, I see a question that is a long string of numbers and letters:

fetch("http://192.168.8.134/cgi-bin/mkv.cgi", {
  "headers": {
    "accept": "*/*",
    "accept-language": "en-US,en;q=0.9",
    "content-type": "application/x-www-form-urlencoded",
    "x-requested-with": "XMLHttpRequest"
  },
  "referrer": "http://192.168.8.134/",
  "referrerPolicy": "strict-origin-when-cross-origin",
  "body": "QUESTION=30020130020330030130030230030a300701300702300703300704300707302105300501300502300503300504300505300507300508300e03300e2a31130131130331130431130531130731130831130931130a31130b31130c31130d31130e31130f31131031131131131231131331131431131531131631131731131831131931131a31131b31131c31131d31131e31131f31132031132131132231132331132431132531132631132731132831132931132a31140131140231140331140431140531140631140731140831140931140a31140b31140c31140d31140e31140f31141031141131141230090130090630091130090730091230090930090a300108",
  "method": "POST",
  "mode": "cors",
  "credentials": "include"
});

Then if I do something like this:

where the function node contains the question:

and the HTTP Request node contains the POST & URL as shown:

then I get a response that looks as long and convoluted as the Question:

I am guessing that somewhere in that response (answer) is the value that I am looking for?

[Steve-Mcl]

It looks like a buffer. You can use node-red-contrib-buffer-parser to convert the data into meaningful values.

How you do that (and what values are what) can be determined from the code in the Questions class and other areas where a value is taken and assigned to a UI element.

Alternatively, you could copy the necessary logic from the web pages and put it together in a function node to "re-use" it.

Both of these solutions will completely negate any need for the likes of puppeteer.

[grant1]

Thank you. I think I got it:

The settings which I used for the Buffer Parser are as follows:

[dceejay]

Love it when a plan comes together...
nice work