SMA EV Charger get data using curl - how to do this in Node-RED #2?

@Steve-Mcl
Hi Stephen,
you helped me already a lot to solve this topic:
SMA EV Charger get data using curl - how to this in Node-RED?

But as I have already announced, I need some more help to read out more data/parameter from my EV Charger using Node-RED.

As before I have an example using Curl, which is already working in my home network using FHEM (fhem.de), a GPL'd perl server for house automation. But I want to implement it using Node-RED.

I tried to "translate" this new curl command on the basis of your solution from the previous topic. But it does not work. Here is a brief description of the result:

Request:

22.4.2022, 14:34:00node: f142feba42f76d5b
msg : Object
object
payload: "[{"queryItems":[{"componentId":"IGULD:SELF"}]}]"
topic: ""
timestamp: 1650630845862
_msgid: "80ba3f2c19e44baf"
method: "post"
url: "https://192.168.178.62/api/v1/parameters/search/"
headers: object
Host: "192.168.178.62"
Connection: "close"
Accept: "application/json, text/plain, */*"
User-Agent: "okhttp/3.10.0"
Sec-Fetch-Site: "same-origin"
Sec-Fetch-Mode: "cors"
Sec-Fetch-Dest: "empty"
Accept-Encoding: "gzip, deflate"
Accept-Language: "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"
Content-Type: "application/json"
Content-Length: "45"
Referer: "https://192.168.178.62/webui/login"
Authorization: "Bearer eyJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE2NTA2MzA3ODYsInN1YiI6IldCQWRtaW4iLCJ1aWQiOiIzY2RkYTc1NS04N2E5LTQ3NzItODBmOC0zODY2NzU4ZTc0YzIiLCJleHAiOjE2NTA2MzQzODZ9.f4tMLbYAYTpMsh03C4G0tKMV9f0mbDFV136MlKfDYF8"
cookies: object
empty
rejectUnauthorized: false

Response:

22.4.2022, 14:34:02node: 0faa33d44135350f
msg : Object
object
payload: ""
topic: ""
timestamp: 1650630845862
_msgid: "80ba3f2c19e44baf"
method: "post"
url: "https://192.168.178.62/api/v1/parameters/search/"
headers: object
connection: "close"
date: "Fri, 22 Apr 2022 12:34:08 GMT"
cache-control: "no-cache,no-store,must-revalidate"
x-node-red-request-node: "141e2fa8"
cookies: object
empty
rejectUnauthorized: false
statusCode: 500
responseUrl: "https://192.168.178.62/api/v1/parameters/search/"
redirectList: array[0]
retry: 0

And here is a more detailed description of the topic including the original Curl command and some other useful information:
https://docs.google.com/spreadsheets/d/1ruGhnYTSvHoEGB1Z1RQCWBWFV5DaOKaz/edit?usp=sharing&ouid=105804670964876172397&rtpof=true&sd=true

I hope you can help me again!
If there is a need to support you in any way to solve this issue again, please let me know.

Thank you for any response in advance and
kind regards
Thomas

Strikes me that most of those headers aren't needed. But the big difference I can see is that the CURL is sending the data as binary but I'm not sure that the request node will do that? (though maybe the content type header triggers that anyway?)

@TotallyInformation
Hi Julian,

thank you for your response!

First of all: I am no SW developer and I have absolutely no clue about Curl and Perl and so on.
I can do a little programming using visual basic, that's all!
May be this is the reason why I do not get in detail what you mean?

Below you can find the two Curl commands, but I can not see what you mean with "the big difference I can see is that the CURL is sending the data as binary".

Can you help me in more detail please?

On top: The old Curl command which was successfully implemented using the http request node with the help from @Steve-Mcl.
On bottom: The new Curl command, which I also want to implement in Node-RED using the http request node.

I can

curl -i -s -k -X 'POST' 
-H 'Host: 192.168.178.62' 
-H 'Connection: close' 
-H 'Accept: application/json, text/plain, */*' 
-H 'User-Agent: okhttp/3.10.0' 
-H 'Sec-Fetch-Site: same-origin' 
-H 'Sec-Fetch-Mode: cors' 
-H 'Sec-Fetch-Dest: empty' 
-H 'Accept-Encoding: gzip, deflate' 
-H 'Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7' 
-H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8' 
-H 'Content-Length: 30' 
-H 'Referer: https://192.168.178.62/webui/login' 
-H 'Authorization: Bearer ey...Ws' -b 'node0neukabkiaw6z1cnfbfy1fhohb37.node0' --data-binary '[{"componentId":"IGULD:SELF"}]' https://192.168.178.62/api/v1/measurements/live/

curl -i -s -k -X 'POST' 
-H 'Host: 192.168.178.62' 
-H 'Connection: close' 
-H 'Accept: application/json, text/plain, */*' 
-H 'User-Agent: okhttp/3.10.0' 
-H 'Sec-Fetch-Site: same-origin' 
-H 'Sec-Fetch-Mode: cors' 
-H 'Sec-Fetch-Dest: empty' 
-H 'Accept-Encoding: gzip, deflate' 
-H 'Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7' 
-H 'Content-Type: application/json' 
-H 'Content-Length: 45' 
-H 'Referer: https://192.168.178.62/webui/Plant:1,IGULD:SELF/configuration/view-parameters' 
-H 'Authorization: Bearer ey...Ws' -b 'node0neukabkiaw6z1cnfbfy1fhohb37.node0' --data-binary '{"queryItems":[{"componentId":"IGULD:SELF"}]}' https://192.168.178.62/api/v1/parameters/search/

Not sure I can help I'm afraid. Feel free to send me an electric vehicle and a charger though I can say with certainty that I will get it working for you :rofl:

I did find the API specs: SMA APIs | SMA Developer Portal but I'm not sure that helps.

@TotallyInformation
Is already on the way :ship:

But seriously: Isn't it easy for a software expert to do reverse engineering using the developer tools of a browser to implement the correct http request (see screenshot below)?

But seriously 2: Is it nowadays really important, that the computer is directly connected to the DUT?
In addition: The parameters can be read without connecting the charger to vehicle.
:rofl:

Well, how can I be helped?

No. Often not. The interactions can be complex and it isn't always clear what headers and security are needed. Maybe for someone more expert than me. But my biggest one was the Drayton Wiser heating system and that was mostly already done by someone else in Python and I was able to translate it.

DUT?

The site that I referenced has all of the API schema's defined. The main thing is to get the connection.

Generally, if you have the authorisation figured then you should be good to go and the rest is just brute-forcing your way through the various API's.

DUT… Device Under Test ?

Ahh, jargon! :rofl:

Not necessarily. But that depends on the device surely. If the device has a network interface, it can be anywhere accessible on your local network. Node-RED has excellent communications capabilities.

DUT: Device Under Test
But may be this is not the correct abbreviation in this case...

it isn't always clear what headers and security are needed.

But headers can be seen in the screenshot of the browser tool. Or not?
And security is already solved I think, because login and token is already working.

The only difference is now to read data from here:

instead from here, which is the same device and working stable:

Using Perl and Curl it is done like this:

Function "login", "refresh_token", "livedata" is already working using http request node.
And now I want to implement "read_params".
But changing only the url in the http request node does not work and leads to status code 500 (as you can see on top of this thread).
Any ideas?

# get all basic infos for the curl command
	my $baseurl = $hash->{HELPER}{BASEURL};
	my $curlcmd = "";
	my $url = "";
	my $special_header = "";
	my $cmd_call = "curl -i -s -k -X ";
	my $method = "'POST' ";
	my $header = "-H 'Host: $hash->{HOST}' -H 'Connection: close' -H 'Accept: application/json, text/plain, */*' -H 'User-Agent: okhttp/3.10.0' -H 'Sec-Fetch-Site: same-origin' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Dest: empty' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7' ";
	my $cookies = "-b '$hash->{HELPER}{SESSIONID}' ";
	my $token = "-H 'Authorization: Bearer $hash->{HELPER}{ACCESS_TOKEN}' " if (defined($hash->{HELPER}{ACCESS_TOKEN}));
	my $len_corr = 0; # maybe there are chars in data which should not be count for header content_len

	Log3 $hash->{NAME}, 5, "$hash->{NAME} -> SMAEVCgarger_getCurlcmd";
	
	#correct string len with count of masking char \
	if (defined ($data))
	{
		$len_corr = () = $data =~ /\\/g;
	}
	
	# check wich command we have to build
	if ($api eq "login")
	{
		$url = $baseurl."/api/v1/token";
		my $content_len = length($data) - $len_corr;
		$data = "--data-binary '$data' ";
		$special_header = "-H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8' -H 'Content-Length: $content_len' -H 'Origin: $baseurl' -H 'Referer: $baseurl/webui/login' ";
		$cookies = "";
		$token = "";
	}
	elsif ($api eq "refresh_token")
	{
		my $refresh_token = $hash->{HELPER}{REFRESH_TOKEN};
		$url = $baseurl."/api/v1/token";
		$data = "grant_type=refresh_token&refresh_token=$refresh_token ";
		my $content_len = length($data);
		$data = " --data-binary '$data' ";
		$special_header = "-H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8' -H 'Content-Length: $content_len' -H 'Origin: $baseurl' -H 'Referer: $baseurl/webui/login' ";
	}
	elsif ($api eq "livedata")
	{
		$url = $baseurl.'/api/v1/measurements/live/';
		$data = '[{"componentId":"IGULD:SELF"}]'; # cmd to get live data from wallbox
		my $content_len = length($data);
		$data = "--data-binary '$data' ";
		$special_header = "-H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8' -H 'Content-Length: $content_len' -H 'Referer: $baseurl/webui/login' ";
	}
	elsif ($api eq "read_params")
	{
		$url = $baseurl.'/api/v1/parameters/search/';
		my $content_len = length($data) - $len_corr;
		$data = "--data-binary '$data' ";
		$special_header = "-H 'Content-Type: application/json' -H 'Content-Length: $content_len' -H 'Referer: $baseurl/webui/Plant:1,IGULD:SELF/configuration/view-parameters' ";
	}

Not really, as I said before, I don't think most of those are needed and some may be more problematic than helpful.

But what I don't know, because I don't own such a device, is whether the token and session cookie is permanent or temporary. You also can't tell from looking whether those are tied to a source device (e.g. IP or some other test). They might always work or might only work today for only for an hour.

OK, from your next part, it looks like you might have that sorted anyway. And you can see from the PERL code that there is indeed a refresh_token function so you might need to find out how often that needs to be called.

Not sure if the request node will calculate the content length header for you if you set the 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8' header, you would need to check and if not, you will need to calculate it yourself.

@TotallyInformation Thank you very much for your answers!!! Everything is helpful for me!

Mmmh....

I don't think this is the problem. I already know that the token has to be refreshed every hour.
And I think the token is valid for the entire website of the EV Charger and not only for a specific webpage of the device. But that is only a feeling and not a concrete knowledge.

The Content-Lenght I have already adapted based on the information from the screenshot of the browser developer tool..

In the meantime I have raised my problem also to the author of the PERL Script. But beware, it's German!
:wink:

https://forum.fhem.de/index.php/topic,116543.msg1219311.html#msg1219311