Hi...im trying to do some scraping and get data on what all devices are connected in my local network and check for internet status.
Firstly i need to login into my router page (I know username and password but can't get it login using POST request method in node red ) ..i tried few but sadly none of those works ...can someone help me to get this authentication done?
My local router webpage is 192.168.0.1
Looks like this
A lot of routers have the login user id (usually "admin") and the password printed on a label on the bottom or back.
Some routers force you to change the password when you first login. If you did that but can't remember it, you probably have to reset the router using a straightened out paperclip.
I can't quite see the connection to Node-red ?
@jbudd I want to do web scraping.. And I know the username and password
For example
Username: admin
Password: admin
I want to do POST request to login into page(192.168.0.1) from node red and get the cookie value and use the cookie to do some web scrapping from router page
Ah, you want to access the router admin from within Node-red.
Have you checked if it supports SNMP? There is a node-red-node-snmp
I will check but can't we just do with help of funtion node and request node?
I am afraid I don't know, I don't really speak http.
No doubt others can help.
You could try to reverse engineer the request it makes;
On a desktop browser:
Go to the login page.
Right click > inspect element.
Go to the network tab
Now login.
You will see all kinds of things appearing. The page it redirected to should be there as well.
Click once on that on that particular link and you should see some details about the headers that were send in the POST request.
example (google produces a major mess)
Yes I get this.. But my router webpage timeouts every 30 mins so I need new cookies every 30 mins.. I need help on how to do POST request with username and password
What part "do you get" ? The post request (apparently) produces cookies, so you can store the cookies and everytime you interact with the page you use the cookie as auth method. There is a ResponseCookies key in the request node response.
If you just want to be able to post, set the headers, example in the documentation.
Problem is its not logging at all... I tried basic and digest authentication and still get response with stateuscode 500
I have never done this, but I think you need to figure out what http requests are made when logging on to the site. You can do that with your browser developer tools.
You can then mimic those http (post) requests in node-red.
... but I am fearing a bit that those requests are encrypted/hashed somehow so that just replaying them might not work.
what model is your router, and what exactly is it your trying to scrape or obtain from the HTML?
I ask, as does the router support SSH or telnet at all?
EDIT:
I Read the full description 
SSH/Telnet might be a little overkill.
So logging in is “no problem at all”, except that you get errors, so you need to determine what headers are expected.
@janvda most of what the browser can do, node-red can do.
thanks for responding @bakman2 ..when i login and this shows up in developer tools ...not sure how to get this same replicated in nodered...
i tried to capture request headers when login with username and password as per the trace
GET /?_type=loginData&*tag=login_token&*=1666200358037 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: _TESTCOOKIESUPPORT=1; SID=6bcbba175bedd4a72d012eeee0e94dc431bd37381470a6199e70a34688b69c65
Host: 192.168.0.1
Referer: [http://192.168.0.1/](http://192.168.0.1/)
User-Agent: Mozilla/5.0 (iPad; CPU OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/87.0.4280.77 Mobile/15E148 Safari/604.1
X-Requested-With: XMLHttpRequest
POST /?_type=loginData&_tag=login_entry HTTP/1.1
Accept: application/json, text/javascript, */*; q=0.01
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 140
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Cookie: _TESTCOOKIESUPPORT=1; SID=6bcbba175bedd4a72d012eeee0e94dc431bd37381470a6199e70a34688b69c65
Host: 192.168.0.1
Origin: [http://192.168.0.1](http://192.168.0.1/)
Referer: [http://192.168.0.1/](http://192.168.0.1/)
User-Agent: Mozilla/5.0 (iPad; CPU OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/87.0.4280.77 Mobile/15E148 Safari/604.1
X-Requested-With: XMLHttpRequest
GET / HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: _TESTCOOKIESUPPORT=1; SID=cb64d7d13ba943112f93a47f2f4a76560b333e9538c64d7b28076858b8fe5b08
Host: 192.168.0.1
Referer: [http://192.168.0.1/](http://192.168.0.1/)
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (iPad; CPU OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/87.0.4280.77 Mobile/15E148 Safari/604.1```
Problem is its exchanging username/password for a token i guess...hoping for some help to get this work
I think the username/password is not visible in the request header output you have shared but it is encoded in the payload (or body) of the POST request.
E.g. when I want to logon to my modem I found following javascript logic
so it makes use of SRP (= Secure Remote Password Protocol ) which javascript is also loaded by the web page as you can see:
So if I want to create a node-red application that automatically logs in into my modem then it should also implement the same secure remote password protocol. It is not impossible but far from easy.
@janvda that is a better approach to secure ssl-less pages.
Looking at op's output, they appear to use a similar approach with the jsencrypt library, the only way around will be by using something nbrowser or puppeteer (that actually launches a headless browser in the background).
Interesting, I wasn't aware there existed an nbrowser node-red node. I need to remember that. 
Copyright OpenJS Foundation and Node-RED contributors. All rights reserved. The OpenJS Foundation has registered trademarks and uses trademarks. For a list of trademarks of the OpenJS Foundation, please see our Trademark Policy and Trademark List. Trademarks and logos not indicated on the list of OpenJS Foundation trademarks are trademarks™ or registered® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.
The OpenJS Foundation | Terms of Use | Privacy Policy | OpenJS Foundation Bylaws | Trademark Policy | Trademark List | Cookie Policy
