Get HTTP Request to ignore the first search bar?

General
1

Hi everyone! I've searched all over the forum here but I have not found exactly what I'm looking for. I'm hoping you all might be able to shed some light on this. Basically I'm setting up an HTTP reuqest node to send a URL to archive.org to have the page archived. I've been getting weird responses from the page and I just realized that the HTTP Request may be getting sent to the search bar in the top right of the page and not the Save Page Now bar in the center of the screen.

Is there a way to specify that my node ignores the first search bar and instead uses the second?
Thank you all for your thoughts on this. I'm learning a ton and I'm loving Node-Red!
Rob.

image
image1339×618 30.6 KB

2

Hi @Guydelombard

the HTTP Request node sends a request to a URL.

It sounds like you are sending it to the url used by the first search box. So instead, you need to identify what url is used by the second box.

One way of doing that will be to open your browser developer tools, and look at the 'network' tab. Enter something into the search box and submit it. Watch what new entry appears in the developer tools - that should help you identify what the url is for your request node.

3

Thank you so much! I'll do exactly that right now.Fingers crossed.

Thanks

4

Thankfully, the form isn't a "smart" JavaScript one.

:grinning:

Spoiler - only open if you are stuck 
<form class="form search-form js-search-form" id="searchform" method="get" role="search" action="https://archive.org/searchresults.php" data-event-form-tracking="Home|SearchForm" data-wayback-machine-search-url="https://web.archive.org/web/*/">
5

OK. Thank you both for your help with this. I'm still rolling along. I may have to look for an alternative method. I'm not seeing a URL listed for the save page now bar. And @TotallyInformation I truly appreciate the hidden spoiler. I got desperate after 4 hours and checked it. It's for the search bar on the top of the page. However it gave me clues to look for the save page url which does not seem to be listed, I did find where that information should be, it's just not there. Possibly because archive.org sells a service similar to what I'm trying to build for free. haha. Thank you both so much for your help. I love technical communities for just this reason. I'll keep working it.

6

Perhaps I wasn't on the same page as you. I went to the link you gave - that has 2 search bars, the form I selected was the one in the middle of the page.

image
image1320×708 77.6 KB

THe highlighted line on the right is the div that contains the search input. Working backwards you get to the form tag that I shared.

Let me know if that was the wrong page.

OK, I must have been tired last night so I see that you were actually trying to save a page not do a search.

Can you share a link to the actual page please?

7

Oh dear lord. I see exactly what you mean, that's my fault. I'm using web.archive.org/save which uses the input box in the middle of the page to get the system to archive a page. The input box in the top right corner is exactly the same as the box on their front page at archive dot org.

I'm sorry for the confusion. I've looked through both and while the urls you listed show fine for the search box. I'm not seeing them for the save box on web.archive.org/save

Thanks!
Rob

8

Hmm, well it is still using a standard form control:

<form class="web-save-form" id="web-save-form" name="wwmform_save" action="/save" method="POST" _lpchecked="1">
                <input type="text" class="form-control web-save-url-input web_input web_text" id="web-save-url-input" name="url" placeholder="https://" autofocus="">
                                <div>
                    <input type="checkbox" name="capture_outlinks" id="capture_outlinks">
                    <label for="capture_outlinks">Save outlinks</label>
                </div>
                                                <div>
                    <input type="checkbox" name="capture_all" id="capture_all" checked="">
                    <label for="capture_all">Save error pages (HTTP Status=4xx, 5xx)</label>
                </div>
                                <div>
                    <input type="checkbox" name="capture_screenshot" id="capture_screenshot">
                    <label for="capture_screenshot">Save screen shot</label>
                </div>
                <div id="wm-save-option" style="display: none">
                    <input type="checkbox" name="wm-save-mywebarchive" id="wm-save-mywebarchive">
                    <label for="wm-save-mywebarchive">Save also in my web archive</label>
                </div>
                                <input type="submit" class="web-save-button web_button web_text" value="SAVE PAGE" style="margin: 16px 0;">
            </form>

So it appears to be using a POST to http://web.archive.org/save and the other fields will be encoded into the body of the POST response.

There are tools that will let you do test GET/POST submissions and you could try that to see if it works though it isn't clear without trawling through a lot of code whether the page is doing anything clever with JavaScript but it doesn't look like it to me.

So in theory, you should be able to use Node-RED with an http-request node set to POST and passing the parameters on the payload. Here is an example just using Node-RED:

image
image728×203 10.5 KB 

[{"id":"c93cb35b.9df52","type":"http request","z":"5cf69d1b.d88c14","name":"","method":"POST","ret":"txt","paytoqs":true,"url":"http://127.0.0.1:1880/nr/testpost","tls":"","persist":false,"proxy":"","authType":"","x":710,"y":960,"wires":[["3f52e332.e2e23c"]]},{"id":"f193ee3d.cb73c","type":"http in","z":"5cf69d1b.d88c14","name":"","url":"/testpost","method":"post","upload":false,"swaggerDoc":"","x":340,"y":1040,"wires":[["f2e9ec2c.c6bfd","34707403.93638c"]]},{"id":"f2e9ec2c.c6bfd","type":"http response","z":"5cf69d1b.d88c14","name":"","statusCode":"","headers":{},"x":800,"y":1040,"wires":[]},{"id":"34707403.93638c","type":"debug","z":"5cf69d1b.d88c14","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":770,"y":1100,"wires":[]},{"id":"bcc33041.382ea","type":"inject","z":"5cf69d1b.d88c14","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":340,"y":960,"wires":[["3a9686e5.13200a"]]},{"id":"3a9686e5.13200a","type":"change","z":"5cf69d1b.d88c14","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"{\"url\":\"http://example.com\",\"capture_outlinks\":false,\"capture_all\":true,\"capture_screenshot\":false,\"wm-save-mywebarchive\":false}","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":960,"wires":[["c93cb35b.9df52"]]},{"id":"3f52e332.e2e23c","type":"debug","z":"5cf69d1b.d88c14","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":870,"y":960,"wires":[]}]
9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.