Need help with curl

bigmac5753 · 30 December 2020 12:32

I have this curl command that I need to somehow run in node-red:

curl -X POST --data-urlencode "url={{url}}" http://{{host}}:8080/youtube-dl/q

I know it's best to use http rather than exec but can't figure out how to do it.

If it helps, i'm also able to run this fetch command instead:

fetch(`http://${host}:8080/youtube-dl/q`, {
  method: "POST",
  body: new URLSearchParams({
    url: url,
    format: "bestvideo"
  }),
});

Thanks for reading.

E1cid · 30 December 2020 12:57

Not sure where host and url are set so had to guess

example http request

[{"id":"63f3e144.1f0478","type":"inject","z":"8d22ae29.7df6d","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{}","payloadType":"json","x":110,"y":2480,"wires":[["dbf18da0.3fef08"]]},{"id":"a808fae1.564ae","type":"http request","z":"8d22ae29.7df6d","name":"","method":"POST","ret":"txt","paytoqs":"body","url":"http://{{host}}:8080/youtube-dl/q","tls":"","persist":false,"proxy":"","authType":"","x":410,"y":2440,"wires":[["d82a6daf.bc593"]]},{"id":"d82a6daf.bc593","type":"debug","z":"8d22ae29.7df6d","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":600,"y":2460,"wires":[]},{"id":"dbf18da0.3fef08","type":"change","z":"8d22ae29.7df6d","name":"","rules":[{"t":"set","p":"payload.url","pt":"msg","to":"http://youtube.com/h455ggghhss","tot":"str"},{"t":"set","p":"host","pt":"msg","to":"youtube.com","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":210,"y":2440,"wires":[["a808fae1.564ae"]]}]

the payload body has to be a json set in msg.payload.i.e msg.payload.url

bigmac5753 · 30 December 2020 13:53

Thanks for your reply.
{{url}} is a link to a youtube video I want to download.
{{host}} is the IP of the server running the youtube-dl software.

Therefore a better command to use would be: curl -X POST --data-urlencode "url={{url}}" http://192.168.0.2:8080/youtube-dl/q

Hope this helps, thanks again.

E1cid · 30 December 2020 14:11

What i posted should work. Export the section of the flow so we can see how you used it.

bigmac5753 · 30 December 2020 20:41

[{"id":"63f3e144.1f0478","type":"inject","z":"dee1ee38.a66f9","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{}","payloadType":"json","x":130,"y":1280,"wires":[["dbf18da0.3fef08"]]},{"id":"a808fae1.564ae","type":"http request","z":"dee1ee38.a66f9","name":"","method":"POST","ret":"txt","paytoqs":"body","url":"http://192.168.0.2:8080/youtube-dl/q","tls":"","persist":false,"proxy":"","authType":"","x":530,"y":1280,"wires":[["d82a6daf.bc593"]]},{"id":"d82a6daf.bc593","type":"debug","z":"dee1ee38.a66f9","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":710,"y":1280,"wires":[]},{"id":"dbf18da0.3fef08","type":"change","z":"dee1ee38.a66f9","name":"","rules":[{"t":"set","p":"payload.url","pt":"msg","to":"https://www.youtube.com/watch?v=6jAb8Bzrs8U","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":340,"y":1280,"wires":[["a808fae1.564ae"]]}]

I've changed the flow slightly as there's no need for {{host}}, i've put the server's IP instead.
I've also linked a valid youtube video - but it will not work, I get this error:

  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 582, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 243, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 54, in app
    response = await func(request)
  File "/usr/src/app/./youtube-dl-server.py", line 35, in q_put
    url = form.get("url").strip()
AttributeError: 'NoneType' object has no attribute 'strip'```

Steve-Mcl · 30 December 2020 20:48

Have you tried setting the msg.payload to be a string containing url=http://youtube.com?xxxxx like in your curl example...

change node...

Also, are you certain you want to return a string from the HTTP request?

bigmac5753 · 30 December 2020 20:54

Just tried these, neither work:

[{"id":"cfa7c917.1c39c8","type":"inject","z":"dee1ee38.a66f9","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{}","payloadType":"json","x":190,"y":1440,"wires":[["3cc4c4ab.c6acbc"]]},{"id":"b568ddb4.8c72a","type":"http request","z":"dee1ee38.a66f9","name":"","method":"POST","ret":"txt","paytoqs":"body","url":"http://192.168.0.2:8080/youtube-dl/q","tls":"","persist":false,"proxy":"","authType":"","x":590,"y":1440,"wires":[["dc2d954d.e94ba8"]]},{"id":"dc2d954d.e94ba8","type":"debug","z":"dee1ee38.a66f9","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":770,"y":1440,"wires":[]},{"id":"3cc4c4ab.c6acbc","type":"change","z":"dee1ee38.a66f9","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"url=https://www.youtube.com/watch?v=6jAb8Bzrs8U","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":360,"y":1440,"wires":[["b568ddb4.8c72a"]]},{"id":"ac55eccc.d2798","type":"inject","z":"dee1ee38.a66f9","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{}","payloadType":"json","x":190,"y":1380,"wires":[["3f19ccbe.7ef084"]]},{"id":"2e0daf3.abed75","type":"http request","z":"dee1ee38.a66f9","name":"","method":"POST","ret":"txt","paytoqs":"body","url":"http://192.168.0.2:8080/youtube-dl/q","tls":"","persist":false,"proxy":"","authType":"","x":590,"y":1380,"wires":[["6326239b.c35fcc"]]},{"id":"6326239b.c35fcc","type":"debug","z":"dee1ee38.a66f9","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":770,"y":1380,"wires":[]},{"id":"3f19ccbe.7ef084","type":"change","z":"dee1ee38.a66f9","name":"","rules":[{"t":"set","p":"payload.url","pt":"msg","to":"url=https://www.youtube.com/watch?v=6jAb8Bzrs8U","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":370,"y":1380,"wires":[["2e0daf3.abed75"]]}]

E1cid · 30 December 2020 21:01

Try urlencoding the url

[{"id":"2475a82e.f6ca18","type":"http request","z":"8d22ae29.7df6d","name":"","method":"POST","ret":"txt","paytoqs":"body","url":"http://192.168.0.2:8080/youtube-dl/q","tls":"","persist":false,"proxy":"","authType":"","x":570,"y":2600,"wires":[["123eba6a.f9aa3e"]]},{"id":"123eba6a.f9aa3e","type":"debug","z":"8d22ae29.7df6d","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":750,"y":2600,"wires":[]},{"id":"d92e3972.96d3e","type":"change","z":"8d22ae29.7df6d","name":"","rules":[{"t":"set","p":"payload.url","pt":"msg","to":"$encodeUrl(\"https://www.youtube.com/watch?v=6jAb8Bzrs8U\")","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":380,"y":2600,"wires":[["2475a82e.f6ca18"]]},{"id":"eb12914e.a980d8","type":"inject","z":"8d22ae29.7df6d","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{}","payloadType":"json","x":170,"y":2600,"wires":[["d92e3972.96d3e"]]}]

you could also try sendng as application/x-www-form-urlencoded , by setting msg.headers["content-type"]

bigmac5753 · 30 December 2020 21:08

nope, tried both and still not working.

I can confirm running this in terminal works:

curl -X POST --data-urlencode "url=https://www.youtube.com/watch?v=6jAb8Bzrs8U" http://192.168.0.2:8080/youtube-dl/q

E1cid · 30 December 2020 21:18

Did you try urlEncoding the string ìn the change node? eg.
set msg.payload to
JSONata expression $encodeUrl("url=https://www.youtube.com/watch?v=6jAb8Bzrs8U")

bigmac5753 · 30 December 2020 21:21

yes

[{"id":"cfa7c917.1c39c8","type":"inject","z":"dee1ee38.a66f9","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{}","payloadType":"json","x":190,"y":1440,"wires":[["3cc4c4ab.c6acbc"]]},{"id":"b568ddb4.8c72a","type":"http request","z":"dee1ee38.a66f9","name":"","method":"POST","ret":"txt","paytoqs":"body","url":"http://192.168.0.2:8080/youtube-dl/q","tls":"","persist":false,"proxy":"","authType":"","x":590,"y":1440,"wires":[["dc2d954d.e94ba8"]]},{"id":"dc2d954d.e94ba8","type":"debug","z":"dee1ee38.a66f9","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":770,"y":1440,"wires":[]},{"id":"3cc4c4ab.c6acbc","type":"change","z":"dee1ee38.a66f9","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"$encodeUrl(\"url=https://www.youtube.com/watch?v=6jAb8Bzrs8U\")","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":360,"y":1440,"wires":[["b568ddb4.8c72a"]]}]

E1cid · 30 December 2020 22:11

one last suggestion try appending it as a query string

[{"id":"63f3e144.1f0478","type":"inject","z":"8d22ae29.7df6d","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":120,"y":2180,"wires":[["a808fae1.564ae"]]},{"id":"a808fae1.564ae","type":"http request","z":"8d22ae29.7df6d","name":"","method":"POST","ret":"txt","paytoqs":"body","url":"http://192.168.0.2:8080/youtube-dl/q?url=https://www.youtube.com/watch?v=6jAb8Bzrs8U","tls":"","persist":false,"proxy":"","authType":"","x":290,"y":2180,"wires":[["d82a6daf.bc593"]]},{"id":"d82a6daf.bc593","type":"debug","z":"8d22ae29.7df6d","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":470,"y":2180,"wires":[]}]

bigmac5753 · 31 December 2020 09:56

Unfortunately that didn't work either.

I asked the dev and got this response...

Steve-Mcl · 31 December 2020 10:00

The dev wrote "the content type should be application/x-www-form-urlencoded"

@E1cid suggested that

Did you add that header?

bigmac5753 · 31 December 2020 10:08

i'd previously used the header, but not with also setting payload to:

$encodeUrl("url=https://www.youtube.com/watch?v=6jAb8Bzrs8U")

So with the above payload combined with the header..... it works!!

Thanks for your help guys, much appreciated.

Colin · 31 December 2020 11:05

I haven't been following this thread, but would one of the youtube nodes be an easier way of achieving this? node-red-contrib-ytdl (node) - Node-RED

bigmac5753 · 31 December 2020 11:07

ok, last question on this...

If I pass the URL as msg.payload.url, how do I enter that into $encodeUrl() ?

I tried $encodeUrl("url={url}") but it didn't work.

bigmac5753 · 31 December 2020 11:10

Those nodes only allow youtube videos, a standard installation of youtube-dl allows downloading and streaming of many sites - which is what I need.

Colin · 31 December 2020 11:13

OK, that makes sense.

E1cid · 31 December 2020 11:38

try

$encodeUrl("url=" & payload.url)

Topic		Replies	Views
Use the HTTP-Request to recreate an curl command General	19	297	30 March 2023
Best way to run a curl command? General	19	13888	22 July 2020
cURL in Node red General	4	1807	20 April 2022
Convert working curl command into Node-Red flow General	8	2616	11 November 2020
First time using cURL - Notion.so API General http-request	16	449	19 January 2023

Need help with curl

Related topics