# For each Line in a text file do

**URL:** <https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720>\
**Category:** General\
**Created:** [6 March 2020 13:43 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720 "2020-03-06T13:43:06Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [7 March 2020 15:16 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/21 "2020-03-07T15:16:34Z")

</div>

Yes and what is he planning to do with the information ? Request urls right ? Those urls will not work to retrieve data from those pages because it is an ajax post request.

@zenofmud i see that mustache works but with 3 curly braces. `{{{`

---

<div class="post-metadata">

**Author:** ![unixneo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/unixneo/32/17441_2.png) [@unixneo](https://discourse.nodered.org/u/unixneo)\
**Post date:** [7 March 2020 15:19 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/22 "2020-03-07T15:19:37Z")

</div>

> [@bakman2](#):
>
> Yes and what is he planning to do with the information ? Request urls right ? Those urls will not work to retrieve data from those pages because it is an ajax post request.

Huh?

I am not following you. I looked at all his URLs he posted. What exact URLs are you talking about? Please repost them.

Thanks!

EDIT: Here is the URL he posted, right:

> and the page is www.radiopopular.pt/pesquisa/ where I can have the product name and the price..

If it was me, I would pull this data with a `python` or `php` script in the `shell`, as I mentioned. it's trivial to do.

What other URLs did he mention? I missed them haha

Funny!

---

<div class="post-metadata">

**Author:** ![LucasSaraivaAzevedo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/lucassaraivaazevedo/32/8686_2.png) [@LucasSaraivaAzevedo](https://discourse.nodered.org/u/LucasSaraivaAzevedo)\
**Post date:** [7 March 2020 15:23 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/23 "2020-03-07T15:23:29Z")

</div>

The info that i want its the price and the description of the Ean product . Example :

[https://www.radiopopular.pt/pesquisa/6901443274383](https://www.radiopopular.pt/pesquisa/6901443274383)

![price](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/a/5/a5415adf5fc9d88a9233d1a5c5b22ebbb483e0a7.png)

SMARTPHONE HUAWEI PSMART 2019 BLK  
159,99 €  
and save this info in a .txt file

---

<div class="post-metadata">

**Author:** ![unixneo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/unixneo/32/17441_2.png) [@unixneo](https://discourse.nodered.org/u/unixneo)\
**Post date:** [7 March 2020 15:24 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/24 "2020-03-07T15:24:26Z")

</div>

You are "screen scraping" right? Or are you using an API?

---

<div class="post-metadata">

**Author:** ![LucasSaraivaAzevedo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/lucassaraivaazevedo/32/8686_2.png) [@LucasSaraivaAzevedo](https://discourse.nodered.org/u/LucasSaraivaAzevedo)\
**Post date:** [7 March 2020 15:26 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/25 "2020-03-07T15:26:17Z")

</div>

Only screen scraping no API ...

---

<div class="post-metadata">

**Author:** ![unixneo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/unixneo/32/17441_2.png) [@unixneo](https://discourse.nodered.org/u/unixneo)\
**Post date:** [7 March 2020 15:30 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/26 "2020-03-07T15:30:38Z")

</div>

That's exactly what I thought you posted .. LOL

Of course you can pull this with `wget, curl, php, python, ajax, anything under the sun` and process the text as you like.

I do this all the time, and as mentioned earlier, I just wrote an app for `coronavirus stats` which grabs the web page (from China, in Chinese) using `php file_get_contents()` and processes the test, publishes the stats we want with `MQTT` and receives via `MQTT` on `Node-RED`.

However, I see you are on Windows Lucas, and sorry, I don't do Window :).

The only reason I am still posting in your thread is because I read you wanted to use `wget` to screen-scrape, like you mentioned before. Thanks for confirming..... 🙂

I've already posted at least twice on how I would easily accomplish your task in the `Linux shell`. I'm "outta here" for now.

Best Regards!!

---

<div class="post-metadata">

**Author:** ![LucasSaraivaAzevedo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/lucassaraivaazevedo/32/8686_2.png) [@LucasSaraivaAzevedo](https://discourse.nodered.org/u/LucasSaraivaAzevedo)\
**Post date:** [7 March 2020 15:31 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/27 "2020-03-07T15:31:44Z")

</div>

It doesn't work @unixneo ... I have a server at home with linux... If this works with linux we can make at my personal server ( I have proxmox with nodered at home )

> [@unixneo](#):
>
> Of course you can pull this with `wget, curl, php`

no you can't because it's like @bakman2 says...

---

<div class="post-metadata">

**Author:** ![unixneo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/unixneo/32/17441_2.png) [@unixneo](https://discourse.nodered.org/u/unixneo)\
**Post date:** [7 March 2020 15:36 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/28 "2020-03-07T15:36:04Z")

</div>

> [@LucasSaraivaAzevedo](#):
>
> It doesn't work @unixneo ... I have a server at home with linux... If this works with linux we can make at my personal server ( I have proxmox with nodered at home )
> 
> no you can't because it's like @bakman2 says...

Ah... I have no idea about `proxmox`

My bad. Sorry. I thought you were on Linux or the PI. I missed that part.

Whoops. Then, I'm outta your thread for good. I have zero idea about `proxmox`, really. LOL, I thought most folks here were PI people ....

I'll stop offering `Linux shell-like solutions` then, for sure, in your case.

:)'

@bakman2 sorry for misunderstanding this was a `Linux/PI kinda` problem.

---

<div class="post-metadata">

**Author:** ![unixneo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/unixneo/32/17441_2.png) [@unixneo](https://discourse.nodered.org/u/unixneo)\
**Post date:** [7 March 2020 15:41 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/29 "2020-03-07T15:41:32Z")

</div>

> [@meeki007](#):
>
> proxmox

But wait... I just Googled proxmox.... it said:

## Server virtualization with support for KVM and LXC

 ![Proxmox is a Debian-based platform](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/f/0/f0bd5273187d8f45eee89635557ca3c93b98c51d.png)Proxmox VE is based on [Debian GNU/Linux](https://www.debian.org/index.en.html) and uses a customized Linux Kernel. The Proxmox VE source code is free, released under the GNU Affero General Public License, v3 ([GNU AGPL, v3](http://www.gnu.org/licenses/agpl-3.0.en.html)). This means that you are free to use the software, inspect the source code at any time or contribute to the project yourself.

Using open-source software guarantees full access to all functionalities at any time as well as a high level of reliability and security. We encourage everybody to contribute to the Proxmox VE project while Proxmox, the company behind it, ensures that the product meets consistent and enterprise-class quality criteria.

[Read more about Proxmox open-source](https://www.proxmox.com/en/about/proxmox-technology)

So why is this NOT a Linux-based solution?

Seems like just another VM, like Linode. Right?

You are not running Linux on proxmox? What are you running then?

---

<div class="post-metadata">

**Author:** ![LucasSaraivaAzevedo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/lucassaraivaazevedo/32/8686_2.png) [@LucasSaraivaAzevedo](https://discourse.nodered.org/u/LucasSaraivaAzevedo)\
**Post date:** [7 March 2020 15:46 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/30 "2020-03-07T15:46:25Z")

</div>

Proxmox it's for vertualization . I have debian , ubuntu, debian with docker on proxmox and other virtual machines... soo yhea i have linux at home. Proxmox = Vmware , wine , qemu , etc ...

we do like your opinions 😉

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [7 March 2020 15:49 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/31 "2020-03-07T15:49:30Z")

</div>

I was trying to reverse engineer the request, but they use multiple mechanisms to protect their data, the main issue seems to be the content-length header and cookies, we can't specify it as we don't know the length.

---

<div class="post-metadata">

**Author:** ![unixneo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/unixneo/32/17441_2.png) [@unixneo](https://discourse.nodered.org/u/unixneo)\
**Post date:** [7 March 2020 15:49 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/32 "2020-03-07T15:49:35Z")

</div>

> [@LucasSaraivaAzevedo](#):
>
> Proxmox it's for vertualization . I have debian , ubuntu, debian with docker on proxmox and other virtual machines... soo yhea i have linux at home. Proxmox = Vmware , wine , qemu , etc ...

Well, I just pulled this:

> ## The Proxmox Linux Kernel
> 
> Our software is based on a rock-solid Linux kernel. Linux is a free Unix-type operating system originally created by Linus Torvalds with the assistance of developers around the world. It is published under the GNU General Public License.
> 
> Linux provides great performance and is an extremely stable and reliable platform for our servers. It also supports a wide range of hardware configurations - from cheap systems for testing to high performance multiprocessor servers connected to SAN solutions.

It's just a VM.....

What does running in a VM have to do with "cannot use wget, curl, (or even PHP, Python, etc)?

Now I am really lost...LOL

I use a lot of VM services, like I assume everyone here,.... that has very little to do with how you `exec` a `shell command or script`.

You are running Node-RED on Windows? No? Yes?

---

<div class="post-metadata">

**Author:** ![LucasSaraivaAzevedo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/lucassaraivaazevedo/32/8686_2.png) [@LucasSaraivaAzevedo](https://discourse.nodered.org/u/LucasSaraivaAzevedo)\
**Post date:** [7 March 2020 15:50 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/33 "2020-03-07T15:50:34Z")

</div>

> [@unixneo](#):
>
> Whoops. Then, I'm outta your thread for good. I have zero idea about `proxmox` , really. LOL, I thought most folks here were PI people ....
> 
> I'll stop offering `Linux shell-like solutions` then, for sure, in your case.
> 
> :)'

Because... This ?

---

<div class="post-metadata">

**Author:** ![LucasSaraivaAzevedo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/lucassaraivaazevedo/32/8686_2.png) [@LucasSaraivaAzevedo](https://discourse.nodered.org/u/LucasSaraivaAzevedo)\
**Post date:** [7 March 2020 15:55 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/34 "2020-03-07T15:55:31Z")

</div>

> [@unixneo](#):
>
> You are running Node-RED on Windows? No? Yes?

I'm running on windows and linux..i have multiple VMs

---

<div class="post-metadata">

**Author:** ![unixneo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/unixneo/32/17441_2.png) [@unixneo](https://discourse.nodered.org/u/unixneo)\
**Post date:** [7 March 2020 15:57 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/35 "2020-03-07T15:57:52Z")

</div>

> [@LucasSaraivaAzevedo](#):
>
> Because... This ?

Yes, and then before my wife called me to sleep, I Googled `promox` and saw it was a Linux based VM..

You said you are a noob... well, I was a Linux noob at Linux Slackware 0.8.... before 1993-07-17 when Slackware 1.0 was released (27 years ago, noob on Linux)

We were `screen scraping` web pages, way back then (believe it or not, before Google).... so what's the big deal? It's not rocket science to take a few URLs, read them a line at a time, get the content, and save the file (with content) to disk or even process in script.

This can be done in more ways I can think of quickly (Javascript XHR, PHP, Python, VueJS Axios, fetch, wget, curl... the list goes on and on).... for miles. This is not rocket science, grabbing data from the web...

Good night....

---

<div class="post-metadata">

**Author:** ![LucasSaraivaAzevedo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/lucassaraivaazevedo/32/8686_2.png) [@LucasSaraivaAzevedo](https://discourse.nodered.org/u/LucasSaraivaAzevedo)\
**Post date:** [7 March 2020 16:07 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/36 "2020-03-07T16:07:19Z")

</div>

sooo @bakman2 .. it's impossible using node-red?

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [7 March 2020 16:07 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/37 "2020-03-07T16:07:51Z")

</div>

> This is not rocket science, grabbing data from the web...

Please enlighten us with a working flow or even script (no difference here) for the page @LucasSaraivaAzevedo is requesting and extract the price - as text, that can be saved to a text file.

> **[Pesquisa](https://www.radiopopular.pt/pesquisa/6901443274383)**

---

<div class="post-metadata">

**Author:** ![meeki007](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/meeki007/32/12499_2.png) [@meeki007](https://discourse.nodered.org/u/meeki007)\
**Post date:** [7 March 2020 17:07 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/38 "2020-03-07T17:07:42Z")

</div>

because your using windows and I gave you a flow to work on linux.

I have not used windows 2003 to now so I cant help you with file access on a windows machine ..... I have no clue...

---

<div class="post-metadata">

**Author:** ![meeki007](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/meeki007/32/12499_2.png) [@meeki007](https://discourse.nodered.org/u/meeki007)\
**Post date:** [7 March 2020 17:14 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/39 "2020-03-07T17:14:39Z")

</div>

these guys

```auto
https://www.radiopopular.pt/pesquisa/6901443274383

```

are real good at obfuscating their data. But if you click the link (the image) the next page that loads has the data you need to scrape.

```auto
    <meta itemprop="price" content="159.99">

```

you need to emulate the click on the image from the page then scrape the result.

---

<div class="post-metadata">

**Author:** ![LucasSaraivaAzevedo](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/lucassaraivaazevedo/32/8686_2.png) [@LucasSaraivaAzevedo](https://discourse.nodered.org/u/LucasSaraivaAzevedo)\
**Post date:** [7 March 2020 23:57 UTC](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720/40 "2020-03-07T23:57:59Z")

</div>

No problems doing on Linux . I have linux

[Previous page](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720.md?page=1)

[Next page](https://discourse.nodered.org/t/for-each-line-in-a-text-file-do/22720.md?page=3)
