I'm using NR to get a csv from a FTP server, run a java program via exec (using command "ant") and then transfer the output file to another FTP.
When I run ant directly via ssh on the server, the complete file with about 500.000 rows gets exported, taking about 18 seconds.
On the other hand, if I run "ant" via NR, it only takes 3 seconds but also only exports 90.000 to 150.000 rows.
Why is there a difference?
It runs the exact same files.
What does your script actually do, it appears to be building the app and then exiting, rather than running it.
When you run the command in a terminal what output to you see, and how long does it take before it returns to the command prompt? I am wondering whether the script builds the app and then spawns a process to run it, but does not wait for it to complete before returning.
Prior to the command "ant" I ran "ant clean jar", which is only necessary one time.
Now, it's sufficient to run "ant" which just runs the program. The program itself is a java program, generated from a third party flow processor.
It takes a csv as input, does some converting and puts out another csv.
Output in terminal is:
Picked up _JAVA_OPTIONS: -Xmx5120m
Buildfile: /root/transfer/stock/build.xml
compile:
test:
[java] Picked up _JAVA_OPTIONS: -Xmx5120m
[java] Mapping Application
[java] Parsing Text...
[java]
[java] Finished
BUILD SUCCESSFUL
Total time: 14 seconds
But yeah, now I see a difference there. Running it from terminal says that it picked up the java option -Xmx5120m which is set under /etc/profile.
Maybe I have to set this in any other location to have it system wide?
If that is an environment option then you should be able to set it in the command in the exec node. Something like cd /root/transfer/stock && _JAVA_OPTIONS=-Xmx5120 ant
What is that file? It suggests that your system time is confused or something like that. Is node-red running in docker? If so then check that the docker time and timezone are the same as the system.
Done. The terminal o/p you showed from the terminal did not show that line, or had you deleted the line? I would still like to know what the file is, as it appears to be part of the compile sequence.
You could try the exec node in Spawn mode. I still suspect it is something to do with the script launching another process, which is then getting killed off early.
Yes starting it via terminal also gives the warning about the date in future.
The file is a file provided by our supplier with some product data. The java process takes some of the columns, filters some rows and the puts out another file.
Why is that appearing under a compile heading? I thought they were coming from Java, building the app. Or does all that output come from the application itself? If from the application, can you add some debug to it to try and work out what is happeining?
That is presumably from output 3. What about outputs 1 and 2? Set the debug nodes to show Complete Message and show us what they all give.
The error appears under the Compiling. Also the option -Xmx5120 is an instruction about how much heap to allocate, so I would have thought that line would be before it actually starts running the app.
Also a but of googling suggests that the three seconds is the time taken to build it, not to run it.
What exactly is the file ant? Is it a shell script of some sort? If so then what is in it.
What, no message at all?
Did you wait some time to make sure nothing is going to arrive?
Did it convert any records?
Did the message on o/p 3 arrive instantly?
Are you sure you changed nothing except the spawn mode?
I'm not an expert by any means in ant but from what I understand is, that it is used to compile and then run java-based programs. So first, you do something like "ant clean jar" to compile it. From that moment on, you just have to do "ant" to run it.
The program is already compiled and therefore, we only need ant to trigger the program, while this just runs some conversion.
In spawn mode, I only have an output in "return code" and none for the others. In exec mode, all outputs provide something. Nothing else changed.
OK, I have no idea why it does not run in spawn mode.
Create a file in the directory called antrun.sh (or whatever you fancy), containing this script
#!/usr/bin/env sh
# run ant and wait 30 seconds
echo "running ant"
_JAVA_OPTIONS=-Xmx5120 ant
echo "ant completed, waiting 30 seconds"
sleep 30
echo "done"
Make it executable, then in the exec node put cd /root/transfer/stock && _JAVA_OPTIONS=-Xmx5120 /root/transfer/stock/antrun.sh
The above is all one line of course. Set the exec node to Exec mode.
That should run ant exactly as before, but will wait 30 seconds before terminating, so if my supposition that the app is running in the background, and it is the shell termination that stops it is correct, then you should get all your data converted. If it works then there is probably a better way of solving it.
Another thought, what version of node-red (at bottom of editor menu), and nodejs (command node -v) are you using? I seem to remember a fix a long time ago that was causing the exec node to terminate early in the case where the command spawned further tasks.
Assuming that it waited the 30 seconds then I am all out of ideas, other than to add some debug into the app in order to determine what is going wrong.
