Workflow with bluemix

Hi
I read the posts by Nick O'Leary including Connecting to IBM Cloud services from Node-RED.
Despite the detailed explanation, I am still confused about files like settings.js.

When I run node-red to develop locally with
$ node-red
the settings.js files in ~/.node-red is used ( and I suppose package.json and flows.js as well ?). This is not part of the versioned project dir.
When my app runs in the cloud, since there is no ~/.node-red, the version from the versioned project dir is
used.
Is this correct ?
If yes, then I still have the problem to keep these setting files in sync, since they share chunks of code common to local dev and to cloud operation. Also, the files in ~/.node-red would need to versioned in a different repo since these files are subject to changes during the lifetime of the project.

Am I missing something ? , What are your recommendations ?

Thanks
Peter

Hi @vittali

When you develop locally, using the Projects feature, then almost everything you need is in the project's git repo. If you want to see if on disk, look under ~/.node-red/projects/<project-name>/

That includes the flows file and package.json file.

The one thing the project doesn't get by default is a settings file.

When you run locally, it will use ~/.node-red/settings.js. But that file is not part of the project. This is why my blog post talks about adding settings.js to the project files.

So any customisations you make to ~/.node-red/settings.js you would also need to make to the settings file you add to the project directory. That is slightly cumbersome, but there are ways around it. For example, when you develop locally, you could tell node-red to use the settings file in the projects directory (rather than ~/.node-red/settings.js) via the -s command line argument.

Nick, thank you, the only problem with your -s option proposal seems to be that the VCAP incantation that you describe in your blog is needed for local dev ( to mimic the situation with env vars on the server if I correctly understood). But this incantation would be wrong on the server because there is no vcap_.json file.
So I still need two different settings.js files with partially shared content for local and deployment mode.
Not that I think that this would be un unsurmountable problem, but I want to make sure that I fully understand your blog post.

Peter

Indeed - you could add some logic to the settings.js file that checks what environment it is in and then loads the VCAP property from the file if it doesn't find the env var already set.

Ok, thanks Nick, I got this. Another point I don't quite understand relates to credentials:

Tell your application your credential secret

Now that your application has been created on the IBM Cloud, one final step is to tell your application the key it should use to decrypt your credentials file.

Go to the IBM Cloud dashboard page for your newly deployed application. On the ‘runtime’ page, go to the ‘environment variable’ section and add a variable called NODE_RED_CREDENTIAL_SECRET set to whatever credential key you set when you created your Node-RED project right at the start of this whole exercise.

... right at the start of this whole exercise. Does this relate to keys required by travis ?

Thanks
Peter

Node-RED encrypts your credentials file. It uses the credentialSecret value in your settings file as the key to encrypt/decrypt the file.

You do not want to put the true value of credentialSecret into version control. So instead, you can configure your settings.js file to use the value of an environment variable.

This is what the following line does:

credentialSecret: process.env.NODE_RED_CREDENTIAL_SECRET

So this part of the blog post tells you to set that environment variable in the IBM Cloud application so it is available to the settings file.

Nick, is there a follow-up to this post where I can see how all this is used to build a local app ?

In the next part of this series, we’ll look at how to start building a simple application using this setup. That’s what I said in the previous post, but I really mean it this time.

Thanks
Peter

No, that post is still on my todo list... but it's only been... (checks dates on the last post)... a year since I said I would write it.

Is there a particular aspect you are interested in? The posts I did publish help setup the deployment pipeline. From there its a case just actually building your Node-RED app as you would normally.

As time flies by (:-

I apologize for these beginner questions, but I am new to node.js, node-red and bluemix ( and javascript ), so this is quite a steep learning curve.

I am still struggling with the workflow and the various credentials.

First, here is my bluemix-settings.js that I got from creating the node-red-starter-kit from the bluemix catalog:

var path = require("path");
var when = require("when");
var util = require("util");
var fs = require("fs");

var cfenv = require("cfenv");
var appEnv = cfenv.getAppEnv();

var userDir = path.join(__dirname,".node-red");
// Ensure userDir exists - something that is normally taken care of by
// localfilesystem storage when running locally
fs.mkdirSync(userDir);
fs.mkdirSync(path.join(userDir,"node_modules"));

var settings = module.exports = {
    uiPort: process.env.PORT || 1880,
    mqttReconnectTime: 15000,
    debugMaxLength: 1000,

    userDir: userDir,

    flowFile: "flows.json",

    // Add the bluemix-specific nodes in
    nodesDir: path.join(__dirname,"nodes"),

    // Blacklist the non-bluemix friendly nodes
    nodesExcludes:['66-mongodb.js','75-exec.js','35-arduino.js','36-rpi-gpio.js','25-serial.js','28-tail.js','50-file.js','31-tcpin.js','32-udp.js','23-watch.js'],

    // Enable module reinstalls on start-up; this ensures modules installed
    // post-deploy are restored after a restage
    autoInstallModules: true,

    // Move the admin UI
    httpAdminRoot: '/red',

    // Serve up the welcome page
    httpStatic: path.join(__dirname,"public"),

    functionGlobalContext: { },

    // Configure the logging output
    logging: {
        // Only console logging is currently supported
        console: {
            // Level of logging to be recorded. Options are:
            // fatal - only those errors which make the application unusable should be recorded
            // error - record errors which are deemed fatal for a particular request + fatal errors
            // warn - record problems which are non fatal + errors + fatal errors
            // info - record information about the general running of the application + warn + error + fatal errors
            // debug - record information which is more verbose than info + info + warn + error + fatal errors
            // trace - record very detailed logging + debug + info + warn + error + fatal errors
            // off - turn off all logging (doesn't affect metrics or audit)
            level: "info",
            // Whether or not to include metric events in the log output
            metrics: false,
            // Whether or not to include audit events in the log output
            audit: true
        }
    }
};

// Look for the attached Cloudant instance to use for storage
settings.couchAppname = appEnv.name;
// NODE_RED_STORAGE_NAME is automatically set by this applications manifest.
var storageServiceName = process.env.NODE_RED_STORAGE_NAME || new RegExp("^"+settings.couchAppname+".cloudantNoSQLDB");
var couchService = appEnv.getService(storageServiceName);

if (!couchService) {
    util.log("Failed to find Cloudant service: "+storageServiceName);
    if (process.env.NODE_RED_STORAGE_NAME) {
        util.log(" - using NODE_RED_STORAGE_NAME environment variable: "+process.env.NODE_RED_STORAGE_NAME);
    }
    //fall back to localfilesystem storage
} else {
    util.log("Using Cloudant service: "+storageServiceName+" : "+settings.couchAppname);
    settings.storageModule = require("./couchstorage");
    settings.couchUrl = couchService.credentials.url;
}

With this I get a working app , connected to a cloudant database and I can run this tutorial. I use a java MQTT client and my events get stored in the cloudant database.

I also use the build-in continuous delivery chain provided by bluemix, for example:
$ ibmcloud cf push
will build and deploy my app , much the same way you have shown in your first blog post ( it uses gitlab rather than github).

So far so good:

BUT:

  • when I open the node-red editor on the cloud, I am asked to provide a username and a password. Each time I re-deploy, I have to reenter username and password.
  • I don't understand how I can use the project feature of node-red when working in the cloud.
  • I have a separate bluemix-settings.js file (not part of the project), where I added:
var fs = require("fs");
var path = require("path");
process.env.VCAP_SERVICES = fs.readFileSync(path.join(__dirname,"EauDuPuid.json"));

(Later I will add some logic (as you suggested) to avoid having a second settings file.)

But when I want to run node-red locally:
$ node-red --settings $NUAGE/dev/node-10/userDir/EauDuPuid/bluemix-settings.js -u .

I get:

Error loading settings file: /media/p/nuage/dev/node-10/userDir/EauDuPuid/bluemix-settings.js
Error: Cannot find module 'when'

I realize that this is newbie stuff, but the problem is that bluemix and node-red evolve very quickly and I simply can't find an uptodate tutorial that gives me:

  • a starter project that runs on bluemix using cloudant and the bluemix provided continous delivery toolchain
  • where I can develop locally and then say
    $ ibmcloud cf push
    and it just works !

Sorry for the long post
Peter