When running on bluemix, I get:
2019-08-16T22:03:54.45+0200 [APP/PROC/WEB/0] OUT 16 Aug 20:03:54 - [debug] red/runtime/nodes/credentials.load : user provided key
2019-08-16T22:03:54.45+0200 [APP/PROC/WEB/0] OUT 16 Aug 20:03:54 - [debug] red/runtime/nodes/credentials.load : default key present. Will migrate
2019-08-16T22:03:54.45+0200 [APP/PROC/WEB/0] OUT 16 Aug 20:03:54 - [warn] Error loading credentials: SyntaxError: Unexpected token in JSON at position 0
2019-08-16T22:03:54.45+0200 [APP/PROC/WEB/0] OUT 16 Aug 20:03:54 - [warn] Error loading flows: Error: Failed to decrypt credentials
Running locally gives me:
16 Aug 22:24:15 - [debug] red/runtime/nodes/credentials.load : user provided key
16 Aug 22:24:15 - [debug] red/runtime/nodes/credentials.load : keyType=user
The key is hardcoded in settings.js, no env vars are involved.
I have deleted the app on bluemix and deleted the node_modules dir in the project dir.
Locally I run:
$ node-red -s settings.js -u .
And my command in manifest.yml is:
command: node --max-old-space-size=160 index.js --settings ./settings.js -v
What does this mean:
default key present. Will migrate.
Why is there a default key ? And why is it even considered when a key provided by the user is recognized ?
Thanks
Peter
How are you running in Bluemix? Is this using the boilerplate or via the deployment pipeline we've been discussing in other threads?
Hi Nick
It uses the boilerplate and the bluemix-provided deployment pipeline.
And I found the problem secs ago: I had to delete the nodered table in the cloundant database.
Perhaps this is a remainder of some previous tests, or is the database used to store credential related info.
From what I understood is that the key and the flow_cred.json should contain all required info to extract
the protected data.
Am I wrong ?
Thanks
Peter
And where exactly did you put your credentialSecret? In which file?
In the settings.js file:
var when = require("when");
var util = require("util");
var cfenv = require("cfenv");
var path = require("path");
var fs = require("fs");
var appEnv = cfenv.getAppEnv();
var userDir = __dirname;
if ("VCAP_SERVICES" in process.env) {
util.log("running in bluemix");
userDir = path.join(__dirname,".node-red");
if(!fs.existsSync(userDir)) fs.mkdirSync(userDir);
if(!fs.existsSync(path.join(userDir,"node_modules"))) fs.mkdirSync(path.join(userDir,"node_modules"));
} else {
util.log("running locally");
var p = path.join(__dirname, "App_vcap.json");
process.env.VCAP_SERVICES = fs.readFileSync(p);
}
var settings = module.exports = {
uiPort: process.env.PORT || 1880,
credentialSecret: "My-Credential",
adminAuth: {
type: "credentials",
users: [{
username: "admin",
password: "xxxxx",
permissions: "*"
}],
},
mqttReconnectTime: 15000,
debugMaxLength: 1000,
userDir: userDir,
useAppmetrics: false,
flowFile: "flow.json",
// Add the bluemix-specific nodes in
nodesDir: path.join(__dirname, "nodes"),
// Blacklist the non-bluemix friendly nodes
nodesExcludes: ['66-mongodb.js', '75-exec.js', '35-arduino.js', '36-rpi-gpio.js', '25-serial.js', '28-tail.js', '50-file.js', '31-tcpin.js', '32-udp.js', '23-watch.js'],
// Enable module reinstalls on start-up; this ensures modules installed
// post-deploy are restored after a restage
autoInstallModules: true,
// Move the admin UI
// httpAdminRoot: '/red',
// Serve up the welcome page
// httpStatic: path.join(__dirname, "public"),
functionGlobalContext: {},
// Configure the logging output
logging: {
// Only console logging is currently supported
console: {
level: "trace",
// Whether or not to include metric events in the log output
metrics: false,
// Whether or not to include audit events in the log output
audit: false
}
}
};
// Look for the attached Cloudant instance to use for storage
settings.couchAppname = appEnv.name;
// NODE_RED_STORAGE_NAME is automatically set by this applications manifest.
var storageServiceName = process.env.NODE_RED_STORAGE_NAME || settings.couchAppname + "-cloudantNoSQLDB";
var couchService = appEnv.getService(storageServiceName);
if (!couchService) {
util.log("Failed to find Cloudant service: " + storageServiceName);
if (process.env.NODE_RED_STORAGE_NAME) {
util.log(" - using NODE_RED_STORAGE_NAME environment variable: " + process.env.NODE_RED_STORAGE_NAME);
}
//fall back to localfilesystem storage
} else {
util.log("Using Cloudant service: " + storageServiceName + " : " + settings.couchAppname);
settings.storageModule = require("./couchstorage");
settings.couchUrl = couchService.credentials.url;
}
Thanks
Peter
Do you mean bluemix-settings.js as that is the settings file the Boilerplate is configured to use? Or have you made further customisations you've not mentioned?
I have changed the name of bluemix-settings.js to settings.js ( and adapted index.js accordingly).
This is to reflect the fact that the same script runs locally and remotely.
Other than that , I have made no customizations , the code I just posted is what created the problem that is the subject of this thread.
Thanks
Peter
So you aren't really following the blog posts on creating a deployment pipeline we discussed before. You wouldn't be using cloudant storage at all if that were the case. So I'm not 100% clear what model you are trying to achieve.
Regardless, if no credentialSecret is provided, then Node-RED generates one and writes it into runtime settings. When running locally that is in ~/.node-red/.config.json. When running in Bluemix with the Cloudant storage plugin, that is in the settings document in Cloudant.
If Node-RED finds a generated key in the runtime settings and a credentialSecret provided by the settings file, then it assumes the user has just provided their own key and needs to migrate from the generate key to the user provided one. It will delete the generate key as part of the migration.
So I assume you had a generated key in cloudant from a previous run. You've then somehow added flows that were encrypted with your generated key and Node-RED has tried to decrypt it with the generated key as part of that migration. That is why it then worked when you deleted the database - you removed any record of the generated key so it wouldn't try to migrate off it.
What I don't fully understand is how you are managing your flows file. Are you uploading that as part of the deployment pipeline or are you storing them in cloudant? Your settings file is configuring the storage module to connect to cloudant, so any local flows file will be ignored.
In fact, as I mentioned, bluemix has introduced its own pipeline, this is pipeline.yml in ./bluemix:
---
stages:
- name: Build
inputs:
- type: git
branch: master
triggers:
- type: commit
jobs:
- name: Build
type: builder
- name: Deploy
inputs:
- type: job
stage: Build
job: Build
triggers:
- type: stage
jobs:
- name: dev
type: deployer
script: |-
#!/bin/bash
CLOUDANT_NAME="node-red-shared-cloudant"
cf push "${CF_APP}" --no-start
cf set-env "${CF_APP}" NODE_RED_STORAGE_NAME "${CLOUDANT_NAME}"
cf create-service cloudantNoSQLDB Lite "${CLOUDANT_NAME}"
cf bind-service "${CF_APP}" "${CLOUDANT_NAME}" -c '{"role":"Manager"}'
cf start "${CF_APP}"
# View logs
cf logs "${CF_APP}" --recent
target:
region_id: ${CF_REGION_ID}
organization: ${CF_ORGANIZATION}
space: ${CF_SPACE}
application: ${CF_APP}
api_key: ${API_KEY}
I used the "node-red-starter-kit" from the IBM cloud catalog and choose to connect to a cloudant database.
This will setup a git repo and a pipeline much the same way you described in your first blog post.
I checked with boilerplate code from the node-red website (the node-red-bluemix-starter.git) and the code is the same.
So the model I want is: develop locally , deploy remotely, don't modify the flows in bluemix.
My flow.json is part of my project dir ( my git repo) that I push to bluemix, which then starts the pipeline in exactly the same way you described it in your post.
You are saying that couch is configured to get the flow files from the database. This is unfortunately true but not what I want. I want local changes to flow.json to be propagated to bluemix when I push-deploy the repository.
Sorry, if my questions/posts are confusing, but as I said in my post yesterday, I find it quite hard to get started as a newbie. There are simply too many moving targets.
Peter
Hi Peter,
The first few paragraphs of the first post explain what the boilerplate is and what it is and isn't suitable for.
The whole point of the blog post was to show how this can be done without using the boilerplate/starter-kit for all of the reasons it outlines.
Yes you can get a git repo and deployment pipeline from the app, but that app is setup to use cloudant - you don't need that. It has httpStatic setup to serve all the static content and landing page etc - I assume you don't want that. So you'd need to strip out all the bits you don't want to get back to just the bits you do.
Or you can follow the blog post and get just the bits you need.
As you are new to this, I'd recommend doing what the blog post says rather than try to create some hybrid approach that isn't what you want.
If there is a part of the blog post you are not clear on, then do ask.
Ok, thanks Nick, I'll get back to the basics (;
Kind regards
Peter
Nick, thank your for that explanation, it was really helpful and I think I now got what I want.
I am putting toegether a small tutorial for a "develop locally - auto deploy remotely" application, based your first blog post and on the bluemix integraged delivery toolchain for an IoT example with a simple dashboard gui.
One point remains: I want my app remotely to only serve the /ui path and completley block access to editor when running remotely.
Can this be done easily ?
Thanks
Peter
