🎉 Refresh tokens in node-red-node-dropbox (Now published as v2.0.0)

[BartButenaers]

Hi folks,

Last week @Paul-Reed reported an issue with the dropbox node in another discussion (and here). Since Paul has always been ready to assist me with my electronics at my house, I was happy to help him in turn...

The issue

In the past you could access the API of a Dropbox app via an access token, and that token would never expire. So this access token had to be simply entered in the config node's config screen:

However starting from September 30th 2021, all new access tokens are only 4 hours valid and then they expire. From that moment on you had to generate a refresh token, by having the user to allow the node to access the API of the Dropbox app (via an oauth2 authorisation flow). That refresh token will be infinite usable, i.e. it will never expire. As soon as a short-life access token expires, the long-life refresh token will be used to generate automatically a new access token. Then that new access token can be used to access the API of the Dropbox app.

Solution

I have created a fork of the dropbox node, that allows user to enter their refresh token instead of an access token:

The step-by-step list on the config screen has been reorganised, containing some hyperlinks that call new endpoints to make the entire process a bit less complex for the users.

Paul has done a series of tests. And now we would like to discuss it here, before we create a pull request.

Description of the implementation

To solve this issue, I have created 3 new endpoints in this node.
Hopefully the following diagram gives a understandable overview of the entire flow:

1. As soon as the config node's config screen is openened, endpoint A will be called automatically to determine the redirect url. That url will be displayed in the tips.
2. The user clicks on the hyperlink to create a new app on the Dropbox developer website.
3. The user manually enters on the Dropbox website the app properties, like e.g. the app permissions
4. The user manually enters on the Dropbox website the redirect url from the config screen.
5. The user manually enters the app key (also called client id) in the field of the config screen.
6. The user clicks on the hyperlink to start an authentication flow, which calls endpoint B to generate a Dropbox authentication url.
7. The config screen opens automatically that authentication url in a popup window. The first time this is being executed, the user manually will have to confirm (in two successive Dropbox windows) that he allows the Node-RED node to access the API of his Dropbox app.
8. At the end of this authentication flow, Dropbox will call endpoint C via the redirect url (from step 4). When the user has allowed the app to access the API of the app, then the refresh token will be passed to endpoint C.
9. Endpoint C will pass html to Dropbox, which will be rendered by Dropbox in the popup window. Via that html, the Node-RED node will display the refresh token.
10. The user manually enters the refresh token in the field of the config screen.
11. Once deployed, this node will use the refresh token to generate new access token. That will happen automatically (by the Dropbox SDK) every time the API of the app is accessed, and the previous access token has become invalid (after 4 hours).

Breaking change

Access token generated before September 30th 2021, will never expire. So if you have such an old token, you won't need (at least currently) a refresh token.

To allow people to keep on using their old config token, the first version of my config node screen still contained that access token field.

But two input fields on the config screen (one for the access token and one for the refresh token) was very confusing for all other users that don't have an old acces token. Because they only have a refresh token, and the access token field on their config screen is not used. Their access tokens are generated (based on their refresh token) on the server side, and will never end up in their config screen...

Therefore we have removed the old access token input field from the config screen. But of course that means that the old access tokens won't be usable anymore. Which is not a real problem, because for all users the refresh tokens are the secure way to go anyway!

The end ...

As usual all constructive feedback is very welcome, and we hope the community likes this proposal.

Thanks!
Bart (read "highly underpaid subcontractor of Paul" )

[Paul-Reed]

Although this introduces a 'breaking change' for the users lucky enough to have access tokens generated before September 30th 2021 (that's me!), Bart's node update to Oauth2 will provide a future for the Dropbox node, which has a diminishing number of users, due to the short-lived-tokens making the node currently unusable.

But the work needed by users to update & use the proposed node update is minimal and easy to implement. No changes to user's flows, just an easy 5 minute change to the node auth.

Contrary to what Bart has said above, I'm his alpha tester, because as a non-programmer, and not really aware of what goes on below the hood, if I can make it work - so can other users

On behalf of the node-RED community, I would like to thank Bart for providing this update, I'm sure it wasn't the easiest, and has no doubt consumed a lot of his family life over the past few days.

[craigcurtin]

@BartButenaers as usual is a bloody Legend !!!

Yes go ahead and break away !!! (Even though i also have one of the old tokens too)

Well done Paul for pulling this all together also

Craig

[dceejay]

Naive question. If the old auth token is still in place can that be detected and used without breakage ? Ie old flows keep running, and only require new token for new flows ?

But yes please Bart to this PR

[Paul-Reed]

@dceejay

If the old auth token is still in place can that be detected and used without breakage

We discussed that point, but the following weighed against it;

1. Dropbox now consider the long-lived-tokens depreciated, and although they haven't currently stopped them being used, it can't be ruled out. When I asked the Dropbox team, they said 'not at this time'. If they did, then user flows using the old tokens could come to a very abrupt end.

2. Behind the scenes, the process is fairly complicated to obtain the access token, but Bart has kept the user config very simple, with just 2 boxes for users to complete. Retaining a 'access token' box for legacy purposes may confuse users, because the new Oauth2 process does not store the access token, it's re-created each time the node is run, and therefore cannot be displayed in the config.
   

3. It's not as though users would not have a upgrade path, which only takes about 5 minutes to complete.

4. API v2 has been introduced to increase security for dropbox users, not just with the addition of PKSE, but with scoped permissions, so access can be limited to just the app storage space allocated, and not the whole account storage. This should be a big motivator to update.

EDIT - if the node is updated where the user is currently using a legacy access token, the debug panel shows;

Maybe a message advising users to check their dropbox node config settings, may be helpful instead.

[dceejay]

Paul

I was meaning not to have both options available in UI - so UI only has new keys, and as soon as you edit you have to upgrade ... but old key would keep working if it was present... but could also log a warning to the log etc... But hey - as this will be a major version bump anyway we can break it now... not the end of the world, and definitely a major step forwards.

@BartButenaers please add your self to the contributors in package.json and bump the version to 2.0.0 before you raise the PR

[HaroldPetersInskipp]

Yes, I think it will be less confusing going forward "not to have both options available in UI" as well.

[Frida]

Works like a dream. First I had to allow popup in safari so I could get my Refresh Token.
After updating, the dropbox node became a small square, but it worked.
A new dropbox node keeps the size.

[Paul-Reed]

@Frida

After updating, the dropbox node became a small square

Hi Frida, did you try refreshing your browser (it sounds like a browser caching issue).

@Frida

First I had to allow popup in safari

Good point!

[Frida]

To get it working on my raspi, I had to open the browser in the raspi because only local is allowed to get my Refresh Token.
Since everything was set up locally, I was able to back up again from mac safari.

[BartButenaers]

@dceejay: indeed my first reaction was also to keep the old access token somehow on the screen, so users could keep on using their old access token. Or that you have a dropdown on the screen where you can select following options:

• "access token": which is by default selected for nodes that have an old access token, so people can keep on using their old token (i.e. not breaking existing flows).
• "refresh token": whick is by default selected for new nodes.

But not sure if it would be a good decision:

• I think it will be confusing, especially since refresh tokens also result in access tokens. But those access tokens are generated under the cover and have nothing to do with the access token field on the config screen.
• Seems a bit wrong to me, to do extra efforts to offer users a less-secure solution. Although Dropbox also allows users to keep on using their old tokens, without expiration. Although that might end some day.

So I am not sure about what would be the best solution.
Personally I like to keep the config screen simple (with only a refresh token field), because I think that most users get nuts of tokens and all related security stuff...

@Frida

the dropbox node became a small square, but it worked.

Heuh, do I see this correctly in your screenshot:

The image that I had share above (to explain how this mechanism) works, appears as an icon in your node (instead of the dropbox.png file). This is the moment that I need to buy new glasses or visit a psychiatrists...

@Frida

First I had to allow popup in safari

Yes I had the same problem in Chrome last week, so I had added this to the config node screen:

But apparently that is not clear enough. If anybody has a better proposal, please let me know.

@Frida

To get it working on my raspi, I had to open the browser in the raspi because only local is allowed to get my Refresh Token.

I wasn't aware that only local was allowed. Will need to to find some time in the weekend to try this myself. Thanks for testing this!!!!

Found some other issue. Tonight I had a "connection lost" on my flow editor, while the config screen was open. This resulted somehow in an alert box with the text "undefined", and the redirect url was empty:

Will need to have a look at that also...

[BartButenaers]

@Frida

To get it working on my raspi, I had to open the browser in the raspi because only local is allowed to get my Refresh Token.

@Frida,
Quickly tried it my Raspberry, which I access here via an ip address in my browser url:

When I try to add that redirect url in my dropbox add, it fails because Dropbox only allows http for localshost:

When I don't add this new url, I cannot work using the localhost url (which I already on my dropbox app for my previous tests on my portable) because it fails:

Is that the problem that you are referring to?

While I was implementing this feature, I had first an idea to use always localhost. But at the end I thought it was better to use the same hostname which you use to navigate to the flow editor. Perhaps it is better if I always use localhost. Not sure if that works however in setups like Docker containers, because I received another ticket some time ago about localhost in Docker (for one of my other nodes).

Background info for techns following this issue: the redirect url is called by the dropbox sdk, which is running of course on the same host as the dropbox node. So the sdk can allways call the endpoint of the dropbox node via a localhost callback url. Just need to make sure that the protocol (http/https) and other basepath abnormalities (adjusted by the user via settings.js file) are taken into account I assume (e.g. https://localhost:9999/some/extra/stuff/dropbox/authenticate).

Now off to the daily job...

[Paul-Reed]

I seem to recall that during early testing, localhost didn't work for me either (we discussed in a DM some days ago)

[Frida]

@BartButenaers

Is that the problem that you are referring to?

"First I had to allow popup in safari"
Your instructions are clear enough. It was just information
about what I did.

"Is that the problem that you are referring to?"
Yes, if I open with 127.0.0.1:1880, or open with localhost:1880,
then it is what I open with that must be inserted in redirect URIs.
And that can only be done in the browser on the local machine.

[BartButenaers]

@Paul-Reed

localhost didn't work for me either

Ah yes, now I remember.
My last proposal above to have the node always to use localhost, was pure nonsense...

As @Frida already experienced, the hostname in the dropbox redirect_uri must match the hostname used to access the flow editor.

1. If you want to use localhost in your redirect_uri, then you need to use the browser that is installed on that host. Because the browser installed e.g. on a raspberry, can access local resources on that same raspberry via localhost.
2. But normally if you access your flow editor via https://my_rpi_name:1880/some_path, then the redirect uri should be https://my_rpi_name:1880/dropbox /authenticate

Am I correct that there is only a problem for Node-RED systems that fulfill the following 2 conditions:

1. The system is not accessible via localhost, because there is no browser installed on that host.
2. And SSL is not enabled (via the Node-RED settings.js file.

Because dropbox requires https, unless you use localhost.

Not sure if there is anything I could do about that, to be honest...

[borpin]

I think this is similar to the issue with the FitBit node not authenticating correctly. I ended up Forking the node from GitHub I found and got it working that way.

If this is a common method used to authenticate, perhaps there should be a more generic way to do the authentication that other nodes could then use.

@BartButenaers - yes FitBit has this same issue (localhost).

[BartButenaers]

@borpin,
Thanks for sharing your experience!

Am I correct that I cannot workaround this?

1. The user navigates first to dropbox (in this case in a popup dialog window), where you allow the app (in this case the dropbox node) to access the files on dropbox.
2. Dropbox redirects the browser to the redirect_uri.
3. The browser (or more specific in this case the popup dialog window) will call the redirect_uri. Which of course will fail if the redirect_uri contains localhost, but the browser is not hosted on that same (local)host.

Which means that you can only get a refresh token when you use https or you use localhost (via a browser hosted on that same host). Once you have a refresh token, the ssl is not needed anymore to get periodically a new access token.

Is that all correct?

[stephenn]

I am confused on how to install your version vs the original. The page you linked for your fork has instructions to install the original version. Newbie here. What am I missing? Is it as simple and downloading the dropbox.html, dropbox.js, and package.json files and pasting them into the node-red-node-dropbox folder to replace previous version then restart Node Red?

[Paul-Reed]

@stephenn

Is it as simple and downloading the dropbox.html, dropbox.js, and package.json files and pasting them into the node-red-node-dropbox folder to replace previous version then restart Node Red?

That's what I did, and you need to replace the following files;

dropbox.html
dropbox.js
package.json
locales/enUS/dropbox.json

It's made difficult because the dropbox folder is in a repo shared with many other nodes
Maybe @dceejay can suggest an easier way.

I've been running @BartButenaers's fork for the past month, with lots of test uploads, backing up folders 3 times a day, and it's worked reliably, and without any intervention whatsoever.

Let us know how to get on

[EDIT - ] @DiverRich - The package.json file has also been updated to include the very latest dropbox library, so after updating the file, run npm update from within the dropbox folder to update the library.

[dceejay]

I thought @BartButenaers had submitted a PR and we had merged it or were we waiting for something ?