Rate Limits for HTTP requests

[lizzardguki]

@lizzardguki

I have a basic OAuth flow to authenticate with 3rd party API, after the authentication is successful, I use a delay node to wait for x seconds (expires_in property - 100 seconds) and then I push the message to refresh the token, and I am trying to keep that loop going, but after a couple of token refreshes, the node-red fails to authenticate. Is there any better way to make the flow?

So the problem I had in the previous post was caused by the API rate limit.
so string to figure out a design for rate limit, after each message I receive back headers X-RateLimit-Reset & X-RateLimit-Limit (Airthings for Business).
Does anyone have any idea?
Here is what i have tried: `

[
    {
        "id": "f300838f7519a85a",
        "type": "link in",
        "z": "f871c3c06ed134b9",
        "name": "http",
        "links": [],
        "x": 55,
        "y": 560,
        "wires": [
            [
                "f0bc4da034fa8851"
            ]
        ]
    },
    {
        "id": "5bb784d67a54953a",
        "type": "switch",
        "z": "f871c3c06ed134b9",
        "name": "",
        "property": "x-ratelimit-remaining",
        "propertyType": "flow",
        "rules": [
            {
                "t": "lt",
                "v": "100",
                "vt": "num"
            },
            {
                "t": "else"
            }
        ],
        "checkall": "true",
        "repair": false,
        "outputs": 2,
        "x": 290,
        "y": 560,
        "wires": [
            [
                "f42159788a0e1452"
            ],
            [
                "6096b02a49a94961"
            ]
        ]
    },
    {
        "id": "f42159788a0e1452",
        "type": "function",
        "z": "f871c3c06ed134b9",
        "name": "",
        "func": "var o = RED.util.cloneMessage(flow.get('o')) || 0\nvar q = msg.payload\nmsg.url = msg.payload.url  \nmsg.method = msg.payload.method\nmsg.headers = msg.payload.headers\nmsg.payload = msg.payload.data\nif (o < 4900){\n    // @ts-ignore\n    o++ \n    flow.set('o',o)\n    msg.delay = 0\n    return msg\n}\nelse {\n    var time = RED.util.cloneMessage(flow.get('x-ratelimit-reset'))\n\n    // @ts-ignore\n    msg.delay = (time * 1000)- parseInt(Date.now())\n    return msg\n}\n",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 420,
        "y": 540,
        "wires": [
            [
                "b698a49fbba89065"
            ]
        ]
    },
    {
        "id": "b698a49fbba89065",
        "type": "delay",
        "z": "f871c3c06ed134b9",
        "name": "",
        "pauseType": "delayv",
        "timeout": "5",
        "timeoutUnits": "seconds",
        "rate": "1",
        "nbRateUnits": "1",
        "rateUnits": "second",
        "randomFirst": "1",
        "randomLast": "5",
        "randomUnits": "seconds",
        "drop": false,
        "allowrate": false,
        "outputs": 1,
        "x": 600,
        "y": 540,
        "wires": [
            [
                "428e8c9f431bf3d3"
            ]
        ]
    },
    {
        "id": "4a554b189c583933",
        "type": "switch",
        "z": "f871c3c06ed134b9",
        "name": "",
        "property": "statusCode",
        "propertyType": "msg",
        "rules": [
            {
                "t": "eq",
                "v": "200",
                "vt": "num"
            },
            {
                "t": "else"
            }
        ],
        "checkall": "true",
        "repair": false,
        "outputs": 2,
        "x": 930,
        "y": 580,
        "wires": [
            [
                "bb7730d81242ae7e"
            ],
            [
                "cf485a655de75706"
            ]
        ]
    },
    {
        "id": "428e8c9f431bf3d3",
        "type": "http request",
        "z": "f871c3c06ed134b9",
        "name": "",
        "method": "use",
        "ret": "obj",
        "paytoqs": "ignore",
        "url": "",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "",
        "senderr": false,
        "x": 770,
        "y": 580,
        "wires": [
            [
                "4a554b189c583933"
            ]
        ]
    },
    {
        "id": "f0bc4da034fa8851",
        "type": "function",
        "z": "f871c3c06ed134b9",
        "name": "",
        "func": "var o = parseInt(RED.util.cloneMessage(flow.get('x-ratelimit-reset')))\nvar e = RED.util.cloneMessage(flow.get('x-ratelimit-reset-old')) || 0\nif (e == 0){\n    e = o\n}\n\nif (o<=e){\nreturn msg;\n}\nelse {\n    flow.set('x-ratelimt-reset-old',o )\n    flow.set('o', 0)\n    return msg;\n}\n",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 160,
        "y": 560,
        "wires": [
            [
                "5bb784d67a54953a"
            ]
        ]
    },
    {
        "id": "cf485a655de75706",
        "type": "link out",
        "z": "f871c3c06ed134b9",
        "name": "",
        "mode": "return",
        "links": [],
        "x": 1235,
        "y": 580,
        "wires": []
    },
    {
        "id": "6096b02a49a94961",
        "type": "function",
        "z": "f871c3c06ed134b9",
        "name": "",
        "func": "msg.url = msg.payload.url\nmsg.method = msg.payload.method\nmsg.headers = msg.payload.headers\nmsg.payload = msg.payload.data\n\nflow.set('o',0)\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 420,
        "y": 580,
        "wires": [
            [
                "428e8c9f431bf3d3"
            ]
        ]
    },
    {
        "id": "bb7730d81242ae7e",
        "type": "function",
        "z": "f871c3c06ed134b9",
        "name": "",
        "func": "\nflow.set('x-ratelimit-remaining', parseInt(msg.headers['x-ratelimit-remaining']))\nflow.set('x-ratelimit-reset', msg.headers['x-ratelimit-reset'])\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 1080,
        "y": 560,
        "wires": [
            [
                "cf485a655de75706"
            ]
        ]
    }
]

[lu4t]

I believe what you're experiencing is how OAuth works. Refresh tokens are supposed to be used just once, and an auth server will typically set measures of control to prevent a leaked token is used by someone which was not the original requester. This is done usually not allowing the usage of the refresh token more than once, setting it to "expired" after used.

Your approach is correct: you should request a new access token when you get close to the expiring time of the one you're using. But bear in mind there's one thing you don't control (neither know): when is the refresh token expiring? you might be using a refresh token already expired... this is something the auth server won't tell you.

When I look at your flow, I don't see where are you handling the fact that the refresh token is changing after a new access token is granted... I hope it makes sense.

[lizzardguki]

I don't think i have the problem refreshing the token more than once. Because i receive a new refresh token when i refresh the token (makes sense?).
The problem i have is i reach the api limit, and when the refresh token flow is triggered i don't have anymore api requests left to refresh the token.

So i need to figure out a way to have a linked flow, called with a link call node, so i can keep track of the number of requests i sent. And if i don't have anymore left, push them to a delay node...