Inserting integers into mongo db with node-red-node-mongodb module

[CreativeWarlock]

Hi guys,

I'm trying to insert a (long) series of timestamps and values into my local mongo db,
but rather than saving the timestamps as integers they are saved as doubles.

The function I am using contains the following:

var now = new Date().getTime();

msg.payload = {
    "symbol": "MySmbol",
    "data": {
        "timestamps": [ now, now+100, now+200],
        "values": [Math.random(), Math.random(), Math.random()]
    }
}
return msg;

According to the mongoDB documentation I can make use of the NumberLong function, see Data Types in the mongo Shell — MongoDB Manual

However, when I use NumberLong in the assigment for timestamp:

        "timestamp": [ NumberLong("'" + now + "'")], // "NumberLong('" + (now+100) + "')", 

... I receive this error in the
"ReferenceError: NumberLong is not defined (line 6, col 22)"

And when I use this line:

"timestamp": [ "NumberLong('" + now + "')", "NumberLong('" + (now+100) + "')", "NumberLong('" + (now+200) + "')"],

the inserted data in each timestamp field looks as follows

timestamp:
[0] - NumberLong('1611424544678')
[1] - NumberLong('1611424544778')
[2] - NumberLong('1611424544878')

...which is what I neither want.

Anyone having an idea how to get integers into the mongo db?

My flow looks as follows:

You can copy the flow here:

[{"id":"1aa86ea6.d665a1","type":"debug","z":"9ba23ed4.19e7f","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":550,"y":540,"wires":[]},{"id":"4207f55c.7ac6e4","type":"function","z":"9ba23ed4.19e7f","name":"insert","func":"var now = new Date().getTime();\n\nmsg.payload = {\n    \"symbol\": \"Usa500\",\n    \"data\": {\n        \"timestamp\": [ NumberLong(\"'\" + now + \"'\"), NumberLong(\"'\" + (now+100) + \"'\"), NumberLong(\"'\" + (now+200) + \"'\")],\n        //\"timestamp\": [ now, now+100, now+200],\n        \"values\": [Math.random(), Math.random(), Math.random()]\n        //\"high\": [Math.random(), Math.random(), Math.random()],\n        //\"low\": [Math.random(), Math.random(), Math.random()],\n        //\"close\": [Math.random(), Math.random(), Math.random()],\n    }\n}\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":310,"y":600,"wires":[["1aa86ea6.d665a1","1d542582.bcb82a"]]},{"id":"28f711dc.7848be","type":"inject","z":"9ba23ed4.19e7f","name":"Inject","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{}","payloadType":"json","x":150,"y":600,"wires":[["4207f55c.7ac6e4"]]},{"id":"1d542582.bcb82a","type":"mongodb out","z":"9ba23ed4.19e7f","mongodb":"bab2560a.6eb3c8","name":"Insert into Test1","collection":"Test1","payonly":true,"upsert":false,"multi":false,"operation":"insert","x":580,"y":600,"wires":[]},{"id":"bab2560a.6eb3c8","type":"mongodb","hostname":"kaspar","topology":"direct","connectOptions":"authSource=admin","port":"27017","db":"SYMBOL_HISTORY","name":"MongoDB"}]

Cheers
Marcel

[UnborN]

Hi @CreativeWarlock

@CreativeWarlock

but rather than saving the timestamps as integers they are saved as doubles.

I think by default mongodb saves big numbers as Double .. Whats wrong with doubles ?

Another query i have is why dont you save your timestamp as a Date and convert it with
.getTime(); afterwords when you read it from the db if you have to.

[CreativeWarlock]

With billions of series values that have to be stored in the mongo DB it's important for me to not waste space with doubles.

If you allow me to get back to my actual problem - anyone having some ideas?

[Steve-Mcl]

@CreativeWarlock

With billions of series values that have to be stored in the mongo DB it's important for me to not waste space with doubles.

You realise MongoDBs NumberLong function creates 64bit values and that JS Double / MongoDB default double is 64 bit also?

As for why NumberLong doesnt work in your 2 examples...
Ver1: The NumberLong function doesnt exist in node-js (nor in node-red run time) hence the error NumberLong is not defined
Ver2: The value of timestamps is an array of strings (they get inserted as an array of strings as you generated e.g. this "NumberLong('1611424544678')" is generating a string)

---

There may be a way around this but my knowledge of MongoDB is quite limited
Options:

• (longshot) Try using 64bit integers e.g. BigInt(now)
  • NOTE: you must be on nodejs V10.4.0 or greater for BigInt support.
  • MongoDB might not understand or support bigint
  • If at any point the JS object is converted using JSON, this will fail (JSON.parse doesnt support BigInt)
• Raise an issue on the github repo requesting support for NumberInt, NumberLong and NumberDecimal functions
• Live with the default double types created when using "timestamps": [ now, now+100, now+200]

[UnborN]

The link you sent above is from Mongo Shell .. mongo Shell has access to those internal mongo functions
like NumberLong.

The node-red Mongo node doesnt (expose) them especially in a preceding Function node.

A workaround would be to edit your settings.js file in the .node-red folder and under functionGlobalContext add the following ..

functionGlobalContext: {

        // os:require('os'),
        // jfive:require("johnny-five"),
        // j5board:require("johnny-five").Board({repl:false})
      //  moment:require('moment'),
        ObjectId:require('mongodb').ObjectID,
        NumberInt:require('mongodb').Int32,
        NumberLong:require('mongodb').Long

    },

And in the Function node in order to use them :

var now = new Date().getTime() //.toString();

let NumberInt = global.get('NumberInt')
let NumberLong = global.get('NumberLong')

msg.payload = {
    "symbol": "Usa500",
    "data": {
        //"timestamp": [ NumberLong("'" + now + "'"), NumberLong("'" + (now+100) + "'"), NumberLong("'" + (now+200) + "'")],
        "timestamp": [ NumberLong(now), NumberLong(now), now],
        "values": [Math.random(), Math.random(), Math.random()]
        //"high": [Math.random(), Math.random(), Math.random()],
        //"low": [Math.random(), Math.random(), Math.random()],
        //"close": [Math.random(), Math.random(), Math.random()],
    }
}
return msg;

The Mongodb node uses the Node js MongoDb Driver and the documentation to it is here instead

Also read this interesting article

I did try all this before commenting yesterday and surely my intension wasnt to sidetrack from the actual problem

[EDIT]
i made a mistake above .. i think you use NumberLong.fromNumber(now) to force save it as Long.

As Steve explained, its the same as using Double .. its the same amount of space in the Db?!
Im new to mongodb and its data types .. for sure Int32 isn't big enough to store your timestamp so ..

[CreativeWarlock]

Thanks so much for your elaborate reply, UnborN! Also thanks to you, Steve.

I just added the required mongodb classes to functionGlobalContext, adjusted my node red function node and voila, everything works perfectly! Timestamps are stored as Int64 in the mongo db.

Being new to mongoDB myself I did not start with the right documentation. Thanks for leading me to the MongoDB driver API for Node.js.

Yes, using the method NumberLong.fromNumber(..) and "forcing" timestamps to be saved in Int64 representation is indeed what I want. So, that there's no need to worry in 2038 where timestamps with Int32 will reach their limit.
(e.g. discussion here: mysql - How should unix timestamps be stored in int columns? - Stack Overflow)

Again, thanks to you all!

Cheers,
Marcel