this works just fine when the numbers are positive for example 0x00000f = 15, but when the numbers are negative values - the hex to decimal changes to something like 0xfffffff1 = 4294967281.

Now I know this is because of a decimal from signed 2's complement (because the hex converter I found online said so), and shows the value -15.

I tried looking how to calculate this value correctly to give me a negative number - I saw an answer online that said to write the function like this:

A couple of points. Firstly just making sure that you realise that the //==-3 is just a comment, presumably related to the example you took this from. In javascript anything on a line after // is ignored. So your code is identical to (and slightly less confusing) written as msg.payload = ~~parseInt(msg.payload,16)
It seems initially odd that this does anything at all as the ~ operator is the bitwise NOT operator which flips all 1s to 0 and all 0s to 1. So ~~ flips them and then flips them back again, the net result of which is nothing when you look at the bit pattern. However javascript defines the result of a bitwise logic operation as being a signed 32 bit integer, so you are starting off with a bit pattern that is interpreted as a positive integer, but after flipping the bits and flipping them back again javascript interprets it as signed, giving the result you want.
It is worth noting that this will only work with 32 bit values, so for a negative numbers the value you start with must be 8 hex characters long.
I am interested in knowing what you actually start with in your calculations as it is possible there is a better way. Does your device give you a string of the forrm "0xfffffff1" or is it "fffffff1" or is it a buffer or what?

The output from the device is a string 0xffffffff starting at negative one, then counting backwards 0xfffffffe -2 etc. Positive numbers are normal 0x00000001 and start counting up.

That first example I found didn't really explain itself as to what exact it was doing - thank you for explaining the tildes function, I did understand the comment marks but wasn't quite sure what the reference to "-3" was (hoped someone could explain that).

I'm really really new to programming and I'm just learning as I go. I still am lost with the 2's compliment and the math to make 4294967295 in to -1.

OK, if you are starting with a string then you do need the parseInt(). In fact you can use msg.payload = ~~parseInt(msg.payload)
without the ,16 because the 0x on the front tells parseInt that it is hex.
The -3 comes from the original example you found which I guess was ~~parseInt('11111111111111111111111111111101',2)// == -3
The reason for the -3 in a comment is just the writer telling those viewing the code that for that bit pattern there the result will be -3. The whole text \\ == -3 is just a comment.

That is not really correct. Consider 0xffffffff, which in binary is 32 bits all 1.
Now consider what happens if you add 1 to that. The result is 32 zeros and a carry of 1, but since we only have 32 bits the carry is dropped off the front and the result is 0. So that means that 0xffffffff + 1 is zero. Similarly if you start with 0 and subract one from it you will find the result is 0xffffffff and again the carry drops off the front. So if 0xffffffff + 1 is zero and zero - 1 is 0xffffffff then the inescapable conclusion is that 0xffffffff is -1. Or at least we can choose to interpret it as that.
So to go back to your statement, when using two's complement values it is more accurate to say that the numbers start (or end) with the maximum positive value which is 0x7fffffff then count down to 1 then 0, and on down through -1 (0xffffffff) and on to the largest -ve number which is 0x80000000.