I don't think the size is necessarily and issue - though check the OS to see if SWAP is being used. If it is then you need more RAM or need to remove other things from memory. Make sure you aren't running a desktop on the Pi.
I suspect that the 100ms is more of an issue.
There are lots of other things you could potentially do to optimise performance but it is hard to comment with such little to go on.
Lets start with the SWAP issue anyway, if you can report back on that, maybe we can help hone in on other things that could be optimised.
