mouse lag

I am chasing some mouse lag that happens after being up for a while. I thought it was due to firefox memory use and memory paging so I reduced those and now have reduced that even though I have 64G of memory. Not only that, but I switched from Weyland to Xorg and I'm still getting some lag a couple of days after a reboot. What else should I be looking at?