not sure if this could be an issue, but is it possible the stick is hitting sheet metal of the floor? I have heard of that issue on otehr vehicles, and that could make the vibrations shift itself
I had this issue when I did a bodylift on my old 2wd hilux