I only click on them when I try to click on something else, and right at the same time something else downloads onto the page (pic or something) and the page shifts under the mouse.
I do think its probably a matter of what it is, and where its at though. Like on pirate, if I see a 12voltguy.com banner, and I'm interested in his stuff, I'd click it mabye, but only if I'm looking/thinkg about it at the time