I used a bearing puller once for one that was really stuck and it came off pretty easy but with some effort.
I have cleaned up the pins before with a little emory cloth but don't make it so the bearing just falls onto the pin.
You still want a fair amount of 'snugness' to the bearing race so it will come apart easily the next time without grenading the upper bearing. So do it slowly and only a little at a time. If you take off too much it's the same as if the bearing were loose or badly worn which will cause you some issues.
Good luck with that.
