I am not an expert on stuff like this, but since you aren't getting much advise I will offer an idea.
Try putting the axle into the diff without an axle seal in the housing. See if the axle looks very centered in the seal bore. You can wiggle the axle around a little and get a feel for where it is centered in the side gear, more or less just taking the weight off the end of it. By the sounds of your problem, it should show if the axle is pretty far off center.