I have this problem, sort of, on both of mine and they're both factory stock.
On mine it only happens when the T-case is in 4-low and they don't "crash into" each other, it's just light "side-swipe" contact but enough that sometimes it knocks the t-case lever back to neutral which is no fun at all when climbing hills...!
On the xtra cab truck which only has 149K on it, I installed the Marline delrin bushings and that pretty much solved it. I put those bushings in the standard cab and it made it better but with 230K on the tranny there was still enough slop that it was still happening occasionally. So I put a W56 in it and problem solved (levers are different on those).
If you're having major interference you probably have the wrong shifter or as suggested the T-case shifter could be backwards.
