I did the same thing. I just used the drive shaft from the 87 toyota that my transmission came out of to put it in my 81. That way no flange swapping and no driveshaft shortening
But if you don't have one of those I'd swap flanges instead of trying to drill it
