Yeah I would build it based off the hitch, but with two supports coming off of each side of it.  I wish I had a picture of the one I built.  But it is basically the same as your ladder idea if I am following it right.  I connected my two side supports with hitch pins. That way I could have the entire setup on and off in less then a minute.  I have towed my truck from CA to UT multiple times without any problems.