I used to tow a 24' ski boat w/ my '82 2wd and never had an issue with my brakes. As mentioned as long as you let the 5-spd drivetrain do the work for you, and give yourself more distance for emergency braking(as with most types of towing), you should be fine.
As far as being "That slow guy towing on a mountain road", when it is me, I just figure it is payback for all the slow towing rigs I get stuck behind.
Just pull over when you can and let those behind you pass, that should keep most of them happy.
