I guess it is all in what you like, I don't think I would mind the extra weight and I would still get better mileage than my stock 3.0. I think the Toy tranny, t-case, and axles would hold up for awhile if you have a light foot. I don't think the power to weight ratio applies with a diesel, my dad has a 6bt Cummins in his Dodge and the motor alone weights in at 1100lb, I can't remember the exact truck weight, but it is damn heavy and I can hit 80MPH no problem and its take off is incredible for a truck of its size.
The 4bt only has 70hp less and 70lb-ft less than the 6bt, so I think it would make a Toyota move alone like snot.