yeah actually you do. you can't run a 2 ohm speaker system with an amp that is only 4 ohm stable, regardless of running hi or low level inputs.
You are correct that you can't run 2 ohm speakers on an amp that is only 4 ohm stable, BUT the low level input into the amp makes no difference. Low level imputs have nothing to do with the load on the amp.
A lower ohm loading will simply produce more heat in the amp and it's the heat that damages the amp. They are rated so that the cooling is adequate to run at a normal operating temp. At low volumes you won't have a problem. The problems come with the power required to push bass.
One of my best friends in an electrical engineer that enjoys both car and home audio so I've picked his brain on this stuff over the years.