Does it also run rough, too? If so, I'd say bad diaphragm in booster causing a vacuum leak.
You can use a vacuum pump (hand held type) and see if the booster hold vacuum.
I seem to remember a "anti-backfire" one way valve in the booster vacuum supply too.....hmmmmm. Maybe is just the plastic one way valve im thinking of.
I'd vacuum check the booster first though.
