As you have seen already by yourself, pullup resistors are vital for the i2c bus, both in the sda and scl lines. It is usually obvious that the sda line is bidirectional, what is not so obvious is that the scl is also somewhat bidirectional, since slaves can maintain the line low to throttle down the speed of the bus.
In many cases the bus can become locked because of a collision or other physical problem like loosing your pullups, in such a case, some slaves don't release the bus properly because they understand that they are still being addressed. This situation can be fixed by the master sending output SCL pulses and finally a stop condition when the lines are released. However if you say that there are no slaves, this probably is not your case.
I'm afraid that your problem might have to do with the master's i2c peripheral not seeing the lines go high when expected because the pullups are disconnected temporarily. The best solution I can think in your case is that you should make sure that you don't loose those pull ups. The best way to achieve this is to connect two or three resistors in parallel in each line, this way all two/three would have to be disconnected to leave the bus "unpulled". Use two 10k resistors in parallel for SDA and another two for SCL.
