Sun cream can cause this (also helps turning the pool green) - make sure you/guests shower before swimming.  

Also general dust - try a pool cover for when the pool isn't in use.

I've ended up doing the same - pump it all to waste as otherwise it just gets pumped straight back into the pool but since using the cover I've not had any problems.

Nigel