When SS says "don't worry about training while sore", they mean "sore from two days ago when you were squatting and oh man it was heavy". They don't mean "go screw with your strength gains by doing something else". That said, if you're OK with sub-optimal strength gains, go ahead and do other stuff. That's your choice. For many people optimal strength gain does not trump all.
I would definitely consider that much swimming immediately after lifting to probably interfere with a lifting program. You should be tired after those squats. Tired enough to not want to go swimming. Tired enough for the swimming to take a bite out of your recovery from said squats. If you want to swim, do it on the off-days after lifting, and recognize that it will throw a wrench in your linear progression fairly quickly.
I consider walking on the off-days a great idea. It's recommended to keep active at a low level on rest days, and this is perfect. You get your joints lubed up, your blood pumping gently, but it's not timed or draining enough to conflict with recovery.