(1)
You have two days of intense muscle building workout, which are upper body and legs. I wouldn't call core-day as a muscle building day (unless you are doing deadlifts on core-day!?). The core regenerates pretty fast so you dont need much rest for that day. For the legs and upper body you definitly need rest. But rest day is more meant like a day where you dont train the muscle hard again. That doesn't mean that you can't do any activities like swimming, skipping, etc. Actually acitvities on rest days are extremely good for your regeneration and recovery. Doing nothing on rest days and sleeping all day long will just slow your metabolism (I've done that wrong for a long time). Remembering your muscle of activity will instead lead to better performances in your next training. So you are doing quite well.
But...
(2) you are doing two half-leg sessions (swimming) before your leg day. So you're probably tired on friday and can't bring your whole strength into your workout. Which means your will not shock your leg muscle as hard as you would do if you wouldn't have done two cardio workouts the days before. This could mean that your legs doesn't have enough time to recover. Nevertheless, you will probably get stronger, but it could end in extrem tiredness. But over time your body will get used to it.
(3) So to conclude it is totally ok to activate your muscles between muscle-building days. But don't overdo it so your muscles can go with full strenght into the next workout. Because if not, your goal of gaining muscle weight will take much more time.