The soreness is a result of you working your muscles in ways they are not used to. The plyo X on P90X is intense, and I felt the same way afterward.
I would say if your job requires you to climb ladders or something strenuous like that, I would say save your plyo routine for the weekend or a day where you don't have to work immediately the next day. P90X is an all-inclusive workout. You're getting HIIT, strength training and warm-ups/cooldowns. You don't really NEED the elliptical, as it's probably not doing much but burning some extra calories.
If you're really trying to get better at the P90X thing, I would drop the elliptical and incorporate some stretching, some lunges, some core work, maybe some shoulder work, until the plyo circuit gets easier. AND IT WILL GET EASIER. It takes consistency, so keep trying, don't give up!