I think it's just based off reputation. "Empty calories" basically means just "all of the calories with none of the benefit".
So in the case of simple sugars, they're more associated with candy. Pop rocks, jellybeans, gummies, rock candy, etc. are all made of pure sugar. Soda is also another source of "empty calories". They do not have any nutritional micro-nutrients which are vitamins and minerals. Yet they are a very concentrated source of calories. Pair that with the fitness industry's demonization of calories, you get the idea that you shouldn't waste your limited calorie budget on "empty calories".
Complex carbs are more often found in fruits and vegetables and hence are not "empty". That's not to say there aren't simple carbs in fruits, but the concentration is considerably less. Also when people think of "sugar", they don't think of "bananas".
Complex carbs do, however, increase insulin. As does protein in fact. The digestion rate is slower than simple carbs, so rather than a spike you get an elevated increase over time. One of the reasons that simple carbs are far less satiating.
There are actually a lot of people who argue that carbs primarily should be avoided regardless of source. Particularly Keto community. Then the IFIYM (It Fits Your Macros) community argues that a "carb is a carb" and the source of it doesn't matter as long as you get your macros in. Though since the body doesn't run on only macros, it's best to get your carbs mainly from "non-empty" food sources.