In general, the average person will experience worsening metabolism with age. What else takes place over time that makes it harder to lose weight at 40 versus say 20?
Basing these on my own experience and observation, obviously not all will apply to everyone.
Appetite doesn't slow down with your metabolism.
For many people, alcohol becomes another huge additional source of calories. The binge drinking in college turns into a few glasses of wine or bottles of beer every night, not to mention many people continue the binge drinking on the weekends.
Activity levels tend to naturally decline as people get busy with jobs and kids and other parts of life and aren't obligated to exercise.
Many people end up with low activity or desk jobs, so their metabolic rate (BMR) goes down.
As people age, muscle mass decreases, again decreasing BMR even more.
Continuously easy access to calorie dense foods starts working against us from an early age. Rome wasn't built in a day and neither are excess subcutaneous and visceral fat.
Poor sleeping habits and stress start taking a toll.
So basically, the way that we live compounded with the physiological changes from aging work against us to maintain healthy weight. It's not like some switch just goes off at an arbitrary age. It's a slow decline in metabolism and outside influences that seems to catch up with many people after their 20's.