I feel like there's a fairly wide variety of potential answers here, so I'll throw a few of them out.
It's not easy at first
As someone who has picked up biking to work probably a half dozen times over the years, that first week or two can be agony or at least great frustration. You find yourself out of breath a few minutes in. You have to dismount to get up a small hill. You find yourself thinking that you're good as long as there's just one more mile, then you realize that you're only halfway along a four mile commute. Some of this is a matter of getting into shape. Some of it is building the right muscle memory (for biking, it's especially important to learn to rely more on a high cadence at a higher gear than to power through a lower gear due to various vagaries of how it expends energy and builds fatigue in the body). Frankly, a good bit of it is psychological; you may find that if you get distracted while riding, it all goes by much faster than you think.
Once you're in better shape, there are exigent circumstances
Once you're in decent shape, that commute will probably be pretty easy. But what about a day when it's unseasonably hot or cold? Or starts raining? Or you had one too many beers the night before or didn't sleep well? One could argue that these exceptional cases shouldn't discourage people, but when they do happen, they feel disproportionately discouraging.
There are additional stressors beyond the physical
Commuting via bike means you need to worry about upkeep. You need to pack supplies for repair, and if your bike does break down, you're now toting along a heavy weight on the walk to work, if you don't wind up having to call for transportation which may or may not accommodate the bike. And lastly, you have to deal with other people on the road whether it's a bike path (the cyclists are usually not that bad, but slow-moving crowds of pedestrians have a tendency to move unpredictably, and a collision with one of them is as likely to injure or kill you as running into a car) or the road (if you're lucky, you just have to deal with traffic, exhaust, and people passing a little too close. If you're unlucky, you get people yelling, people spitting, and occasionally people swerving to try to make you crash, or outright hitting you).
Even in good shape, there is a price to be paid
There was an excellent essay I read a few years back arguing that people should be willing to pay an extra $2000 for every mile closer a house was to their place of work. Time is money, as the saying goes, and more importantly, your time is worth something to you. When I ride public transportation, not only do I get to work more quickly (and return more quickly), letting me use that saved time productively, but I can also set an alarm on my phone and catch a catnap, or read a book, during my commute.
