Short answer: I would agree with your doctor's assessment unless you already have healthy, strong feet. These would probably be able to take (and perhaps even profit from) hard concrete surfaces. If your feet are not in their best of health, do foot strengthening exercises, and/or walk on softer, natural, rugged surfaces to get your feet into shape first.
Long answer: (See also sources below.)
As others have already said, walking on hard surfaces without any shoes (i.e. cushioning under your foot soles) means an additional impact shock force on your body every time you land one of your feet on the ground. Our bodies apparently need some kind of feedback from taking a step, so that impact force isn't all bad; but it probably shouldn't get too strong. If you hit the ground on your heels, that force will be felt all the way up to your neck; that is, it travels unhindered across all the bony structures and joints in your legs and spine. It's virtually not absorbed by your body at all. That's probably going to give you some back pain very soon, so I would advise you avoid this kind of walking step.
Many people who start to walk barefoot soon automatically develop a different kind of step pattern: They strike the ground with the forefoot, or mid-foot. Because the foot has two arch structures (one along the length of the foot, one across the forefoot, both held in place by ligaments and muscles), these will help absorb the impact shock force. Another place where the shock gets absorbed with this kind of stepping is in the ankle joint.
(Try the difference between walking on your heels and forefoot for yourself: Walk barefoot on a hard surface right now, and you will notice that with a forefoot walking pattern, you don't feel, and hear, each step you're taking nearly as much (perhaps not at all) as the "bump", "bump" of heel walking.)
Now, assuming you've adopted a forefoot step pattern, and that your feet aren't in best health — let's say you have splayfeet (i.e. the arch across your forefoot has collapsed) and an unstable/weak ankle joint, then walking on hard surfaces like pavement could easily lead to problems:
Because of the splayfoot, the forefoot won't be able to absorb much of the step impact, and your ankle joint will have to do more of that work. This could easily lead to an overworked angle, and from there to ankle pain and inflammation.
Again, with splay feet, meaning the forefoot arch won't absorb any of the impact force of a step, this might not just overload the ankle, but also other structures in your forefoot. You might start to feel pain in your foot "knuckles" (sorry, I don't know the proper word for it). When you've been walking barefoot for a while, when you take a look at your foot soles, you should notice that your skin is more used, and therefore thicker, right beneath the big toe "knuckle" and beneach the pinky toe "knuckle". That's how it should be for healthy feet. However, if you also notice considerably thicker skin (pads) beneath the middle toe "knuckles" (which is likely with splayfeet), that might indicate problems to come.
So my advice would be:
Listen to your doctor and exercise your feet mostly on natural, rugged surfaces. (Flat stone doesn't count, even when it's not concrete. :-) Walk on grass, in forests, on sand (but build it up slowly here!); if the terrain is uneven and you have to go up and down, that's even better (but again don't overdo sloped terrain in the beginning).
Perhaps do some foot muscle, ankle, and lower leg strengthening exercises. Strong muscles help relieve and protect your ligaments and joints.
Do go walking on pavement and other hard surfaces, but let your feet get used to it slowly.
Myself. I have been walking barefoot, at some time in the past, for approx. 1 1/2 years non-stop. This included lots of walking in cities, forest walks, climbing a volcano, and some other strange locations. And I've experienced all of the problems I described above.
A wonderful book called "Born to Run" by Christopher McDougall. This is quite likely the best book on barefoot running outside the academic world. It's both well-researched and some good story telling. You can't go wrong with that book.