"Fitness" is a fairly broad term, but I'd provide a boiled down definition that it means your body's ability to handle physical exertion. Exertion comes in all shapes and sizes (moving a piano, walking across a city, etc), so being "fit" enough to do those things depends on the type of training and conditioning you're doing.
Most people want to train in such a way that maximizes the benefits across multiple categories like fat loss, improved image, being less injury prone, reducing illness, and increased general athleticism.
Additionally, training is built on the concept of adaptation, and adaption is specific. Again, most people are best served by picking the training (that requires your body to adapt) that yields the most benefits with the shortest amount of time. This is what divides effective fitness programs from less effective ones.
If I continue to [walk 5 flights] once a day will the breathlessness improve
(I'm not sure of the official term for this, but what I mean is, will
normal breathing return more quickly, will I sweat less, that sort of
thing)?
Yes. Your body will adapt to the workload, and you will get better at it. There will also be some carry over to walking and anything requiring step-ups. Adaptation is specific.
Would I need to be performing this small exercise a lot more
frequently for it to make a difference?
The more you do it (barring injuries and overtraining) the bigger and faster the adaptation will happen. Adaptation is specific, so someone walking 20 flights of steps will be adapted to those, versus you doing your 5.
In practical terms though, even if you want to walk those stairs ten times a day, you're probably best served initially by doing it once or twice per day and ratcheting up from there. Another aspect of training is that you want to put yourself right into "too much" without going into "ouch, I'm hurt" mode. It's often nicknamed "good pain" vs "bad pain".
