I've searched for a reference for you, online and on my bookshelf, that would clearly state for you that 'there is no direct link between physical fitness or fitness gains and amount of body hair'.
But using my work experience as a reference then, as a sport massage therapist, I deal with clients who either epilate or grow their body hair for diverse reasons in relation to their sport. Entirely aside from the adjustments I make in the bodywork I do depending on amount of body hair a particular client has -- I've only found what is commonly known: that weight trainers epilate because they obviously want their musculature to be better seen. Cyclists say having less hair reduces irritation and wind resistance and, if they have a fall and cut their skin, being hairless is safer and allows for being able to get the wound cleaned up better. Swimmers epilate to decrease drag in the water. Women athletes of all kinds epilate because of all the above type of reasons and the general cultural expectations that women be generally hairless.
But in my work, I have never met anyone whose amount of hair either improved or damaged the key physical fitness markers that I look for when working with them (muscle tone, strength, flexibility, circulation, recovery, etc.) --- though a large amount of hair could affect an individual's performance depending on any perceived/psychological issues or actual/measurable issues with things like irritation and wind or water resistance.