Research is pretty divided. On one end, there are claims that it reduces performance and muscle growth by slowing muscle reaction and displacing calcium:
Regular intake of caffeine reduces your muscles strength and resistance training performance by slowing down the muscle contraction and relaxation cycle.
High intake of caffeine also displaces calcium absorption, which may partially explain the adverse effect to muscle contractions, as calcium plays an important role in nervous system function, as well as bone growth and structure.
Others claim that it improves performance:
The primary effect that caffeine has is on your central nervous system. The drug increases the firing of your neurons and triggers your pituitary gland to release adrenaline. American Council on Exercise sports nutritionist Fabio Comana states that reputable studies have shown this stimulating effect to improve endurance exercise performance. By allowing you to perform exercise for longer periods of time and a sustained intensity level, caffeine may aid in muscle growth.
A second mechanism through which caffeine may enhance your muscle growth is through the release of fatty acids into your bloodstream. In a review of energy beverages presented in 2010 in the "Mayo Clinic Proceedings," Dr. John Higgins and colleagues noted that your muscles will use fatty acids as an energy source before the stored glycogen in your muscles. Preserving your muscle glycogen may help you exercise for longer periods of time without reaching muscle exhaustion.
Overall, I'd say that 3-4 cups of coffee isn't really going to break the bank either way. If it keeps you moving to perform more exercise, that's good. If the caffeine is disrupting your sleep habits, that's bad. The creamer is less than 10 calories per teaspoon, so 3-4 cups will neither result in massive weight gain or much extra fuel for muscle-building. It's pretty much a wash.