This is a great question, and I think it's something everyone has struggled with. First, just take a peek at this answer about gym etiquette, which should set your mind at ease in many respects.
When starting out at the gym, everyone will feel self-conscious. This is part of training anything. If you take dance lessons or archery lessons, you will feel the same thing. It takes time to build confidence. At the gym though, we want to start using different machines and different equipment in order to have some variation and keep our workouts fun and effectual. So what do we do?
First, we should make an outline of our goals. What do we want to do in the gym? Do we want to be able to bench press 405 lbs? Do we want to be able to do muscle-ups? With this knowledge we'll be able to figure out what equipment we need to use. If we want a bigger upper body, knowing how to use the leg curl machine might not be the most valuable knowledge -- but knowing how to bench press and bent-over row would be necessary.
Next, we need to find a routine that will help us achieve our goals. An olympic weightlifting routine that involves front squats and overhead presses won't do much for us if we're trying to have a body like Arnold. If that's what we're shooting for, we should be looking for a bodybuilding routine that keeps our rep ranges at in the right thresholds. On the other hand, if we just want to have a beach body, maybe a crossfit-type "high intensity interval" routine is what we're looking for. In any case, we need to know what we're supposed to be doing in the gym while we're there.
Finally, we need to practice the exercises in our routine. As an anecdotal estimate, maybe 90% of training is just knowing how to move your body in that way. After that, it's a matter of how we're stimulating the body. This means weight per rep, duration, intensity, rest time, and so on. There are terms like "greasing the groove" that speak to this type of neuromuscular connection. There are also studies that suggest just going through the range of motion many times per day, even under no resistance, will contribute greatly to performance.
Also, if there are exercises we're just too self-conscious to do it at the gym, it's easy to practice anything at home first: just find the exercise online. Exrx.net is just absolutely invaluable for any imaginable exercise. Look at the exercise directory here. YouTube is also great for any conceivable exercise -- especially compound movements like powerclean or snatch that require tons and tons of practice.
As an aside: three workouts is not enough to really make much a difference. Many advanced lifters haven't changed their program in years and still see progress month after month. As a beginner, it should take at least weeks before you see any significant changes in your physique that last beyond "the pump". Though, you should see fast improvements in strength training as your body learns how to perform exercises.