Your goals and your current level of progress determine the number of sets and reps.
Training for strength, power, endurance and hypertrophy all require a different number of sets and reps:
- Strength (how much your muscle can move) is best developed by lifting as much as possible. This is probably best achieved with 5 or less reps. Strength is expressed in how much you can lift one time, so the closer you train to 1 rep at a time, the more specific you are training for strength.
- Power (how much your muscle can move quickly) is best developed by moving heavy weights very fast. This is best achieved with very few reps, something like 2 or 3 in a set, but you have to use slightly less weight so that you can move it faster.
- Endurance (how long your muscle can keep doing its job) is best developed by lifting a weight many times, which requires many, many more reps: at least 15. You'll have to use a lot less weight in order for this to be possible.
- Hypertrophy (how big your muscles are) is best developed by achieving momentary muscular failure, and with overall training volume. Moderate weight works best for this purpose, since it is heavy enough to quickly make one unable to lift it, but not so heavy that failure to lift is dangerous. Most people use 6-12 reps for this purpose, but there may or may not be anything special about this range. Multiple sets and exercises are useful for achieving high total workout volume.
This is well explained by a chart in the article I linked to:
Three sets of 7 at 25kg will demand and develop more strength than three sets of 10 at 20kg would. The sets of 10 would promote greater hypertrophy and require more strength-endurance and conditioning. The different is not going to be terribly significant, however, since 7 and 10 aren't too far apart.
Doing fewer reps with heavier weights requires and develops more strength and less conditioning than more reps with somewhat lighter weights. Doing more reps with slightly lighter weights may, in some circumstances, for some exercises, produce more hypertrophy (mass gain). Fewer heavier reps is better for strength; more reps (but still as heavy as possible) is better for size.
Though the original on page 60 of Rippetoe & Kilgore's Practical Programming is better (the gradations from range to range are less stark) this chart from reddit does an excellent job explaining the effects of different rep schemes:
Doing fewer repetitions with heavier weights builds strength most effectively. Doing more repetitions (circa 4 to 12), with weight that is still challenging, builds mass most efficiently. Doing more than 12 repetitions in a single set is generally best for endurance as opposed to strength. (See this answer for more information.) Does this mean that someone who is diligent with a 12-rep program can't get strong? Heck no! People get strong with 12-rep sets all the time. But if raw strength is their goal, they could probably achieve that goal faster with sets of, say, 3 or 6.
A novice generally does best with a focus on strength and some hypertrophy. Three sets of five or five sets of five are the two most common set/rep schemes. The weight must be heavy enough to make more than 5 sets very difficult.
Note that there is very little about these rep ranges in and of themselves that produces desired attributes. It is how these rep ranges relate to elements of training such as volume, intensity (that is, proximity to 1RM), and muscular exhaustion that determines the effects of training. Notice also that hypertrophy produces strength and power, and strength and endurance enable hypertrophy.
The following "map", from the Starting Strength site, explains the relationship of this volume-to-purpose relationship to sports. It refers to the three metabolic pathways: phosphagenic, glycolytic, and oxidative. The phosphagenic pathway is used when we do a very small number of reps in a set (3 or less, whereas higher-rep sets use the glycolytic pathway (approximately 4 to 12, though it depends on how vigorously one is working out). The oxidative pathway is for even higher rep ranges (e.g. 20), and is more commonly associated with longer duration, repetitive exercise like distance running or bicycling. It is explained in more detail in this article.