If my goal is size.
EDIT: Since other people with potentially different goals than mines are likely to see this question, it would be good if your answer includes the optimal rest times for different goals (strength, endurance, etc)
If you are looking to build mass, you should rest 45-60 seconds between sets of 8-12 repetitions for optimal gains. This timing builds optimal muscle mass and hypertrophy.
If instead you are looking to improve your strength or endurance, you should be looking at a 3-4 minute break between sets. You'll want 4-6 repetitions with heavier resistance for strength or 16-24 repetitions with lower resistance for endurance.
The Art of Manliness published a great article with an extremely simple strength/endurance training routine.
If you want size, you don't want to rest very long - perhaps under a minute between sets. Your goal in the gym is to get your muscles as fatigued as possible as quickly as possible (and then go home to eat!)
I'm currently doing a hybrid workout to gain size (but I still want to work on strength a little, which is generally my long-term goal) and my workout is:
1 set of 5 x heavy (80-90% one-rep-max). Then as many sets of 5 as possible (at 80% of the first set) while still doing good form. 40-60 seconds of rest between these.
I say this is a hybrid, because if I were going purely for size and not worried about strength at all I would probably increase the number of reps to 8-12 (and make the weights accordingly lighter).
In order to apply the principle of metabolic stress 30'' or lower rest time are needed, metabolic stress does build muscle too.
If you are doing 1 or 2 rep maxes, then take your time, 2 or 3 minutes of rest time is reasonable.