If I find myself wanting more time between warm-up sets than it takes to change the plates, I know something is wrong. Most likely I'm not recovered enough, indicating my program or eating or sleeping or stress is messed up. It's not a big problem, but it's something to note. I should be getting warm and mentally ready for my work sets, not getting tired.
Between work sets, I calculate time in minutes, or at a minimum, 30 second chunks. So for heavy sets of five I'll try to stay at three to five minutes of rest, whereas for heavy triples I'll try to keep it to two or three minutes, and for singles I sometimes go on the minute or 30 seconds.
Between warm-up sets, I count time in thirty-second blocks at a maximum. If I need a breather, I'll take ten or twenty seconds to write something in my training log, but I'll also take a mental note to get myself psyched up, because being winded after a 50% warm-up set is not something I want to do. This generally helps.