Eating Fat Does Not Make You Fat
Excess fat on the body is an entirely different thing from the fat that you eat. Science and experience agree on this: here's a study entitled Dietary fat is not a major determinant of body fat, and here's Gary Taubes' overview on the subject, appropriate for the layperson. It's a common misconception that eating fat somehow turns into adipose tissue.
Milk might be bad for weight-loss (or it might not), but it wouldn't be because it has a significant proportion of fat. In fact, since fat is satiating--it makes you feel full--I am of the opinion that if one's goal is losing body fat, then fat (like whole-fat milk, or even cream) should be one of the things you eat.