To quote William Roberts, MD sports medicine physician at the University of Minnesota,
“Muscle fatigue takes away all your protective mechanisms and really increases your risk of all injuries.”
Granted he was referring to specifically athletes and sports-related injuries in his quote, but I feel there is enough parallel here to apply the advice to your situation.
The advice makes sense when you think about it, when your muscles are tired you are more likely to use poor form and thus more prone to injury. Make no mistake, if you are in the Starting Strength program... your muscles will be fatigued after the workout.
The solution to this is, quite obviously, to give your muscles enough time and nutrients to properly recover before you do something overly strenuous.
For the problem of time, it really comes down to proper planning and scheduling of activities. Have to do manual labor in the evening? Do your workout in the morning!
For nutrition, it comes down to feeding your muscles enough so they can recover from the workout. Have a protein drink 10-60 minutes after the workout. Have a meal 1-2 hours hours after the workout.
Taking these steps should ensure that you are properly recovered post-workout, but if you absolutely cannot avoid hard labor whole fatigued, always remember to use proper form.