I think a lot of the problem is with the nature of the job itself. You have to know how to drive, but there's also a lot of physical work on top of it. You can get injured fairly easily because you're often doing a lot of heavy lifting and handling.
Traffic can really get to some people. The roads can really get congested. Driving's definitely not for everyone, and trucking companies can certainly be a little bit pickier when it comes to whom they hire.