I thought the term came from the resemblance of the glove to a "work" glove used by the typical coal miner or farm hand in the late 19th or early 20th century. Similarly, the "brakeman's glove" term describes a glove that a train brakeman would have used in his work in the engine cab of a steam locomotive.
I have a bucolic, pastoral image in my mind of a group of farm boys rolling up one glove and putting it in the bib of their overalls, leaving one glove on their hands and going to the county fair grounds for a game of "base ball." Maybe the train came in, and the brakeman, wearing his train engineer's cap and one of his gloves, will be pitching for the visiting team today ...
When we work on our barbed-wire (bob-wahr) fences down here in Texas, we use leather working gloves, but I'll be dogged if I try to catch a line drive with one of them!
Yes, I still have my first glove.