Labor Day is celebrated in the U.S. on the first Monday of September, marked the traditional end of summer. It also holds a fashion myth that is as old as time and no one can ever put a reason or origination on this myth.
The best explanation that I have heard was due to the fact that white is worn during the summer to keep cool. So it's more of end one season, start a new season marker.
At the same time, the fashion forward lived in big cities where were usually in northern climates that had noticeable seasons. Down south, seasons all blend so well together. Setting a date to not wear a certain garment doesn't make sense.