The symbolism of "putting on Christ" by putting on a white robe is certainly appropriate dress for one who emerges from the cleansing waters of baptism. The white robe or alb also can be a valuable reminder that the pastor occupies an "office" and functions not on her own but as a representative of Christ's church. However, this symbolism is often lost on people to whom it is rarely explained, becoming instead a uniform connoting a distancing other worldly separation or presumed perfection.
In contrast, the black robe's symbolic reminder of human sinfulness is usually eclipsed by the robe's more commonly understood association with the judge or academic -- hardly pastoral images.