Just out of curiosity: Did you have what others saw as an Alabama or a "Southern" accent (though that has many variations)? If so, was it a factor in any of this?

In my experience, the dialect can modify both black and white speech and affect how others see you.