'You don't sound Welsh' - people question my identity because of my accent

Accents vary often village to village - but do people often think there is just one Welsh accent?