I don't know who said it, but there is a quote - "There is no such thing as an ugly woman, only a lazy one".
The thing is, as somebody else pointed out, women will find all kinds of bizarre men attractive (serial killers being the obvious example), often it's about how the man feels about himself (probably true for women too). I think very generally speaking women just aren't interested in guys who consider themselves to be losers.
There are things that destroy a man's confidence, like being short, or being bald (also being overweight, but you can do something about that one). Sure, there are some guys who are short, or who are bald that would be considered 'unconventionally handsome', but not a lot.
But, when I was in my early 20s I knew a guy who was short (5'3"-ish) AND bald, he was mid-40s, and he was a killer hit with the ladies, including young ladies (his gf was 21 at the time, I recall) because he was confident, charming and funny, 'gift of the gab' type.
See, me, I'm tall and physically not un-masculine, as in, I have a heavy beard and a deep voice, but I'm definitely not a 'masculine alpha-male', and in fact, I can't think of many things that would make me more uncomfortable than pretending to be one of those type guys.
Women I think may be more clustered around the average, because while most women obviously aren't Naomi Campbell, they don't need to be, being short etc doesn't matter for a woman, a lot depends on how much effort she wants to make to be physically attractive. I shy away from this example a little, but I used to work with a lady whose husband very tragically died aged only about 40, and within a year the lady had lost a ton of weight, had her hair-did etc, because she was 'back on the market', and she looked great.