I used to not pay much attention to fashion, I used to think it was just following trends and overpaying just to fit in, but then I realized a few things.
Your style is a form of self-expression. It allows you to express your interests, your individuality, and also (whether you are aware of it or not) your "with-it-ness" for lack of a better term, and how you feel about yourself.
In school, I wasn't very expressive, for fear of getting teased about my interests (fandoms - Star Wars, Transformers, etc.). So I just wore what was around. I didn't really make conscious style choices, and as a result, I didn't really express a coherent "me". Rather, I expressed that I didn't know what I was doing/didn't have a fashion sense, and that I was either afraid to express myself, or that I had no "self" to express.
I needed to not be afraid to express myself and my interests. At the same time, I needed to have a more well-rounded mix of interests, including "doing" interests instead of just "liking" interests, so that I had more to express. I needed to be more balanced and complete, and to show it.
I also realized that I like the way I look in different things. I like my fandom and band/other interest shirts, but I like dressier button shirts too, and I want to try this "mod" stuff I found on Facebook. And I've even come to find out that I like how I look in a suit, too. I used to think "I don't want to dress professionally, I don't want to conform blah blah". But I've come to find I like dressing up a little from time to time. It makes me feel crisp and clean, and I do want to be taken seriously.
The way I see it, fashion sense is another box to tick off in making yourself a complete person, but the important thing is to pick things that are authentic to you. Fashion isn't necessarily shallow/silly/a waste of money - it's only like that if you're hopping on a bandwagon and trying to be "cool", instead of using fashion to express your true self.
Nevertheless, I feel like the most important thing when trying to attract someone, will always be substance/conversation material, and your mind - being likeminded. It won't matter what you look like if you run out of interesting things to talk about, and can't show commonality.