I think the best way for someone to date who is new to dating is to not really "date", in a traditional sense, but just socialize. Don't do it when you're not feeling up to it, or you won't be yourself, but just socialize with everyone and anyone you can. Chat with the guy behind the counter at the convenience store, even if it's just a couple sentences. Don't just follow the nurse into the room quietly, tell her a little bit about your day or something you saw outside that was interesting, or complement her on her hair. Talk to the girl or guy (it doesn't matter who it is.. girls are better, yes, but anyone!) at the cash register at the grocery store about something you observe at the store that's different, or tell them what you're making tonight with the ingredients you have purchased and how excited you are to cook tonight (if it's genuine!).
Don't expect any sort of real dates or relationships to happen, but just socialize more with strangers. Once you're comfortable with that, and you're just being yourself, take a look at yourself in the mirror and don't buy a new wardrobe or make drastic changes, but yes, see if you can do some minor things to enhance your appearance or at least show that you understand current fashion and aren't wearing jeans from 1994 or have the exact same haircut from when you graduated from high school, but don't do something drastic if you aren't comfortable with it either.
Now you're ready. Keep socializing, but be showered, brush your teeth, carry gum, and your appearance is at least a little more sharp now... and just keep socializing. When it happens to be that you are socializing with a girl you find attractive, finish up with something like "Hey, I have to run, but do you want to talk more sometime later? I can give you me cell?" or whatever. If she accepts, don't expect a call/text, since she might not be right for you. If she declines, a happy smile and a "You can't blame me for asking!" and you're on your way, no heart crushed!
This is a socialization game where you can't force things unfortunately. Most things in life we can try harder and make it work, but not here. You just will be talking to too many people who already have boyfriends (but don't assume it!), or just aren't into you that way, but you have to keep socializing and then asking them if they want to talk more or even get coffee or grab something to eat some time, and just be ready for you happy smile and "You can't blame me for asking!" natural warm response.
So I guess yea, there is a fine line here between what people mean when you need to try, but then you're trying to hard. You have to at least put forth the effort to talk to people, and there is nothing wrong with wanting to be in a relationship. God I wish I was back in one right now and it sucks to not be, but you can't make it happen by trying harder and harder, being someone you are not, etc. That is what people mean by trying too hard. You shouldn't think you are necessarily doing something wrong just because you aren't having success. Like baseball, you fail 70% of the time and you are great and successful at hitting. When socializing with women and then trying to get a date and move forward from there, you can be "great" at it and still "fail" a very high percentage of the time. It's not like college where almost everyone is single any more. People also tend to get more selective when they are older and have had experience with what they do and don't like on top of that, but don't think of it as a "fail", it is a success if you "played your cards right" and got more experience. You can only control putting your best foot forward and putting yourself out there at the end to let her know you're interested, but you really can't expect to "succeed" very often at all... but you'll never succeed without trying at this level at least. You don't have to try any harder than this though!
If you are starting out just now, you can't cram in decades of dating experience and socializing and talking to women in a year's time either. That's just something you have to accept. It's like going into martial arts and wanting to work harder than everyone else so you can become a black belt in 6 months. You do have to accept that some things take time, and this is one of them, but working towards that goal and seeing progress should be satisfying enough.
Does this make sense at all, or shine any light on where the line is between needing to put forth some effort vs. trying to hard and going overboard? Believe me, I wish you could just try harder and end up in a happy relationship quicker, because I'd be a far happier person right now, but I have experience dating and know it just doesn't work that way.