I'd say as far as young men go, that yes, there's a bit of a "Why didn't he go on a mission?" attitude. That's not necessarily the way it should be, but it's the way it is. Starting about 25 years ago, there has been a big push that "every worthy young man" should go on a mission.

There's no such attitude towards young women who do not serve a mission.