I wouldn't say so. I have a good friend, he is completely bald, a little chubby, not that facially attractive AND he's had relationships with really attractive girls.
Greek philosophers defined desire as "eros" (from the mythological god of desire with the same name); in one of the dialogues from the book "symposium" Socrates explains that physical attraction is the easiest way to awaken "eros" as a young person, as attractive bodies are the most evident expression of Beauty; but they are not the only way Beauty can manifest itself, specially when you consider that beauty is something extremely subjective. There are also beautiful souls (defining soul as the conscious and subconscious components of the mind, which includes personality, intelligence, spirit or character, etc.). Feeling attraction to someone's soul can easily awaken physical desire.
Many philosophers saw carnal desire as banal, or even as unhealthy; for them, casual sexual encounters based only in carnal desire were harming to one's soul. Socrates didn't see carnal desire as something insignificant, in fact, he was attracted to the most beautiful and young boys of Athens; and even though he was old and known for his ugliness, the young boys would fall in love with him.
Socrates would only consummate his carnal desire with those with whom he also felt a deep intellectual and spiritual connection... He would only satisfy his desire for beautiful bodies when those bodies were accompanied by beautiful souls, and the young boys would feel so attracted to the old man's soul, they'd feel a strong physical desire. The beauty of the soul was so important to him because it can not only awaken the other expressions of eros, but it will allow sexual intimacy to be transcendent, instead of banal.
