Geekdom rules!
Given up trying to defend my reading tastes. I always go back to my old English teacher who told me that 'I would be good if I could just stop reading 'that Sf stuff' '.
30+ years on - still reading and enjoying, and educating others!
The tide has turned a little in recent years - SF/Fantasy has now become big sellers (unlike many other novels and genres), so there are more people around reading more SF/Fantasy. Consequently it's not so much of a surprise as the genres become more mainstream that SF/Fantasy becomes more widely known. There's enough of us out there to make a difference, I guess.
As a side note - when people see some of the 'stuff' out there, I can partly sympathise with their misguided view. There is a lot of 'troll-by-numbers' dross out there.
But as a general rule of thumb, Sf/Fantasy readers are intelligent and deep-thinkers who care a lot about what they read, IMHO. They are usually people with the greatest imaginations/visions, they have a greater ability to think 'outside the box' and usually have a greater IQ than the majority of 'readers'. (Have read this somewhere following a Time/Life sort of poll, but can't remember where, sorry).
Alternatively, we're the people with the least interest in Fashion (unless 'Hobbit-dressing' is in this year)and gossip - other worldly concerns...
So, if others don't want to read it - their loss. I prefer to try something first, then deride it.
An opinion is an opinion, after all.
Hobbit