It's what is or isn't vital.
A strong test is whether the elements pertinent to a given genre are in fact necessary to the tale. Not a few "genre" stories and novels could easily be rewritten with only minimal changes in some other genre, or even as "mainstream" (which is, of course, itself a genre). In such tales, the "genre" elements are merely wallpaper, and could be changed at will. Only if the genre-relative elements are essential to the tale--that is, the tale is basically impossible without them--is the tale of that genre.
Originally Posted by SR_Seldon
Many less-able writers write "genre" fiction that is just a plain story with fancy curtains and wallpaper. An example that always pops up in my mind is David Gemmell's "Jerusalem Man" series, always listed as "fantasy" but in plain fact just westerns. Glen Cook's "Black Company" tales, after the first set (which is basically one extended novel") are effectively Vietnam war stories. And so on.
Well, examples were mentioned.
The "test" is a shorthand way of stating that the chief proper reason for writing "speculative fiction" tales (or tales in any genre, including "mainstream") is that the rules of that world, and to a lesser extent the conventions of that genre, best allow the author to examine, dissect, and make statements about The Human Condition. The peculiar rules of the world in a speculative-fiction tale are essentially a sort of spotlight that the author can shine on those aspects of Life, The Universe, And Everything that he or she is interested in examining. Just right off the top of my head, I would say that Ursula K. Le Guin's novel The Left Hand of Darkness is a fine example of both meeting the criteria of the test (absent the peculiar nature of the inhabitants of that world, the story cannot be told) and of the idea of using special rules as a way of focussing on some aspect of The Human Condition.