I think it is the imagination factor as well. When someone sets out to write within the real and reconizable world, they immediately hamper themselves with implicit rules and regulations. If the author writes about Rome, then they have to stick to what is real and right about Rome, inserting their own fabric of imagination within the fabric that those of us who know Rome won't find wrong or disconcerting. It can work, but it detracts in many author's works. And many authors do not take the time to deal with this at all. They think they know or that their readers will not care. In many cases this can be overlooked if the prose and plot are well done. Too bad those are few and far between.
Fantasy is about risks in some ways as well. The author has to take chances, they have to do the unexpected. Not in terms of plots (though this can help), but rather by introducing a new aspect that comes directly from their own imagination (or cribbed from someone else's though that is another thread entirely!). Fantasy is more personal and good and great fantasy works because that person clearly shared their inner vision with the reader. Writing this vision in a readable and approachable manner is of course essential as well.
Fantasy takes leaps where regular fiction takes steps. If you want to take a nice walk, read regular fiction. If you want to fly and sweep and soar then read fantasy.