This is a common problem in the fantasy genre. It's not a matter of developing the world. It's a matter of explaining it, so that the reader will have all the relevant understanding. Without a properly developed world, the use of magic for example will seem very "deux ex machina."
On the other hand, you don't have to get bogged down with the details. Think of the "Three Little Pigs." I'm pretty sure that in most versions there's no mention of why pigs can talk, or where the third little pig got the concrete mix to build his house with bricks. But as kids we still understood the world.
One thing you may want to try (hopefully this won't mean tossing a whole pile of work away) is to figure out the story first. Then develop the characters and the world around it so that they all work to tell the story.
Example:
You decide to write a story something like the 3 little pigs - a tale about work ethic and survival, but you want a fantasy twist. You basically need a character with a good work ethic (GWE), one with a bad work ethic (BWE), and a threat.
This is a fantasy setting, so lets make the threat a barbarian horde. (The Big Bad Horde?)
Then you develop your characters to exaggerate these traits. BWE needs to be lazy. Maybe he relies on a magic of some sort that he doesn't understand to protect his village. He teases GWE and flaunts his extra time.
GWE is the passionate character that instinctively knows something the others do not. He is efficient and headstrong.
You have to think of ways to contrast the personalities - so you need to establish a setting in which they can interact and in which the threat can loom ever closer.
In a nutshell, rather than starting with the world, and hoping a story will evolve, try starting with a story and build the world around it.
)
