Sometimes I fear for traditional publishing and hope that self-pubs will save the industry. To elaborate, I see publishing going the same way as Hollywood, i.e. endless sequels and churning out what is essentially the same movie with a new title and/or cast. The corporates are extremely risk-averse and the whole industry suffers and stagnates because of it. This is where I hope the self-pubs can save the day, as these authors aren't afraid to go in new directions without worrying about which category Marketing will slot them into. And, if they manage to build a following, the contracts will (hopefully) follow, assuming that's what the author wants.
Fiction publishing is nothing like Hollywood and self-pubs don't need to "save" anything -- they are just another sector of the market. And your fears are the exact same fears that they whine about in the 1980's when the bestseller lists had much less turnover and fewer authors tended to dominate the top spots because of the power of the wholesale market. (And in the 1960's, when they proclaimed the death of literature, and the oughts and the 1930's, and pick a decade.) It's the mythical imaginary fear where we pretend there are not giant "literary" bestsellers that stay on the list for over a year and then sell for decades to universities and high schools, and that every book that does well on the list is somehow there because of a commercial "formula," ignoring the thousands of other books that have "formulas" (otherwise known as mystery stories, romance stories, epic war stories, family dramas, etc.) that do not do well or certainly don't make the lists. Where we pretend that larger publishers are "the Man," enforcing sterile conformity and rejecting anything "unconventional," while citing for praise daring, original authors -- who are published by the larger publishers, and usually are bestsellers. I'm sorry, but the fairy tale world of the soulless commercial machine and the serious artist in the poverty gutter as the model of fiction publishing doesn't really exist except in Hollywood movies. Eighty percent of fiction authors, no matter who publishers them, are poor on writing income. Of the remaining twenty percent, only about half can still afford to quit their day jobs.
The fiction publishing market relies on variety to sell -- if they don't have variety, they don't have enough readers buying, and their profit margins and cash reserves are so thin, they can't afford that. Fiction publishing is the land of a thousand niches, because the combined total of the niches is far more successful than just relying on the bestsellers -- most of which they can't predict will be bestsellers or by how much (unlike movies.) New books from bestselling authors lose money because of the discounts, and publishers use them to sell the authors' backlist, which makes the steady (and unexciting) money. Larger publishers must publish a maximum number of niches in order to make bank. Phenomena authors, which are rare and largely can't be predicted and certainly not engineered, get all the media attention and can effect markets mostly by expansion, not contraction, but while they can be a huge financial support to the particular house that publishes them, they aren't the steady money, and all the glamor that comes with them, or rather their film/tv adaptations, doesn't really apply to the regular business of fiction publishing. There are no focus groups, marketing data, etc., because they aren't much use to publishers.
The books that appear in the self-publishing market -- thousands and thousands of them -- are not any more unconventional than the ones put out by publishers. They are dominated by romance, just like the fiction paperback and e-book markets, followed by fantasy, mystery and science fiction. They do not have a lot of contemporary dramas, serious historical epics or international sagas -- the books that tend to get called "literary" that publishers put out and often sell well. Instead, the self-publishing market is dominated by what we call "genre" fiction -- the stuff that you are calling commercial formulaic fiction. A small percentage of publishers' titles sell well, the rest not that much. A small percentage of the self-published titles sell well, the rest not that much. An author may have been told by various editors that they don't want his novel because X is not moving for them right now, because publishing pros can be as stupid as the next person (and sometimes because they think it's a gentler blow than saying that they just weren't that interested in the author's writing.) And the author may then self-publish and it may sell. That doesn't tell you anything about publishers or the market, because books rejected by one publisher are picked up by another publisher all the time. Anything might sell, but if it does sell, it will be by word of mouth -- not by formulas, not by ads, not by who publishes it (because readers really don't care.) And publishers know this well, but other businesses can't believe it's true and get very frustrated when they own book publishers.
Self-publishing existed for a long time before the e-book market developed into a viable retail market, in paper and then in e-books before the Kindle. The development of retail e-books is not a revolution, it's not going to save anything -- it's simply a market expansion, and it seems to have largely just taken over mass market paperback sales from the shrunken wholesale market, now leveling off. (Which is not at all a bad thing.) That expansion from e-books has been mainly from the publishers' sales of e-books, but does involve the sea of self-publishing.
RichardWolanski said:
Why can't LeGuin be like LeGuin?
She can and she did. If they were really chuffed about it, they'd have just cancelled the contract. Le Guin is famed in children's/YA for her Earthsea books, as well as her adult SF, which are firmly cemented into school curriculums and sell steadily each year. (The SF is firmly cemented into high school and university curriculums.) Her books are bestsellers and she has the potential to attract a lot of attention for each one. The three last books mentioned are middle school novels aimed at 5th to 9th graders. So somebody, probably younger someones, said, "hey, we could potentially have a Harry Potter effect, so why don't you have throw in (whatever they think magically made Harry Potter sell,) because the booksellers want it (because everything sounds more official if you say the booksellers want it.) And Le Guin being Le Guin, she patted them on the head. Annals of the Western Shore was unlikely to sell as much as Earthsea, it being late in her career, but they were children's bestsellers, published internationally, lead titles for their publishers, won her a Nebula and a PEN award, helped sell another wave of Earthsea backlist titles and further increased her children's backlist, which will keep selling for decades. Publishers would like every book to sell at Harry Potter levels, (as would self-publishing authors,) but as is perfectly clear, that doesn't happen. No one has sold as much as the Harry Potter series, except the Bible and perhaps Agatha Christie. Le Guin's sales path for a middle school series in late age is more sedate -- but quite financially successful. But it's not sexy enough to get media attention. Because the non-SFF media doesn't like written fiction, especially the visual media. They are usually little interested unless it's a phenom or there's a film adaptation. (Le Guin had to deal with a bad tv adaptation of Earthsea recently, but did get media coverage out of it.)
Angelo Nero said:
I chose to do a sci fi novel, limiting my audience
Science fiction is doing great guns right now, especially in YA, but also in the adult sector, with quite a lot of adaptation deals of older and newer properties too. (We just had the nice announcement, for instance, that John Scalzi's best-selling satirical SF novel Redshirts will be a limited t.v. series on FX.) Science fiction is one of the most international genres at this point. So no, you're not limiting your audience.
I chose to do a dark novel with little romance, limiting its appeal once again
Well then you're in luck, because science fiction readers mostly don't like romance. They looovvveeee dark stories, from SF horror to grim military SF and dystopias (post-apocalypse or other,) and noir cyberpunk crime novels. In fact, every genre loves dark novels, so all well and good.
And finally decided to make my lead female character an australian, again limiting my potential readership to such a small portion of the market
The Australians are now very insulted. Contrary to what is often folk wisdom, readers in America don't care much if the protagonist is American or not. The Australian market is a bit more limited because their SF sector has been smaller than fantasy, so they aren't as known internationally for their SF as they are for fantasy, and because their market is small, their authors need international sales. But many of them are getting those sales, with distance less of a problem. Led by the venerable bestselling Greg Egan, a whole crop of Australian authors are doing really well. If you're located in another territory than Australia, having an Australian protagonist is not necessarily a hindrance, though you may run into people who claim that it is. The female protagonist will also get you people telling you that you can't sell as well. Just wave the Honor Harrington series in their faces.
RichardWolanski said:
Romance is easily the the highest selling genre.
1. Suspense is the highest selling genre. But romance owns paperback -- but mainly in the North Americas. Category romance novels tend not to sell as much internationally. Romance has done well in e-books, being the earliest genre market in e-books.
2. There is a difference between a romance novel and putting a romance sub-plot in a story. Nearly every novel has a romance sub-plot -- it doesn't make them romances and it doesn't necessarily help them sell. A romance novel is a specific thing. A lot of books get called romance that aren't -- because a woman wrote them and they are stereotyped. The category romance market has its own set of publishers, media outlets, etc., though they do crossover marketing.
3. Lots and lots of romance novels sell only relatively low amounts, but romance authors are often a lot more prolific, having several books out a year. Category romance novels get fewer reviews than say SFF, and less library sales, and they are typically given less time on the bookstore shelves than other types of books, but they can do quite well. Erotica is a separate category market but overlaps with romance.
But because there are a lot of offerings, people have to make choices -- but we do have the tools to make fairly well informed ones, even for self-publishing books. They are riskier because they are many, but there are vehicles in place -- at Amazon particularly -- to sort out some and the small press offerings as well. 
