It's a hardy perennial of a question among writers, because it matters, from which agent might represent you to what cover your book might get. The writers' forums seethe with arguments about "any book worth reading" to "would you call Dickens literary?" to "pointless pretentious rubbish" or even (seriously) "a book only academics will like".
The latter can't be true, because there can't be enough academics in the world to account for the combined sales of Margaret Atwood, Ian McEwan, Barbara Kingsolver, Helen Dunmore, Hilary Mantel, Peter Ackroyd, Barry Unsworth, Philip Roth, Jonathan Franzen, Martin Amis, Beryl Bainbridge, Gabriel Garcia Marquez, Umberto Eco... But it is worth remembering that if you like and read literary fiction, in the world at large you are a tiny minority of book readers, and an even tinier minority of human beings. That's the truth.
But I've noticed that inside the book trade (and don't forget that, as Harry Bingham says in Getting Published, writers by definition aren't inside it) no one angsts about this question at all. They might argue about whether a book really classes as "literary", for all those purposes from imprint to bookshop promotion, but they don't argue about what "literary" is. Not that they could tell you, at least only by talking about specific examples; they just know. They're also less and less inclined to tolerate low sales in return for literary prestige, so although the market of readers is undeniably there, the publishers are more and more stringent about whether they can afford to publish a literary-seeming novel. So I've been trying to work it all out.
The first thing to say is that it's always talked about as an either-or but actually it's a spectrum as wide as the number of books published. And a little way from the right-hand, literary end of the spectrum there's a ditch, after which books count as "commercial". I know that ditch isn't very wide because I write with one foot on each side of it. But it IS, undeniably, there.
The second thing to say (since lots of the points I'm about to make are about the original versus the familiar), is that all fiction works by integrating the familiar (the world that readers experience themselves) with the new (what they don't know themselves), in varying proportions. This is as true of category erotica or the thriller which still finds a new twist while ticking the genre boxes utterly reliably, as it is of the uber-literary blockbuster novel which is like nothing you've ever read before but is, among other things, playing sophisticated games with the great and familiar literature of the past. So, here is my attempt to help you decide which side of the ditch you're on.
"Literary fiction" for the book trade is just another genre, another place on the bookshop shelf; one which carries more weight in literary prestige than it does in sales, and so gets covers, blurb, promotion and marketing to suit that market. It overlaps with, but is not the same as, "literary fiction" as something for writers to think about when they're imagining their readers And it gets confused with "literature", which (I would say, roughly) means the books that, over time, have stayed alive and read beyond the time and place of their writing.
So, if we're talking about writers now and the readers who might buy their work, I would suggest that for a book to be classed as literary fction it needs to fulfil a good few (though not necessarily all) of the following criteria. So, a literary novel may do several of the following:
1) pay as much attention to the originality and quality of the prose, and the ideas it expresses, as it will to the demands of plot-driven storytelling.
2) ask harder work of the reader (think contemporary poetry), in terms of the prose, (whether it's lavishly baroque or fish-bone spare); in terms of assembling the story out of the way it's told (think Possession or Time's Arrow); in terms of the ideas it explores
3) ask the reader to tolerate not getting everything there is to be got, in the first read. Some readers love this, and feel a book's superficial if they get it all immediately. Some hate it, and feel frustrated that they're missing stuff.
4) when it comes to the proportions of originality to familiarity, there's more originality, in more aspects (plot, character, prose, ideas etc. etc.) than in commercial fiction. The more that's original and the more original it is, the more challenging it will be to read. It may or may not actually then be harder to understand the story at the basic, factual level.
5) ask the reader to tolerate, say, an unreliable narrator who may be lying; to read between the lines for the truth which may not be the truth actually, or not all of it; to appreciate an inadequate narrator (The Curious Incident, What Maisie Knew) and assemble the truth from what they transmit but can't understand.
6) ask the reader to read between the lines of extremely bare prose (think Hemingway and acolytes) which makes no judgements and leaves very large gaps in which the reader has to do the thinking for themselves.
7) ask the reader to be patient with prose which deliberately doesn't play by the normal conventions of spelling, grammar and punctuation, for good reasons. Or maybe it's richly packed with metaphors and references, which mean that absorbing the action of the story is a slower process.
8) take the reader into the heads of unlikeable characters (American Psycho, God's Own Country) and enjoy being there for the human interest. Glamorous baddies, who do all the evil things we don't feel able to, have an honoured place in commercial fiction (Alexis in Dynasty) but obscurely unpleasant and disgusting ones don't.
9) expect the reader to enjoy a story where "less happens", not because nothing happens but because "what happens" is less straighforward terms of action and emotion, and may be more elliptically expressed or less clearly formed in terms of growth-and-change.
10) expect the reader to enjoy a story where what you're supposed to think is less clear: both characters and outcomes are not obviously good or bad, desirable or undesirable, and it may not be obvious what the reader should want to happen. This is a point excellently made in Francis Spufford's The Child that Books Built.
11) work with explicit and/or implicit references to literature (not just Pride & Prejudice and Anne of Green Gables), art, film, philosophy, music or a few of the gazillion other bits of culture/language/ideas which you can't rely on all readers to have at their fingertips - or even play games with those things in ways which, again, make the basics of story harder to grasp and enjoy.
12) have more equivocal or unresolved explorations of the subject and themes, with more morally complex characters, with more challenging (because less satisfying) outcomes of the story.
13) expect the reader to be happy not to have the basic genre-boxes ticked: this book may not deliver the satisfactions we want from a thriller, a romance, a whodunnit or an adventure-story: what kind of action, what matters, what ending we're hoping for, what ending we're dreading.
14) if it does work as a genre story, it works with it in challenging, questioning or playful ways: both providing and refusing to provide the genre pleasures, being conscious of them, making implied or explicit jokes about them
14) if it uses a genre setting - history, space, fantasy, exotic location - to use it in challenging and original ways. Yes, a non-western-contemporary setting doesn't mean the book isn't about us here and now - a novel is always about that at some level, or why would you be writing it? But literary historical fiction isn't just modern stories with swords and nicer frocks, literary sci-fi isn't just space ships instead of cars.
And yes, I'm sure you're now boiling with examples of commercial novels which do some of these, and literary novels which don't. The point I'm trying to establish is that the more of these a novel does, the more literary it'll seem to most readers. Notice how much I'm saying "more" and "less" and "most"? That's because it's a spectrum, not a binary either/or. But the trade has to decide where on the spectrum a book is, and what sort of readers cluster there, because thereby hang many decisions about the best way to get it under the noses of reader who might buy it, and like it, whether it's covers, or review copies or prizes.
