A few things I've noticed about the line between literary fiction and genre fiction, collected over the years without any particular order.
Donna Tartt writes literary fiction with the pacing of a thriller and outsells most thriller writers. The Secret History is a campus murder mystery. The Goldfinch is a Dickensian adventure novel with a stolen painting at its center. Both get shelved in the literature section. If she'd published under a pseudonym and a different cover design, bookstores might have filed them somewhere else entirely.
"Literary fiction" is a marketing category. Publishers invented it in the mid-twentieth century to distinguish "serious" novels from the paperback racks. Before that, Dickens was popular entertainment. Dostoevsky published in serial magazines alongside sensational crime stories. The category we now treat as timeless is younger than most people's grandparents.
Cormac McCarthy wrote westerns and post-apocalyptic novels that sit on university syllabi next to Joyce and Faulkner. Blood Meridian follows a scalp-hunting gang across the Mexican border. The Road is a survival story about a father and son walking through ash. No Country for Old Men became a Coen Brothers film with a coin-flipping villain. He never explained how any of it counted as literature, because he rarely explained anything. He never did interviews if he could help it. He died in 2023, and the work still doesn't fit neatly into a single shelf.
The literary vs genre fiction debate gets loudest in MFA workshops and quietest in bookstores. Readers pick up what interests them. The anxiety over categories tends to live in the people who teach or review books, less so in the people who buy them.
Some genre writers write better sentences than some literary writers. This shouldn't be a controversial observation, but in certain rooms it still is.
Margaret Atwood has always insisted that The Handmaid's Tale and Oryx and Crake are speculative fiction, not science fiction. She's drawn the line at plausibility, arguing that science fiction includes "talking squids in outer space" while speculative fiction stays within the bounds of what could actually happen. Whether you agree with the line she's drawing, the fact that she feels the need to draw it tells you something about the anxiety that still runs through literary culture around genre labels. Atwood has won the Booker Prize twice. She shouldn't have to defend her shelf placement, and she knows it, and she does it anyway.
I know people who read Kazuo Ishiguro and Stephen King in the same month and don't see a contradiction. I think they're right. Never Let Me Go is a science fiction novel that got reviewed as literary fiction because Ishiguro wrote it. If a debut genre author had published the same book with the same sentences, it would have landed in a different section of the store.
The Pulitzer Prize for Fiction has gone to genre-inflected novels more and more in the past two decades. The Road won it. So did The Amazing Adventures of Kavalier & Clay, and later The Goldfinch. The award's taste has been quietly shifting, even as the conversation about literary vs genre fiction stays stuck in the same old positions.
I'm honestly not sure the distinction between literary fiction and genre fiction means anything coherent anymore, or if it ever did, and I say that as someone who writes about literary fiction every week and probably should have a firmer opinion on the matter by now.