I find it curious that whenever this comes up, 99% of the time the "example" book that supposedly wouldn't be published today happens to be a rule-breaking and/or idiosyncratic and/or conventional-wisdom-defying classic. Ulysses or The Sound and the Fury or Infinite Jest or Moby-Dick etc. etc. etc. And more curiously still, the thing that most of these books have in common is that they were written by an author who had already established huge followings and credibility the old fashioned way.
The Hits Before the Hits
J.K. Rowling did not start off writing 200,000+ word books for middle graders where important beloved characters, ya know, die. By the time she wrote Goblet of Fire and Order of the Phoenix, the longest in the series, she had the audience's trust to delve extremely deeply into the world of her novels and to explore a deep emotional palette, deeper than may have have been possible with a debut.
J.R.R. Tolkien wrote The Hobbit, a more conventionally told tale, before dropping the epic The Lord of the Rings.
James Joyce wrote Dubliners before Ulysses. David Foster Wallace wrote the relatively trim (467 pages) The Broom of the System and had a zillion short stories published everywhere that matters before he wrote Infinite Jest. Herman Melville wrote conventional travel memoirs before publishing Moby-Dick (which famously was a bust at first).
In other words, these writers built their audience before they tried to break all the rules.
The Keith Hernandez Rule
There is a classic Seinfeld episode where former Mets baseball star Keith Hernandez is on a date with Elaine, and he's worrying about whether he should kiss her. Then he thinks to himself, "Wait a second, I'm Keith Hernandez!"
The Sports Guy calls this a Keith Hernandez Moment -- when a sports star realizes, wait a second, I'm LeBron James, I'm going to score the next 25 points.
Writers who have achieved "I'm Keith Hernandez!" status haven't just achieved the trust of the publishing industry, they've achieved the trust of readers, who will stick with them longer than they would have otherwise if it were a debut. They've earned the ability to delve in deeper into a world or into an idiosyncratic style than would normally be possible because they have gained the authority to do it.
But for the most part, if you're going to take a journey with someone deep into the wilderness, the first step is convincing the other person that you're a good guide.