There’s definitely a relationship between the size of the advance and how well publishers think a book would sell. But publishers don’t have to pay royalties on most books, anyway, because most books don’t earn back their advances, and you don’t start earning royalties until they do.

One figure I’ve seen is that 70% of books don’t earn back their advances (i.e., earn no royalties). That figure is impossible to verify, given that there are hundreds (maybe thousands) of publishers who have different approaches to advances and royalty structures. But every editor or literary I’ve talked to about this has agreed that most books don’t “earn out”: in other words, at least 50% of books don’t earn royalties.