If a fantasy book had an unhappy ending, would that affect how it was received by publishers and readers?
This is an interesting question to be asking! I'm going to preface this by saying a couple things.
First, there is a difference between UNHAPPY and UNSATISFYING. These are two completely different things. For example: many classic tragedies are enitre stories with momentum pushing toward the tragic. A modern fantasy example would be some of George R. R. martin's work, where the books often have tragic endings, with the protagonists losing or dying. (Granted, his series isn't done yet, so there's no way to know yet if the final ending will be tragic or triumphant.)
These books are still satisfying, however. The tone of these stories implies that tragic events will occur--and sadness is a powerful emotion. Stories exist, in part, to explore emotion. If the Story is built well, and handled expertly, the reader will be SATISFIED with the ending even if it's tragic. You will feel, "This is where the story was supposed to go. Even if I don't like what happened, it's beautiful in its tragic nature."
Many long form stories also tend to have a balance bittersweet ending. Some things are accomplished, some things are lost. As one might say on Roshar, it's not about the last page--it's about whether the journey there was worthwhile.
In response to your question, then, my instinct says that the sadness of the ending doesn't have a direct correlation with sales, goodreads rating, etc. Quality and deft ahndling of the material will certainly affect these things--but not specifically if the ending is happy or not. Publishers would certainly publish one with a sad ending. Note that if you take the bodies of work by some creators (Including both Shakespeare and Star Wars) the most popular and most successful installments WERE the ones with the sad endings.
(Note that I DO think certain readers are going to dislike an ending that is sad, while others are going to dislike an ending that is too neat and happy. Individual certainly will have opinions. I just think the balance, at the end, will probably be around the same.)
That said, you do focus on a "Bad" ending, equating it with sad. So in the interest of discussion, I'll call this a sad ending to an otherwise upbeat book--a twist of tone that happens right at the end, unexpectedly, leaving the reader frustrated. This would be an ending that completely defies genre conventions. The heroic adventure story where the hero unexpectedly dies at the end, or the Jane Austen style romance that ends with the love interest running off with some other woman.
There would be a subset of people who would just love this, but I think if the book doesn't give the proper tone promises at the start, it would create a less commercially viable work. I don't think this is a reason not to try something like that as a writer, but I do think you might have more trouble finding an audience.