I've heard this expressed many times on here, but from my experience I just do not believe it to be true. Perhaps it is true in the United Kingdom where Morrissey is a much bigger cultural phenomena than in the US and where Quarry probably was a lot more culturally significant/was his bestselling album. However, pretty nearly everyone I've met that skews younger and knows of Morrissey generally either associates him pretty much exclusively with the Smiths or with (at best) his very early solo work like Suedehead and Everyday is Like Sunday.
I mean if you think about it if you ever hear Morrissey on the radio, in a Starbucks, at a bar, etc. it is either going to be the Smiths (most likely) and occasionally Suedehead or Everyday is Like Sunday. I don't think I've ever heard First of the Gang to Die played anywhere other than at the Smiths v. Cure dance party I go to. And I certainly have been surprised by the number of places I've heard The Smiths (over a mall intercom, at a Chipotles, in a Starbucks) and there used to be a bar that I would go to that played quite a bit of Smiths stuff (it was kinda hipstery) and would also frequently play Suedehead/Sunday.
I think this would explain why his Smiths albums/early solo albums did not sell that well upon initial release in the US, especially compared to Your Arsenal, Vauxhall, even his more recent albums, yet have all well outsold the rest of his catalogue, because they continue to consistently sell copies year after year--albeit a small number.
Your Arsenal and Vauxhall does seem to be the peak of his commercial accessibility in the US--going on Jay Leno, getting attention from MTV, etc.--and I think that is why there is this subgroup of fans who hold onto this notion that those are his best works and show much less interest in the Smiths/early-solo work and have a great bit of disdain for his post-2004. I think they also tend be overrepresented here.