Today while theming Views, I came across a View that was displaying bizarrely. Basically, our Book Clubs block display showed multiple instances of the same entries. It was the only display that did this, all the other block views filtered by taxonomy term and worked fine.
Connor, our intern, observed that the default Fields for the View included 'taxonomy: term', which in turn showed up in the book clubs block, even though we weren't using that field for that block. I removed the 'taxonomy: term' Field from the Book Clubs Block display and the problem went away.
The moral of this story: For each View display, use ONLY the necessary elements for the query. Don't leave Fields, Filters, etc in the display that aren't being used.