It will not harm your site at all, The Panda update was targeted towards "content farms" that contained junk and spam content. The irrelevant pages may not help you much as far as rankings but they certainly will not hurt. What you should be looking at is the "quality" of your content.
Is your content of high quality ? Before you answer that and say yes, do you understand what high quality content is ? Let me explain what it is so you do understand. In order to create quality content, your articles must contain proper spelling and grammar, semantically related words, and read at a proper grade level that corresponds to your niche, according the Flesch-Kincaid readability test.
Tweak the quality of your content and make sure it is offering your visitors and search engines what they want. Once you have tweaked this content then build some backlinks and wait and see what happens. This is a part of marketing, testing!
disclaimer: the above is the opinion of Ord Allenbea