My answer to the question will be NO.
Never mind whether the backlinks is from pr0, unrelated sites, sig links, dead sites, porn sites, maybe it does not help you as much, but it will not hurt you.
Otherwise anyone can do harm to competitors by just linking to them.
Someone earlier mentioned about google banning your site for bad backlinks,
that's only when you do link spamming, eg having thousands of backlinks in a small period of time say a few days.
That will trigger google algo to think you're doing something fishy and penalise you.
But if your site is really getting naturally famous, don't worry.
Just focus on getting more links, especially with your targeted keywords.
In fact, even low
PR link from unrelated sites will help you.