When choosing to link to a website, do so because you feel it is truly a quality resource for your visitors and the odds are very good that you won't go wrong. If you are linking to a website that is obviously outside the
quality guidelines of Google, then you might have reason for concern, but don't worry about sites that are linking to you, as this is outside of your control, only worry about those that YOU are linking to.