When your rankings go down, there are 2 possible things that happened: 1. Your competitors did well than you; or 2. Your effort of gathering links to boost your keywords didn't work (so you need to work harder)...

That's just some possibilities... Just continue building your links for that particular keywords, and make sure that you monitor everything..
