I have wondered the same thing Anet-M...but i have recently seen some sites with horrible designs who are ranking regularly on page 1. So I'm starting to think that the sites design doesn't have quite as much effect as I previously thought.
I have done quite a bit of design changes to my site and have maintained my ranking in several keywords.
In some cases yes, major changes to the websites structure, can shake the rankings very hard. For instance, assume dozens of your internal content pages have their own incoming links and then suddenly you change their URLs, move from HTML to PHP. Naturally the links will not seem that relevant in such cases. Or when you quickly alter titles and descriptions of dozens of pages on your website, the ones already cashed and indexed by Google and use completely new stuff, you can expect a drop in rankings in such cases.
Well, I think it's not harm cause seo. Because you only and only change the Design not content. Your site harm if you change in content and meta details. If you do so than you have new keywords and start work from beginning.