No, it isn't. You can find Matt Cutts confirming this as well. Basically, if all you have is 4 sites, you're in safe territory.
If you have thousands of sites and you're interlinking them, you'll very likely going to get into trouble.
If you're not interlinking you're sites at all, then you're certainly safe. The vast majority of websites use shared hosting, where hundreds or thousands of domains/sites share a single IP - this is perfectly normal, and Google is well aware of it.