High Server CPU Usage Due to JRun

[twalters84]

Hey there,

My client just bought a dedicated server for his website lukebouman.com. The only other website on this server is greenhappenshere.com.

We have been having problems where the CPU Usage gets to about 50% to 70% due to JRun. We are able to see this much under the task manager.

I noticed if I restart the coldfusion 8 application server service the CPU usage drops to almost 0 percent. When I do that, everything goes smoothly for a few hours. However, the high CPU usage issue reoccurs after a few hours.

One developer told me this could be due to session variables. However, the application.cfm file is the only place where session variables are being used. Everytime a session variable is read or written to I am using cflocks.

I also noticed the log files have quite a few exceptions in them. Most of the exceptions are requests timing out.

I really need to figure out the cause of this high CPU usage due to JRun. If anybody has experienced a similar issue, please contact me either on here or via direct email.

Thank you all in advance for any information you can provide me with.

Sincerely,
Travis Walters
admin@codebuyers.com

[~ServerPoint~]

Looks like web application hosted is the main reason. BTW what are changes were made?

[ManageMyServices]

have you checked the network connections, possibly a small DDOS on the server?

[twalters84]

Hey there,

Thanks for the response.

Lukebouman.com use to be hosted with dailyrazor.com. We switched to a dedicated server because the hosting company was complaining we were using too many resources. This referred to the sheer number of photos my client has (close to 500,000). We resized a few thousand new photos each day.

The new dedicated server was able to resize 800 photos / minute when we first put the lukebouman.com website on there. It might have been able to go higher but there was no reason to even try. We were able to resize all of his photos within a couple hours.

So when we got the dedicated server, the lukebouman.com website was super fast. A few weeks later we added the greenhappenshere.com website to the server. The greenhappenshere.com website had no issues on the previous server.

Right before we added the greenhappenshere.com website, lukebouman.com was attacked with SQL Injections. That issue has been fixed recently. I dont think that fix has anything to do with the lag problem though.

My client is posting a description of this problem on rentacoder.com. If anybody is able to identify the reasons for the lag and give suggestions how to fix the issues, we are willing to pay up to 100 dollars US. If anybody is interested in this, let me know and I will give you the link to the rentacoder.com project. The project is not for actually fixing the problem unless it is something small. We just need the problem identified really.

Thanks for anymore assistance you can provide me with.

Sincerely,
Travis Walters
admin@codebuyers.com

[twalters84]

Hey ManageMyServices,

I am not sure how to check network connections for denial of service attacks.

That is an interesting theory though. You see this lag issue started happening after we fixed the SQL Injection issue for lukebouman.com. Somebody was attacking his site recently but we fixed the SQL Injection issue. Maybe the attacker moved onto a new type of attack?

If you can explain how we would check network connections for this type of attack, that would be really helpful. If your theory turns out to be correct, we will compensate you for your time. We would some suggestions to defeat this type of attack.

Sincerely,
Travis Walters
admin@codebuyers.com

[twalters84]

Hey there,

My hosting provider advised me that if we were being DDos attacked that the bandwidth would be through the roof and that is not the case.

I guess the lag issue must be something else. It was a good theory though.

Any more ideas? Thanks again.

Sincerely,
Travis Walters

[SiberForum]

Maybe something sort of Digg effect.
Just another guess from me

[twalters84]

Hey there,

By the Digg Effect, you mean popular websites linking to a smaller website, and the increased traffic slows the site down or renders it temporarily unavailable.

Lukebouman.com does do a lot of linking to news stories from sites such as Digg. However, the content is just west michigan real estate. The other website on the server has not been released to the public yet. We are on a dedicated server and the bandwidth hasnt been all that extensive so I dont think its the Digg Effect.

I did make an update to lukebouman.com yesterday. I used the cfsilent tag to get rid of the white space before every file. I also noticed a few exceptions in the coldfusion logs where certain pages were timing out due to larger queries. I used the appropriate coldfusion tag to request a bit more time to process those pages. So over the next couple days I will see if that has any effect on the lag. I also restarted the server. I dont think that has been done for a couple weeks.

I will let you guys know if I find out anything more.

Thanks again for the ideas.

Sincerely,
Travis Walters

[twalters84]

Greetings,

As you all may know, I have been working the last couple weeks to try to figure out what is causing the lag issue.

I thought I made a breakthrough when I came across this article:

http://www.bennadel.com/blog/154-Col...Spider-III.htm

Basically, when users come to the websites, session variables are created to store common elements such as host name and geo data. Session (cookie data) is stored in two places - on users personal computers and on the server. The problem started occurring because cookies are disabled in web crawlers. So each time a web crawler was crawling a webpage, the server would attempt to initiate new session variables. Over a couple hours,
the memory and CPU usage would increase dramatically.

So the solution to this problem was to differentiate web crawlers from regular users. I updated the code on all websites in the application.cfm file. When a web crawler is detected, I disabled session management. When a regular user is detected, I enabled session management. Thus, session information is only created for regular users now.

We also put an additional 1 GB of RAM on the server.

The lags seems to have disappeared, but with closer investigation, it still looks like memory increases over time. I will explain more momentarily...

Sincerely,
Travis Walters

[twalters84]

Hello again,

I recently installed fusionReactor on the server.

Here is a graph of the CPU usage over time:

http://www.lukebouman.com/temp/server_cpu.bmp

Here is a graph of the Memory Usage over time:

http://www.lukebouman.com/temp/server_memory.bmp

The CPU usage seems to have a pattern to it. It shoots up the same amount about every hour and a half.

With the memory graph, the blue area is used memory. It increases over time and drops off. However, when it increases it also increases the allocated memory and the allocated memory never decreases unless I restart coldfusion.

Does anybody have an ideas why this could be?

Sincerely,
Travis Walters
admin@codebuyers.com