As you all may know, I have been working the last couple weeks to try to figure out what is causing the lag issue.
I thought I made a breakthrough when I came across this article:
the memory and CPU usage would increase dramatically.
So the solution to this problem was to differentiate web crawlers from regular users. I updated the code on all websites in the application.cfm file. When a web crawler is detected, I disabled session management. When a regular user is detected, I enabled session management. Thus, session information is only created for regular users now.
We also put an additional 1 GB of RAM on the server.
The lags seems to have disappeared, but with closer investigation, it still looks like memory increases over time. I will explain more momentarily...