[OSM-Devserver] we need to reduce the server load!

[Jochen Topf]

On Fri, Mar 19, 2010 at 10:14:04PM +0200, Nic Roets wrote:
> This graph tells me that many of the CPUs are idle while Linux is
> waiting for something to be swapped in:
> http://munin.openstreetmap.de/gauss/gauss-cpu.html
> 
> So it looks like we are running too many things at the same time and
> have too little RAM. Fortunately we can reduce the amount of RAM
> required by running the batch processes in serial. So instead of given
> each project it's own cron job, we must have one shell script that
> calls the batch processes one at a time. The shell script must 'time'
> each process and post the results in a public place. Then we see who
> are "good" and who need to optimize more and / or scale back.

That sounds like a good approach. We have to take into account, though,
when files appear upstream, for instance the europe extract or whatever.

So maybe we need one batch script for each input file. And then a master
batch script for all of them. So something like this:

* master runs in a loop for each input script
  * each input script
    * checkes for new input file upstream, exists if no new file
    * runs all dependend batch scripts

In praxis its a bit more complex, because some things should run more
often than others etc. 

Jochen
-- 
Jochen Topf  jochen at remote.org  http://www.remote.org/jochen/  +49-721-388298