[OSM-Devserver] proposal for a poor mans batch processing system :)

[Kai Krueger]

On 04/27/2010 09:11 AM, Sven Geggus wrote:
> Hello,	
>
> here is my idea of how to reduce the problem of different IO-bound processing
> applications blocking each other.
>
> While there are no traditional batch processing systems for Linux (which
> came as a bit of a surpise to me)

I do wonder a bit why you are still claiming this after I have pointed 
out twice the existence of Sun Grid Engine and Torque/PBS, both quite 
capable OpenSource batch processing systems that have packages in 
Ubuntu. There are presumably more too (GNUbatch sounds like another 
one). They might turn out to be an overkill for the situation on Gauss, 
or not worth the admin effort, but they do exist. And getting a basic 
version up and running doesn't seem that difficult to experiment with 
either.

there is the batch(1) command which should
> help us to prevent jobs from getting started while others are still running.
>
> So here is how I think it could work.
>
> * Do not run your scripts directly using cron/incron
> * Instead use a script which installs a batch job which will run your job
>    as soon as the System load is below 5[1]
> * Inside you cron/incron script check if your old batch
>    job is still pending. If so, do nothing, just terminate[2]
> * directly run batch processing applications from cron/incron are prohibited
>    by admin policy in future and can be killed without warning instead the
>    proposed script should be used by anybody.
>
> What do you think?
>
> Sven
>
> [1] 5 is an estimate which might be tuned a little bit higher or lower.
> [2] If this happens too often we need to reduce the processing frequency or
> get a couple of additional machines :)
>