[OSM-Devserver] proposal for a poor mans batch processing system :)

[Sven Geggus]

Hello,

here is my idea of how to reduce the problem of different IO-bound processing
applications blocking each other.

While there are no traditional batch processing systems for Linux (which
came as a bit of a surpise to me) there is the batch(1) command which should
help us to prevent jobs from getting started while others are still running.

So here is how I think it could work.

* Do not run your scripts directly using cron/incron 
* Instead use a script which installs a batch job which will run your job
  as soon as the System load is below 5[1]
* Inside you cron/incron script check if your old batch
  job is still pending. If so, do nothing, just terminate[2]
* directly run batch processing applications from cron/incron are prohibited
  by admin policy in future and can be killed without warning instead the
  proposed script should be used by anybody.

What do you think?

Sven

[1] 5 is an estimate which might be tuned a little bit higher or lower.
[2] If this happens too often we need to reduce the processing frequency or
get a couple of additional machines :)

-- 
Das allgemeine Persönlichkeitsrecht (Art. 2 Abs.1 i.V.m. Art.1 Abs. 1GG)
umfasst das Grundrecht auf Gewährleistung der Vertraulichkeit und Integrität
informationstechnischer Systeme. (BVerfG, 1BvR 370/07)
/me is giggls at ircnet, http://sven.gegg.us/ on the Web