[OSM-Devserver] proposal for a poor mans?batch?processing?system ?:)

[Nic Roets]

> Nic Roets <nroets at gmail.com> wrote:
> But if my tasks are started at random time, there is a chance that
> it's peak memory usage coincides with peak memory usage of another
> task.

2010/4/27 Sven Geggus <lists at fuchsschwanzdomain.de>:
but this is exactly what I try to _avoid_ using some kind of batch
mechanism. Unfortunately all of them seem to either be unsuitable for our
task or overkill like Sun gridengine.

2010/4/27 Sven Geggus <lists at fuchsschwanzdomain.de>:
>
>> How will Batch know that the memory usage of BboxSplit will peak at 3
>> GB while the memory usage of Gosmore will peak at only 750MB ?
>
> I don't care what your scrtips are doing, I just care about the fact that
> different people do not interfere with each other.
>

Hello Sven,

I know that.

I'm just trying to point out that a finding a 3rd party job scheduling
system that can optimize the things that are important to us may not
exist.

Whether or not we use a 3rd party system, we need to decide which
projects are more important. Perhaps based on the number of distinct
web hits.

I would like to see a "budget" or a "target" for each project, with a
spare / reserve amount, e.g. projects X is allocated Xi CPU hours per
week*. Not a hard quota, but a guideline. Then we can measure it and
refine it. Then we don't have the Tragedy of The Commons. Frederik
hinted at this in his email.
http://en.wikipedia.org/wiki/The_Tragedy_of_the_Commons

Furthermore I would like see fixed times and days of a week* reserved
for the batch projects. During those times those projects should have
priority over other batch projects. It makes it easier for project
owners to be logged in and catch any bugs. If a project overruns its
allocated time, it should be niced, so that it does not interfere with
subsequent processes.

weeks*: Substitute with 2 or 3 week cycles, whichever will be more appropriate.

Regards,
Nic