[OSM-Devserver] we need to reduce the server load!

[Nic Roets]

Hello Sven,

This graph tells me that many of the CPUs are idle while Linux is
waiting for something to be swapped in:
http://munin.openstreetmap.de/gauss/gauss-cpu.html

So it looks like we are running too many things at the same time and
have too little RAM. Fortunately we can reduce the amount of RAM
required by running the batch processes in serial. So instead of given
each project it's own cron job, we must have one shell script that
calls the batch processes one at a time. The shell script must 'time'
each process and post the results in a public place. Then we see who
are "good" and who need to optimize more and / or scale back.

A similar idea is to give each batch process a slot (window). A fixed
amount of run time (at night). But filling the slots may not be
straight forward, because the user may want to processes a different
part of the map each night and not all parts of the map may be the
same size.

Personally I don't mind scaling gosmore back a little bit. I killed it
right now. But I do think Gosmore is important because the prebuilt
files now covers the US and it takes time to build a user base there.
(Very few OSM projects cover the US). Gosmore also supports a very
extensive list of OSM tags: Everything from surveillance cameras and
trees to multipolygons and turn restrictions.

Regards,
Nic

2010/3/19 Sven Geggus <lists at fuchsschwanzdomain.de>:
> Hello altogether,
>
> I'm going to write this in english because its important to anybody
> using the devserver machine (gauss).
>
> Currently we have at least two relevant projects (in terms of users)
> which is the Reit- und Wanderkarte (TOPO Map ->
> http://topo.geofabrik.de and the "All in One Garmin Map".
>
> I think that in future the processing power needs to be scheduled in
> a matter of project importance!
>
> Sorry guys I would like to provide a seperate Machine to any one of
> you, but I don't have them :(
>
> The TOPO Map does currently have insufficient resources for tile
> rendering. This needs to be changed!
>
> Just as an example how important or unimportant the different
> projects are, here are some access statistics (counted by quick and
> dirty shell one-liners) from March 14th to March 19th:
>
> Reit- und Wanderkarte: 2484
> AIO Garmin Map: 1562
> Navipowmmaps: 56
> Gosmore Maps: 90
>
> This is access from unique ip-addresses. Multiple downloads from one
> ip-address are counted as one. Only real downloads are counted in case of
> the Maps.
>
> So we have a real disproportion between processing power used and
> overall importance of the single projects!  OSM2POWM is using rougly
> the same processing power as the Garmin Map build process every day
> while the latter has an importance which is about 25 times higher!
>
> I also think that good Garmin maps are very important for the project
> (AIO includes Mapper features like Openstreetbugs, Fixme etc.) and
> the download numbers approve this.
>
> This does not mean that I think of the others to be unimportant, but
> I think it should be enough to produce up to date Navipowm and
> Gosmore maps once a week.
>
> The trouble is mainly with data conversion tasks and bulk database
> imports that could be processed in a batch job like manner in future.
>
> Probably its also a problem with the current lack of some kind of batch
> processing scheduler (any hints for a suitable software?).
>
> Currently all these jobs are started by cron when the users thought it
> would be the best time to have them running.
>
> Maybee the two important projects can also get migrated to the
> machine which is currently not yet that busy (wms aka humboldt), but
> I think this would only defer the problem rather than resolving it.
>
> So what do you think?
>
> Regards
>
> Sven
>
> --
> Den Rechtsstaat macht aus, dass Unschuldige wieder frei kommen
> (Wolfgang Schäuble)
>
> /me is giggls at ircnet, http://sven.gegg.us/ on the Web
> _______________________________________________
> Devserver mailing list
> Devserver at lists.openstreetmap.de
> http://lists.openstreetmap.de/mailman/listinfo/devserver
>