Complete rewrite of create_index_pages

The old version was very hard on all of CPU, memory,
and I/O. The new version tries to at least lessen the
burden on memory (and to a lesser extend on CPU), in the
hopes that this will indirectly (by lesser swapping) also
reduce the stress on I/O.

In the old version we first collected all the information
about all packages and then wrote the index files one after
one with TT.

The newer version tries to write all of the index files while
collecting the package information, therefor eliminating the
need to store any information longer than needed. Because the
amount of files we can open simulaniously is limited though,
we can't write all the index files at once. Instead we write
one index files for all languages, and in a second step then
use these files to write one file per language. I tried to
use the "slice" program for that, but it also computes all
the results in memory, while I wanted to have a concurrent
approach here as well.
Since we only use a very limited subset of the features of slice
I wrote a simple replacement that supports the bare minimum of
what we need with the price that we need to be a bit careful
about the format of the slice source files.

I also don't use TT for the package entries anymore since here
as well we only used a very limited set of features and I wanted
to make this script as fast as possible (without having to rewrite
it in C ;). The header and footer still use TT since this part isn't
speed critical and we can reuse the templates we have for the dynamic
pages.

1 files changed, 21 insertions, 0 deletions

diff --git a/cron.d/900index_pages b/cron.d/900index_pages
index 1213e36..6268c12 100755
--- a/cron.d/900index_pages
+++ b/cron.d/900index_pages
@@ -4,8 +4,29 @@
 
 cd "$topdir"
 
+set -e
+
 date
 ./bin/create_suite_index_pages
 date
 ./bin/create_index_pages
 date
+
+# for file in $(find www/ -name *.slices)
+# do
+#     dir=$(dirname $file)
+#     base=$(basename $file .slices)
+#     target=
+#     for l in $polangs
+#     do
+# 	lu=$(echo $l|tr /a-z/ /A-Z/)
+# 	target="$target -o ${lu}uUNDEF:$dir/$base.$l.html.tmp"
+#     done
+    
+#     echo slice $target $file
+#     slice $target $file
+# done 
+
+./bin/trivial_slice en $polangs -- $(find www/ -name *.slices)
+
+date