tcwg-llvmbot monitor: Add caching at the point after status is read
So originally I was going to cache all the API responses as JSON,
which would be great if we want to test this thing more later, however:
1. Buildkite uses a wrapper library, so it'd take some hacking.
2. JSON can't encode anything but strings as keys.
So instead I've cached the status data using Python's pickle
just after we've read from both sources. This means you can test
changes to the HTML generation in seconds.
Disadvantge being that the pickle object can't be hand constructed.
Switching to JSON or YAML would mean we have to change a bunch of our
keys though, so I don't see the point in that at the moment.
There is one new optional option, --cachefile. If the path exists,
load from it, if it doesn't read the status and save to it.
Change-Id: I45e6edc875271bf258f66655e03d933cbbf02dee
diff --git a/monitor/bot-status.py b/monitor/bot-status.py
index 16b63c1..f858ef6 100755
--- a/monitor/bot-status.py
+++ b/monitor/bot-status.py
@@ -14,6 +14,7 @@
import json
import tempfile
import logging
+import pickle
import shutil
import time
from datetime import datetime, timedelta
@@ -327,6 +328,12 @@
parser.add_argument(
"-d", dest="debug", action="store_true", help="show debug log messages"
)
+ parser.add_argument(
+ "--cachefile",
+ required=False,
+ help="Location of bot status data cache file (a pickled Python object). If it exists use it, "
+ "if it does not, read the status from the network and write it to this path.",
+ )
parser.add_argument("config_file", help="Bots description in JSON format")
parser.add_argument("output_file", help="output HTML path")
args = parser.parse_args()
@@ -341,6 +348,17 @@
print("error: failed to read {} config file: {}".format(args.config_file, e))
sys.exit(os.EX_CONFIG)
- status = get_buildbot_bots_status(config)
- status.update(get_buildkite_bots_status(config))
+ status = None
+ if args.cachefile and os.path.exists(args.cachefile):
+ logging.debug("Using cache file {}".format(args.cachefile))
+ with open(args.cachefile, "rb") as f:
+ status = pickle.load(f)
+ else:
+ status = get_buildbot_bots_status(config)
+ status.update(get_buildkite_bots_status(config))
+ if args.cachefile:
+ logging.debug("Writing status to cache file {}".format(args.cachefile))
+ with open(args.cachefile, "wb") as f:
+ pickle.dump(status, f)
+
write_bot_status(config, args.output_file, status)