monitor/README.txt

LLVM Buildbot Monitor
=====================

This is to be left running on a server or desktop as monitoring for your buildbots.
It purely reports the status of the builds. If you want hardware monitoring, look
elsewhere.

It supports Buildbot (as used by LLVM) and Buildkite (https://buildkite.com/, used by LLVM's
libcxx project). It does not support LLVM Green Dragon (https://green.lab.llvm.org/green/).

Buildkite does require an API key to be available, see the script for how to do that
and contact your Builkite org admin to get one.

Currently we have one monitor running at http://llvm.validation.linaro.org/.
Bookmark this if you have a need to check bot status at a glance.

JSON Format
-----------

The JSON file describes the bots we want to monitor and which master/build service
they connect to.

Buildbot JSON Format
--------------------

The base structure is a list of masters, which has a few properties and a list
of builder groups, which in turn also have some properties and a list of bots
(which in Buildbot terms are actually called "Builders" but we ended up calling
them bots here).

Master properties:

    "name": Name of the master, which will appear as the section title.
    "base_url": The base URL of the master, which will be used to make API calls.
                For example for LLVM this might be "https://lab.llvm.org/buildbot".
    "builder_url": The part of the URL that refers to the list of builders.
                   Will be added to base_url when making API calls.
    "build_url": Part of the URL that refers to the list of builds. Added to base_url
                 when making API calls.
    "ignore" : Set to "true" to hide the master from the page.
    "builders": [ ...a list of builder groups as detailed below... ]

Builder group properties:

    "name": Name of this group. "fast bots", "self-hosting", etc. Used as the section title.
    "ignore" : Set to "true" to hide this builder group from the page.
    "bots": [ ...a list of bots as decribed below... ]

Bot properties:

    "name": The exact name of the buildbot. This will be used to build URLs for API calls.
    "ignore": Set to "true" to ignore the status of this bot.

Notes on bots:
  * Bots may be repeated across builder groups if they fall into multiple categories
    (this does not slow down the monitor as results are cached).
  * The same bot name on 2 different masters refers to 2 different bots.

Note that "ignore" has two different behaviours:

 * On masters and builder groups, it omits the entire section from the output.
 * On bots it shows the bot but ignores their status. Meaning that an ignored bot failing
   does not make the overall page status failed.

Buildkite JSON Format
---------------------

The Buildkite format follows the Buildbot format closely with some differences.

  * Since Buildkite is a centralised service the "name" is always "Buildkite".
  * The "base_url" is always "https://www.buildkite.com".
  * There is a new key "buildkite_org". This is used to find our particular bots
    via the API.
  * The "builder_url" and "build_url" keys are used to form clickable links,
    and are always "builders" and "builds".
  * Bots have an extra key "buildkite_pipeline" (explained below).

When querying Buildkite we make a request like this:
"show me the results of the pipeline <buildkite_pipeline> in the organisation <buildkite_org>"

From there the script looks for the bot's name in the last finished build. After that
all the processing is the same as the Buildbot results.

HTML Page
---------

The script will generate an HTML page. This page is separated into blocks:
  * Masters which contain...
    * Builder Groups which contain...
      * Bots

The date is printed at the top of the page so you know when the results were generated.

Bots that are offline or partially fail to read via the API will show up with a message
along the lines of "<bot name> is offline!". The page should still update correctly
for the rest of the bots.

Each listed bot has these columns:

  * "Buildbot": This shows the name and a link to the master's web interface for the bot.
  * "Status": The status of the last finished build. PASS or FAIL (currently cancelled
    is also treated as a failure).
  * "T Since": The time since the last build finished. This is useful for spotting bots
    that have gotten disconnected. If this time is greater than 24 hours, it will be shown
    in red.
  * "Duration": The length of the last build.
  * "Latest": The build number of the last finished build, which itself will be a link
    to the results page for that build.
  * "Failing steps": The failed build steps, if it was a failed build.
  * "Build In Progress": This will be "Yes" if there is a build currently running, and
    the text will link to the build. Or "No" if there is not. If we cannot determine this,
    it will be left blank and you should check the builder's status page instead.
  * "1st Failing": The number of the first failed build, if the bot fails for several
    builds.
  * "Failing Since": The time since the first failed build finished. This is useful for
    spotting bots that fail for a long time.

Note: "finished" here refers to the build ending be that by success, cancellation or
failure.