diff options
| author | Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org> | 2024-04-17 14:50:19 +0000 |
|---|---|---|
| committer | Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org> | 2024-04-18 15:15:21 +0000 |
| commit | efc2d53716f93044bcaceb47e00f4a4bd7c4bdb4 (patch) | |
| tree | fd3b81dd4ff03cf3ee00104a4c58f5348780d19e | |
| parent | 6611a5de48fedd269fb170d517ac9e89daf22ba9 (diff) | |
This patch improves survivability of flaky entries in base-artifacts
history. It improves handling of the following scenario:
1. Trim_base_artifacts() has removed some really old "non-forced"
history entries, say, from 6 months ago. With it, it removed
a 6-month-old flaky entry.
2. We trigger re-write of history for some good reason.
3. A "non-forced" entry from 5 months ago no longer passes because
it is missing a flaky entry from the removed revision -- note
the time difference between removed 6-month old entry and this
5-month old entry is within flaky active period.
4. The "non-forced" entry from 5 months ago is removed. With it,
it removed a 5-month-old flaky entry.
5. And so on into the present time.
It appears that we can mitigate the above behavior by adding more
flaky entries to the "forced" history entries. E.g., we could stop
passing known flaky list to ABE in "forced" builds -- this will
make ABE work hard to re-detect as many flaky tests as possible.
These flaky entries will then remain in history forever because
of update_baseline==force setting.
Change-Id: Ie6bf3e0058a10f316a5cfd45dcf6bf5dbec474f9
| -rw-r--r-- | round-robin.sh | 25 |
1 files changed, 17 insertions, 8 deletions
diff --git a/round-robin.sh b/round-robin.sh index f7fb019d..ed23bf00 100644 --- a/round-robin.sh +++ b/round-robin.sh @@ -479,17 +479,26 @@ build_abe () "--flaky-failures" "$flaky_tests" "--expected-failures" "$baseline_fails") - if [ "${rr[update_baseline]}" != "ignore" ]; then + if [ "${rr[update_baseline]}" = "onsuccess" ]; then # If we have a chance to commit $new_flaky into base-artifacts.git - # then pretend it is 8 weeks into the future and ignore flaky entries - # that will be expired by then. This, effectively, gives us 8 weeks - # to re-detect/confirm flaky tests without no_regression_p() - # noticing anything. + # then pretend it is 8 weeks into the future and ignore flaky + # entries that will be expired by then. This, effectively, gives + # us 8 weeks to re-detect/confirm flaky tests without + # no_regression_p() noticing anything. # We will then set expiration date to "now+12 weeks" (see below) # for entries in $new_flaky/"sumfiles/flaky.xfail". - local week_from_now - week_from_now=$(date -d "now+8 weeks" +%Y%m%d) - rerun_failed_tests+=("--failures-expiration-date" "$week_from_now") + local weeks_from_now + weeks_from_now=$(date -d "now+8 weeks" +%Y%m%d) + rerun_failed_tests+=("--failures-expiration-date" "$weeks_from_now") + elif [ "${rr[update_baseline]}" = "force" ]; then + # If we know that we will commit $new_flaky into base-artifacts.git, + # then ignore known flaky entries to try and detect as many + # flaky tests as possible. "Forced" history entries will remain + # in base-artifacts more-or-less forever, while "onsuccess" entries + # are eventually discarded to keep history size reasonable. + # For "update_baseline==init" we have this behavior naturally, + # since there is no history to pull flaky entries from. + echo "# Ignoring known flaky entries" > "$flaky_tests" fi case "${rr[ci_project]}" in |