path: root/kernel/sched/fair.c
diff options
authorPeter Zijlstra <a.p.zijlstra@chello.nl>2012-08-08 21:46:40 +0200
committerThomas Gleixner <tglx@linutronix.de>2012-08-13 18:41:54 +0200
commita35b6466aabb051568b844e8c63f87a356d3d129 (patch)
treea5d38cce8290a60f6729f97591cfa25d545c6474 /kernel/sched/fair.c
parentb9403130a5350fca59a50ed11c198cb8c7e54119 (diff)
sched, cgroup: Reduce rq->lock hold times for large cgroup hierarchies
Peter Portante reported that for large cgroup hierarchies (and or on large CPU counts) we get immense lock contention on rq->lock and stuff stops working properly. His workload was a ton of processes, each in their own cgroup, everybody idling except for a sporadic wakeup once every so often. It was found that: schedule() idle_balance() load_balance() local_irq_save() double_rq_lock() update_h_load() walk_tg_tree(tg_load_down) tg_load_down() Results in an entire cgroup hierarchy walk under rq->lock for every new-idle balance and since new-idle balance isn't throttled this results in a lot of work while holding the rq->lock. This patch does two things, it removes the work from under rq->lock based on the good principle of race and pray which is widely employed in the load-balancer as a whole. And secondly it throttles the update_h_load() calculation to max once per jiffy. I considered excluding update_h_load() for new-idle balance all-together, but purely relying on regular balance passes to update this data might not work out under some rare circumstances where the new-idle busiest isn't the regular busiest for a while (unlikely, but a nightmare to debug if someone hits it and suffers). Cc: pjt@google.com Cc: Larry Woodman <lwoodman@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Reported-by: Peter Portante <pportant@redhat.com> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Link: http://lkml.kernel.org/n/tip-aaarrzfpnaam7pqrekofu8a6@git.kernel.org Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Diffstat (limited to 'kernel/sched/fair.c')
1 files changed, 9 insertions, 2 deletions
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d0cc03b3e70..c219bf8d704 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3387,6 +3387,14 @@ static int tg_load_down(struct task_group *tg, void *data)
static void update_h_load(long cpu)
+ struct rq *rq = cpu_rq(cpu);
+ unsigned long now = jiffies;
+ if (rq->h_load_throttle == now)
+ return;
+ rq->h_load_throttle = now;
walk_tg_tree(tg_load_down, tg_nop, (void *)cpu);
@@ -4293,11 +4301,10 @@ redo:
env.src_rq = busiest;
env.loop_max = min(sysctl_sched_nr_migrate, busiest->nr_running);
+ update_h_load(env.src_cpu);
double_rq_lock(this_rq, busiest);
- if (!env.loop)
- update_h_load(env.src_cpu);
* cur_ld_moved - load moved in current iteration