diff options
| author | Mark Rutland <mark.rutland@arm.com> | 2017-01-03 18:27:01 +0000 |
|---|---|---|
| committer | Alex Shi <alex.shi@linaro.org> | 2017-05-27 11:47:16 +0800 |
| commit | 9f72bfbc3b0786e2025b177e959f8bc68659fe07 (patch) | |
| tree | 7944f307abe7a3f8336d697cd3fe5405226edb04 | |
| parent | f2082852632e850bb15e9985a833d87b7455d731 (diff) | |
arm64: restore get_current() optimisation
Commit c02433dd6de32f04 ("arm64: split thread_info from task stack")
inverted the relationship between get_current() and
current_thread_info(), with sp_el0 now holding the current task_struct
rather than the current thead_info. The new implementation of
get_current() prevents the compiler from being able to optimize repeated
calls to either, resulting in a noticeable penalty in some
microbenchmarks.
This patch restores the previous optimisation by implementing
get_current() in the same way as our old current_thread_info(), using a
non-volatile asm statement.
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reported-by: Davidlohr Bueso <dbueso@suse.de>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 9d84fb27fa135c99c9fe3de33628774a336a70a8)
Signed-off-by: Alex Shi <alex.shi@linaro.org>
| -rw-r--r-- | arch/arm64/include/asm/current.h | 10 |
1 files changed, 9 insertions, 1 deletions
diff --git a/arch/arm64/include/asm/current.h b/arch/arm64/include/asm/current.h index f2bcbe2d9889..86c404171305 100644 --- a/arch/arm64/include/asm/current.h +++ b/arch/arm64/include/asm/current.h @@ -9,9 +9,17 @@ struct task_struct; +/* + * We don't use read_sysreg() as we want the compiler to cache the value where + * possible. + */ static __always_inline struct task_struct *get_current(void) { - return (struct task_struct *)read_sysreg(sp_el0); + unsigned long sp_el0; + + asm ("mrs %0, sp_el0" : "=r" (sp_el0)); + + return (struct task_struct *)sp_el0; } #define current get_current() |