1. 16 Oct, 2016 1 commit
  2. 21 Jul, 2016 1 commit
  3. 15 Jul, 2016 3 commits
  4. 04 Jul, 2016 1 commit
    • Shreyas B. Prabhu's avatar
      cpuidle: Fix last_residency division · dbd1b8ea
      Shreyas B. Prabhu authored
      Snooze is a poll idle state in powernv and pseries platforms. Snooze
      has a timeout so that if a CPU stays in snooze for more than target
      residency of the next available idle state, then it would exit
      thereby giving chance to the cpuidle governor to re-evaluate and
      promote the CPU to a deeper idle state. Therefore whenever snooze
      exits due to this timeout, its last_residency will be target_residency
      of the next deeper state.
      
      Commit e93e59ce "cpuidle: Replace ktime_get() with local_clock()"
      changed the math around last_residency calculation. Specifically,
      while converting last_residency value from nano- to microseconds, it
      carries out right shift by 10. Because of that, in snooze timeout
      exit scenarios last_residency calculated is roughly 2.3% less than
      target_residency of the next available state. This pattern is picked
      up by get_typical_interval() in the menu governor and therefore
      expected_interval in menu_select() is frequently less than the
      target_residency of any state other than snooze.
      
      Due to this we are entering snooze at a higher rate, thereby
      affecting the single thread performance.
      
      Fix this by using more precise division via ktime_us_delta().
      
      Fixes: e93e59ce "cpuidle: Replace ktime_get() with local_clock()"
      Reported-by: default avatarAnton Blanchard <anton@samba.org>
      Bisected-by: default avatarShilpasri G Bhat <shilpa.bhat@linux.vnet.ibm.com>
      Signed-off-by: default avatarShreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
      Acked-by: default avatarDaniel Lezcano <daniel.lezcano@linaro.org>
      Acked-by: default avatarBalbir Singh <bsingharora@gmail.com>
      Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
      dbd1b8ea
  5. 17 May, 2016 1 commit
    • Daniel Lezcano's avatar
      cpuidle: Fix cpuidle_state_is_coupled() argument in cpuidle_enter() · e7387da5
      Daniel Lezcano authored
      Commit 0b89e9aa (cpuidle: delay enabling interrupts until all
      coupled CPUs leave idle) rightfully fixed a regression by letting
      the coupled idle state framework to handle local interrupt enabling
      when the CPU is exiting an idle state.
      
      The current code checks if the idle state is coupled and, if so, it
      will let the coupled code to enable interrupts. This way, it can
      decrement the ready-count before handling the interrupt. This
      mechanism prevents the other CPUs from waiting for a CPU which is
      handling interrupts.
      
      But the check is done against the state index returned by the back
      end driver's ->enter functions which could be different from the
      initial index passed as parameter to the cpuidle_enter_state()
      function.
      
       entered_state = target_state->enter(dev, drv, index);
      
       [ ... ]
      
       if (!cpuidle_state_is_coupled(drv, entered_state))
      	local_irq_enable();
      
       [ ... ]
      
      If the 'index' is referring to a coupled idle state but the
      'entered_state' is *not* coupled, then the interrupts are enabled
      again. All CPUs blocked on the sync barrier may busy loop longer
      if the CPU has interrupts to handle before decrementing the
      ready-count. That's consuming more energy than saving.
      
      Fixes: 0b89e9aa (cpuidle: delay enabling interrupts until all coupled CPUs leave idle)
      Signed-off-by: default avatarDaniel Lezcano <daniel.lezcano@linaro.org>
      Cc: 3.15+ <stable@vger.kernel.org> # 3.15+
      [ rjw: Subject & changelog ]
      Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
      e7387da5
  6. 28 Apr, 2016 1 commit
  7. 25 Apr, 2016 1 commit
    • Daniel Lezcano's avatar
      cpuidle: Replace ktime_get() with local_clock() · e93e59ce
      Daniel Lezcano authored
      The ktime_get() can have a non negligeable overhead, use local_clock()
      instead.
      
      In order to test the difference between ktime_get() and local_clock(),
      a quick hack has been added to trigger, via debugfs, 10000 times a
      call to ktime_get() and local_clock() and measure the elapsed time.
      
      Then the average value, the min and max is computed for each call.
      
      From userspace, the test above was called 100 times every 2 seconds.
      
      So, ktime_get() and local_clock() have been called 1000000 times in
      total.
      
      The results are:
      
      ktime_get():
      ============
       * average: 101 ns (stddev: 27.4)
       * maximum: 38313 ns
       * minimum: 65 ns
      
      local_clock():
      ==============
       * average: 60 ns (stddev: 9.8)
       * maximum: 13487 ns
       * minimum: 46 ns
      
      The local_clock() is faster and more stable.
      
      Even if it is a drop in the ocean, changing the ktime_get() by the
      local_clock() allows to save 80ns at idle time (entry + exit). And
      in some circumstances, especially when there are several CPUs racing
      for the clock access, we save tens of microseconds.
      
      The idle duration resulting from a diff is converted from nanosec to
      microsec. This could be done with integer division (div 1000) - which is
      an expensive operation or by 10 bits shifting (div 1024) - which is fast
      but unprecise.
      
      The following table gives some results at the limits.
      
       ------------------------------------------
      |   nsec   |   div(1000)   |   div(1024)   |
       ------------------------------------------
      |   1e3    |        1 usec |      976 nsec |
       ------------------------------------------
      |   1e6    |     1000 usec |      976 usec |
       ------------------------------------------
      |   1e9    |  1000000 usec |   976562 usec |
       ------------------------------------------
      
      There is a linear deviation of 2.34%. This loss of precision is acceptable
      in the context of the resulting diff which is used for statistics. These
      ones are processed to guess estimate an approximation of the duration of the
      next idle period which ends up into an idle state selection. The selection
      criteria takes into account the next duration based on large intervals,
      represented by the idle state's target residency.
      
      The 2^10 division is enough because the approximation regarding the 1e3
      division is lost in all the approximations done for the next idle duration
      computation.
      Signed-off-by: default avatarDaniel Lezcano <daniel.lezcano@linaro.org>
      Acked-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      [ rjw: Subject ]
      Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
      e93e59ce
  8. 08 Apr, 2016 1 commit
  9. 21 Mar, 2016 1 commit
    • Rafael J. Wysocki's avatar
      cpuidle: menu: Fall back to polling if next timer event is near · 0c313cb2
      Rafael J. Wysocki authored
      Commit a9ceb78b (cpuidle,menu: use interactivity_req to disable
      polling) changed the behavior of the fallback state selection part
      of menu_select() so it looks at interactivity_req instead of
      data->next_timer_us when it makes its decision.  That effectively
      caused polling to be used more often as fallback idle which led to
      significant increases of energy consumption in some cases.
      
      Commit e132b9b3 (cpuidle: menu: use high confidence factors
      only when considering polling) changed that logic again to be more
      predictable, but that didn't help with the increased energy
      consumption problem.
      
      For this reason, go back to making decisions on which state to fall
      back to based on data->next_timer_us which is the time we know for
      sure something will happen rather than a prediction (which may be
      inaccurate and turns out to be so often enough to be problematic).
      However, take the target residency of the first proper idle state
      (C1) into account, so that state is not used as the fallback one
      if its target residency is greater than data->next_timer_us.
      
      Fixes: a9ceb78b (cpuidle,menu: use interactivity_req to disable polling)
      Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
      Reported-and-tested-by: default avatarDoug Smythies <dsmythies@telus.net>
      0c313cb2
  10. 16 Mar, 2016 1 commit
    • Rik van Riel's avatar
      cpuidle: menu: use high confidence factors only when considering polling · e132b9b3
      Rik van Riel authored
      The menu governor uses five different factors to pick the
      idle state:
       - the user configured latency_req
       - the time until the next timer (next_timer_us)
       - the typical sleep interval, as measured recently
       - an estimate of sleep time by dividing next_timer_us by an observed factor
       - a load corrected version of the above, divided again by load
      
      Only the first three items are known with enough confidence that
      we can use them to consider polling, instead of an actual CPU
      idle state, because the cost of being wrong about polling can be
      excessive power use.
      
      The latter two are used in the menu governor's main selection
      loop, and can result in choosing a shallower idle state when
      the system is expected to be busy again soon.
      
      This pushes a busy system in the "performance" direction of
      the performance<>power tradeoff, when choosing between idle
      states, but stays more strictly on the "power" state when
      deciding between polling and C1.
      Signed-off-by: default avatarRik van Riel <riel@redhat.com>
      Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
      e132b9b3
  11. 16 Feb, 2016 2 commits
  12. 27 Jan, 2016 1 commit
  13. 21 Jan, 2016 1 commit
  14. 19 Jan, 2016 2 commits
  15. 15 Jan, 2016 2 commits
  16. 14 Jan, 2016 1 commit
    • Rafael J. Wysocki's avatar
      cpuidle: menu: Fix menu_select() for CPUIDLE_DRIVER_STATE_START == 0 · 9c4b2867
      Rafael J. Wysocki authored
      Commit a9ceb78b (cpuidle,menu: use interactivity_req to disable
      polling) exposed a bug in menu_select() causing it to return -1
      on systems with CPUIDLE_DRIVER_STATE_START equal to zero, although
      it should have returned 0.  As a result, idle states are not entered
      by CPUs on those systems.
      
      Namely, on the systems in question data->last_state_idx is initially
      equal to -1 and the above commit modified the condition that would
      have caused it to be changed to 0 to be less likely to trigger which
      exposed the problem.  However, setting data->last_state_idx initially
      to -1 doesn't make sense at all and on the affected systems it should
      always be set to CPUIDLE_DRIVER_STATE_START (ie. 0) unconditionally,
      so make that happen.
      
      Fixes: a9ceb78b (cpuidle,menu: use interactivity_req to disable polling)
      Reported-and-tested-by: default avatarSudeep Holla <sudeep.holla@arm.com>
      Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
      9c4b2867
  17. 17 Dec, 2015 1 commit
  18. 14 Dec, 2015 3 commits
  19. 16 Nov, 2015 3 commits
    • Rik van Riel's avatar
      cpuidle,menu: smooth out measured_us calculation · efddfd90
      Rik van Riel authored
      The cpuidle state tables contain the maximum exit latency for each
      cpuidle state. On x86, that is the exit latency for when the entire
      package goes into that same idle state.
      
      However, a lot of the time we only go into the core idle state,
      not the package idle state. This means we see a much smaller exit
      latency.
      
      We have no way to detect whether we went into the core or package
      idle state while idle, and that is ok.
      
      However, the current menu_update logic does have the potential to
      trip up the repeating pattern detection in get_typical_interval.
      If the system is experiencing an exit latency near the idle state's
      exit latency, some of the samples will have exit_us subtracted,
      while others will not. This turns a repeating pattern into mush,
      potentially breaking get_typical_interval.
      
      Furthermore, for smaller sleep intervals, we know the chance that
      all the cores in the package went to the same idle state are fairly
      small. Dividing the measured_us by two, instead of subtracting the
      full exit latency when hitting a small measured_us, will reduce the
      error.
      Signed-off-by: default avatarRik van Riel <riel@redhat.com>
      Acked-by: default avatarArjan van de Ven <arjan@linux.intel.com>
      Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
      efddfd90
    • Rik van Riel's avatar
      cpuidle,menu: use interactivity_req to disable polling · a9ceb78b
      Rik van Riel authored
      The menu governor carefully figures out how much time we typically
      sleep for an estimated sleep interval, or whether there is a repeating
      pattern going on, and corrects that estimate for the CPU load.
      
      Then it proceeds to ignore that information when determining whether
      or not to consider polling. This is not a big deal on most x86 CPUs,
      which have very low C1 latencies, and the patch should not have any
      effect on those CPUs.
      
      However, certain CPUs (eg. Atom) have much higher C1 latencies, and
      it would be good to not waste performance and power on those CPUs if
      we are expecting a very low wakeup latency.
      
      Disable polling based on the estimated interactivity requirement, not
      on the time to the next timer interrupt.
      Signed-off-by: default avatarRik van Riel <riel@redhat.com>
      Acked-by: default avatarArjan van de Ven <arjan@linux.intel.com>
      Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
      a9ceb78b
    • Rik van Riel's avatar
      cpuidle,x86: increase forced cut-off for polling to 20us · 7884084f
      Rik van Riel authored
      The cpuidle menu governor has a forced cut-off for polling at 5us,
      in order to deal with firmware that gives the OS bad information
      on cpuidle states, leading to the system spending way too much time
      in polling.
      
      However, at least one x86 CPU family (Atom) has chips that have
      a 20us break-even point for C1. Forcing the polling cut-off to
      less than that wastes performance and power.
      
      Increase the polling cut-off to 20us.
      
      Systems with a lower C1 latency will be found in the states table by
      the menu governor, which will pick those states as appropriate.
      Signed-off-by: default avatarRik van Riel <riel@redhat.com>
      Acked-by: default avatarArjan van de Ven <arjan@linux.intel.com>
      Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
      7884084f
  20. 23 Oct, 2015 2 commits
  21. 02 Sep, 2015 1 commit
  22. 28 Aug, 2015 2 commits
  23. 03 Aug, 2015 1 commit
    • Mark Rutland's avatar
      ARM: migrate to common PSCI client code · be120397
      Mark Rutland authored
      Now that the common PSCI client code has been factored out to
      drivers/firmware, and made safe for 32-bit use, move the 32-bit ARM code
      over to it. This results in a moderate reduction of duplicated lines,
      and will prevent further duplication as the PSCI client code is updated
      for PSCI 1.0 and beyond.
      
      The two legacy platform users of the PSCI invocation code are updated to
      account for interface changes. In both cases the power state parameter
      (which is constant) is now generated using macros, so that the
      pack/unpack logic can be killed in preparation for PSCI 1.0 power state
      changes.
      Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
      Acked-by: default avatarRob Herring <robh@kernel.org>
      Cc: Catalin Marinas <catalin.marinas@arm.com>
      Cc: Ashwin Chaugule <ashwin.chaugule@linaro.org>
      Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
      Cc: Russell King <rmk+kernel@arm.linux.org.uk>
      Cc: Will Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      be120397
  24. 21 Jul, 2015 1 commit
  25. 09 Jul, 2015 1 commit
  26. 25 Jun, 2015 1 commit
    • preeti's avatar
      tick/idle/powerpc: Do not register idle states with CPUIDLE_FLAG_TIMER_STOP set in periodic mode · cc5a2f7b
      preeti authored
      On some archs, the local clockevent device stops in deep cpuidle states.
      The broadcast framework is used to wakeup cpus in these idle states, in
      which either an external clockevent device is used to send wakeup ipis
      or the hrtimer broadcast framework kicks in in the absence of such a
      device. One cpu is nominated as the broadcast cpu and this cpu sends
      wakeup ipis to sleeping cpus at the appropriate time. This is the
      implementation in the oneshot mode of broadcast.
      
      In periodic mode of broadcast however, the presence of such cpuidle
      states results in the cpuidle driver calling tick_broadcast_enable()
      which shuts down the local clockevent devices of all the cpus and
      appoints the tick broadcast device as the clockevent device for each of
      them. This works on those archs where the tick broadcast device is a
      real clockevent device.  But on archs which depend on the hrtimer mode
      of broadcast, the tick broadcast device hapens to be a pseudo device.
      The consequence is that the local clockevent devices of all cpus are
      shutdown and the kernel hangs at boot time in periodic mode.
      
      Let us thus not register the cpuidle states which have
      CPUIDLE_FLAG_TIMER_STOP flag set, on archs which depend on the hrtimer
      mode of broadcast in periodic mode. This patch takes care of doing this
      on powerpc. The cpus would not have entered into such deep cpuidle
      states in periodic mode on powerpc anyway. So there is no loss here.
      Signed-off-by: default avatarPreeti U Murthy <preeti@linux.vnet.ibm.com>
      Cc: 3.19+ <stable@vger.kernel.org> # 3.19+
      Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
      cc5a2f7b
  27. 22 Jun, 2015 1 commit
  28. 16 Jun, 2015 1 commit
    • Paul Gortmaker's avatar
      drivers/cpuidle: Convert non-modular drivers to use builtin_platform_driver · 090d1cf1
      Paul Gortmaker authored
      All these drivers are configured with Kconfig options that are
      declared as bool.  Hence it is not possible for the code
      to be built as modular.  However the code is currently using the
      module_platform_driver() macro for driver registration.
      
      While this currently works, we really don't want to be including
      the module.h header in non-modular code, which we'll be forced
      to do, pending some upcoming code relocation from init.h into
      module.h.  So we fix it now by using the non-modular equivalent.
      
      Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
      Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
      Cc: Michal Simek <michal.simek@xilinx.com>
      Cc: linux-pm@vger.kernel.org
      Cc: linux-arm-kernel@lists.infradead.org
      Signed-off-by: default avatarPaul Gortmaker <paul.gortmaker@windriver.com>
      090d1cf1
  29. 29 May, 2015 1 commit
    • Rafael J. Wysocki's avatar
      cpuidle: Do not use CPUIDLE_DRIVER_STATE_START in cpuidle.c · 7d51d979
      Rafael J. Wysocki authored
      The CPUIDLE_DRIVER_STATE_START symbol is defined as 1 only if
      CONFIG_ARCH_HAS_CPU_RELAX is set, otherwise it is defined as 0.
      However, if CONFIG_ARCH_HAS_CPU_RELAX is set, the first (index 0)
      entry in the cpuidle driver's table of states is overwritten with
      the default "poll" entry by the core.  The "state" defined by the
      "poll" entry doesn't provide ->enter_dead and ->enter_freeze
      callbacks and its exit_latency is 0.
      
      For this reason, it is not necessary to use CPUIDLE_DRIVER_STATE_START
      in cpuidle_play_dead() (->enter_dead is NULL, so the "poll state"
      will be skipped by the loop).
      
      It also is arguably unuseful to return states with exit_latency
      equal to 0 from find_deepest_state(), so the function can be modified
      to start the loop from index 0 and the "poll state" will be skipped by
      it as a result of the check against latency_req.
      Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
      Reviewed-by: default avatarPreeti U Murthy <preeti@linux.vnet.ibm.com>
      7d51d979