Opened on 03/16/15 at 08:29:54
Closed on 05/20/15 at 07:14:55
#2307 closed enhancement (fixed)
Improved watchdog implementation
Reported by: | Sebastian Huber | Owned by: | Sebastian Huber |
---|---|---|---|
Priority: | normal | Milestone: | 4.11 |
Component: | score | Version: | 4.11 |
Severity: | normal | Keywords: | |
Cc: | Alexander Krutwig | Blocked By: | |
Blocking: |
Description (last modified by Sebastian Huber)
Benefit
Improved average-case and worst-case performance. Uni-processor configurations will also benefit from some changes, e.g. the watchdog insert without restarts.
Problem Description
The timekeeping is an important part of an operating system. It includes
- timer services,
- timeout options for operating system operations.
On RTEMS the timekeeping is implemented using
- watchdog delta chains for timer and timeout services, and
- a clock tick function
rtems_clock_tick()
.
Global Watchdog Delta Chain
RTEMS uses two global watchdog delta chains. One for clock tick based timers and one for seconds based timers. This approach is not scalable, since each additional processor will add more load to the watchdog handler making it a bottleneck in the system. Inter-processor interrupts must be issued to propagate scheduling decisions from the processor serving the clock tick interrupt to the processor assigned to execute a thread. The insert operation has O(n) time complexity so it is desirable to have short watchdog delta chains. Also removal of watchdog controls leads to a restart of the insert procedure.
Giant Lock for Watchdog Delta Chain
The watchdog handler disables interrupts to protect critical sections. Since this is insufficient on SMP configurations the complete clock tick function is executed under Giant lock protection on SMP. Thus the Giant lock section time depends on the execution time of all timer services in one clock tick, which can be arbitrarily long and is out of control of the operating system.
Problem Solution
Move the global watchdog state variables
_Watchdog_Sync_level
,_Watchdog_Sync_count
,_Watchdog_Ticks_chain
, and_Watchdog_Seconds_chain
into a watchdog context structure and modify the watchdog operations to use a watchdog context instead of global variables directly.
Add an interrupt lock to protect the watchdog state changes.
Replace the watchdog synchronization level and count, because the current approach does not work on SMP, since the interrupts can happen not only on the local processor. The watchdog operations are
- watchdog insert (requires a forward iteration of the delta chain, O(n); due to the possibility of restarts, it has an essentially unbounded execution time with the current implementation),
- watchdog removal (constant time operation, O(1)), and
- watchdog adjust (requires a forward iteration of the delta chain in the worst-case, O(n)).
The watchdog synchronization level and count is used to detect the removal of watchdogs during a watchdog insert procedure. In case a removal is detected the iteration restarts. This can be avoided using a technique similar to the SMP lock statistics iteration (SMP_lock_Stats_iteration_context
). This would turn all watchdog operations into worst-case time O(n) operations. For insert and adjust n is the count of watchdogs in the chain. For removal n is the count of threads performing an insert operation.
Change History (8)
comment:1 Changed on 03/16/15 at 08:33:36 by Sebastian Huber
Component: | SMP → cpukit |
---|---|
Summary: | Improved watchdog implementation for SMP → Improved watchdog implementation |
comment:2 Changed on 04/14/15 at 06:22:35 by Sebastian Huber <sebastian.huber@…>
comment:3 Changed on 04/15/15 at 07:03:19 by Sebastian Huber
comment:4 Changed on 05/16/15 at 09:31:59 by Sebastian Huber
Milestone: | 4.11.1 → 4.11 |
---|
comment:5 Changed on 05/16/15 at 09:38:47 by Sebastian Huber
Status: | new → accepted |
---|
comment:6 Changed on 05/20/15 at 07:13:01 by Sebastian Huber
Description: | modified (diff) |
---|
comment:7 Changed on 05/20/15 at 07:14:42 by Sebastian Huber
comment:8 Changed on 05/20/15 at 07:14:55 by Sebastian Huber
Resolution: | → fixed |
---|---|
Status: | accepted → closed |
In 1dc66622762fa9921a62fc0cd73f12641278f23f/rtems: