Changes between Version 22 and Version 23 of TBR/Review/Debugging/Start

Nov 19, 2018, 8:22:35 PM (5 months ago)
Joel Sherrill

Removed obviously out of date material that has current content elsewhere


  • TBR/Review/Debugging/Start

    v22 v23  
    11= Debugging =
    3 = Symbolic Debug Information =
    6 RTEMS and the RTEMS tool chains support symbolic debugging. The application, RTEMS and any libraries you wish to debug need to be compiled with the '''gcc''' option '''-g'''. This option causes the compiler to emit debugging information, while the code generated should not change. The '''GNU gcc''' debugging options can be found here -
    8 <center>
    10 </center>
    12 Using the '''-g''' option results in larger object and executable file sizes, and for C++ this can be quite large. For example a M68K C++ application can have a ELF executable file size of 19M bytes yet the code size is only 1.6M bytes. The actual memory foot print can be seen by using the '''size''' tool on an object file or the final executable -
    14 {{{
    15 $ m68k-rtems-size myfile.o
    16 }}}
    18 For RTEMS you typically do not need to strip the executable. The loading of the executable into the target memory or conversion to S records, Intel Hex, for binary format will automatically strip the debug information. Keeping the excutable with the debugging information is recommended. If a problem appears with code in the field and you recieve a some sort of dump or trace you can use the '''objdump''' tool to help locate the problem -
    20 {{{
    21 $ m68k-rtems-objdump -D --line-numbers --source --demangle  myapp.elf | less
    22 }}}
    234= Hardware Assisted Debugging =
    5839 * Use the [wiki:TBR/UserManual/Capture_Engine Capture Engine] to aid the debugging and verfication of your real-time design.
    59 = GDB and RTEMS =
    62 Currently GDB is not RTEMS aware. GDB scripts exist that can help by providing presenting kernel structures in a user friendly manner. These can be found here: [wiki:Debugging/GDBScripts GdbScripts].
    64 The issue with making GDB aware of RTEMS is where the knowledge of the kernel structures is located. A version of GDB that contains a specific kernel structure layout will break if RTEMS changes. The ideal would be for GDB to find the structure elements using the applications debug information.
    65 = Eclipse Plug-in =
    67 Eclipse can be used as GUI for GDB. General Eclipse related information can be found at [wiki:Developer/Eclipse/Information RTEMS Eclipse Information]. An more specific example of how serial port is utilized for remote debugging can be found at [wiki:Developer/Eclipse/Plugin RTEMS Eclipse Plug-in].
    68 = How much memory is left in the C Program Heap? =
    71 The C heap is a region so this should work:
    73 {{{
    74   (gdb) p ((Region_Control *)_Region_Information->local_table[1])->Memory->first
    75   $9 {back_flag=1, front_flag=8058280, next=0x7ea5b4, previous=0x7ea5b0}
    76 }}}
    78 Let's look at that gdb command in more detail.
    80  * ''_Region_Information'' contains the information used to manage Region objects.
    81  * ''_Region_Information->local_table'' is the object pointer table for Regions.  It is
    82   indexed by the ''object index'' portion of the object ID.
    83  * ''Region_Information->local_table'' points to the first Region object.  It is of type ''(Region_Control *)''.
    84  * ''((Region_Control *)_Region_Information->local_table[1])->Memory'' points to the Heap control portion of this Region's control block.
    85  * ''((Region_Control *)_Region_Information->local_table[1])->Memory->first'' references the contents of the first ''heap block'' on this Heap.
    87 Notice that the ''front_flag'' is displayed as ''8058280''.  This is in decimal since we used ''p'' not ''p/x'' to gdb.  Since this number is even, we know the in use bit is 0 and the block is free.  Thus the first block on the heap is 8,058,280 bytes and there are at least that many bytes left.
    89 '''NOTE:''' This is really a crude estimate.
    91 If you have compiled RTEMS libraries with -DRTEMS_DEBUG, malloc will maintain statistics. From an RTEMS application:
    93 {{{
    94 #include <libcsupport.h>
    95     .....
    96   malloc_dump ();
    97 }}}
    99 will print malloc's heap statistics to ''stdout''. Example usage can be found in <tt>c/src/tests/frye/libcleak</tt> and <tt>c/src/tests/libtests/malloctest</tt>.
    101 The following GDB macro may also be of use. If provided with an argument of '''0''' for a summary, or '''1''' to explicitly list the regions.
    103 {{{
    104  define rtems-mallocheap-walk
    105    printf "walking the heap:\n"
    106    set $heapstart = ((Region_Control *)_Region_Information->local_table[RTEMS_Malloc_Heap&0xffff])->Memory->start
    107    set $currentblock = $heapstart
    108    set $used = 0
    109    set $numused = 0
    110    set $free = 0
    111    set $numfree = 0
    112    while $currentblock->front_flag != 1
    113      if $currentblock->front_flag & 1
    114        if $arg0 != 0
    115         printf "USED: %d\n", $currentblock->front_flag & ~1
    116        else
    117          printf "*"
    118        end
    119        set $used = $used + $currentblock->front_flag & ~1
    120        set $numused = $numused + 1
    121      else
    122        if $arg0 != 0
    123         printf "FREE: %d\n", $currentblock->front_flag & ~1
    124        else
    125          printf "."
    126        end
    127        set $free = $free + $currentblock->front_flag & ~1
    128        set $numfree = $numfree + 1
    129      end
    130      set $currentblock = (Heap_Block *)((char *)$currentblock + ($currentblock->front_flag&~1))
    131    end
    132    if $arg0 == 0
    133      printf "\n"
    134    end
    135    printf "TOTAL: %d (%d)\tUSED: %d (%d) \tFREE: %d (%d)\n", \
    136      $used + $free, $numused + $numfree, \
    137      $used, $numused, \
    138      $free, $numfree
    139  end
    140 }}}
    141 = How much memory is left in the RTEMS Workspace? =
    144 An RTEMS workspace overage can be fairly easily spotted with a debugger. Look at ''Workspace''Area. If first == last, then there is only one free block of memory in the workspace (very likely if no task or message queue deletions). Then do this:
    146 {{{
    147   (gdb) p (Heap_Block *) Workspace_Area->first
    148   $3 = {back_flag=1, front_flag=68552, next=0x1e260, previous=0x1e25c}
    149 }}}
    151  * ''Workspace_Area'' is the variable name of the RTEMS Workspace Heap control block.
    152  * ''(Heap_Block *) _Workspace_Area->first'' is the contents of the first heap block information.
    154 Just as with the C Program Heap, the number was even indicating it is free.  In this case, I had 68552 bytes left in the workspace.
    156 '''NOTE:''' This is really a crude estimate.  GDB 5.0 and newer support a macro language that provides the features necessary to write a function which would walk a heap structure and print out accurate statistics.  If you write this, submit it. :)
    15742= BSP rtems_initialize_executive_late call dies =
    17257The initialization task is a real task in RTEMS. The ''rtems_initialize_executive_late'' creates it and switches context to it. This means your BSP environment is another context that RTEMS will switch back to when RTEMS is shut down. This allows your BSP to take control again, then perform any specific functions it needs. A typical operational thing to do is to reboot as embedded targets should not stop.
    17359= GDB Cannot Find My Source Files =
    176 If you find the source code paths in your executable as seen by GDB are missing, you may find using an absolute path to invoke the RTEMS configure script may help. When RTEMS libraries get built, nested ''Makefiles'' are executed that '''walk''' through the build directory structure. Therefore, each file is compiled from a certain point in the build directory structure that lies in paralllel to the source directory structure.
     62If you find the source code paths in your executable as seen by GDB are missing, you may find using an absolute path to invoke the RTEMS configure script may help. When RTEMS libraries get built, nested ''Makefiles'' are executed that '''walk''' through the build directory structure. Therefore, each file is compiled from a certain point in the build directory structure that lies in parallel to the source directory structure.
    178 As a consequece of this, you get a varing number of "dots", depending on how deep the corresponding directory is inside the build tree. There is no common location, from which the source file pointers in the debug info is correct for all object files.
     64As a consequence of this, you get a varying number of "dots", depending on how deep the corresponding directory is inside the build tree. There is no common location, from which the source file pointers in the debug info is correct for all object files.
    18066If you call the initial <tt>configure</tt> in an absolute way such as :
    19278GDB should find the source.
    19380= Starting With Hello World =
    21299As you add to your program, you may have to increase the number of objects configured as well.
    213101= Standard IO and File Issues =
    218106The stdio functions in newlib depend on both initialised and uninitialised data.  If you find they are returning -1, ensure your .bss and .data sections are correctly setup. Check your linkcmds file is creating the correct memory map and that your bsp boot process is copying/zeroing all appropriate sections in ram.  It's also worth double checking that your ram and other hardware is working correctly!
    219108= open, fopen, and socket creation fail =
    228117Note: Most of what follows has been gleaned from the associated README files; I haven't added much original content, partly due to sloth, but more justifiably because I haven't yet used these tools extensively enough to be able to do any better!  My primary purpose, at this point, in including these is so others, especially "nuBees" are aware of them.
    229 =  RTEMS Monitor  =
    232 The RTEMS Monitor is run as a high-priority task, and provides a useful window into the operation of your system.  From the README:
    234 {{{
    235 monitor task
    237 The monitor task is an optional task that knows about RTEMS
    238 data structures and can print out information about them.
    239 It is a work-in-progress and needs many more commands, but
    240 is useful now.
    242 The monitor works best when it is the highest priority task,
    243 so all your other tasks should ideally be at some priority
    244 greater than 1.
    246 To use the monitor:
    247 -------------------
    249     #include <rtems/monitor.h>
    251     ...
    253     rtems_monitor_init(0);
    255     The parameter to rtems_monitor_init() tells the monitor whether
    256     to suspend itself on startup.  A value of 0 causes the monitor
    257     to immediately enter command mode; a non-zero value causes the
    258     monitor to suspend itself after creation and wait for explicit
    259     wakeup.
    262     rtems_monitor_wakeup();
    264     wakes up a suspended monitor and causes it to reenter command mode.
    266 Monitor commands
    267 ----------------
    269     The monitor prompt is 'rtems> '.
    270     Can abbreviate commands to "uniquity"
    271     There is a 'help' command.  Here is the output from various
    272     help commands:
    274         Commands (may be abbreviated)
    276           help      -- get this message or command specific help
    277           task      -- show task information
    278           queue     -- show message queue information
    279           symbol    -- show entries from symbol table
    280           pause     -- pause monitor for a specified number of ticks
    281           fatal     -- invoke a fatal RTEMS error
    283         task [id [id ...] ]
    284           display information about the specified tasks.
    285           Default is to display information about all tasks on this node
    287         queue [id [id ... ] ]
    288           display information about the specified message queues
    289           Default is to display information about all queues on this node
    291         symbol [ symbolname [symbolname ... ] ]
    292           display value associated with specified symbol.
    293           Defaults to displaying all known symbols.
    295         pause [ticks]
    296           monitor goes to "sleep" for specified ticks (default is 1)
    297           monitor will resume at end of period or if explicitly awakened
    299         fatal [status]
    300           Invoke 'rtems_fatal_error_occurred' with 'status'
    301           (default is RTEMS_INTERNAL_ERROR)
    303         continue
    304           put the monitor to sleep waiting for an explicit wakeup from the
    305           program running.
    308 Sample output from 'task' command
    309 ---------------------------------
    311     rtems> task
    313     ------------------------------------------------------------------------
    314     00010001   UI1     2    READY    P:T:nA    NONE15: 0x40606348
    315     00010002   RMON    1    READY    nP    NONE15: 0x40604110
    317     'RMON' is the monitor itself, so we have 1 "user" task.
    318     Its modes are P:T:nA which translate to:
    320         preemptable
    321         timesliced
    322         no ASRS
    324     It has no events.
    325     It has a notepad value for notepad 15 which is 0x40606348
    326     (this is the libc thread state)
    327 }}}
    329 Note that this README is quite dated - it hasn't been changed in 11 '''years'''!  In fact, it provides much more information; here is the output from the help command on a recent (July 2007 HEAD) session:
    331 {{{
    332 rtems $ help
    333 config     - Show the system configuration.
    334 itask      - List init tasks for the system
    335 mpci       - Show the MPCI system configuration, if configured.
    336 pause      - Monitor goes to "sleep" for specified ticks (default is 1).
    337              Monitor will resume at end of period or if explicitly
    338              awakened
    339               pause [ticks]
    340 continue   - Put the monitor to sleep waiting for an explicit wakeup from
    341              the program running.
    342 go         - Alias for 'continue'
    343 node       - Specify default node number for commands that take id's.
    344               node [ node number ]
    345 symbol     - Display value associated with specified symbol. Defaults to
    346              displaying all known symbols.
    347               symbol [ symbolname [symbolname ... ] ]
    348 extension  - Display information about specified extensions. Default is to
    349              display information about all extensions on this node.
    350               extension [id [id ...] ]
    351 task       - Display information about the specified tasks. Default is to
    352              display information about all tasks on this node.
    353               task [id [id ...] ]
    354 queue      - Display information about the specified message queues.
    355              Default is to display information about all queues on this
    356              node.
    357               queue [id [id ... ] ]
    358 object     - Display information about specified RTEMS objects. Object id's
    359              must include 'type' information. (which may normally be
    360              defaulted)
    361               object [id [id ...] ]
    362 driver     - Display the RTEMS device driver table.
    363               driver [ major [ major ... ] ]
    364 dname      - Displays information about named drivers.
    365 exit       - Invoke 'rtems_fatal_error_occurred' with 'status' (default is
    366              RTEMS_SUCCESSFUL)
    367               exit [status]
    368 fatal      - 'exit' with fatal error; default error is RTEMS_TASK_EXITTED
    369               fatal [status]
    370 quit       - Alias for 'exit'
    371 help       - Provide information about commands. Default is show basic
    372              command summary.
    373             help [ command [ command ] ]
    374            - 'i"a
    375 rtems $
    376 }}}
    377 =  Capture Engine  =
    380 The [wiki:TBR/UserManual/Capture_Engine Capture Engine] is another neat tool.  Unlike the RTEMS Monitor, this one is already documented on the Wiki, though you have to know to search for it.  Check it out!
    381 =  CPU Usage Monitoring  =
    384 There is a CPU usage monitoring facility available in cpukit/libmisc/cpuuse.  Again, from the README:
    386 This directory contains code to report and reset per-task CPU usage.
    387 If the BSP supports nanosecond timestamp granularity, this this information
    388 is very accurate.  Otherwise, it is dependendent on the tick granularity.
    390 It provides two primary features:
    392  * Generate a CPU Usage Report
    393  * Reset CPU Usage Information
    394 ==  NOTES  ==
    397 #If configured for tick granularity, CPU usage is "docked" by a clock tick at each context switch.
    398 #If configured for nanosecond granularity, no work is done at each clock tick.  All bookkeeping is done as part of a context switch.
    399 =  Stack Checker  =
    401 ==  Introduction  ==
    404 This directory contains a stack bounds checker.  It provides two
    405 primary features:
    407 #check for stack overflow at each context switch
    408 #provides an educated guess at each task's stack usage
    409 ==  Enabling  ==
    412 Add the stack checker extension to the initial user extension set.
    413 If using confdefs.h to build your configuration table, this is
    414 as simple as adding -DSTACK_CHECK_ON to the gcc command line which
    415 compiles the file defining the configuration table.  In the RTEMS
    416 test suites and samples, this is always init.c.  Another way to
    417 enable it is to include the following prior to including confdefs.h:
    419 {{{
    420 #define STACK_CHECKER_ON
    421 }}}
    423 Once you've enabled the stack checker when building your application, it the stack checker
    424 runs automatically as part of a context switch.  Additionally, you can call
    426 {{{
    427 boolean rtems_stack_checker_is_blown(void);
    428 }}}
    430 at any time to check yourself; it returns FALSE if the stack appears okay,
    431 or TRUE if the stack pointer is out of range or the pattern marker has
    432 been corrupted.
    433 ==  Background  ==
    436 The stack overflow check at context switch works by looking for
    437 a 16 byte pattern at the logical end of the stack to be corrupted.
    438 The "guesser" assumes that the entire stack was prefilled with a known
    439 pattern and assumes that the pattern is still in place if the memory
    440 has not been used as a stack.
    442 Both of these can be fooled by pushing large holes onto the stack
    443 and not writing to them... or (much more unlikely) writing the
    444 magic patterns into memory.
    446 This code is provided as a tool for RTEMS users to catch the most common mistake in multitasking
    447 systems ... too little stack space.  Suggestions and comments are appreciated.
    448 ==  Optional Compile-time Selections  ==
    450 If someone ever gets VERY VERY desperate, Joel recently added some conditionals which can turn on walking
    451 the heap and checking the stack EVERY time you enter an RTEMS dispatching disabled critical section.
    452 You have to recompile but since this is such a heavy handed thing to have on, that seemed a fair trade off.
    453 The "heavy stack check" feature is enabled by defining RTEMS_HEAVY_STACK_DEBUG, and the "heavy malloc check"
    454 by defining RTEMS_HEAVY_MALLOC_DEBUG; in both cases, do this before you build and install RTEMS.
    455 ==  NOTES  ==
    458 #Stack usage information is questionable on CPUs which push large holes on stack.
    459 #Prior to, the stack checker used printf() instead of printk().  This often resulted in a fault when trying to print the helpful diagnostic message.  If using the older printf() stack checker and it comes out, congratulations. If not, then the variable Stack_check_Blown_task contains a pointer to the TCB of the offending task.  This is usually enough to go on.  Now that it is using printk(), it should be able to get messages out.
    460120=  = FUTURE ====