Debugging

Symbolic Debug Information

RTEMS and the RTEMS tool chains support symbolic debugging. The application, RTEMS and any libraries you wish to debug need to be compiled with the gcc option -g. This option causes the compiler to emit debugging information, while the code generated should not change. The GNU gcc debugging options can be found here -

<center> ​http://gcc.gnu.org/onlinedocs/gcc-3.4.3/gcc/Debugging-Options.html#Debugging-Options </center>

Using the -g option results in larger object and executable file sizes, and for C++ this can be quite large. For example a M68K C++ application can have a ELF executable file size of 19M bytes yet the code size is only 1.6M bytes. The actual memory foot print can be seen by using the size tool on an object file or the final executable -

$ m68k-rtems-size myfile.o

For RTEMS you typically do not need to strip the executable. The loading of the executable into the target memory or conversion to S records, Intel Hex, for binary format will automatically strip the debug information. Keeping the excutable with the debugging information is recommended. If a problem appears with code in the field and you recieve a some sort of dump or trace you can use the objdump tool to help locate the problem -

$ m68k-rtems-objdump -D --line-numbers --source --demangle  myapp.elf | less

Debugging a BSP

To test a BSP you need an application. The samples and tests provide a proven set of applications that allow you to test your BSP. Select a sample application such as the Hello World to try first, then move to an application that uses more interrupt sources.

The following is a list of debugging tools and setups that we recommended. Time spent on this will be rewarded further into the project:

• Try to create a working debugger environment for the target hardware. The GNU debugger, GDB, supports a number of different ways to connect to a target. The interface can be a simple serial port using a GdbSerialMonitor, or an expensive TCP/IP type hardware probe.

• If your target processor supports a simulator it is recommended you learn how to use it. It will give you a stable debugging enviroment.

• Implement a printk interface in your BSP. For target hardware that does not have a video interface the printk outputs to a serial port. The driver is normally a simple polled UART driver.

• Use the Capture Engine to aid the debugging and verfication of your real-time design.

GDB and RTEMS

Currently GDB is not RTEMS aware. GDB scripts exist that can help by providing presenting kernel structures in a user friendly manner. These can be found here: GdbScripts.

The issue with making GDB aware of RTEMS is where the knowledge of the kernel structures is located. A version of GDB that contains a specific kernel structure layout will break if RTEMS changes. The ideal would be for GDB to find the structure elements using the applications debug information.

How much memory is left in the C Program Heap?

The C heap is a region so this should work:

  (gdb) p ((Region_Control *)_Region_Information->local_table[1])->Memory->first
  $9 {back_flag=1, front_flag=8058280, next=0x7ea5b4, previous=0x7ea5b0}

Let's look at that gdb command in more detail.

• _Region_Information contains the information used to manage Region objects.
• _Region_Information->local_table is the object pointer table for Regions. It is indexed by the object index portion of the object ID.
• Region_Information->local_table points to the first Region object. It is of type (Region_Control *).
• ((Region_Control *)_Region_Information->local_table[1])->Memory points to the Heap control portion of this Region's control block.
• ((Region_Control *)_Region_Information->local_table[1])->Memory->first references the contents of the first heap block on this Heap.

Notice that the front_flag is displayed as 8058280. This is in decimal since we used p not p/x to gdb. Since this number is even, we know the in use bit is 0 and the block is free. Thus the first block on the heap is 8,058,280 bytes and there are at least that many bytes left.

NOTE: This is really a crude estimate.

If you have compiled RTEMS libraries with -DRTEMS_DEBUG, malloc will maintain statistics. From an RTEMS application:

#include <libcsupport.h>
    .....
  malloc_dump ();

will print malloc's heap statistics to stdout. Example usage can be found in <tt>c/src/tests/frye/libcleak</tt> and <tt>c/src/tests/libtests/malloctest</tt>.

The following GDB macro may also be of use. If provided with an argument of 0 for a summary, or 1 to explicitly list the regions.

 define rtems-mallocheap-walk
   printf "walking the heap:\n"
   set $heapstart = ((Region_Control *)_Region_Information->local_table[RTEMS_Malloc_Heap&0xffff])->Memory->start
   set $currentblock = $heapstart
   set $used = 0
   set $numused = 0
   set $free = 0
   set $numfree = 0
   while $currentblock->front_flag != 1
     if $currentblock->front_flag & 1
       if $arg0 != 0
 	printf "USED: %d\n", $currentblock->front_flag & ~1
       else
         printf "*"
       end
       set $used = $used + $currentblock->front_flag & ~1
       set $numused = $numused + 1
     else
       if $arg0 != 0
 	printf "FREE: %d\n", $currentblock->front_flag & ~1
       else
         printf "."
       end
       set $free = $free + $currentblock->front_flag & ~1
       set $numfree = $numfree + 1
     end
     set $currentblock = (Heap_Block *)((char *)$currentblock + ($currentblock->front_flag&~1))
   end
   if $arg0 == 0
     printf "\n"
   end
   printf "TOTAL: %d (%d)\tUSED: %d (%d) \tFREE: %d (%d)\n", \
     $used + $free, $numused + $numfree, \
     $used, $numused, \
     $free, $numfree
 end

How much memory is left in the RTEMS Workspace?

An RTEMS workspace overage can be fairly easily spotted with a debugger. Look at WorkspaceArea. If first == last, then there is only one free block of memory in the workspace (very likely if no task or message queue deletions). Then do this:

  (gdb) p (Heap_Block *) Workspace_Area->first
  $3 = {back_flag=1, front_flag=68552, next=0x1e260, previous=0x1e25c}

• Workspace_Area is the variable name of the RTEMS Workspace Heap control block.
• (Heap_Block *) _Workspace_Area->first is the contents of the first heap block information.

Just as with the C Program Heap, the number was even indicating it is free. In this case, I had 68552 bytes left in the workspace.

NOTE: This is really a crude estimate. GDB 5.0 and newer support a macro language that provides the features necessary to write a function which would walk a heap structure and print out accurate statistics. If you write this, submit it. :)

BSP rtems_initialize_executive_late call dies

Your target is booting and the BSP is initialising but a call to rtems_initialize_executive_late results in an exception, or the target locking up. This can be due to a few reasons that you will need to work through.

• The memory map is not correct. RTEMS use the WorkSpace? to create the initialization task, it's stack, the interrupt stack, plus more. If the WorkSpace? is wrong the target will die. The exact way depends on the specific processor and mapping error. If you have implemented printk it may be a good idea to add some code to bsp_pretasking_hook to show you the base and size:

          printk ("Heap : %5d KB @ 0x%08x\n   ", heap_size >> 10, heap_start);

• The initialization process encountered an error. The set of these is documented in the Classic API C User's Guide. For RTEMS 4.6.1, this was documented in the ​Initialization Manager Failure section.

• Interrupts are pending and no interrupt handler is present, or a bug exists in the interrupt handler, or vector table layout. RTEMS will enable interrupts when it switches to the initialization task rtems_initialize_executive_late creates. This problem can be found by getting to the context switch call, _CPU_Context_Switch, then the crash or lockup. The fix is to make sure the BSP has masked all sources of interrupts in the hardware around the processor. This allows RTEMS and the application an ordered and controlled initialization of driver and therefore interrupts.

The initialization task is a real task in RTEMS. The rtems_initialize_executive_late creates it and switches context to it. This means your BSP environment is another context that RTEMS will switch back to when RTEMS is shut down. This allows your BSP to take control again, then perform any specific functions it needs. A typical operational thing to do is to reboot as embedded targets should not stop.

GDB Cannot Find My Source Files

If you find the source code paths in your executable as seen by GDB are missing, you may find using an absolute path to invoke the RTEMS configure script may help. When RTEMS libraries get built, nested Makefiles are executed that walk through the build directory structure. Therefore, each file is compiled from a certain point in the build directory structure that lies in paralllel to the source directory structure.

As a consequece of this, you get a varing number of "dots", depending on how deep the corresponding directory is inside the build tree. There is no common location, from which the source file pointers in the debug info is correct for all object files.

If you call the initial <tt>configure</tt> in an absolute way such as :

/usr/local/src/rtems/tools/rtems-4.6.0/configure

rather that a relative way :

../rtems-4.6.0/configure

GDB should find the source.

Newlib's Stdio Functions return -1/EOF

The stdio functions in newlib depend on both initialised and uninitialised data. If you find they are returning -1, ensure your .bss and .data sections are correctly setup. Check your linkcmds file is creating the correct memory map and that your bsp boot process is copying/zeroing all appropriate sections in ram. It's also worth double checking that your ram and other hardware is working correctly!