#3170 closed defect (fixed)

Use BSP_output_char via RTEMS printer or simple console driver for test output by default

Reported by: Chris Johns Owned by: Chris Johns
Priority: high Milestone: 5.1
Component: unspecified Version: 5
Severity: critical Keywords: zedboard testing
Cc: Blocked By:
Blocking:

Description (last modified by Sebastian Huber)

Test runs with a interrupt driven console driver result in unreliable test outcomes.

Problem was noticed with test runs on Microzed, for example libtest/block08:

The test prints:

** END OF TEST BLOCK 8 ***

The rtems-test command marks the result as a failure. There is a single * missing from the start of the line. I attach the full test trace.

Attachments (1)

microzed-block8.txt (5.0 KB) - added by Chris Johns on Oct 6, 2017 at 4:03:09 AM.
RTEMS Tester test log for block8 on a Microzed.

Download all attachments as: .zip

Change History (39)

Changed on Oct 6, 2017 at 4:03:09 AM by Chris Johns

Attachment: microzed-block8.txt added

RTEMS Tester test log for block8 on a Microzed.

comment:1 Changed on Oct 6, 2017 at 9:48:30 PM by Chris Johns

Severity: majorcritical
Summary: Microzed libtest/block8 fails to print end of test correctly.Microzed libtest/block08 fails to print end of test correctly.

I changed all printk calls in the block08 test to printf and the output from the test is now correct and rtems-test passes the test.

There must be something wrong with the Zync UART drivers. I have escalated this to critical because I would like the Zynq to be a tier 1 device with the Beaglebone Black so we have Cortex-A9 and Cortex-A8 devices in tier 1.

There are 233 printk references in the testsuite. I am sure some are needed however block08's printk calls were not needed so I am now wondering how many are?

comment:2 Changed on Oct 9, 2017 at 7:19:29 AM by Sebastian Huber

Does the following patch alter the output:

diff --git a/c/src/lib/libbsp/arm/xilinx-zynq/console/zynq-uart.c b/c/src/lib/libbsp/arm/xilinx-zynq/console/zynq-uart.c
index fa91f3f46e..3c42be428f 100644
--- a/c/src/lib/libbsp/arm/xilinx-zynq/console/zynq-uart.c
+++ b/c/src/lib/libbsp/arm/xilinx-zynq/console/zynq-uart.c
@@ -226,12 +226,18 @@ void zynq_uart_write_polled(
 {
   zynq_uart_context *ctx = (zynq_uart_context *) base;
   volatile zynq_uart *regs = ctx->regs;
+  rtems_interrupt_level level;
+
+  rtems_interrupt_local_disable(level);
 
   while ((regs->channel_sts & ZYNQ_UART_CHANNEL_STS_TFUL) != 0) {
     /* Wait */
+    rtems_interrupt_local_enable(level);
+    rtems_interrupt_local_disable(level);
   }
 
   regs->tx_rx_fifo = ZYNQ_UART_TX_RX_FIFO_FIFO(c);
+  rtems_interrupt_local_enable(level);
 }
 
 static void zynq_uart_write_support(

comment:3 Changed on Oct 10, 2017 at 6:46:55 AM by Sebastian Huber

Component: testingunspecified

comment:4 Changed on Oct 11, 2017 at 11:11:59 PM by Chris Johns

Owner: changed from joel.sherrill@… to Chris Johns
Status: newassigned

comment:5 in reply to:  2 Changed on Oct 12, 2017 at 5:19:10 PM by Chris Johns

Replying to Sebastian Huber:

Does the following patch alter the output:

No it did not make a difference.

See https://lists.rtems.org/pipermail/build/2017-October/000007.html. Search for Result: invalid Time: 0:00:23.859291 dumpbuf01.exe.img. The rtems_print_buffer() call uses rtems_putc and that uses printk or the putc equivalent.

Reviewing the failures there looks to be something wrong in the BSP. There are failures which should not or do not fail on other BSP. I cannot debug this remotely so it may have to wait. Are you able to run the tests on a Zynq?

comment:6 Changed on Oct 13, 2017 at 6:48:49 AM by Sebastian Huber

Do you use a polled or interrupt driven console driver? The tests need a polled console driver.

comment:7 in reply to:  6 ; Changed on Oct 13, 2017 at 12:51:02 PM by Chris Johns

Replying to Sebastian Huber:

Do you use a polled or interrupt driven console driver?

I use the default configuration:

configure --target=arm-rtems4.12 --prefix=/opt/work/chris/rtems/kernel/4.12 --disable-networking --enable-rtemsbsp=xilinx_zynq_zedboard --enable-maintainer-mode --enable-tests

I have always assumed the default configuration for any BSP lets the tests run. Is this not the case?

The tests need a polled console driver.

What do I do to make this happen?

comment:8 in reply to:  7 ; Changed on Oct 13, 2017 at 12:54:16 PM by Sebastian Huber

Replying to Chris Johns:

Replying to Sebastian Huber:

Do you use a polled or interrupt driven console driver?

I use the default configuration:

configure --target=arm-rtems4.12 --prefix=/opt/work/chris/rtems/kernel/4.12 --disable-networking --enable-rtemsbsp=xilinx_zynq_zedboard --enable-maintainer-mode --enable-tests

I have always assumed the default configuration for any BSP lets the tests run. Is this not the case?

No, this is not the case for the real hardware BSPs which usually use an interrupt driven console driver.

The tests need a polled console driver.

What do I do to make this happen?

This is BSP-specific. Try, to add "ZYNQ_CONSOLE_USE_INTERRUPTS=" to the configure command line.

comment:9 in reply to:  8 ; Changed on Oct 13, 2017 at 1:21:40 PM by Chris Johns

Replying to Sebastian Huber:

Replying to Chris Johns:

I have always assumed the default configuration for any BSP lets the tests run. Is this not the case?

No, this is not the case for the real hardware BSPs which usually use an interrupt driven console driver.

This seems like a new requirement. Has something changed in RTEMS to cause this? I do not remember having this problem with previous versions.

This is confusing because each BSP is different and has different options. The tests should manage this internally at runtime and not be dependent on low level BSP specific driver implementations. We have polled console driver support in BSPs, why not get the tests to use it?

Is this documented anywhere? FWIW I see adding some now as hiding the real issue.

What do I do to make this happen?

This is BSP-specific. Try, to add "ZYNQ_CONSOLE_USE_INTERRUPTS=" to the configure command line.

<sigh> I will consider the blocker status for this issue and discuss it with Joel today. He told me yesterday no tests should be using printk unless configured to do so, ie small memory targets. There seems to be some confusion.

comment:10 in reply to:  9 ; Changed on Oct 13, 2017 at 1:28:24 PM by Sebastian Huber

Replying to Chris Johns:

Replying to Sebastian Huber:

Replying to Chris Johns:

I have always assumed the default configuration for any BSP lets the tests run. Is this not the case?

No, this is not the case for the real hardware BSPs which usually use an interrupt driven console driver.

This seems like a new requirement. Has something changed in RTEMS to cause this? I do not remember having this problem with previous versions.

At least since I use RTEMS, this was the case. I asked questions about this on the mailing list.

This is confusing because each BSP is different and has different options. The tests should manage this internally at runtime and not be dependent on low level BSP specific driver implementations. We have polled console driver support in BSPs, why not get the tests to use it?

Is this documented anywhere? FWIW I see adding some now as hiding the real issue.

The console drivers are a real mess. We have no link-time configuration options/mechanism for drivers. Its not so easy.

Changing the tests so that they work with an interrupt driven console driver is hard. I don't think it makes sense. What we need is a non-blocking test output support.

What do I do to make this happen?

This is BSP-specific. Try, to add "ZYNQ_CONSOLE_USE_INTERRUPTS=" to the configure command line.

<sigh> I will consider the blocker status for this issue and discuss it with Joel today. He told me yesterday no tests should be using printk unless configured to do so, ie small memory targets. There seems to be some confusion.

Some tests must not use printf, due to execution environment constraints, e.g. interrupt context, no drivers, early system state, fatal error, etc.

If you consider this as a blocker, then you possibly delay the release for several months.

comment:11 Changed on Oct 13, 2017 at 1:38:59 PM by Sebastian Huber

An short term option would be a unification of the BSP option to select a polled console driver.

comment:12 Changed on Oct 13, 2017 at 1:41:11 PM by Joel Sherrill

There is a buffered test up option for the tests which will help those that use the infrastructure. Should be in testsuites/configure.ac

comment:13 in reply to:  10 ; Changed on Oct 13, 2017 at 1:42:07 PM by Chris Johns

Replying to Sebastian Huber:

Replying to Chris Johns:

Replying to Sebastian Huber:

Replying to Chris Johns:

I have always assumed the default configuration for any BSP lets the tests run. Is this not the case?

No, this is not the case for the real hardware BSPs which usually use an interrupt driven console driver.

This seems like a new requirement. Has something changed in RTEMS to cause this? I do not remember having this problem with previous versions.

At least since I use RTEMS, this was the case. I asked questions about this on the mailing list.

The beaglebone black does not have the same problems as the Zynq. I checked the drivers for the BBB and there is an interrupt version so I wonder if the polled driver is used by default. I did not check this.

The console drivers are a real mess. We have no link-time configuration options/mechanism for drivers. Its not so easy.

If consoles provide interrupt and poll devs, why have the test close the console and reopen it with the polled driver? Yeah ok It is easy to say this and no I have not looked into the detail and what complexity there is. :)

Changing the tests so that they work with an interrupt driven console driver is hard. I don't think it makes sense. What we need is a non-blocking test output support.

Agreed, I was not considering changing the tests to do this rather how output is handled. There is the TESTS_BUFFER_OUTPUT and TESTS_USE_PRINTK can they be forced on and all stdout and stderr paths switched to what is used?

If you consider this as a blocker, then you possibly delay the release for several months.

This is being a little dramatic. I am not after a rewrite, I am looking for a simple pragmatic solution. We need the ability to test on hardware in a repeatable manner for the life of the release branch.

comment:14 in reply to:  13 Changed on Oct 13, 2017 at 1:44:54 PM by Sebastian Huber

Replying to Chris Johns:

Replying to Sebastian Huber:

Replying to Chris Johns:

Replying to Sebastian Huber:

Replying to Chris Johns:

I have always assumed the default configuration for any BSP lets the tests run. Is this not the case?

No, this is not the case for the real hardware BSPs which usually use an interrupt driven console driver.

This seems like a new requirement. Has something changed in RTEMS to cause this? I do not remember having this problem with previous versions.

At least since I use RTEMS, this was the case. I asked questions about this on the mailing list.

The beaglebone black does not have the same problems as the Zynq. I checked the drivers for the BBB and there is an interrupt version so I wonder if the polled driver is used by default. I did not check this.

It depends also on the hardware, FIFO depth, DMA support, etc.

comment:15 in reply to:  11 Changed on Oct 13, 2017 at 1:47:42 PM by Chris Johns

Replying to Sebastian Huber:

An short term option would be a unification of the BSP option to select a polled console driver.

We have an option --enable-tests=all? Should this force polled drivers?

comment:16 Changed on Oct 16, 2017 at 5:22:23 AM by Sebastian Huber

Its not really transparent to the user that --enable-tests=all tinkers with the console driver.

comment:17 Changed on Oct 23, 2017 at 11:49:42 AM by Sebastian Huber <sebastian.huber@…>

In 88e84c22/rtems:

testsuite: Fix build

Updates #3170.

comment:18 Changed on Oct 24, 2017 at 9:33:16 PM by Chris Johns <chrisj@…>

In 4c70110/rtems:

testsuite: Fix build

Updates #3170.

comment:19 Changed on Oct 26, 2017 at 12:06:21 PM by Sebastian Huber

Tests must be audited to ensure that they don't call the standard output functions, e.g.

nm find -name '*.exe' | grep 'exe:\|\<f\?printf\>\|\<puts\>'

comment:20 Changed on Oct 27, 2017 at 5:01:07 AM by Sebastian Huber

Description: modified (diff)
Summary: Microzed libtest/block08 fails to print end of test correctly.Use BSP_output_char via RTEMS printer for test output by default

If we use BSP_output_char for all tests, then the console driver is not tested. We probably need a special test for it.

We should also remove CONFIGURE_APPLICATION_NEEDS_CONSOLE_DRIVER from all other tests.

comment:21 Changed on Oct 28, 2017 at 12:08:04 PM by Sebastian Huber <sebastian.huber@…>

In f703e7f/rtems:

tests: Move rtems_test_printer definition

Statically initialize it to use printk().

Update #3170.
Update #3199.

comment:22 Changed on Oct 28, 2017 at 12:08:16 PM by Sebastian Huber <sebastian.huber@…>

In 7bec7f27/rtems:

rtems: Add rtems_print_printer_fprintf_putc()

Update #3170.
Update #3199.

comment:23 Changed on Oct 28, 2017 at 12:08:27 PM by Sebastian Huber <sebastian.huber@…>

In 73d892d8/rtems:

tests: Use rtems_test_printer

Update #3170.
Update #3199.

comment:24 Changed on Oct 28, 2017 at 12:08:39 PM by Sebastian Huber <sebastian.huber@…>

In 46ddc3c5/rtems:

tests: Use rtems_print_printer_fprintf_putc()

Use rtems_print_printer_fprintf_putc() instead of
rtems_print_printer_printf() to output via rtems_putc().

Update #3170.
Update #3199.

comment:25 Changed on Oct 28, 2017 at 12:08:51 PM by Sebastian Huber <sebastian.huber@…>

In 7e10291/rtems:

tests: Use rtems_test_printer in general

Update #3170.
Update #3199.

comment:26 Changed on Oct 28, 2017 at 12:09:02 PM by Sebastian Huber <sebastian.huber@…>

In acc9d064/rtems:

tests: Remove obsolete TESTS_USE_PRINTK

Update #3170.
Update #3199.

comment:27 Changed on Oct 28, 2017 at 12:09:15 PM by Sebastian Huber <sebastian.huber@…>

In af43554/rtems:

tests: Remove TEST_INIT

The TEST_EXTERN is a used only by the system.h style tests and they use
CONFIGURE_INIT appropriately.

Update #3170.
Update #3199.

comment:28 Changed on Oct 30, 2017 at 5:31:17 AM by Chris Johns <chrisj@…>

In 2126438a/rtems:

testsuite: Add bspIo for a local printk.

Update #3170.
Update #3199.

comment:29 Changed on Nov 2, 2017 at 1:25:12 PM by Sebastian Huber <sebastian.huber@…>

In 0d796d6/rtems:

tests: Delete obsolete TESTS_USE_PRINTF

Update #3170.
Update #3199.

comment:30 Changed on Nov 2, 2017 at 1:26:51 PM by Sebastian Huber <sebastian.huber@…>

In 8c1f4064/rtems:

tests: Use printf() instead of fprintf()

Update #3170.
Update #3199.

comment:31 Changed on Nov 6, 2017 at 6:29:25 AM by Sebastian Huber <sebastian.huber@…>

In ac28f15/rtems:

Add simple console driver

Update #3170.
Update #3199.

comment:32 Changed on Nov 6, 2017 at 6:29:41 AM by Sebastian Huber <sebastian.huber@…>

In c4b8b147/rtems:

tests: Use simple console driver

Update #3170.
Update #3199.

comment:33 Changed on Nov 7, 2017 at 6:09:15 AM by Sebastian Huber <sebastian.huber@…>

In 7b00c2fa/rtems:

tests: Use <tmacros.h> in all tests

Update #3170.
Update #3199.

comment:34 Changed on Nov 7, 2017 at 7:32:57 AM by Sebastian Huber <sebastian.huber@…>

In 32ceb38/rtems:

tests: Use <tmacros.h>

Update #3170.
Update #3199.

comment:35 Changed on Nov 9, 2017 at 6:27:14 AM by Sebastian Huber

Milestone: 4.12.05.1

Milestone renamed

comment:36 Changed on Nov 13, 2017 at 9:01:06 AM by Sebastian Huber

Resolution: fixed
Status: assignedclosed

comment:37 Changed on Nov 13, 2017 at 9:02:07 AM by Sebastian Huber

Summary: Use BSP_output_char via RTEMS printer for test output by defaultUse BSP_output_char via RTEMS printer or simple console driver for test output by default

comment:38 Changed on Feb 5, 2018 at 9:48:37 AM by Sebastian Huber <sebastian.huber@…>

In d078405/rtems-docs:

CONFIGURE_APPLICATION_NEEDS_SIMPLE_CONSOLE_DRIVER

Close #3170.
Update #3199.

Note: See TracTickets for help on using tickets.