wiki:GSoC/2019/POSIX_Compliance

POSIX Compliance

Student(s): Vaibhav Gupta.

Mentors: Joel Sherrill, Aditya Upadhyay, Hesham Almatary .

Ticket: POSIX Compliance

Development Blog: My GSoC 2019 Journey

1) - Introduction to Project

1.1) - Project Description

Developers and community have made the Real Time Executive for Multiprocessor Systems (RTEMS) well compliant with various standards like POSIX (6) IEEE Standard 1003.1™ , C99, C11, and various profiles of FACE (8) and SCA. This makes it possible for RTEMS to be executed on a large variety of architectures, devices and environments.

But this is an indefinite task as with advancement in technology and necessity, these standards keep updating and hence creates the need for updating the compliance status too.

Due to physical resource constraints some real time systems like small embedded system needs limited set of operating system functionality. For these type of system it is necessary that the standards allow implementation to support only a particular subset of POSIX functions. For instance, there is an approved change request against FACE 3.0 to make support for multiple processes optional. Any method listed in the FACE Technical Standard Edition 3.0, in Appendix A.1 as POSIX_MULTI_PROCESS is now optional. This is logical for embedded systems like RTEMS which follows Single Process Multi Threading Concept.

1.2) - Project Goal

The goal of the project is to update the compliance status of RTEMS with POSIX IEEE and FACE General purpose profile. Various functions and headers were required to be implemented. As the project will advance, many headers, stubs, documentation and test-suites will be added.

2) - Project Setup

Steps I followed to setup RTEMS on host OS (my system):

3) - Project Sandboxing On My System

Home Folder (~): /home/varodek

~/development
	|
	|____/rtems
	|	|
	|	|____/5 	#- Directory for toolchain for rtems5
	|	|____/rsb 	#- contains RSB
	|	|____/kernel
	|		|____/erc32 			#- contains RTEMS kernel for SPARC
	|		|____/xilinx_zynq_a9_qemu 	#- contains RTEMS kernel for ARM
	|		|____/rtems			#- rtems clone from git repository
	|
	|
	|
	|
	|____/newlib
		|____/b-sparc-rtems5-newlib	#- newlib compiled for SPARC using rtems5 toolchain
		|____/b-arm-rtems5-newlib	#- newlib compiled fro ARM using rtems5 toolchain
                |____/newlib-cygwin             #- Clone of newlib from git repository

4) - Managing autoconf Versions In Sub-Directories Of Newlib Source Tree

Newlib is the RTEMS's choice for its C library, hence, most of the the contribution will be done in Newlib source tree. As other good projects, Newlib also uses Autoconf tools for producing “Makefile.in” and various other useful scripts. The situation is, inside its source tree, newlib-cygwin/ uses autoconf tools version 2.64. Whereas newlib-cygwin/newlib/libc uses autoconf tools version 2.69 or above.

I wrote a blog for this: How To Handle Two Versions of autoconf?

5) - Apply Newlib Patches To RTEMS Source Builder

Once you make changes or add new code to Newlib source tree, you are required to send the patch to their mailing list. Here, org admins will review the patch and if everything goes good, it will merged in the master branch.

But this does not go that simple, first you need to be sure that your contribution is not breaking the master branch and/or is working properly. For example If you added a header file in Newlib, you need to first make sure that all the methods are working fine. To do this, we need to use that header somewhere. Since, RTEMS is already using Newlib, we can test some RTEMS codes (teast-suite), which will use that header.

Now how to do as discussed above? As Newlib master branch does not have your code yet but RTEMS uses their master branch for its library. Thanks to developers, the RSB has the solution. RSB allows developers to test the local patches. We can make changes in newlib and create patch, RSB will address that patch and build your new library if we follow some steps.

6) - Task: Add Test-Suite For <inttypes.h> Methods

This was a pending task, original work belongs to the GSoC 2017 student of same project. The working directory is : rtems/testsuites/psxtests.

6.1) - Modify The Test-Suite psxinttypes01/init.c

There was need to modify the original code to add some more tests and fix some bugs. inttypes uses strtoimax and wcstoimax. The 'opengroup' page contains every information to write testcases. For a good testsuite, every output (expected or error value) has to be verified.

The patch: https://devel.rtems.org/changeset/d9fcb22/rtems

6.2) - Add psxinttypes01/psxinttypes01.doc And psxinttypes01/psxinttypes01.scn

6.3) - Modify configure.ac

Entry had to be made for psxinttypes01 in configure.ac, so that it gets build with rest of the test-suites of RTEMS.

Add the line RTEMS_TEST_CHECK([psxinttypes01]).

Note, changes has to be made without disturbing lexicographical order.

diff --git a/testsuites/psxtests/configure.ac b/testsuites/psxtests/configure.ac
index cdd6ee7e4e..85559e4aa5 100644
--- a/testsuites/psxtests/configure.ac
+++ b/testsuites/psxtests/configure.ac
@@ -91,6 +91,7 @@ RTEMS_TEST_CHECK([psxid01])
 RTEMS_TEST_CHECK([psximfs01])
 RTEMS_TEST_CHECK([psximfs02])
 RTEMS_TEST_CHECK([psxintrcritical01])
+RTEMS_TEST_CHECK([psxinttypes01])
 RTEMS_TEST_CHECK([psxitimer])
 RTEMS_TEST_CHECK([psxkey01])
 RTEMS_TEST_CHECK([psxkey02])

6.4) - Modify Makefile.am

This entry is made to generate make commands inside Makefile.in. Note $(support_includes) CPPFLAG is necesaary to include required header files.

This diff can be used as a template to modify/add Makefile.am enteries in fututre.

diff --git a/testsuites/psxtests/Makefile.am b/testsuites/psxtests/Makefile.am
index 1e354c0df7..59c9f2085b 100755
--- a/testsuites/psxtests/Makefile.am
+++ b/testsuites/psxtests/Makefile.am
@@ -523,6 +523,13 @@ psxintrcritical01_CPPFLAGS = $(AM_CPPFLAGS) \
 endif
 endif

+if TEST_psxinttypes01
+psx_tests += psxinttypes01
+psxinttypes01_SOURCES = psxinttypes01/init.c
+psxinttypes01_CPPFLAGS = $(AM_CPPFLAGS) $(TEST_FLAGS_psxinttypes01) \
+       $(support_includes)
+endif
+
 if HAS_POSIX
 if TEST_psxitimer
 psx_tests += psxitimer

7) - Task: Port ndbm From FreeBSD To Newlib

7.1) - Figure Out The Location to Place The Declarations And Definitions

This was necessary because according to the definition, ndbm.h has to be placed in libc\include and ndbm.c should be placed in libc/posix directory but according to environment and support, ndbm.c was shifted to libc/search directory.

8) - Task: Develop Test-Suite For ndbm

8.1) - Apply ndbm Patch From Newlib to RSB

The porting was complete and the patch was sent to Newlib Mailing list. But it could take time to be pushed. Since, to make test-suite, we need ndbm in RTEMS tool-chain, we applied the ndbm patch to RSB.

8.2) - Develop Test-Suite

8.3) - Add psxndbm01.scn And psxndbm01.doc

8.4) - Modify Makefile.am

The ndbm port was successful but the latest Newlib updates in libm were causing some troubles in RTEMS. Hence Dr Joel ported the test-suite in conditional way. With this test-suite will be compiled for a specific BSP only when it is supported by it.

The if HAS_NDBM conditional statement allows BSP to first check if it supports ndbm or not.

This will get more clear with diff of configure.ac.

diff --git a/testsuites/psxtests/Makefile.am b/testsuites/psxtests/Makefile.am
index 59c9f20..c12b036 100755
--- a/testsuites/psxtests/Makefile.am
+++ b/testsuites/psxtests/Makefile.am
@@ -694,6 +694,17 @@ psxmutexattr01_CPPFLAGS = $(AM_CPPFLAGS) $(TEST_FLAGS_psxmutexattr01) \
     $(support_includes) -I$(top_srcdir)/include
 endif
 
+if HAS_NDBM
+if TEST_psxndbm01
+psx_tests += psxndbm01
+psx_screens += psxndbm01/psxndbm01.scn
+psx_docs += psxndbm01/psxndbm01.doc
+psxndbm01_SOURCES = psxndbm01/init.c
+psxndbm01_CPPFLAGS = $(AM_CPPFLAGS) $(TEST_FLAGS_psxndbm01) \
+       $(support_includes)
+endif
+endif
+
 if TEST_psxobj01
 psx_tests += psxobj01
 psx_screens += psxobj01/psxobj01.scn

8.5) - Modify configure.ac

If we talk in terms of GCC, to make things easy to understand, the AC_CHECK_HEADER comes under the family of "Generic Header Checks". The precise definition and description can be found here. In this case, it will simple check if ndbm.h is present in the BSP or not. The value of HAS_NDBM variable, which was used by if directive in Makefile.am too, gets changed accordingly.

The AM_CONDITIONAL is a macro used by automake. The precise definition and description can be found here. Here, the condition given to this macro is the value of HAS_NDBM, and the process that it affects is the generation of configuration scripts for ndbm.

diff --git a/testsuites/psxtests/configure.ac b/testsuites/psxtests/configure.ac
index 85559e4..bb44bb8 100644
--- a/testsuites/psxtests/configure.ac
+++ b/testsuites/psxtests/configure.ac
@@ -36,6 +36,9 @@ AM_CONDITIONAL([HAS_CPLUSPLUS],[test x"$HAS_CPLUSPLUS" = x"yes"])
 RTEMS_CHECK_CPUOPTS([RTEMS_POSIX_API])
 AM_CONDITIONAL(HAS_POSIX,test x"${rtems_cv_RTEMS_POSIX_API}" = x"yes")
 
+AC_CHECK_HEADER([ndbm.h]. [HAS_NDBM=yes], [HAS_NDBM=no])
+AM_CONDITIONAL(HAS_NDBM,test x"${ac_cv_header_ndbm_h__HAS_NDBM_yes}" = x"yes")
+
 # BSP Test configuration
 RTEMS_TEST_CHECK([psx01])
 RTEMS_TEST_CHECK([psx02])
@@ -110,6 +113,7 @@ RTEMS_TEST_CHECK([psxmsgq02])
 RTEMS_TEST_CHECK([psxmsgq03])
 RTEMS_TEST_CHECK([psxmsgq04])
 RTEMS_TEST_CHECK([psxmutexattr01])
+RTEMS_TEST_CHECK([psxndbm01])
 RTEMS_TEST_CHECK([psxobj01])
 RTEMS_TEST_CHECK([psxonce01])
 RTEMS_TEST_CHECK([psxpasswd01])

9) - Task: Port fenv For ARM, PPC And SPARC

The task started with finding the sources to port the header. FreeBSD and NetBSD are most suitable for this as they are under favorable license.

Now, the situation was, when explored for fenv, some architectures can port from FreeBSD and some from NetBSD. Hence, the sources were mixed and thus difficult to follow one code structure. So, the first task was to decide a code structure. Dr Joel took the responsibility to define templates for the same. RISC-V already had a correct version of a fenv for it, so its generic header was modified and used as the template.

Now we have libc/include/fenv.h which is the template for libc/machine/<arch>/include/fenv.h and libc/include/sys/fenv.h which is the template for libc/machine/<arch>/sys/fenv.h. Note - <arch> is the name of the architecture.

The libc/machine/<arch>/include/fenv.h has function declarations. The libc/machine/<arch>/sys/fenv.h has macro declarations which are architecture specific. He also made templates of fenv stubs in libm/fenv/ directory, which are meant to be overridden by the stubs present in libm/machine/<arch>/ directory.

Attempts made to develop testsuite by testing it for RISC-V failied initially as exceptions are not being raised by the BSP as expected, hence developed the testsuite for Linux environment. It can be easily ported to RTEMS once we have atleast one architecture with fenv working.

10) - Links To My Contribution

11) - New Methods Ported:

  • int dbm_clearerr(DBM *);
  • void dbm_close(DBM *);
  • int dbm_delete(DBM *, datum);
  • int dbm_error(DBM *);
  • datum dbm_fetch(DBM *, datum);
  • datum dbm_firstkey(DBM *);
  • datum dbm_nextkey(DBM *);
  • DBM *dbm_open(const char *, int, mode_t);
  • int dbm_store(DBM *, datum, datum, int);
  • int dbm_dirfno(DBM *);

12) - Test-Suites Developed:

  • psxinttypes01 - inttypes.h Test-Suite.
  • psxndbm01 - ndbm.h Test-Suite.

References

Last modified on Aug 24, 2019 at 4:46:05 AM Last modified on Aug 24, 2019, 4:46:05 AM