= OpenLDAP = [[TOC(Projects/OpenLDAP, depth=2)]] '''Status:''' Exploratory Light Weight Directory Access Protocol (LDAP) is used to manage distributed directories over a network. According to Wikipedia: ''Historically the OpenLDAP server (slapd, the Standalone LDAP Daemon) architecture was split between a frontend which handles network access and protocol processing, and a backend which deals strictly with data storage. The architecture is modular and many different backends are available for interfacing to other technologies, not just traditional databases.'' OpenLDAP has been identified by Dr. Sherrill as a "high-priority" for RTEMS. According to Beyond Linux from Scratch ''The OpenLDAP package provides an open source implementation of the Lightweight Directory Access Protocol,''. In addition to helping me learn valuable porting skills, this project is on of the first steps towards adding needed modern protocols to RTEMS, so that RTEMS can use LDAP. One of the more major first steps towards modernizing the networking protocols in RTEMS is the [http://git.rtems.org/rtems-libbsd.git/ RTEMS BSD Porting project]. = Goal = Concise statement of the overall goal of the project. Refine this initial statement to include: project deliverable (code, docs, testing), required/suggested methodology, standards of quality, possible goal extensions beyond the main objective. C Rempel's main goal for porting OpenLDAP to RTEMS is to develop a method of porting semi-complex existing open source packages to RTEMS that is efficient, elegant and complete. Efficient meaning the person porting the software can do so in a matter of days or weeks. Elegant meaning the essence of the original source package, and the RTEMS source are preserved: as few lines of code are modified as possible in both the original source package and the RTEMS source as possible, as well as libraries, programs, and executable are installed in the same locations as they would be if they were built in the RTEMS source tree. Completeness meaning not only the libraries are ported, but the programs and tests as well. == Deliverables == Just some ideas from C Rempel: == = RTEMS Webkit === # A Makefile.openldap that builds the SlapD libraries for RTEMS applications # A Makefile that builds the programs. # A program that will run the LDAP testing suite provided by OpenLDAP. ''Should be automated, as additional tests will be used in the future.'' == Suggested Methodology == Start by porting to RTEMS Addon Packages, then adding a "network-demo style test" using a quick start as a guide. == Standards of Quality == The standards of quality will be a SLAPD server, with a simple webkit demonstration. Eventually, there should be tests that demonstrate how the SLAPD implementation meets [tools.ietf.org/html/rfc4510 RFC 4510]. == Possible Goal Extensions == Explore and (possibly port to RTEMS) the additional packages used by OpenLDAP: # Integrate [http://www.rtems.org/wiki/index.php/RTEMS_DBKit Database Support.] One possibility to explore is the Berkely Database [http://www.linuxfromscratch.org/blfs/view/svn/server/db.html BDB.] Although BDB falls under the sleepy cat license, BDB is the default database for OpenLDAP. # Authentication (such as Cyrus SASL) [http://cyrusimap.web.cmu.edu/mediawiki/index.php/Downloads#SASL_Library website][http://git.cyrusimap.org/cyrus-sasl/ git.] Since ldap-bind is being deprecated in favor of ldap-sasl-bind, this extention is needed for a longterm port. # Additional encryption (when configuring openLDAP just type ./configure --help) Explore and (possibly port to RTEMS) the additional packages that use OpenLDAP: # Apache (did this already, but could be ported more efficiently, elegantly, and completely) # Others Explore adding other [http://www.zytrax.com/books/ldap/ opensource LDAP implementations] to RTEMS, the [http://directory.fedoraproject.org/wiki/Main_Page 389 Directory Server] looks like it's written in C (as opposed to Java), so may be a preferred LDAP implementation to port, but additional research is needed to determine such issues as how feasible is it to "embed", and what are the license implications? Add an OpenLDAP test to the RTEMS Networking Demos = Requirements = == Required Programming Languages == C, Bash, Makefile == Specific Areas of RTEMS or tools == # RTEMS Addon Packages # RTEMS Networking Demos == Level of Familiarity of RTEMS == # Ability to configure RTEMS # Ability to start the Hello, World RTEMS Application from the Examples v2. # Ability to build the AVL library using the RTEMS Addon Packages == Cross-development == # Experience using Makefiles and using configurations for cross development. == Gnu/Linux == Ability to use the command line, and modify Makefiles. Knowing "grep -r", "find -name", and a text search feature REALLY helpful. == Development / Documentation / Testing Tools == '''Development Tools:''' RTEMS Addon Packages, RTEMS Source Tree, RTEMS Toolset, Qemu '''Documentation Tools:''' RTEMS Wiki, comments '''Testing Tools:''' Tests provided by GCC and OpenLDAP, possibly other third party open source LDAP testing software... == Mathematical/algorithmic background == None: this is simply putting two pieces of existing software together. == Other Desirable Skills == # Code reuse skills: can research and use other programmer's code. # Exploration skills: a willingness to peruse the RTEMS git, and other projects, to find code maintained by other developers (to make this port successful after the developer moves on). = Resources = Current RTEMS developers, papers, etc that may help you in this project. C Rempel is willing to help = Other sections = If you have more to say about the project that doesn't fit in the proposed sections of this template, feel free to add other sections at will. = Acknowledgements = * who helped add did work = Miscellaneous Sections = = A first stab at building Open LDAP = As the project progresses, you will need to add build instructions, etc and this page will evolve from a project description into a HOWTO. I am building this package because it is simpler than the Apache Runtime, and so is a good way to gain needed experience. I started with using the RTEMS Addon Packages. How to use the RTEMS Addon Packages is outlined [http://www.rtems.com/wiki/index.php/RTEMS_Add-On_Packages]. == Makefile.openldap == I put the following in rtems-addon-packages/Makefile.openldap. On the first pass I didn't want to generate the example configuration files, but on the second pass, I added the sysconfdir and localstatedir switches to make the example files. Then I removed the sysconfdir and localstatedir, so my customizations would not be overwritten. # # Declare supported terminal types. # This value can be augmented/overridden by the site-configuration file # include ../RTEMS_Makefiles/Makefile.common # # The following will work only if you have the latest ncurses version # of infocmp installed before trying to build for an RTEMS target. # The cf_cv_type_of_bool hack works around a bug when configuring # for a cross-target. # # Two options needed to generate example files in rootfs # --sysconfdir=/home/joel/rtems-addon-packages/network-demos/ldap/rootfs/etc # --localstatedir=/home/joel/rtems-addon-packages/network-demos/ldap/rootfs/var all: ac_cv_member_struct_msghdr_msg_control=no \ ac_cv_func_memcmp_working=yes \ ac_cv_func_initgroups=yes \ ac_cv_func_setuid=no \ ./configure \ --host=$(RTEMS_CPU)-rtems4.11 \ --prefix=$(exec_prefix) \ --libdir=$(exec_prefix)/$(RTEMS_BSP)/lib \ --includedir=$(exec_prefix)/$(RTEMS_BSP)/lib/include \ --without-tls \ --with-yielding_select=yes \ --disable-mdb \ --disable-hdb \ --disable-cleartext \ --disable-overlays \ --enable-proctitle \ --enable-debug \ --disable-shared make make install Next: $ cd ~/rtems-addon-packages $ git clone git://git.openldap.org/openldap.git openldap $ cd openldap = Binary Relocatability = When I tried linking slapd directly into RTEMS, Qemu bailed out with address outside of range... So, to avoid having the address outside of range, I decided to try making SLADP a relocatable binary (or library). Unfortuneately, when I did that, the convienience libraries were no longer included. However, I will walk through the process of turning an executable into a library... First, I modified the openldap/servers/slapd/Makefile.in to build a library. PROGRAMS = ... + LIBRARY = libslapd.a XPROGRAMS = ... Second, I commented out the original openldap/servers/slapd/Makefile.in libslapd.a: Makefile target, as that was designed for a Windows host. ! # libslapd.a: symdummy.o ! # dlltool ... Third, I added a openldap/servers/slapd/Makefile.in libslapd.a: target by "cutting and pasting" the sources from the slapd: Makefile target, and using ar to build the library, and getting the ar flags from my target.cfg. The linked objects were taken from the slapd: Makefile target. Note: $@ means libslapd.a + libslapd.a: $(SLAPD_DEPENDS) @LIBSLAPI@ + $(AR) ruv $@ $(SLAPD_OBJECTS) $(LIB) $(WRAP_LIBS) Fourth, I added an install-local Makefile target to the install-local-srv. The install-local Makefile target was inspired by the openldap/libraries/lib*/Makefile.in install-local Makefile targets (which are used to install libraries). The install-local Makefile target in the install-local-srv tells the Makefile to actually build the library. ! install-local-srv: install-slapd ... \ install-conf ... install-local Fifth, I added the install-local Makefile target. I was inspired by openldap/libraries/liblutil/Makefile.in and openldap/include/Makefile.in. I not onlu have libslapi.a installed, but main.h, which is needed to call main from my test. + install-local: FORCE $(LIBRARY) + -$(MKDIR) $(DESTDIR)$(libdir) + $(LTINSTALL) $(INSTALLFLAGS) -m 644 $(LIBRARY) $(DESTDIR)$(libdir) + $(LTFINISH) $(DESTDIR)$(libdir) + -$(MKDIR) $(DESTDIR)$(includedir) + for header in $(srcdir)/main.h ; \ + do \ + $(INSTALL) $(INSTALLFLAGS) -m 644 $$header \ + $(DESTDIR)$(includedir); \ + done + Finally, write a main.h int main(int argc, char ***argc); Now, you can make a relocatible libslapd.a! $ make -f RTEMS_Makefiles/Makefile.openldap Note: Very insecure! But builds OK. To make this port a long-term solution, I will need to port SASL to RTEMS as well (ldap-bind is deprecated in favor of ldap-sasl-bind)... that will be a hobby project for months from now... = network-demos/ldap/Makefile = This subsection details out how to call a relocatiable application using the network-demos as an infrastucture. == network-demos/ldap == Get a working demo by copying the http demo rtems-addon-packages$ cd network-demos rtems-addon-packages/network-demos$ cp -r http ldap rtems-addon-packages/network-demos$ cd ldap == network-demos/ldap/doit.sh == Write a compiling and loading script. rtems-addon-packages/network-demos/ldap$ nano doit.sh #!/bin/sh make clean make qemu ... \ ... \ ... Run the script (to make sure it works). rtems-addon-packages/network-demos/ldap$ chmod +x doit.sh rtems-addon-packages/network-demos/ldap$ ./doit.sh Qemu should load with the http network demo == network-demos/ldap/init.c == Because invoking executables on a linux system are basically calls to main, we add a call to main in the init.c To call main, the following were necessary: # get rtems-addon-packages/openldap/servers/slapd/main.c to repeat back the arguments passed it # form strings with the arguments to pass # point argv to each of the arguments # the zeroth argument is the first argument passed to the function # have the last string be the empty string, to tell main that's the last argument # pass the ''address of argv'' &argv to main + #include rtems_task_Init(...) ... + char **argv; + *argv0="-d"; + *argv1="1"; + *argv2="-f"; + *argv3="/etc/openldap/slapd.conf"; [http://www.zytrax.com/books/ldap/ch6/slapd-config.html //openldap guide] + *argv4="-F/etc/openldap"; + *argv5=""; + *argv[0]=&argv0; + *argv[1]=&argv1; + *argv[2]=&argv2; + *argv[3]=&argv3; + *argv[4]=&argv4; + *argv[5]=&argv5; ... + main(5, &argv); ... So far, I've found it helpful to tell RTEMS-openldap that the configuration directory is the /etc/openldap directory instead of the directory hard-coded in the build... == network-demos/ldap/Makefile == This is where binary relocatability gets REALLY interesting! Because libslapd.a is a library and not a program, the dependencies have to be listed from most dependent to least dependent. So, I added the following to the network-demos/ldap/Makefile ! override LD_LIBS += -lmghttpd -lslapd \ + ../../openldap/servers/slapd/back-ldif/libback_ldif.a \ + ../../openldap/servers/slapd/back-monitor/libback_monitor.a \ + ../../openldap/servers/slapd/back-null/libback_null.a \ + ../../openldap/servers/slapd/back-relay/libback_relay.a \ + ../../openldap/servers/slapd/liboverlays.a \ + ../../openldap/libraries/liblunicode/liblunicode.a \ + ../../openldap/libraries/librewrite/librewrite.a \ + ../../openldap/libraries/liblutil/liblutil.a \ + -lldap_r -lldap -llber With all this, I am now able to call the slapd server, but now to figure out how to use the options, the config file, etc... == Using SLAPD == After struggling to call main, I was determined the following: # some configurations that were hard-coded at compile-time would need to be over-ridden at runtime. ## the configuration directory ## others (as dicovered) # additional openLDAP specific knowledge is required. [http://www.zytrax.com/books/ldap/ch6/slapd-config.html An example site for openLDAP.] == examples/openldap/BuildTests.sh == I'll probably strip out the ''examples'' section altogether... it was interesting and I learned about RTEMS Makefile.leaf s, but I'm thinking a network-demo is a better testing choice, because the openLDAP tests are written in shell script, and SLAPD may not require a shell to run... To perform a very basic "smoke" test on the build I modified the BuildTest.sh and Makefile to build the programs needed to run the tests, not the tests themselves. I learned I how to link convenience libraries, and how to use BuildTests.sh, Makefile.leaf and change leaf.cfg. Interestingly, I found it necessary to symbolically link the slapd-common header and source to each test, as well as the libutil library. ### examples/openldap/BuildTests.sh ### #!/bin/sh set -x rm -f rtems-grub.cfg SRCDIR=../../openldap/tests/progs Make the common source and header files available to all the test programs ln -s $SRCDIR/slapd-common.h . ln -s $SRCDIR/slapd-common.c . Make the lutility convenience library available to all the test programs ln -s $SRCDIR/../../libraries/liblutil/liblutil.a . For every test, delete the old link, make a new link, make the test, delete the link and the object file. for TEST in \ slapd-tester \ slapd-search \ slapd-read \ slapd-addel \ slapd-modrdn \ slapd-modify \ slapd-bind \ slapd-mtread \ ldif-filter do rm -f $TEST.c ln -s $SRCDIR/$TEST.c . make TEST=$TEST SRCDIR=$SRCDIR rm -f $TEST.c $TEST.o done After making the test programs, remove the links to the common header, source, and lutility library. rm -f slapd-common.h rm -f slapd-common.c rm -f liblutil.a (Optional) Next, to make it easier for the end-user, automate building rtems-grub.cfg # Generate rtems-grub.cfg Add a comment of what grub is configured to load $(basename $PWD) prints the name of the directory. echo \# RTEMS Grub configuration for $(basename $PWD) >> rtems-grub.cfg This part was rather interesting... I've never used a pipe inside of a shell-script call before... list everything in o-optimize, that ends with .exe and count them. # The number of executables echo set default = $(ls o-optimize | \ grep '\.'exe | wc -l) >> rtems-grub.cfg Used the & to use the name of the executable more than once. ''Note that sed"s/[a-zA-Z\-]/... has to be all on one line'' # The menuentries ls o-optimize | \ grep '\.'exe | \ sed "s/[a-zA-Z\-]/menuentry \"RTEMS \-\" \{\n set root=(hd1,1)\n multiboot (hd1,1)\/o-optimize\/&/" | \ sed "s/\.exe/\.exe\n\}\n/" >> rtems-grub.cfg # List everything in o-optimize... # that is an executable... # select text composed of letters and format it ... # add a line a right brace, and # another line after any .exe ... = More tips = For integrating legacy libraries/applications to RTEMS... I was playing around with adding the sqlite3 command to the multiio shell, and found the following general ideas very helpful. # tear and rebuild ALL applications within the kit every time you make a small change. # link the library into EVERY directory with an RTEMS shell application using your new commands. To tear down and rebuild rtems-addon-packages$ nano doit.sh #!/bin/sh make clean make qemu... To link add an entry to the Makefile: you can put the library in with the COBJS, ../openldap/.libs/libslapd.a or link with LD_LIBS LD_LIBS += -lslapd or LD_LIBS += ../openldap/.libs/libslapd.a == examples/openldap/Makefile == I modified the rtems-addon-packages/examples/ncurses/Makefile to turn openldap programs used to test openldap into stand-alone executables. I had to add slapd-common.c to the C_PIECES, and slapd-common.h to the headers. Also, to link the liblutil.a convienience library, I had to add liblutil.a to C_O_FILES (it's an object file after-all :), I linked the ldap libraries using LD_LIBS, and increased the HeapSize by a factor of ten (because it said networking required more -- online research is needed to get an exact size required). Finally, it's worth noting the $(TEST) parameter is used to identify which test to build, and PGMS stands for programs. # # $Id$ # # Templates/Makefile.leaf # Template leaf node Makefile # SRCDIR=../../openldap/tests/progs # C source names, if any, go here -- minus the .c C_PIECES=slapd-common $(TEST) C_FILES=$(C_PIECES:%=%.c) C_O_FILES=$(C_PIECES:%=${ARCH}/%.o) liblutil.a # C++ source names, if any, go here -- minus the .cc CC_PIECES= CC_FILES=$(CC_PIECES:%=%.cc) CC_O_FILES=$(CC_PIECES:%=${ARCH}/%.o) H_FILES= slapd-common.h # Assembly source names, if any, go here -- minus the .s S_PIECES= S_FILES=$(S_PIECES:%=%.s) S_O_FILES=$(S_FILES:%.s=${ARCH}/%.o) SRCS=$(C_FILES) $(CC_FILES) $(H_FILES) $(S_FILES) OBJS=$(C_O_FILES) $(CC_O_FILES) $(S_O_FILES) PGMS=${ARCH}/$(TEST) # # RTEMS managers go here # MANAGERS=io event message semaphore include $(RTEMS_MAKEFILE_PATH)/Makefile.inc include $(RTEMS_CUSTOM) include $(PROJECT_ROOT)/make/leaf.cfg # # (OPTIONAL) Add local stuff here using += # DEFINES += CPPFLAGS += -I$(SRCDIR)/../../include CFLAGS += LD_PATHS += LD_LIBS += -lldap -llber -lldap_r CFLAGS_LD += -Wl,--defsym -Wl,HeapSize=0x200000 # network needs more space CFLAGS_DEBUG_V += # # Add your list of files to delete here. The config files # already know how to delete some stuff, so you may want # to just run 'make clean' first to see what gets missed. # 'make clobber' already includes 'make clean' # CLEAN_ADDITIONS += CLOBBER_ADDITIONS += all: ${ARCH} $(SRCS) $(PGMS) ${ARCH}/$(TEST): ${OBJS} ${LINK_FILES} $(make-exe) # Install the program(s), appending _g or _p as appropriate. # for include files, just use $(INSTALL) install: all $(INSTALL_VARIANT) -m 555 ${PGMS} /usr/local/tftpboot/bootfiles/m68360/ Then, I ran BuildTests.sh ~/rtems-addon-packages/examples/openldap$ ./BuildTests.sh Again this only built the programs used to run the tests as stand-alone executables... == SLAPD Test Idea == The OpenLDAP [http://www.openldap.org/doc/admin24/quickstart.html quickstart guide] has some interesting ideas. If the test is written based on step 8 forwards, things that would be required would be: # an appropriate configuration file in the rootfs # a call to slapd with the -d option # adding initial entries to a directory # verify the entry exists One general outline of a demo is: [/] slapd -d [/] ldapadd -x -D "cn=Manager,dc=ldapdemo,dc=org" -W -f ldapdemo.ldif [/] ldapsearch -x -b 'dc=ldapdemo,dc=org' '(objectclass=*)' The approach currently being explored is to call "main functions". To find "main functions" openldap$ grep -r "int argc" servers So, the C implementation could look like: typedef char * string; void Init(void) { string slapdstr[2]; // Make 2 strings string ldapaddstr[7]; // Make 7 strings string ldapsearchstr[5]; // Make 5 strings slapdstr[0] = "slapd"; slapdstr[1] = "-d"; ldapaddstr[0] = "ldapadd"; ldapaddstr[1] = "-x"; ldapaddstr[2] = "-D"; ldapaddstr[3] = "\"cn=Manager,dc=ldapdemo,dc=org\""; ldapaddstr[4] = "-W"; ldapaddstr[5] = "-f"; ldapaddstr[6] = "ldapdemo.ldif"; ldapsearchstr[0] = "ldapsearch"; ldapsearchstr[1] = "-x"; ldapsearchstr[2] = "-b"; ldapsearchstr[3] = "\'dc=ldapdemo,dc=org\'"; ldapsearchstr[4] = "\'(objectclass=*)\'"; main(2, slapdstr); slapadd(7, ldapaddstr); slapindex(5, ldapsearchstr); return 0; } [http://cboard.cprogramming.com/c-programming/62187-array-strings-c.html] Another possibility was to look into the ldap C API, but that looks mostly like it's for use with clients, which is not what the requirements called for, and so is not currently being explored. = Current / Future Directions = # Polish the demo, by ## making appropriate calls to main ## possibly renaming main to a more openldap specific function ## adding the appropriate support files (network-demos/ldap/rootfs/etc/...) # Try to get an LDAP connection between RTEMS on Qemu and the build machine # Port SASL to RTEMS and link SASL to the demo # Polish the openldap port by either having all the convienience libraries get installed into a /opt/.../lib/ldap subdirectory and linking from there, or "rolling them into" libslapd.a I explored implementing ldap, ldapsearch, and ldapadd by taking the multiio application from the RTEMS git, and removing as many commands as possible, then adding back in one command at a time. The RTEMS Shell User's guide was very helpful. However, there were "redefiniton issues" (probably because both slapd and the network-demos used init.c). One problem I'm anticipating is the filesystem, but for now, figuring out what calls to main I want, and how to make them is a challenge. Hope this helps! = References = * [http://www.linuxfromscratch.org/blfs/view/svn/server/openldap.html Beyond Linux From Scratch] * [http://www.openldap.org/ OpenLDAP] * [http://milkymist.org/wiki/index.php?title=Run_RTEMS_sample_applications_on_the_Milkymist_One_board_and_on_Qemu Milky Mist.org: How to get an RTEMS Shell -- possibly useful] * [http://rtemsramblings.blogspot.com/2011/02/rtems-shell-as-debug-aid.html RTEMS Ramblings: Another, possibly useful, RTEMS Shell page] * [http://www.rtems.org/wiki/index.php/File_Systems File Systems: The File IO example application uses the RTEMS Shell] * [http://www.openldap.org/doc/admin24/appendix-recommended-versions.html Recommended dependencies: will be VERY useful once testing has been set up are deciphered] * [http://www-archive.mozilla.org/directory/csdk-docs/example.htm Sample Client code] * [http://www.fosteringlinux.com/2010/03/getting-started-with-openldap-part-1 Getting Started with OpenLDAP ]