Add reference counting to file descriptors

[Sebastian Huber]

The use of a file descriptor after or during a close() operation may result in a ​use after free. Finding such errors in applications is difficult. Especially in SMP systems using the highly dynamic libbsd network stack.

The file descriptor objects reside in a table with a application configuration defined size. So, the storage of a file descriptor object is always present, only the referenced file system node may change over time. The file system nodes may use an internal reference counting, which is independent of the file descriptors.

To implement reference counting for the file descriptors add a bit field for the reference count to the rtems_libio_t::flags and use atomic operations to maintain the flags.

Each operation using a file descriptor should perform a sequence like this:

int op( int fd, ... )
{
  rtems_libio_t *iop;
  unsigned int   flags;

  if ( (uint32_t) fd >= rtems_libio_number_iops ) {
    rtems_set_errno_and_return_minus_one( EBADF );
  }

  iop = rtems_libio_iop( fd );
  flags = rtems_libio_iop_hold( iop );

  if ( ( flags & LIBIO_FLAGS_OPEN ) == 0 ) {
    rtems_libio_iop_drop( _iop );
    rtems_set_errno_and_return_minus_one( EBADF );
  }

  do_op( iop, ... );
  rtems_libio_iop_drop( iop );
  return 0;
}

A close() should return -1 with EBUSY in case the file descriptor is referenced. In this case, no close operation will be performed.

[Sebastian Huber <sebastian.huber@…>]

In ead1281f/rtems:

bsp/mrm332: Remove dead code

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In 7b45202/rtems:

libio: Simplify rtems_libio_iop()

Remove the file descriptor validation. This is the job of
rtems_libio_check_fd(). Use an inline function instread of a macro.

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In 4b759b1/rtems:

libio: Avoid direct use of rtems_libio_iops

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In 0169e90/rtems:

libio: Do simple parameter checks early

This simplifies error handling later.

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In 48dbb6cf/rtems:

libio: Remove rtems_libio_check_permissions()

Remove rtems_libio_check_permissions() and convert single user to
rtems_libio_check_permissions_with_error().

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In ec10d26/rtems:

libio: rtems_libio_check_permissions_with_error()

Rename rtems_libio_check_permissions_with_error() in
rtems_libio_check_permissions().

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In 856ede4f/rtems:

libio: Add iop set/clear flags

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In ca90c6c/rtems:

libio: Add rtems_libio_iop_flags_initialize()

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In e2b1db23/rtems:

libio: Add rtems_libio_iop_flags()

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In bbcdc302/rtems:

libio: Add rtems_libio_iop_is_no_delay()

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In a937a5a/rtems:

libio: Add rtems_libio_iop_is_readable()

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In 3cffd66d/rtems:

libio: Add rtems_libio_iop_is_writeable()

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In d4c5441/rtems:

libio: Add rtems_libio_iop_is_append()

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In 9012db8/rtems:

libio: LIBIO_GET_IOP() LIBIO_GET_IOP_WITH_ACCESS()

Replace rtems_libio_check_fd(), rtems_libio_iop(),
rtems_libio_check_open() and rtems_libio_check_permissions()
combinations with new LIBIO_GET_IOP() and LIBIO_GET_IOP_WITH_ACCESS()
macros.

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In 98041b68/rtems:

libio: Unify readv() and writev()

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In baef823c/rtems:

libio: Add hold/drop iop reference

Check iop reference count in close() and return -1 with errno set to
EBUSY in case the file descriptor is still in use.

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In 894c965/rtems-libbsd:

Support reference counting for file descriptors

Close #3132.

[Sebastian Huber]

Milestone renamed

[Sebastian Huber <sebastian.huber@…>]

In d4b99ae/rtems:

libio: Add assert to rtems_libio_iop_drop()

This assert helps to detect an invalid reference counting in RTEMS_DEBUG
configurations.

Update #3132.

[Sebastian Huber <sebastian.huber@…>]

In b03a1c0/rtems-libbsd:

Fix file descriptor reference counting in accept()

Update #3132.