C plugin system: dlopen fails

Question

as a continuation to this post C pluginsystem: symbol lookup error, I am still writing my plugin system and encounter new bugs.

To recap what the plugins are, the pr

Answer 1

The problem is that when you compile the plugin system (i.e. functions called by plugins), and link it to the final executable, the linker does not export the symbols used by the plugins in the dynamic symbol table.

There are two options:

1. Use -rdynamic when linking the final executable, adding all symbols to the dynamic symbol table.

2. Use -Wl,-dynamic-list,plugin-system.list when linking the final executable, adding symbols listed in file plugin-system.list to the dynamic symbol table.

   The file format is simple:

    {
        sendappmessage_all;
        plugin_*;
    };
   

   In other words, you can list either each symbol name (function or data structure), or a glob pattern that matches the desired symbol names. Remember the semicolon after each symbol, and after the closing brace, or you'll get a "syntax error in dynamic list" error at link time.

Note that just marking a function "used" via __attribute__((used)) is not sufficient to make the linker include it in the dynamic symbol table (with GCC 4.8.4 and GNU ld 2.24, at least).

---

Since the OP thinks what I wrote above is incorrect, here is a fully verifiable proof of the above.

First, a simple main.c that loads plugin files named on the command line, and executes their const char *register_plugin(void); function. Because the function name is shared across all plugins, we need to link them locally (RTLD_LOCAL).

#include <stdlib.h>
#include <string.h>
#include <dlfcn.h>
#include <stdio.h>

static const char *load_plugin(const char *pathname)
{
    const char    *errmsg;
    void          *handle; /* We deliberately leak the handle */
    const char * (*initfunc)(void);

    if (!pathname || !*pathname)
        return "No path specified";

    dlerror();
    handle = dlopen(pathname, RTLD_NOW | RTLD_LOCAL);
    errmsg = dlerror();
    if (errmsg)
        return errmsg;

    initfunc = dlsym(handle, "register_plugin");
    errmsg = dlerror();
    if (errmsg)
        return errmsg;

    return initfunc();
}

int main(int argc, char *argv[])
{
    const char *errmsg;
    int         arg;

    if (argc < 1 || !strcmp(argv[1], "-h") || !strcmp(argv[1], "--help")) {
        fprintf(stderr, "\n");
        fprintf(stderr, "Usage: %s [ -h | --help ]\n", argv[0]);
        fprintf(stderr, "       %s plugin [ plugin ... ]\n", argv[0]);
        fprintf(stderr, "\n");
        return EXIT_SUCCESS;
    }

    for (arg = 1; arg < argc; arg++) {
        errmsg = load_plugin(argv[arg]);
        if (errmsg) {
            fflush(stdout);
            fprintf(stderr, "%s: %s.\n", argv[arg], errmsg);
            return EXIT_FAILURE;
        }
    }

    fflush(stdout);
    fprintf(stderr, "All plugins loaded successfully.\n");
    return EXIT_SUCCESS;
}

The plugins will have access via certain functions (and/or variables), declared in plugin_system.h:

#ifndef   PLUGIN_SYSTEM_H
#define   PLUGIN_SYSTEM_H

extern void plugin_message(const char *);

#endif /* PLUGIN_SYSTEM_H */

They are implemented in plugin_system.c:

#include <stdio.h>

void plugin_message(const char *msg)
{
    fputs(msg, stderr);
}

and listed as dynamic symbols in plugin_system.list:

{
    plugin_message;
};

We'll also need a plugin, plugin_foo.c:

#include <stdlib.h>
#include "plugin_system.h"

const char *register_plugin(void) __attribute__((used));
const char *register_plugin(void)
{
    plugin_message("Plugin 'foo' is here.\n");
    return NULL;
}

and just to remove any confusion about what effect there is having each plugin a registration function by the same name, another plugin named plugin_bar.c:

#include <stdlib.h>
#include "plugin_system.h"

const char *register_plugin(void) __attribute__((used));
const char *register_plugin(void)
{
    plugin_message("Plugin 'bar' is here.\n");
    return NULL;
}

To make all of this easy to compile, we'll need a Makefile:

CC              := gcc
CFLAGS          := -Wall -Wextra -O2
LDFLAGS         := -ldl -Wl,-dynamic-list,plugin_system.list
PLUGIN_CFLAGS   := $(CFLAGS)
PLUGIN_LDFLAGS  := -fPIC
PLUGINS         := plugin_foo.so plugin_bar.so
PROGS           := example

.phony: all clean progs plugins

all: clean progs plugins

clean:
    rm -f *.o $(PLUGINS) $(PROGS)

%.so: %.c
    $(CC) $(PLUGIN_CFLAGS) $^ $(PLUGIN_LDFLAGS) -shared -Wl,-soname,$@ -o $@

%.o: %.c
    $(CC) $(CFLAGS) -c $^

plugins: $(PLUGINS)

progs: $(PROGS)

example: main.o plugin_system.o
    $(CC) $(CFLAGS) $^ $(LDFLAGS) -o $@

Note that Makefiles require intendation by tabs, not spaces; listing the file here always converts them to spaces. So, if you paste the above to a file, you'll need to fix the indentation, via e.g.

sed -e 's|^  *|\t|' -i Makefile

It is safe to run that more than once; the worst it can do, is mess up your "human-readable" layout.

Compile the above using e.g.

make

and run it via e.g.

./example ./plugin_bar.so ./plugin_foo.so

which shall output

Plugin 'bar' is here.
Plugin 'foo' is here.
All plugins loaded successfully.

to standard error.

Personally, I prefer to register my plugins via a structure, with a version number, and at least one function pointer (to the initialization function). This lets me load all plugins before initializing them, and resolve e.g. interplugin conflicts or dependencies. (In other words, I use a structure with a fixed name, rather than a function with a fixed name, to identify plugins.)

Now, as to __attribute__((used)). If you modify plugin_system.c into

#include <stdio.h>

void plugin_message(const char *msg) __attribute__((used));

void plugin_message(const char *msg)
{
    fputs(msg, stderr);
}

and modify the Makefile to have LDFLAGS := -ldl only, the example program and plugins will compile just fine, but running it will yield

./plugin_bar.so: ./plugin_bar.so: undefined symbol: plugin_message.

In other words, if the API exported to plugins is compiled in a separate compilation unit, you will need to use either -rdynamic or -Wl,-dynamic-list,plugin-system.list to ensure the functions are included in the dynamic symbol table in the final executable; the used attribute does not suffice.

---

If you want all and only non-static functions and symbols in plugin_system.o included in dynamic symbol table in the final binary, you can e.g. modify the end of the Makefile into

example: main.o plugin_system.o
    @rm -f plugin_system.list
    ./list_globals.sh plugin_system.o > plugin_system.list
    $(CC) $(CFLAGS) $^ $(LDFLAGS) -o $@

with list_globals.sh:

#!/bin/sh
[ $# -ge 1 ] || exit 0
export LANG=C LC_ALL=C
IFS=:
IFS="$(printf '\t ')"

printf '{\n'
readelf -s "$@" | while read Num Value Size Type Bind Vis Ndx Name Dummy ; do
    [ -n "$Name" ] || continue
    if [ "$Bind:$Type" = "GLOBAL:FUNC" ]; then
        printf '    %s;\n' "$Name"
    elif [ "$Bind:$Type:$Ndx" = "GLOBAL:OBJECT:COM" ]; then
        printf '    %s;\n' "$Name"
    fi
done
printf '};\n'

Remember to make the script executable, chmod u+x list_globals.sh.