get callstack of c++ application including shared libraries

Question

I wrote a small function to create a callstack in c++, using bfd to resolve the addresses. It works fine and I get detailed information (source file and line) for all functions inside the current application, but I get no information what-so-ever about shared libraries that are included in my application.

For example:

callstack:
[0x00002b54229ba6d3 .none] <not sectioned address>
[0x00002b5422927907 .none] <not sectioned address>
[0x00002b54229286d0 .none] <not sectioned address>
[0x00000000004f8608 .text] tensorNetwork.hxx:63 (inside operator())
[0x00000000005528da .text] /usr/include/c++/4.8/functional:2058 (inside _M_invoke)
[0x000000000058231c .text] /usr/include/c++/4.8/functional:2469 (inside std::function<bool ()>::operator()() const)
[0x00000000005806c0 .text] test.cpp:26 (inside ___test(std::pair<std::string, std::function<bool ()> > const&))
[0x0000000000581693 .text] test.cpp:119 (inside main)
[0x00002b5423fdebe5 .none] <not sectioned address>
[0x000000000042c129 .text] /home/abuild/rpmbuild/BUILD/glibc-2.18/csu/../sysdeps/x86_64/start.S:125 (inside _start)

As you can see the symbols of the executable and the static object that got linked with the application are resolved correctly, but the address in the high ranges (eg. 0x00002b54229ba6d3) are not. These addresses are not part of either my application or my shared library file. Further tools like addr2line thus cannot reconstruct the position of that instruction either.

That I cannot resolve these addresses with the bfd tools is somewhat expected as long as I only open the file on disc to get the symbols (what I currently do is abfd = bfd_openr("/proc/self/exe", 0);), so is there a way to get the bfd of the currently running process (thus including the sections of the shared libraries)? If not: how can I get a list of loaded shared objects and their offset and how can I relate these offsets to the shared object files on disc (such that I may load the bfd of the .so file separately)?

Answer 1

The glibc adds a function to the dl library called dladdr. With it it is possible to find the filename of the shared object and its loaded memory offset. Loading these object files with bfd_openr allowed me to dump the source-files and lines similar to how addr2line would do it (but still dump some information in case these are not available). For example (notice that the libc6.so does not contain debug symbols on my system and thus only the closest exported symbol is shown):

callstack:
[0x00002b3d744b4340 .text] /homes/numerik/huber/store/code/tensorDev/algorithm/als.hpp:11 (inside xerus::ALSVariant::lapack_solver(xerus::TensorNetwork const&, xerus::Tensor&, xerus::Tensor const&))
[0x0000000000571f0a .text] /usr/include/c++/4.8/functional:2073 (inside std::_Function_handler<void (xerus::TensorNetwork const&, xerus::Tensor&, xerus::Tensor const&), void (*)(xerus::TensorNetwork const&, xerus::Tensor&, xerus::Tensor const&)>::_M_invoke(std::_Any_data const&, xerus::TensorNetwork const&, xerus::Tensor&, xerus::Tensor const&))
[0x00002b3d744ddcba .text] /usr/include/c++/4.8/functional:2469 (inside std::function<void (xerus::TensorNetwork const&, xerus::Tensor&, xerus::Tensor const&)>::operator()(xerus::TensorNetwork const&, xerus::Tensor&, xerus::Tensor const&) const)
[0x00002b3d744b98bc .text] /homes/numerik/huber/store/code/tensorDev/algorithm/als.hpp:117 (inside xerus::ALSVariant::operator()(xerus::TTNetwork<true> const&, xerus::TTNetwork<false>&, xerus::TTNetwork<false> const&, double, std::vector<double, std::allocator<double> >*) const)
[0x0000000000547ac3 .text] /homes/numerik/huber/store/code/tensorDev/unitTests/als.hxx:4 (inside operator())
[0x0000000000554744 .text] /usr/include/c++/4.8/functional:2058 (inside _M_invoke)
[0x00000000005827c8 .text] /usr/include/c++/4.8/functional:2469 (inside std::function<bool ()>::operator()() const)
[0x0000000000580b6c .text] /homes/numerik/huber/store/code/tensorDev/misc/test.cpp:26 (inside ___test(std::pair<std::string, std::function<bool ()> > const&))
[0x0000000000581b3f .text] /homes/numerik/huber/store/code/tensorDev/misc/test.cpp:119 (inside main)
[0x00002b3d75b5dbe5 .text] ??:? (inside __libc_start_main +0x245)
[0x000000000042c269 .text] /home/abuild/rpmbuild/BUILD/glibc-2.18/csu/../sysdeps/x86_64/start.S:125 (inside _start)

In case someone needs the same functionality (I found it annoyingly difficult to create a nice callstack) here is the sourcecode (as part of our library licensed under the AGPLv3):

// Xerus - A General Purpose Tensor Library
// Copyright (C) 2014-2015 Benjamin Huber and Sebastian Wolf. 
// 
// Xerus is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published
// by the Free Software Foundation, either version 3 of the License,
// or (at your option) any later version.
// 
// Xerus is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
// 
// You should have received a copy of the GNU Affero General Public License
// along with Xerus. If not, see <http://www.gnu.org/licenses/>.
//
// For further information on Xerus visit https://libXerus.org 
// or contact us at [email protected].

std::string demangle_cxa(const std::string &_cxa) {
    int status;
    std::unique_ptr<char[]> realname;
    realname.reset(abi::__cxa_demangle(_cxa.data(), 0, 0, &status));
    if (status != 0) {
        return _cxa;
    }
    if (realname) {
        return std::string(realname.get());
    } else {
        return "";
    }
}

struct bfdResolver {
    struct storedBfd {
        bfd* abfd;
        asymbol** symbols;
        intptr_t offset;
    };
    static std::map<void *, storedBfd> bfds;
    static bool bfd_initialized;


    static std::string resolve(void *address) {
        if (!bfd_initialized) {
            bfd_init();
            bfd_initialized = true;
        }

        std::stringstream res;
        res << "[0x" << std::setw((int)sizeof(void*)*2) << std::setfill('0') << std::hex << (uintptr_t)address;

        // get path and offset of shared object that contains this address
        Dl_info info;
        dladdr(address, &info);
        if (info.dli_fbase == nullptr) {
            return res.str()+" .?] <object to address not found>";
        }

        // load the corresponding bfd file (from file or map)
        if (bfds.count(info.dli_fbase) == 0) {
            std::unique_ptr<storedBfd> newBfd(new storedBfd);
            newBfd->abfd = bfd_openr(info.dli_fname, 0);
            if (!newBfd->abfd) {
                return res.str()+" .?] <could not open object file>";
            }
            bfd_check_format(newBfd->abfd,bfd_object);
            size_t storage_needed = bfd_get_symtab_upper_bound(newBfd->abfd);
            newBfd->symbols =reinterpret_cast<asymbol**>(new char[storage_needed]);
            /*size_t numSymbols = */bfd_canonicalize_symtab(newBfd->abfd, newBfd->symbols );

            newBfd->offset = (intptr_t)info.dli_fbase;

            bfds.insert(std::pair<void *, storedBfd>(info.dli_fbase, *newBfd.release()));
        } 

        storedBfd &currBfd = bfds.at(info.dli_fbase);

        asection *section = currBfd.abfd->sections;
        bool relative = section->vma < (uintptr_t)currBfd.offset;
//      std::cout << '\n' << "sections:\n";
        while (section != nullptr) {
            intptr_t offset = ((intptr_t)address) - (relative?currBfd.offset:0) - section->vma;
//          std::cout << section->name << " " << section->id << " file: " << section->filepos << " flags: " << section->flags 
//                      << " vma: " << std::hex << section->vma << " - " << std::hex << (section->vma+section->size) << std::endl;

            if (offset < 0 || (size_t)offset > section->size) {
                section = section->next;
                continue;
            }
            res << ' ' << section->name;
            if (!(section->flags | SEC_CODE)) {
                return res.str()+"] <non executable address>";
            }
            // get more info on legal addresses
            const char *file;
            const char *func;
            unsigned line;
            if (bfd_find_nearest_line(currBfd.abfd, section, currBfd.symbols, offset, &file, &func, &line)) {
                if (file) {
                    return res.str()+"] "+std::string(file)+":"+to_string(line)+" (inside "+demangle_cxa(func)+")";
                } else {
                    if (info.dli_saddr) {
                        return res.str()+"] ??:? (inside "+demangle_cxa(func)+" +0x"+std::to_string((intptr_t)address-(intptr_t)info.dli_saddr)+")";
                    } else {
                        return res.str()+"] ??:? (inside "+demangle_cxa(func)+")";
                    }
                }
            } else {
                return res.str()+"] <bfd_error> (inside "+demangle_cxa((info.dli_sname?info.dli_sname:""))+")";
            }
        }
//      std::cout << " ---- sections end ------ " << std::endl;
        return res.str()+" .none] <not sectioned address>";
    }
};

std::map<void *, bfdResolver::storedBfd> bfdResolver::bfds;
bool bfdResolver::bfd_initialized = false;






std::string get_call_stack() {
    const size_t MAX_FRAMES = 100;
    std::vector<void *> stack(MAX_FRAMES);
    int num = backtrace(&stack[0], MAX_FRAMES);
    if (num <= 0) {
        return "Callstack could not be built.";
    }
    stack.resize((size_t) num);
    std::string res;
    //NOTE i=0 corresponds to get_call_stack and is omitted
    for (size_t i=1; i<(size_t)num; ++i) {
        res += bfdResolver::resolve(stack[i]) + '\n';
    }
    return res;
}