docutils/sphinx custom directive creating sibling section rather than child

Question

Consider a reStructuredText document with this skeleton:

Main Title
==========

text text text text text

Subsection
----------

text text text text text

.. my-import-from:: file1
.. my-import-from:: file2

The my-import-from directive is provided by a document-specific Sphinx extension, which is supposed to read the file provided as its argument, parse reST embedded in it, and inject the result as a section in the current input file. (Like autodoc, but for a different file format.) The code I have for that, right now, looks like this:

class MyImportFromDirective(Directive):
    required_arguments = 1
    def run(self):
        src, srcline = self.state_machine.get_source_and_line()
        doc_file = os.path.normpath(os.path.join(os.path.dirname(src),
                                                 self.arguments[0]))
        self.state.document.settings.record_dependencies.add(doc_file)
        doc_text  = ViewList()

        try:
            doc_text = extract_doc_from_file(doc_file)
        except EnvironmentError as e:
            raise self.error(e.filename + ": " + e.strerror) from e

        doc_section = nodes.section()
        doc_section.document = self.state.document

        # report line numbers within the nested parse correctly
        old_reporter = self.state.memo.reporter
        self.state.memo.reporter = AutodocReporter(doc_text,
                                                   self.state.memo.reporter)
        nested_parse_with_titles(self.state, doc_text, doc_section)
        self.state.memo.reporter = old_reporter

        if len(doc_section) == 1 and isinstance(doc_section[0], nodes.section):
            doc_section = doc_section[0]

        # If there was no title, synthesize one from the name of the file.
        if len(doc_section) == 0 or not isinstance(doc_section[0], nodes.title):
            doc_title = nodes.title()
            doc_title.append(make_title_text(doc_file))
            doc_section.insert(0, doc_title)

        return [doc_section]

This works, except that the new section is injected as a child of the current section, rather than a sibling. In other words, the example document above produces a TOC tree like this:

• Main Title
  • Subsection
    • File1
    • File2

instead of the desired

• Main Title
  • Subsection
  • File1
  • File2

How do I fix this? The Docutils documentation is ... inadequate, particularly regarding control of section depth. One obvious thing I have tried is returning doc_section.children instead of [doc_section]; that completely removes File1 and File2 from the TOC tree (but does make the section headers in the body of the document appear to be for the right nesting level).

Answer 1

I don't think it is possible to do this by returning the section from the directive (without doing something along the lines of what Florian suggested), as it will get appended to the 'current' section. You can, however, add the section via self.state.section as I do in the following (handling of options removed for brevity)

class FauxHeading(object):
    """
    A heading level that is not defined by a string. We need this to work with
    the mechanics of
    :py:meth:`docutils.parsers.rst.states.RSTState.check_subsection`.

    The important thing is that the length can vary, but it must be equal to
    any other instance of FauxHeading.
    """

    def __init__(self, length):
        self.length = length

    def __len__(self):
        return self.length

    def __eq__(self, other):
        return isinstance(other, FauxHeading)


class ParmDirective(Directive):

    required_arguments = 1
    optional_arguments = 0
    has_content = True
    option_spec = {
        'type':          directives.unchanged,
        'precision':     directives.nonnegative_int,
        'scale':         directives.nonnegative_int,
        'length':        directives.nonnegative_int}

    def run(self):
        variableName = self.arguments[0]
        lineno = self.state_machine.abs_line_number()
        secBody = None
        block_length = 0

        # added for some space
        lineBlock = nodes.line('', '', nodes.line_block())

        # parse the body of the directive
        if self.has_content and len(self.content):
            secBody = nodes.container()
            block_length += nested_parse_with_titles(
                self.state, self.content, secBody)

        # keeping track of the level seems to be required if we want to allow
        # nested content. Not sure why, but fits with the pattern in
        # :py:meth:`docutils.parsers.rst.states.RSTState.new_subsection`
        myLevel = self.state.memo.section_level
        self.state.section(
            variableName,
            '',
            FauxHeading(2 + len(self.options) + block_length),
            lineno,
            [lineBlock] if secBody is None else [lineBlock, secBody])
        self.state.memo.section_level = myLevel

        return []

Answer 2

I don't know how to do it directly inside your custom directive. However, you can use a custom transform to raise the File1 and File2 nodes in the tree after parsing. For example, see the transforms in the docutils.transforms.frontmatter module.

In your Sphinx extension, use the Sphinx.add_transform method to register the custom transform.

Update: You can also directly register the transform in your directive by returning one or more instances of the docutils.nodes.pending class in your node list. Make sure to call the note_pending method of the document in that case (in your directive you can get the document via self.state_machine.document).