Parsing XML in Ansible with nodes at different levels of hierarchy

Question

Trying to parse Juniper inventory that looks like this:

<chassis-inventory>
  <chassis style="inventory">
    <name>Chassis</name>
    <serial-number>T8382</serial-number>
    <description>MX5-T</description>
    <chassis-module>
      <name>Midplane</name>
      <version>REV 08</version>
      <part-number>711-038215</part-number>
      <serial-number>ABBZ6152</serial-number>
      <description>MX5-T</description>
      <clei-code>IPMUA00ARA</clei-code>
      <model-number>CHAS-MX5-T-S</model-number>
    </chassis-module>
    <chassis-module>
      <name>PEM 0</name>
      <version>Rev 05</version>
      <part-number>740-028288</part-number>
      <serial-number>WI07395</serial-number>
      <description>AC Power Entry Module</description>
      <clei-code>COUPAFAEAB</clei-code>
      <model-number>PWR-MX80-AC-S</model-number>
    </chassis-module>
    <chassis-module>
      <name>PEM 1</name>
      <version>Rev 05</version>
      <part-number>740-028288</part-number>
      <serial-number>WI07740</serial-number>
      <description>AC Power Entry Module</description>
      <clei-code>COUPAFAEAB</clei-code>
      <model-number>PWR-MX80-AC-S</model-number>
    </chassis-module>
    <chassis-module>
      <name>FPC 0</name>
      <part-number>BUILTIN</part-number>
      <serial-number>BUILTIN</serial-number>
      <description>MPC BUILTIN</description>
      <chassis-sub-module>
        <name>MIC 0</name>
        <part-number>BUILTIN</part-number>
        <serial-number>BUILTIN</serial-number>
        <description>4x 10GE XFP</description>
        <chassis-sub-sub-module>
          <name>PIC 0</name>
          <part-number>BUILTIN</part-number>
          <serial-number>BUILTIN</serial-number>
          <description>4x 10GE XFP</description>
        </chassis-sub-sub-module>
      </chassis-sub-module>
      <chassis-sub-module>
        <name>MIC 1</name>
        <version>REV 18</version>
        <part-number>750-043688</part-number>
        <serial-number>CADE8507</serial-number>
        <description>MS-MIC-16G</description>
        <clei-code>COUIBC4BAA</clei-code>
        <model-number>MS-MIC-16G</model-number>
        <chassis-sub-sub-module>
          <name>PIC 2</name>
          <part-number>BUILTIN</part-number>
          <serial-number>BUILTIN</serial-number>
          <description>MS-MIC-16G</description>
        </chassis-sub-sub-module>
      </chassis-sub-module>
    </chassis-module>
  </chassis>
</chassis-inventory>

The playbook is simple:

- name: Read XML
  set_fact:
    parsed_xml_data: "{{ junos_result.stdout | parse_xml('./templates/parse_inventory.yaml') }}"

And the template:

vars:
    module:
        name: "{{ item.name }}"
        serial_number: "{{ item.serial_number }}"
        description: "{{ item.description }}"
keys:
    dev_inventory:
        value: "{{ module }}"
        top: 'chassis'
        items:
            name: "name"
            serial_number: "serial-number"
            description: "description"

The result of this task gives me chassis attributes as expected:

ok: [lns01.lab] => {
    "ansible_facts": {
        "parsed_xml_data": {
            "dev_inventory": [
                {
                    "description": "MX5-T",
                    "name": "Chassis",
                    "serial_number": "T8382"
                }
            ]
        }
    },
    "changed": false
}

But I would like to find all modules and sub-modules, like with 'chassis*' xpath where serial-number is not "BUILTIN". Is there a way to use a wildcard here to match paths with variable length and use a condition to filter out BUILTIN serial numbers?

Answer 1

Here's a minimal (note in particular the complete lack of error checking) filter plugin that does what you want:

from lxml import etree


def parse_junos_xml(xmlstring):
    doc = etree.fromstring(xmlstring)
    result = []
    for chassis in doc.xpath("//chassis"):
        entry = {"submodules": []}
        result.append(entry)
        for attr in ["name", "serial-number", "description"]:
            entry[attr] = chassis.xpath(attr)[0].text

        for submodule in chassis.xpath("chassis-module"):
            module_entry = {}
            entry["submodules"].append(module_entry)
            for attr in submodule.xpath("*"):
                module_entry[attr.tag] = attr.text

    return result


class FilterModule:
    def filters(self):
        return {"parse_junos_xml": parse_junos_xml}

If we drop this into filter_plugins/junos.py, we can write a playbook like this:

- hosts: localhost
  gather_facts: false

  tasks:
    # replace this with the actual command to get data
    # from your junos device
    - command: cat data.xml
      register: junos_result

    - name: Parse XML
      set_fact:
        parsed_xml_data: "{{ junos_result.stdout | parse_junos_xml }}"

    - debug:
        var: parsed_xml_data

This will produce output that looks like:

ok: [localhost] => {
    "parsed_xml_data": [
        {
            "description": "MX5-T",
            "name": "Chassis",
            "serial-number": "T8382",
            "submodules": [
                {
                    "clei-code": "IPMUA00ARA",
                    "description": "MX5-T",
                    "model-number": "CHAS-MX5-T-S",
                    "name": "Midplane",
                    "part-number": "711-038215",
                    "serial-number": "ABBZ6152",
                    "version": "REV 08"
                },
                {
                    "clei-code": "COUPAFAEAB",
                    "description": "AC Power Entry Module",
                    "model-number": "PWR-MX80-AC-S",
                    "name": "PEM 0",
                    "part-number": "740-028288",
                    "serial-number": "WI07395",
                    "version": "Rev 05"
                },
                {
                    "clei-code": "COUPAFAEAB",
                    "description": "AC Power Entry Module",
                    "model-number": "PWR-MX80-AC-S",
                    "name": "PEM 1",
                    "part-number": "740-028288",
                    "serial-number": "WI07740",
                    "version": "Rev 05"
                },
                {
                    "chassis-sub-module": "\n        ",
                    "description": "MPC BUILTIN",
                    "name": "FPC 0",
                    "part-number": "BUILTIN",
                    "serial-number": "BUILTIN"
                }
            ]
        }
    ]
}

Answer 2

Thanks for the suggestion. I also found how to do it with parse_xml. Here is my playbook:

  tasks:
    - name: Collect HW inventory
      juniper.device.command:
        commands:
          - "show chassis hardware"
        formats:
          - "xml"
        dest: "../configs/{{ inventory_hostname }}-inventory.json"
      register: junos_result

    - name: Print response
      debug:
        var: junos_result.stdout

    - name: Read XML
      set_fact:
        parsed_xml_data: "{{ junos_result.stdout | parse_xml('./templates/parse_inventory.yaml') }}"

    - name: Save output in CSV file
      lineinfile:
        path: "../configs/juniper-inventory.csv"
        line: "{{ inventory_hostname }},{{item.description}},{{item.name}},{{item.serial_number}}"
        create: yes
      with_items: "{{ parsed_xml_data.dev_inventory }}"
      when: item.name is not match('^Xcvr.*$') and item.serial_number != 'BUILTIN'
      #when: item.serial_number != 'BUILTIN'  # Filter only BUILTIN items

The template for parse_xml action:

vars:
    module:
        name: "{{ item.name }}"
        serial_number: "{{ item.serial_number }}"
        description: "{{ item.description }}"
keys:
    dev_inventory:
        value: "{{ module }}"
        top: 'chassis//serial-number/..'
        items:
            name: "name"
            serial_number: "serial-number"
            description: "description"

The result of playbook's tasks execution:

TASK [Read XML] **********************************************************************************************************************************************
ok: [lns01.lab]

TASK [Save output in CSV file] *******************************************************************************************************************************
changed: [lns01.lab] => (item={'name': 'Chassis', 'serial_number': 'T8382', 'description': 'MX5-T'})
changed: [lns01.lab] => (item={'name': 'Midplane', 'serial_number': 'ABBZ6152', 'description': 'MX5-T'})
changed: [lns01.lab] => (item={'name': 'PEM 0', 'serial_number': 'WI07395', 'description': 'AC Power Entry Module'})
changed: [lns01.lab] => (item={'name': 'PEM 1', 'serial_number': 'WI07740', 'description': 'AC Power Entry Module'})
skipping: [lns01.lab] => (item={'name': 'Routing Engine', 'serial_number': 'BUILTIN', 'description': 'Routing Engine'})
skipping: [lns01.lab] => (item={'name': 'TFEB 0', 'serial_number': 'BUILTIN', 'description': 'Forwarding Engine Processor'})
changed: [lns01.lab] => (item={'name': 'QXM 0', 'serial_number': 'ABBZ6460', 'description': 'MPC QXM'})
skipping: [lns01.lab] => (item={'name': 'FPC 0', 'serial_number': 'BUILTIN', 'description': 'MPC BUILTIN'})
skipping: [lns01.lab] => (item={'name': 'MIC 0', 'serial_number': 'BUILTIN', 'description': '4x 10GE XFP'})
skipping: [lns01.lab] => (item={'name': 'PIC 0', 'serial_number': 'BUILTIN', 'description': '4x 10GE XFP'})
changed: [lns01.lab] => (item={'name': 'MIC 1', 'serial_number': 'CADE8507', 'description': 'MS-MIC-16G'})
skipping: [lns01.lab] => (item={'name': 'PIC 2', 'serial_number': 'BUILTIN', 'description': 'MS-MIC-16G'})
skipping: [lns01.lab] => (item={'name': 'FPC 1', 'serial_number': 'BUILTIN', 'description': 'MPC BUILTIN'})
changed: [lns01.lab] => (item={'name': 'MIC 0', 'serial_number': 'ABBY8576', 'description': '3D 20x 1GE(LAN) SFP'})
skipping: [lns01.lab] => (item={'name': 'PIC 0', 'serial_number': 'BUILTIN', 'description': '10x 1GE(LAN) SFP'})
skipping: [lns01.lab] => (item={'name': 'PIC 1', 'serial_number': 'BUILTIN', 'description': '10x 1GE(LAN) SFP'})
skipping: [lns01.lab] => (item={'name': 'Xcvr 0', 'serial_number': 'F7AKU26', 'description': 'SFP-LX10'})
skipping: [lns01.lab] => (item={'name': 'Xcvr 1', 'serial_number': 'F7AKU25', 'description': 'SFP-LX10'})
skipping: [lns01.lab] => (item={'name': 'Xcvr 2', 'serial_number': 'F7AKU27', 'description': 'SFP-LX10'})
skipping: [lns01.lab] => (item={'name': 'Xcvr 3', 'serial_number': 'HL9C310717005', 'description': 'SFP-LX10'})

PLAY RECAP ***************************************************************************************************************************************************
lns01.lab                  : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The file with inventory items contains:

lns01.lab,MX5-T,Chassis,T8382
lns01.lab,MX5-T,Midplane,ABBZ6152
lns01.lab,AC Power Entry Module,PEM 0,WI07395
lns01.lab,AC Power Entry Module,PEM 1,WI07740
lns01.lab,MPC QXM,QXM 0,ABBZ6460
lns01.lab,MS-MIC-16G,MIC 1,CADE8507
lns01.lab,3D 20x 1GE(LAN) SFP,MIC 0,ABBY8576