Disclaimer: – I am not an expert in Saltstack. I have been spending some good time to understand and unwrap bits of it primarily focused on Network Automation use-cases. This note was written by me (Gaurav Agrawal) in my personal capacity. The opinions expressed in this article are solely my own and do not reflect the view of my employer or my preference towards any of the OEMs.

In the previous post we looked at the Installation & configuration of basic necessary steps required to start with Salt. In this post, we will try to unwrap some of the key salt nomenclature used. This is a very next step on Network automation using salt.

Pillar

Pillar is used to organize the configuration data. e.g. NTP server details, DNS server details, Syslog server details, interface details etc.

Since, most of the network device available today – don’t support of native minion agent on the device – hence we need to create a proxy minion which can SSH to the device and get the required information. By default Cisco IOS device are supported by “NAPALM” proxy and other different OS types such as Cisco AirOS are supported by “netmiko” proxy.

Below example demonstrate to configure pillar for “NAPALM” & “NETMIKO” proxy.

Configuring pillar for NAPALM proxy i.e. router.sls

root@mrcissp-master-1:/# cat /etc/salt/pillar/router.sls
proxy:
  proxytype: napalm
  driver: ios
  host: 192.168.200.1
  username: mrcissp
  passwd: Nvidia@123

Refer to NAPALM proxy module for more details.

Configuring NETMIKO pillar i.e. wlc.sls

root@mrcissp-master-1:/# cat /etc/salt/pillar/wlc.sls
proxy:
  proxytype: netmiko
  device_type: cisco_wlc
  username: mrcissp
  password: Nvidia@123
  ip: 192.168.241.2
root@mrcissp-master-1:/#

Refer to NETMIKO proxy module for more details.

Grains

Grains represents static data(i.e. information which is very unlikely to change or does not change often) collected from devices. To collect all the grains from Minions/Proxy-minion use command [salt ‘*’ grains.items]. Below are the grains discovered on the running Minion i.e. “mrcissp-minion-1“.

root@mrcissp-master-1:/# salt '*' grains.items
mrcissp-minion-1:
    ----------
    SSDs:
    biosreleasedate:
        07/29/2019
    biosversion:
        6.00
    cpu_flags:
        - fpu
        - vme
        - de
        - pse
        - tsc
        - msr
        - pae
        - mce
        - cx8
        - apic
        - sep
        - mtrr
        - pge
        - mca
        - cmov
        - pat
        - pse36
        - clflush
        - mmx
        - fxsr
        - sse
        - sse2
        - ss
        - ht
        - syscall
        - nx
        - pdpe1gb
        - rdtscp
        - lm
        - constant_tsc
        - arch_perfmon
        - nopl
        - xtopology
        - tsc_reliable
        - nonstop_tsc
        - cpuid
        - pni
        - pclmulqdq
        - vmx
        - ssse3
        - fma
        - cx16
        - pcid
        - sse4_1
        - sse4_2
        - x2apic
        - movbe
        - popcnt
        - aes
        - xsave
        - avx
        - f16c
        - rdrand
        - hypervisor
        - lahf_lm
        - 3dnowprefetch
        - cpuid_fault
        - pti
        - ssbd
        - ibrs
        - ibpb
        - stibp
        - tpr_shadow
        - vnmi
        - ept
        - vpid
        - fsgsbase
        - smep
        - arat
        - flush_l1d
        - arch_capabilities
    cpu_model:
        Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
    cpuarch:
        x86_64
    disks:
        - loop1
        - sdb
        - loop6
        - loop4
        - sr0
        - loop2
        - loop0
        - loop7
        - sda
        - loop5
        - loop3
    dns:
        ----------
        domain:
        ip4_nameservers:
            - 8.8.8.8
        ip6_nameservers:
        nameservers:
            - 8.8.8.8
        options:
        search:
        sortlist:
    domain:
    fqdn:
        mrcissp-minion-1
    fqdn_ip4:
        - 127.0.1.1
    fqdn_ip6:
    fqdns:
    gid:
        0
    gpus:
    groupname:
        root
    host:
        mrcissp-minion-1
    hwaddr_interfaces:
        ----------
        eth0:
            2a:ed:fc:79:7f:6f
    id:
        mrcissp-minion-1
    init:
        unknown
    ip4_interfaces:
        ----------
        eth0:
            - 192.168.100.3
        lo:
            - 127.0.0.1
    ip6_interfaces:
        ----------
        eth0:
            - fe80::28ed:fcff:fe79:7f6f
        lo:
            - ::1
    ip_interfaces:
        ----------
        eth0:
            - 192.168.100.3
            - fe80::28ed:fcff:fe79:7f6f
        lo:
            - 127.0.0.1
            - ::1
    ipv4:
        - 127.0.0.1
        - 192.168.100.3
    ipv6:
        - ::1
        - fe80::28ed:fcff:fe79:7f6f
    kernel:
        Linux
    kernelrelease:
        4.15.0-55-generic
    kernelversion:
        #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019
    locale_info:
        ----------
        defaultencoding:
            None
        defaultlanguage:
            None
        detectedencoding:
            ANSI_X3.4-1968
        timezone:
            unknown
    localhost:
        mrcissp-minion-1
    lsb_distrib_codename:
        bionic
    lsb_distrib_description:
        Ubuntu 18.04.3 LTS
    lsb_distrib_id:
        Ubuntu
    lsb_distrib_release:
        18.04
    machine_id:
        578962dbb63ae45b159330245dd26e77
    manufacturer:
        VMware, Inc.
    master:
        192.168.100.2
    mdadm:
    mem_total:
        3944
    nodename:
        mrcissp-minion-1
    num_cpus:
        4
    num_gpus:
        0
    os:
        Ubuntu
    os_family:
        Debian
    osarch:
        amd64
    oscodename:
        bionic
    osfinger:
        Ubuntu-18.04
    osfullname:
        Ubuntu
    osmajorrelease:
        18
    osrelease:
        18.04
    osrelease_info:
        - 18
        - 4
    path:
        /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    pid:
        4538
    productname:
        VMware Virtual Platform
    ps:
        ps -efHww
    pythonexecutable:
        /usr/bin/python
    pythonpath:
        - /usr/local/bin
        - /usr/lib/python2.7
        - /usr/lib/python2.7/plat-x86_64-linux-gnu
        - /usr/lib/python2.7/lib-tk
        - /usr/lib/python2.7/lib-old
        - /usr/lib/python2.7/lib-dynload
        - /usr/local/lib/python2.7/dist-packages
        - /usr/lib/python2.7/dist-packages
    pythonversion:
        - 2
        - 7
        - 15
        - final
        - 0
    saltpath:
        /usr/local/lib/python2.7/dist-packages/salt
    saltversion:
        2019.2.2
    saltversioninfo:
        - 2019
        - 2
        - 2
        - 0
    serialnumber:
        VMware-56 4d e4 6c d3 e5 53 d5-0c 20 c1 55 a4 0e b9 4e
    server_id:
        822305722
    shell:
        /bin/sh
    swap_total:
        924
    systemd:
        ----------
        features:
            +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid
        version:
            237
    uid:
        0
    username:
        root
    uuid:
        564de46c-d3e5-53d5-0c20-c155a40eb94e
    virtual:
        VMware
    virtual_subtype:
        Docker
    zfs_feature_flags:
        False
    zfs_support:
        False
    zmqversion:
        4.3.2

Additional Master configuration

File Roots

Primarily it is used to isolate the environment e.g. we have a test environment, development environment, production environment served by common master.
Navigate to “Master” configuration file i.e. “nano /etc/salt/master” & add following details. In our test bed – we are referring to “base” environment. 

file_roots:
  base:
    - /etc/salt/pillar
    - /etc/salt/states
    - /etc/salt/reactors
    - /etc/salt/templates

Pillar Roots

Used to map environment with the appropriate directories of pillar “sls” files.
Navigate to “Master” configuration file i.e. “nano /etc/salt/master” & add following details. In our test bed – we are referring to “base” environment.

pillar_roots:
  base:
    - /etc/salt/pillar

Proxy configuration on a Minion

As the proxy minion is a subset of the regular minion, it inherits the same configuration options, as discussed in the minion configuration documentation. But there are additional configuration required for SSH based proxies to work properly.
Navigate to “Minion” proxy configuration file i.e. “nano /etc/salt/proxy” & add following details.

master: 192.168.100.2
pki_dir: /etc/salt/pki/proxy
cachedir: /var/cache/salt/proxy
multiprocessing: False
mine_enabled: True

Note: Multiprocessing is set to FALSE because in our example we are using SSH based proxies to connect with Router R1 and WLC. In case if we have to use SALT for REST based API for NX-OS, we must set this to TRUE. 

Pillar Top File

A very important configuration – Objective of pillar “top.sls” file is to tell a Minion ID to use which SLS file defined in Master.

Note: The top file is another SLS file named top.sls found under one of the paths defined in the file_roots.

• “ntp_config.sls” could be assigned to all the minion_id’s
• “syslog_config.sls” could be assigned to all the minion_id’s
• However, “ap_config” must be assigned to only WLC specific minion_id’s
• Similarly, “bgp_config” must be assigned to only Router specific minion_id’s

Navigate to “Master” top file i.e. “nano /etc/salt/pillar/top.sls” & add following details. In our test bed – we are referring to “base” environment e.g. Router* represents minion_id’s starting with keyword “Router”

base:
  Router*:
    - router
  wlc*:
    - wlc

Starting “salt-proxy”

To start a salt-proxy – use below command

salt-proxy –proxyid=<proxy_minion_id> -l debug 

root@mrcissp-minion-1:/# salt-proxy --proxyid=Router1 -d
root@mrcissp-minion-1:/# salt-proxy --proxyid=wlc1 -d

Once, proxy minion’s are started – we are required to accept their respective key’s

root@mrcissp-master-1:/# salt-key
Accepted Keys:
mrcissp-minion-1
Denied Keys:
Unaccepted Keys:
Router1
wlc1
Rejected Keys:
root@mrcissp-master-1:/# salt-key -A
The following keys are going to be accepted:
Unaccepted Keys:
Router1
wlc1
Proceed? [n/Y] Y
Key for minion Router1 accepted.
Key for minion wlc1 accepted.
root@mrcissp-master-1:/# salt-key
Accepted Keys:
Router1
mrcissp-minion-1
wlc1
Denied Keys:
Unaccepted Keys:
Rejected Keys:

Verification

Verify the connectivity between Master & proxy Minion. To do this, use below command

root@mrcissp-master-1:/# salt '*' test.ping
Router1:
    True
wlc1:
    True
mrcissp-minion-1:
    True
root@mrcissp-master-1:/#

Please remember :- In our case, “Router1” minion is managing R1 with “napalm” proxy. “wlc1” minion is managed by “netmiko” proxy.