SaltStack Orchestration: Beacons and Reactors

Introduction

This is a simple orchestration lab. Related resources and notes are collected at the end of the guide.


Let’s configure and orchestrate rsyslog and learn about the following:

  • Jinja templates
  • Jinja macros
  • Minion Beacons
  • Master Reactors


Please refer to and run the SaltStack installer provided in SaltStack Quick Start Guide: 


https://linuxacademy.com/cp/socialize/index/type/community_post/id/15792


You should have lab machines #1 (master), #2 (minion), and #3 (minion) setup with salt. 


Objective

On each minion, create a file, /tmp/salt-notes.txt, and use SaltStack to monitor any changes to that file. When the file is modified, push the last line in the file, on only that minion, to the rsyslog service on master, where it should be filtered into a special log file /var/log/notes/handy-salt.log.


Getting Started

Setup

All of the source files below are placed in the master(machine #1) salt filesystem /srv/salt.

Place the following library in /srv/salt/lib/loggerlib.sls


{% macro loggerbundle(path2logfile,tag) -%}
/opt/logger.local/{{ tag }}-logger.sh:
file.managed:
- makedirs: True
- source: salt://files/rsyslog/logger.sh.jinja
- user: root
- group: root
- mode: 755
- template: jinja
- defaults:
path2logfile: {{ path2logfile }}
tag: {{ tag }}
{%- endmacro %}


Place the following 3 files in a template directory for rsyslog

/srv/salt/files/rsyslog:


rsyslog-listener.conf.jinja

$ModLoad imudp
$UDPServerRun 514

logsplitter.conf.jinja

if $syslogtag contains 'handy-salt' then /var/log/notes/handy-salt.log

logger.sh.jinja

{% set masterip = salt['cmd.run']('grep salt /etc/hosts | awk "{ print \$1; }"') -%}
/usr/bin/tail -1 {{ path2logfile }} | /usr/bin/logger -n {{ masterip }} -P 514 -t {{ tag }}


Place the following state in /srv/salt/notelog.sls

{% from 'lib/loggerlib.sls' import loggerbundle with context %}
{{ loggerbundle('/tmp/salt-notes.txt','handy-salt') }}

/tmp/salt-notes.txt:
file.touch


Place the following beacon in /srv/salt/files/minion.d/beacon.conf.jinja

beacons:
inotify:
/tmp/salt-notes.txt:
mask:
- modify
disable_during_state_run: True


Add these state files to /srv/salt:

1) rsyslog.sls

/etc/rsyslog.d/rsyslog-splitter.conf:
file.managed:
- source: salt://files/rsyslog/logsplitter.conf.jinja
- template: jinja

/etc/rsyslog.d/rsyslog-listener.conf:
file.managed:
- source: salt://files/rsyslog/rsyslog-listener.conf.jinja
- template: jinja

2) beacons.sls

/etc/salt/minion.d/beacon.conf:
file.managed:
- source: salt://files/minion.d/beacon.conf.jinja
- template: jinja
- makedirs: True


Beacons are a way to make the salt event bus available through simple configuration to translate system events into salt events, which can be reacted to by the Salt master in Reactors. Minions are instrumented with beacon modules, there are a lot these days:


https://docs.saltstack.com/en/latest/ref/beacons/all/index.html



Let’s run this stuff


1) Apply the rsyslog state to master (your machine #1)

salt yourusername1.* state.sls rsyslog


2) Apply the notelog state to all

salt \* state.sls notelog


3) Restart rsyslog

- confirm it’s there 

salt \* service.get_all | grep rsyslog 

- restart it 

salt \* service.restart rsyslog


4) Test the notelog

salt \* cmd.run 'echo 56789 >> /tmp/salt-notes.txt'
salt \* cmd.run /opt/logger.local/handy-salt-logger.sh
tail /var/log/notes/handy-salt.log


5) Deploy the beacon, and restart minions to pick up the beacon (use salt-ssh because the minion can’t restart itself without failing to return something from the operation, so there is a timeout)

salt \* state.sls beacons
salt-ssh minion[123] service.restart salt-minion


6) Configure the reactor quick and dirty  :)  You need to watch the event bus to figure out what the tags are.  

On master:

salt-run state.event pretty=true

If you modify the salt-notes.txt file,  the event bus will show that you need something that looks like this:

‘salt/beacon/username2.mylabserver.com/inotify//tmp/salt-notes.txt’

You just need a few more files to tie together individual minions to their individual reactors – run this quick and dirty script to generate reactor configuration and sls files directly into the master configuration.  See how the alternative format --out=text on the salt command simplifies the output to make it easier to grab a list of available minions. Crude but effective.

generate-reactor-code.sh:

#!/bin/bash
#
# Quick and dirty script to generate reactor conf and state files
# Run this on master.
# Reactor SLS is a little different. See
# https://docs.saltstack.com/en/getstarted/event/reactor.html
#
echo "reactor:" > /etc/salt/master.d/reactor.conf
for i in `salt \* test.ping --out=text | awk '{ print $1; }' | sed -e 's/.$//'`
do
cat << REACT01 >> /etc/salt/master.d/reactor.conf
- 'salt/beacon/${i}/inotify//tmp/salt-notes.txt':
- salt://reactor-${i}.sls
REACT01
cat << REACT02 > /srv/salt/reactor-${i}.sls
send ${i} note to syslog:
local.cmd.run:
- tgt: '${i}'
- arg:
- /opt/logger.local/handy-salt-logger.sh
REACT02
done


7) Test the reactor config.  Restart master in debug  just to make sure it parses the new file. It should show no errors. Verify, ^C to stop the debug master, then restart the service.


systemctl stop salt-master
salt-master -l debug
...
^C
systemctl start salt-master

8) Finally

Pick one of the minions and either edit or echo >> some text at the bottom of /tmp/salt-notes.txt

On master check the log file /var/log/notes/handy-salt.log, you should see your text logged.

vi will generate two events for inotify modify, so you will see duplicate log entries

echo >> generates only one event for inotify modify

If you want to see the beacon/reactor in action, start both the master and a minion in debug mode (-l debug) in two separate terminals. On a third terminal on master listen to the event log(step 6 above). Then use a fourth terminal to log into the minon and edit the /tmp/salt-notes.txt file.


THE END

There are many more aspects to SaltStack not covered here, such as requisites, salt-cloud, the actual orchestrate runner, etc. that might be covered on future posts.   I hope you had success with this guide and continue to delve into SaltStack.

Sources / Resources

Debugging and Tweaking the Install

IRC

http://irclog.perlgeek.de/salt/ (archives)

http://webchat.freenode.net/?channels=salt (join chat)

This guide was almost derailed by an inoperable beacon system. Fortunately because we have IRC logs for Salt it only cost me a couple hours sleep.

Shutting down the salt-minion and starting in debug is easy enough:


systemctl stop salt-minion
salt-minion -l debug

What this revealed is a failure to load the beacon file.

See this IRC log for relevant conversation, ending with the fact that if you pip install the dependency the problem is fixed:

https://irclog.perlgeek.de/salt/2016-04-06#i_12297672


pip install pyinotify

You’ll see that this line has been added to the minion installer in saltstack-init.sh in the SaltStack Quick Start Guide.

There are other options, such as specifying which version that bootstrap-salt.sh should load, thereby avoiding the newer dependency. What likely happened is that features were added to the inotify part of the beacon system, creating a requirement for pyinotify update and the Centos 7 yum repo was no longer adequate to run it. If you know package systems and python well, you could possibly update your yum repos to load an updated pyinotify. Fortunately the alternative package system for python pip delivered what was required.


https://pypi.python.org/pypi/pyinotify

https://docs.saltstack.com/en/latest/ref/beacons/all/salt.beacons.inotify.html

depends:    pyinotify Python module >= 0.9.5

Beacon/Reactor documentation is online:

https://docs.saltstack.com/en/latest/topics/reactor/

https://docs.saltstack.com/en/latest/topics/event/index.html

https://docs.saltstack.com/en/latest/topics/beacons/index.html

https://docs.saltstack.com/en/getstarted/event/reactor.html

Advanced orchestration 

https://github.com/kucerarichard/SaltedJBoss


Resources for rsyslog

http://download.rsyslog.com/design.pdf

https://media.readthedocs.org/pdf/rsyslog/latest/rsyslog.pdf

http://unix.stackexchange.com/questions/128156/remote-syslog-command-line-client

http://www.rsyslog.com/sending-messages-to-a-remote-syslog-server/

http://kwlug.org/sites/kwlug.org/files/2009-08-10-syslog-servers.pdf

http://lists.adiscon.net/pipermail/rsyslog/2013-August/033421.html


https://docs.saltstack.com/en/latest/topics/beacons/index.html


  • post-author-pic
    Treva W
    03-23-2017

    Wow, this looks awesome. Thanks Richard!

  • post-author-pic
    Christophe L
    03-25-2017

    Thanks for sharing this with our community!

  • post-author-pic
    Alexandru S
    03-26-2017

    hi guys,

    i did a little improvment in bash -please check out  http://pastebin.com/fQ9PJEuH, the code.
    on the salt master create a new file, paste the content, give appropiate permisions and run it
    What the script does ? all the steps in the tutorial UNTILL step : 

    6) Configure the reactor quick and dirty :) You need to watch the event bus to figure out what the tags are.  

    also i attached the little improvment also here : 


    #!/bin/bash

    # please run the first part in the as root or user with sudo and don't forget #about permission

    # define global variables

    USERID=$(uname -a | awk '{ print $2 }' | cut -d '.' -f 1 | sed 's/.$//')

    set -e

    set -v

    set -x

    mkdir -p /srv/salt/lib/

    touch /srv/salt/lib/loggerlib.sls

    cat << VALUES >> /srv/salt/lib/loggerlib.sls

    {% macro loggerbundle(path2logfile,tag) -%}

    /opt/logger.local/{{ tag }}-logger.sh:

    file.managed:

    - makedirs: True

    - source: salt://files/rsyslog/logger.sh.jinja

    - user: root

    - group: root

    - mode: 755

    - template: jinja

    - defaults:

    path2logfile: {{ path2logfile }}

    tag: {{ tag }}

    {%- endmacro %}

    VALUES

    mkdir -p /srv/salt/files/rsyslog

    touch /srv/salt/files/rsyslog/rsyslog-listener.conf.jinja

    cat << VALUES >> /srv/salt/files/rsyslog/rsyslog-listener.conf.jinja

    $ModLoad imudp

    $UDPServerRun 514

    VALUES

    touch /srv/salt/files/rsyslog/logsplitter.conf.jinja

    cat << VALUES >> /srv/salt/files/rsyslog/logsplitter.conf.jinja

    if $syslogtag contains 'handy-salt' then /var/log/notes/handy-salt.log

    VALUES

    touch /srv/salt/files/rsyslog/logger.sh.jinja

    cat << VALUES >> /srv/salt/files/rsyslog/logger.sh.jinja

    {% set masterip = salt['cmd.run']('grep salt /etc/hosts | awk "{ print \$1; }"') -%}

    /usr/bin/tail -1 {{ path2logfile }} | /usr/bin/logger -n {{ masterip }} -P 514 -t {{ tag }}

    VALUES

    touch /srv/salt/notelog.sls

    cat << VALUES >> /srv/salt/notelog.sls

    {% from 'lib/loggerlib.sls' import loggerbundle with context %}

    {{ loggerbundle('/tmp/salt-notes.txt','handy-salt') }}

    /tmp/salt-notes.txt:

    file.touch

    VALUES

    mkdir -p /srv/salt/files/minion.d

    touch /srv/salt/files/minion.d/beacon.conf.jinja

    cat << VALUES >> /srv/salt/files/minion.d/beacon.conf.jinja

    beacons:

    inotify:

    /tmp/salt-notes.txt:

    mask:

    - modify

    disable_during_state_run: True

    VALUES

    touch /srv/salt/rsyslog.sls

    cat << VALUES >> /srv/salt/rsyslog.sls

    /etc/rsyslog.d/rsyslog-splitter.conf:

    file.managed:

    - source: salt://files/rsyslog/logsplitter.conf.jinja

    - template: jinja

    /etc/rsyslog.d/rsyslog-listener.conf:

    file.managed:

    - source: salt://files/rsyslog/rsyslog-listener.conf.jinja

    - template: jinja

    VALUES

    touch /srv/salt/beacons.sls

    cat << VALUES >> /srv/salt/beacons.sls

    /etc/salt/minion.d/beacon.conf:

    file.managed:

    - source: salt://files/minion.d/beacon.conf.jinja

    - template: jinja

    - makedirs: True

    VALUES

    salt "${USERID}".* state.sls rsyslog

    salt \* state.sls notelog

    salt \* service.get_all | grep rsyslog && echo "TRUE "2>&1

    salt \* service.restart rsyslog

    salt \* cmd.run 'echo 56789 >> /tmp/salt-notes.txt'

    salt \* cmd.run /opt/logger.local/handy-salt-logger.sh

    bash timed-run 10 tail /var/log/notes/handy-salt.log

    salt \* state.sls beacons

    salt-ssh minion[123] service.restart salt-minion

    salt-run state.event pretty=true



  • post-author-pic
    Michael M
    03-28-2017

    Thank you Alex!

    Well that's everything in the guide you put into that bash :)

Looking For Team Training?

Learn More