Ambiguous term in salt documentation, overused to describe any piece of salt logic.
The *.sls files contains state definitions, this is just the data file (YAML by default) the Salt translates into State Module execution

Core Modules

Execution Modules

salt/modules
Custom: salt://_modules

The salt or salt-call command executes exactly the Execution Modules
Contrary to State Modules, Execution Modules don't check any state, they just perform action.

salt '*' cmd.run "ls /" cwd=/

cmd - the module name, the module name is either the salt/modules/cmd.py or salt/modules/any_name.py containing __virtualname__ = "cmd"

run - the actual function definition, everything after function definition is the args and kwargs (for the actual function or for the salt state compiler)

"ls /" - first positional argument of run function

cwd=/ - keyword argument

State Modules

salt/states
Custom: salt://_states

Enforces desired states on the remote. Under the hood they usually use Execution Modules

pkgs_pip:
  pip.installed:
    - name: pip_packages
    - pkgs:
      - google-auth
    - reload_modules: True

pkgs_pip - state ID

pip - State Module name, maps directly to either salt/states/pip.py or salt/states/any_name.py containing __virtualname__ = "pip"

installed - the actual function name, naming convention: past tense.

- name: pip_packages the first positional argument, by convention called name. When the name is not passed the state ID is used as the name

Ordering states

Salt supports two ordering modes: 1. Definition order: in the order of appearance in top.sls file, thus it is the filename that determines order. Then the states are executed by the order of appearance in the state file itself. The included files that were defined after including file are executed first. inculde takes precedence. 2. Lexicographic order: states are sorted by: their name, function and then by state ID. Enable with configuration option: state_auto_order: False, the include statement in sls files doesn't affect the order at all.

Both options respect the requisite statements.
The requisites take precedence over configured ordering.

Full list of requisite statements and their usage.

If multiple states happen to be assigned same order number (the internal number that formally determines execution order), then Salt fallbacks to Lexicographic ordering.

Evaluating states

By default sls files are written using in YAML format with Jinja templates. Both engines are not aware of each other. Jinja templating starts before validating YAML.

In other words: 1. Jinja must produce valid YAML file 2. YAML file must be a valid highstate data structure 3. Highstate is compiled to lowstate (lowstate is a sorted list of state executions) 4. Salt executes the list in the order

The full evaluation and execution order:
Jinja -> YAML -> highstate -> lowstate -> execution

It is very easy to misuse jinja. When the state starts to be unreadable, it is possible candidate to switch to different renderer (usually #!py).
However user desired logic may be too complex still. Then writing custom Execution Module or State Module is a better idea.

Additionally as the State Tree grows, it is easy to fall into following trap (depicted by example):

clone_repo:
  git.latest:
    - name: https://github.com/kiemlicz/util
    - target: /tmp/util/
    - branch: master

{% for f in '/tmp/util/' | list_files %}
{% if f == '/tmp/util/README.md' %}

add_developer_{{ f }}:
  file.append:
  - name: {{ f }}
  - text: "added contributor: bla@o2.pl"
  - require:
    - git: clone_repo

{% endif %}
{% endfor %}

User assumed that the jinja will 'see' clone_repo changes. It is not true.
Jinja is evaluated first, thus when this sls file is applied first time, effectively the YAML looks like:

clone_repo:
  git.latest:
    - name: https://github.com/kiemlicz/util
    - target: /tmp/util/
    - branch: master

However during next run, the jinja will 'see' the changes (as they are already applied). Thus the output YAML will become:

clone_repo:
  git.latest:
    - name: https://github.com/kiemlicz/util
    - target: /tmp/util/
    - branch: master

add_developer_/tmp/util/README.md:
  file.append:
  - name: /tmp/util/README.md
  - text: "added contributor: bla@o2.pl"
  - require:
    - git: clone_repo

There are couple of options how to overcome such situation, most common involve: - writing custom Execution Module or State Module - using Slots (if you know what you are doing)

Slots

Relatively new Salt feature, allows to store the result of Execution Module and use it in next Modules (during same run).
Example:

dnsutils:
  pkg.latest:
  - name: dnsutils

find_domain:        # works
  cmd.run:
  - name: "nslookup google.com"

the_state_id:       # fails
  test.show_notification:
  - name: some name
  - text: "the server ip: {{ salt['cmd.run']("nslookup google.com") }}"

Will fail because not-yet-existing command output is used as part of state definition.
Using slots we can overcome this:

dnsutils:
  pkg.latest:
  - name: dnsutils

find_domain:        # works
  cmd.run:
  - name: "nslookup google.com"

the_state_id:       # works
  test.show_notification:
  - name: some name
  - text: __slot__:salt:cmd.run("nslookup google.com")

Idempotence

It is tempting to wonder: is the state execution idempotent?

Is depends, e.g., the states like:

run_scripts:
    cmd.script:
        - name: do_dangerous_stuff.sh

are not guaranteed to be idempotent because everything depends on underlying: do_dangerous_stuff.sh script, so in general it is wise to assume that such state is not idempotent.
Moreover adding jinja constructs that modify underlying system also doesn't help (not advised though).

However Salt provides requisite constructs that can be added to (not all) states: onlyif or unless

Thus it is possible to make Salt states idempotent

Best practices

Check these two documents, they provide excellent details about how to create state properly: - best practices - formulas best practices

Data Modules

Contains runtime configuration, variables, secret data... data...

Data\Authority	Salt Master	Salt Minion	Other
Secrets	Pillar	SDB	SDB
Config	Pillar	Grains	SDB

Grains Modules

salt/grains
Custom: salt://_grains

Minion specific data, can be also specified in configuration file grains: {}, or set by master.
Grains are refreshed on a very limited basis and are largely static data. If there is some minion specific data that needs to be updated on the master then the Salt Mine is the place to go.

Pillar Modules

salt/pillar
Custom: salt://_pillar

Salt Master is authoritative over pillar data. Pushes pillar to minions that cache it. Minion may request pillar data on its own.

SDB Modules

salt/sdb
Custom: salt://_sdb

Used when neither Salt Master nor Salt Minion is authoritative over data. It could be used to pull secrets from HashiCorp Vault or other keystores. If it is Salt Minion that makes the call to sdb it calls directly the third party entity. It is possible to use SDB modules in the Salt Master/Minion config files:

client_id: sdb://module_name/secret_client_id

However it's impossible to bootstrap Salt with custom SDB modules used in config already. During the Salt Minion/Master startup the full config is read and parsed, thus any sdb://<profile>/key are evaluated Use of custom SDB modules requires preceding: sync_sdb, which doesn't happen during initial bootstrap

Event Modules and Reactor System

Salt Master and Salt Minion have their own event buses. Depending on the Module's function used to fire event, event may or may not be propagated to other event buses (e.g. from Minion to Master and vice-versa).

Event always comprises of two things: 1. event tag 2. data dictionary

Salt event system that uses Event Modules is described in separate section

Beacon Modules

salt/beacons
Custom: salt://_beacons
Way to notify the master about anything. Works like a probe/sensor, e.g. disk is going full. Notifications use the Salt Minion's event bus and are propagated to Salt Master.
Internally beacons work in the following way: - minion's scheduler starts the beacon module's beacon function - the function fetches desired data in the most lightweight way possible - data is forwarded to Salt Minion event bus

Queue Modules

salt/queues
Custom: n/a
Helps to handle the events, sometimes it is desirable to enqueue incoming events and pop them sequentially instead of allowing asynchronous reactions to happen.

Engine Modules

salt/engines
Custom: salt://_engines
Can be run on Salt Master or Salt Minion, once started: runs forever in separate process. Commonly used to integrate with external systems (like sending the notifications to slack) or fetching the external systems data under the Salt infrastructure.

Thorium Modules - experimental

salt/thorium
Custom: salt://_thorium
Primarily created to add event aggregation, requires additional configuration. Sometimes it is desirable to start a reaction once the set of N minions complete their highstate logic, not every time each of the minions completes. This can be achieved with Thorium. Example of thorium state file that fires the event only when two salt/custom/event are received:

something:
  reg.list:
    - add: "some_field"
    - match: 'salt/custom/event'
  check.len_eq:
    - value: 2
send_when:
  runner.cmd:
  - func: event.send
  - arg:
    - thor/works
  - require:
      - check: something

Result Modules

TODO

Output Modules

Result Modules

Admin Modules

Wheel Modules

salt/wheel
Custom: n/a
Dealing with Salt infrastructure itself, e.g., accept Salt Minion key.

Runner Modules

salt/runners
Custom: salt://_runners (runner_dirs configuration option)
Used exclusively by salt-run command. They are pure Salt Master Modules, designed to run on master only.

Cache Modules

salt/cache Custom: salt://_cache

Netapi Modules

salt/netapi Modules that expose Salt API, require configuration to enable

Python client API

This is not the part of Salt's netapi, but a library for interaction with Salt
Same that is used by Salt commands

Integration Modules

TODO

Utility Modules

TODO