MOOSE: Salt Grains

A fundamental principle of MOOSE is that servers are treated like “cattle” as much as possible, but with their own individual “pet” personality. To achieve this we built Salt “states” to consistently deploy servers, but those templates are configurable through “grains”. The actual grain data is stored in YAML format in /etc/salt/grains on your server.

The grains interface presents Salt with “grains of information” including default values such as domain name, IP address, kernel version, OS type, memory and many other properties of the system. We have extended this interface to allow common systems administration tasks to be simplified, and those customisations to be easily copied across a fleet of similar servers.

Using Grains

Setting a Grain

The easiest way to set a grain is from the command-line while logged in as root:

root@moose:~# salt-call grains.set moose:owner:name "Monty the Network Moose"

This sets the name key of the owner array within the moose array to the string value Monty the Network Moose. The command will return the structure of the topmost array (in this case moose):

local:
    ----------
    changes:
        ----------
        moose:
            ----------
            owner:
                ----------
                name:
                    Monty the Network Moose
                url:
                    https://twitter.com/NetworkMoose

Applying States

Once grains have been set or changed the entire set of templates can be re-applied to the server with one command. This wil take a several seconds to run, depending on how large the set of changes are:

root@moose:~# salt-call state.highstate --state-output=mixed

This “highstate” command (which applies all relevant states to the server) will output brief information about all the individual configuration items which have been applied, as well as a summary at the end:

Summary for local
 -------------
Succeeded: 316 (changed=32)
Failed:      0
 -------------
Total states run:     316
Total run time:   191.106 s

Showing Grains

To show all the grains:

root@moose:~# salt-call grains.items

To show one specific part of the grains:

root@moose:~# salt-call grains.get moose:owner

Unsetting a Grain

To remove the value set for a particular grain:

root@moose:~# salt-call grains.delkey moose:owner:url

You may need to force this:

root@moose:~# salt-call grains.delkey firewall:tcp4:12345 force=True

System

`fqdn` and `domain`

The fully-qualified domain name is used as the main identifier for your server. Your server overview page will be made available at this address.

fqdn: server.example.com
domain: example.com

`moose:packages`

This is a list of extra packages to install. We recommend using this if you want to be able to deploy an identical instance of your machine from a barebones installation

moose:
  packages:
    - vim
    - emacs24-nox

To set a list of values like this, the syntax is:

root@moose:~# salt-call grains.set moose:packages '["vim","emacs24-nox"]'

`cron`

Contains a mapping of hours and minutes (past the hour) for hourly, daily, weekly, and monthly cronjobs. These are set randomly the first time your server is provisioned. This helps spread the workload and prevent a “thundering herd” during daily maintenance tasks such as backups. Weekly jobs always run on Sunday. Monthly jobs always run on the first day of the month.

cron:
  hourly:
    minute: '17'
  daily:
    hour: '6'
    minute: '25'
  weekly:
    hour: '6'
    minute: '47'
  monthly:
    hour: '6'
    minute: '52'

`email`

A list of email addresses to receive automated information from server. Some packages only support one recipient, and so only the first item in the list will be used.

email:
  default:
    - webmaster@example.com
    - alice@example.com
    - bob@example.com
  security:
    - hostmaster@example.com
    - charlie@example.com

`moose:owner`

It is possible to show the owner of your server on its overview page.

moose:
  owner:
    name: Network Moose
    url: https://twitter.com/NetworkMoose

Security

`firewall`

Deploy a firewall on using netfilter-persistent. On modern hosts this will generate /etc/nftables.conf which will be applied at system startup.

firewall:
  tcp4:
    22:
      46.227.200.128/28: FAELIX Admin VPN
      185.134.196.22/28: FAELIX Admin VPN
      192.0.2.0/24: our network
    80: {}
    443: {}
  udp4:
    53:
      192.0.2.0/24: our network
    443: {}
  tcp6:
    22:
      2a01:9e00:a217:fa00::/56: FAELIX Admin VPN
      2a01:9e01:a217:fa00::/56: FAELIX Admin VPN
      2001:db8::/32: our network
    80: {}
    443: {}
  udp6:
    53:
      2001:db8::/32: our network
    443: {}

Each port number is an array mapping each permitted subnet to a comment.

root@moose:~# salt-call grains.set firewall:tcp4:12345:192.0.2.0/24 "permit our own network"

To open the port to the world, use an empty array (i.e. no restrictions).

root@moose:~# salt-call grains.set firewall:tcp4:80 '{}'

To close the port back down again, delete the reference to the port:

root@moose:~# salt-call grains.delkey firewall:tcp4:12345

`root`

root:
  authorized_keys:
    faelix_noc: True
    faelix_soc: True

`sshd`

sshd:
  installed: True
  port: 22
  listenaddress:
    - '0.0.0.0'
    - '::'
  permitrootlogin: 'prohibit-password'
  passwordauthentication: 'yes'
  challengeresponseauthentication: 'yes'
  maxauthtries: 3
  maxsessions: 10
  x11forwarding: 'no'
  allowtcpforwarding: 'yes'
  allowagentforwarding: 'yes'
  tcpkeepalive: 'yes'
  permittunnel: 'yes'
  printmotd: 'no'
  printlastlog: 'yes'
  compression: 'yes'
  banner: null
  usedns: 'yes'
  logingracetime: '10m'
  loglevel: 'INFO'

`lynis`

lynis:
  skip-tests:
  - SSH-7408:permitrootlogin
  - SSH-7408:port
  - FILE-6310
  - FIRE-4513
  - LOGG-2190

`email:security`

A list of email addresses to receive alerts about the security posture of your server. Some packages only support one recipient, and so only the first item in the list will be used.

email:
  security:
    - hostmaster@example.com
    - security@example.com

If unset, email:default will be used instead.

Hostname Resolution

`moose:network`

If set to “dynamic” then /etc/resolv.conf will not be autoconfigured.

`moose:nameservers`

A list of nameservers. Defaults to Faelix’s standard ones.

moose:
  nameservers:
    - "46.227.200.54"
    - "46.227.200.55"

`moose:hosts`

Static hostname to IP address mappings.

moose:
  hosts:
    "192.0.2.1":
      - myrouter.example.com
      - myrouter
    "2001:db8::42":
      - myrouter.example.com
      - myrouter

The above example will become the following in /etc/hosts:

192.0.2.1      myrouter.example.com myrouter
2001:db8::42   myrouter.example.com myrouter

Backup

`backup:borg`

Use borgbackup to automatically store snapshots of the server. This is used to generate the /etc/cron.daily/borgbackup script and its configuration file /etc/faelix/moose/borgbackup.

At a minimum both backup:borg:destination and backup:borg:passphrase need to be set. Extra backup repositories can be set under the repo key, with their own retention policy and exclusion lists.

backup:
  borg:
    destination: borg1.w.faelix.net
    user: borg
    passphrase: gibberishJGVRM8L3ZYAQWAJTpassword
    repo:
      owncloud:
        path: /var/www/owncloud
        exclude:
        - /var/www/owncloud/data/*.log
        keep_daily: 7
        keep_monthly: 1
        keep_weekly: 1

Monitoring

`collectd:faelix`

Explicitly set this to False if you do not with for statistics to be sent to FAELIX’s collectd service.

collectd:
  faelix: True

`collectd:servers`

Other servers to send collectd statistics to.

collectd:
  servers:
    192.0.2.1:
      username: foo
      password: bar
      securitylevel: Encrypt
      port: 25826

`collectd:user_suffix` and `collectd:password`

The username and password used to send packets to Faelix’s monitoring systems is specified by these two grains. Changing them will probably prevent expected functioning.

`collectd:ping`

Specify extra hosts you would like to ping from your server as a list.

collectd:
  ping:
    hosts:
      - "192.0.2.1"
      - "192.0.2.2"
      - "2001:db8::13"
    interval: 8
    timeout: 0.9
    source_address: "192.0.2.254"
    device: eth0

Web Server

`php`

php:
  max_execution_time: 30
  error_reporting: "E_ALL & ~E_DEPRECATED & ~E_STRICT"
  memory_limit: "128M"
  max_input_time: 60
  post_max_size: "128M"
  upload_max_filesize: "128M"
  max_file_uploads: 32
  default_socket_timeout: 60
  date_timezone: UTC

`xcache`

xcache:
  admin:
    password: FxU35eJ2Ow6vsNh1
  cacher: False

`xcache:cacher`

Set this to False on systems where you encounter problems caused by XCache.

Mail Server

`postfixadmin`

When Postfix Admin is installed, databases are automatically created and the credentials recorded here. Three different MySQL users are created, one for each of the various components that need to query the postfixadmin database.

postfixadmin:
  pfa-dovecot:
    password: YnQquVWHy3G52fE6
  pfa-postfix:
    password: a34Iv36SsEP7D7Do
  pfa-vacation:
    password: K23SQCK0na6Ye0qU

`postfix`

postfix:
  smtp_tls_security_level: may
  smtpd_tls_security_level: may
  smtpd_tls_eecdh_grade: strong
  tls_eecdh_strong_curve: prime256v1
  tls_eecdh_ultra_curve: secp384r1
  tls:
    v12: False
  tls_policy_map:
    example.com: encrypt

`roundcube`

roundcube:
  config:
    des_key: ot4wAP3yYQzAIXHz3hF4lhxd
  mysql:
    password: k24Nxy66U0aeIQ0r

`amavis`

amavis:
  sa_spam_subject_tag: "***SPAM***"
  sa_tag_level_deflt: -999
  sa_tag2_level_deflt: 6.31
  sa_kill_level_deflt: 6.31
  sa_dsn_cutoff_level: 10
  sa_mail_body_size_limit: "200*1024"
  sa_local_tests_only: False
  final_virus_destiny: D_DISCARD
  final_banned_destiny: D_REJECT
  final_spam_destiny:  D_DISCARD
  final_bad_header_destiny: D_PASS
  cron_tidy_quarantine_days: 28

`ntp`

ntp:
  servers:
    ntp2.inrim.it: {}
    time-a.nist.gov: {}
    stdtime.gov.hk: {}
    ntp.ix.ru: {}
    46.227.200.70:
      flags:
        - iburst
    46.227.200.72:
      flags:
        - iburst
    185.134.196.166:
      flags:
        - iburst
        - peer
    127.127.1.0:
      fudge:
        stratum: 11

`nginx`

nginx:
  release: vendor
  ssl_protocols: "TLSv1 TLSv1.1 TLSv1.2"
  client_max_body_size: "128m"

`php`

php:
  release: distribution

distribution for the version of PHP included in your server’s operation system distribution
7.0 for PHP 7.0 from packages.sury.org
7.2 for PHP 7.2 from packages.sury.org
7.3 for PHP 7.3 from packages.sury.org
7.4 for PHP 7.4 from packages.sury.org
8.0 for PHP 8.0 from packages.sury.org

`nextcloud`

nextcloud:
  host: cloud.example.com

Salt Grains

Using Grains

Setting a Grain

Applying States

Showing Grains

Unsetting a Grain

System

fqdn and domain

moose:packages

cron

email

moose:owner

Security

firewall

root

sshd

lynis

email:security

Hostname Resolution

moose:network

moose:nameservers

moose:hosts

Backup

backup:borg

Monitoring

collectd:faelix

collectd:servers

collectd:user_suffix and collectd:password

collectd:ping

Web Server

php

xcache

xcache:cacher

Mail Server

postfixadmin

postfix

roundcube

amavis

ntp

nginx

php

nextcloud

`fqdn` and `domain`

`moose:packages`

`cron`

`email`

`moose:owner`

`firewall`

`root`

`sshd`

`lynis`

`email:security`

`moose:network`

`moose:nameservers`

`moose:hosts`

`backup:borg`

`collectd:faelix`

`collectd:servers`

`collectd:user_suffix` and `collectd:password`

`collectd:ping`

`php`

`xcache`

`xcache:cacher`

`postfixadmin`

`postfix`

`roundcube`

`amavis`

`ntp`

`nginx`

`php`

`nextcloud`