Ansible AWX demo environment -

As I was planning an Ansible Meetup about the Ansible Automation Platform (AAP, the successor of Ansible Tower) I was contemplating about a demo environment for the attendants. This can be done with ease as it it nothing more than clicky-di-click.

You can imagine that’s not the way I went :-). There is a simple rule in life:
If you can automate it, automate it.

The complete environment is going to be a multipart environment, so that the attendants can experiment and have an environment that slightly resembles real live.

So, what’s needed:

Ansible AWX
A Git server
A managed node to connect to (per attendee)

Ansible AWX

As the Meetup is about Ansible Automation Platform, it seems a weird choice to install AWX. But do not be fooled. AWX is the open source upstream of Ansible AAP. There is some functionality missing, but nothing that hurts us for the meetup. And, for AAP a license is required from Red Hat and I think it to be a waste of money to pay for a license that’s only used for a couple of hours. So, AWX it is.

As this is only a demo environment I’m not to concerned with performance and scalability. So a standard GCP machine is used with 4 cores and 16GB memory running CentOS Stream 8.

As AWX runs containerized, it can be installed in K8S, Minikube or in K3S. As K3S is the easiest way to go, I have decided to do it like that. Installing AWX in K3s is well described by kurokobo on Github. If these instructions are followed an AWX will be up and running in no-time.
https://github.com/kurokobo/awx-on-k3s

A Git server

I want to run a local Git server, that’s not to heavy and that I can destroy and rebuild at will. As I am a big fan of Forgejo (https://forgejo.org/) that is what I went for.

Download the binary package and install it. I created a separate user for the Git repos and also created a service file. I didn’t bother with SSL and a webserver in front of it, just let Forgejo handle all requests directly.

useradd -c "Git user" -m git
mkdir /data/git
chown git:git /data/git
chmod 700 /data/git
su - git
wget https://codeberg.org/forgejo/forgejo/releases/download/v1.19.3-0/forgejo-1.19.3-0-linux-amd64
chmod 755 forgejo-1.19.3-0-linux-amd64
mkdir run
cd run
cp -p ../forgejo-1.19.3-0-linux-amd64 forgejo
ln -s forgejo git

Configure the Git server to your liking. This is my git.ini

APP_NAME = Git
RUN_USER = git
RUN_MODE = prod

[repository]
ROOT                = /data/git/repos
FORCE_PRIVATE       = false
MAX_CREATION_LIMIT  = 50
MIRROR_QUEUE_LENGTH = 1000
DISABLE_HTTP_GIT    = false

[ui]
SHOW_USER_EMAIL = false
THEMES          = git,arc-green
DEFAULT_THEME   = git

[server]
PROTOCOL         = http
DOMAIN           = <your_domain_name>
ROOT_URL         = http://git.<your_domain_name>:3000
HTTP_ADDR        = 0.0.0.0
HTTP_PORT        = 3000
DISABLE_SSH      = false
START_SSH_SERVER = true
SSH_DOMAIN       = localhost
SSH_LISTEN_HOST  = 0.0.0.0
SSH_PORT         = 2222
SSH_LISTEN_PORT  = %(SSH_PORT)s
OFFLINE_MODE     = true
APP_DATA_PATH    = /data/git/data

[database]
DB_TYPE  = sqlite3
HOST     = 127.0.0.0:3306
NAME     = root
USER     = git
PASSWD   = lel
SSL_MODE = disable
PATH     = /data/git/data/git.db

[indexer]
ISSUE_INDEXER_PATH            = /data/git/indexers/issues.bleve
ISSUE_INDEXER_QUEUE_TYPE      = levelqueue
ISSUE_INDEXER_QUEUE_DIR       = /data/git/indexers/issues.queue
REPO_INDEXER_ENABLED          = False
REPO_INDEXER_PATH             = /data/git/indexers/repos.bleve
REPO_INDEXER_INCLUDE          =
REPO_INDEXER_EXCLUDE          =
REPO_INDEXER_EXCLUDE_VENDORED = True
MAX_FILE_SIZE                 = 1048576

[security]
INSTALL_LOCK        = true
SECRET_KEY          = <very_secret_key>
INTERNAL_TOKEN      = <very_secret_token>
LOGIN_REMEMBER_DAYS = 7
DISABLE_GIT_HOOKS   = true

[service]
DISABLE_REGISTRATION             = true
REQUIRE_SIGNIN_VIEW              = true
ENABLE_CAPTCHA                   = true
CAPTCHA_TYPE                     = image
RECAPTCHA_SECRET                 =
RECAPTCHA_SITEKEY                =
SHOW_REGISTRATION_BUTTON         = False
ALLOW_ONLY_EXTERNAL_REGISTRATION = False
ENABLE_NOTIFY_MAIL               = False
DEFAULT_EMAIL_NOTIFICATIONS      = onmention
AUTO_WATCH_NEW_REPOS             = False
AUTO_WATCH_ON_CHANGES            = True
SHOW_MILESTONES_DASHBOARD_PAGE   = True

[mailer]
ENABLED            = false
HOST               = localhost:25
SKIP_VERIFY        = False
USE_CERTIFICATE    = false
CERT_FILE          = /data/git/custom/mailer/cert.pem
KEY_FILE           = /data/git/custom/mailer/key.pem
IS_TLS_ENABLED     = true
FROM               = noreply@your.domain
USER               =
PASSWD             =
SEND_AS_PLAIN_TEXT = false
MAILER_TYPE        = smtp
SENDMAIL_PATH      = sendmail

[session]
PROVIDER        = file
PROVIDER_CONFIG = /data/git/data/sessions

[picture]
AVATAR_UPLOAD_PATH = /data/git/data/avatars
DISABLE_GRAVATAR   = true

[attachment]
ENABLED = true
PATH    = /data/git/data/attachments

[log]
ROOT_PATH            = /data/git/log
MODE                 = file
BUFFER_LEN           = 10000
LEVEL                = Info
REDIRECT_MACARON_LOG = false

[oauth2]
ENABLE     = True
JWT_SECRET = <very_secret_jwt_secret>

[metrics]
ENABLED = False
TOKEN   =

And, of course, a systemd service file is needed.

cat <<- @EOF > /etc/systemd/system/git.service
    [Unit]
    Description=Git
    After=syslog.target
    After=network.target
    #After=mariadb.service mysqld.service postgresql.service memcached.service redis.service

    [Service]
    Type=simple
    User=git
    Group=git
    WorkingDirectory=/home/git/run
    ExecStart=/home/git/run/git web
    Restart=always
    Environment=USER=git HOME=/home/git PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin

    [Install]
    WantedBy=multi-user.target
@EOF
systemctl daemon-reload
systemctl enable --now git

Managed node

I also need a managed node for every attendee, but there are a couple of things I want:

A single container image, used by all containers
No persistent storage, if I remove the container no residue should be left
A static and predictable IP address per container
systemd and sshd should be running inside the container, so Ansible can be used with SSH

These demands result in a Containerfile that starts sshd through systemd. The /etc/ssh/ssh_host_* key files are generated once and after that copied out of the container. The reason for this is that by default a new installation of SSH does not have this set of keys. These keys are automatically generated at first boot. But with a large amount of containers, all starting at the same time, this could render the system completely useless. Adding the keys to the container at built-time prevents the thundering herd.

#
# Build with:
#     podman build --squash-all --tag=ansinode .
#
FROM almalinux:9
LABEL maintainer="Ton Kersten <tonk@ansiblelab.nl>"

# Ansible home
ARG ansihome="/home/ansible"

# We are in Europe but we use UTC to be global
ENV TZ=UTC

# Ensure needed packages and enable SSH (Needed for Ansible)
RUN    dnf -y --refresh install        \
       epel-release                    \
       systemd                         \
       sudo                            \
       passwd                          \
       iproute                         \
       openssh-server                  \
       openssh-clients               ; \
    dnf clean all                    ; \
    systemctl enable sshd.service    ; \
    echo "root" | passwd --stdin root

# Add the Ansible user
RUN groupadd -g 10000 ansible                              ; \
    useradd -u 10000 -g ansible -m -d $ansihome ansible    ; \
    mkdir -m 0700 $ansihome/.ssh                           ; \
    chown ansible:ansible $ansihome/.ssh                   ; \
    echo "ansible" | passwd --stdin ansible

# Add some needed files and fix rights
ADD podfiles/id_rsa.pub    $ansihome/.ssh/authorized_keys
ADD podfiles/sudoers       /etc/sudoers.d/ansible
RUN chown ansible:ansible $ansihome/.ssh/authorized_keys   ; \
    chmod 640 $ansihome/.ssh/authorized_keys               ; \
    chown root:root /etc/sudoers.d/ansible                 ; \
    chmod 400 /etc/sudoers.d/ansible

# Add the systems host keys
ADD podfiles/ssh_host_ecdsa_key        /etc/ssh/ssh_host_ecdsa_key
ADD podfiles/ssh_host_ecdsa_key.pub    /etc/ssh/ssh_host_ecdsa_key.pub
ADD podfiles/ssh_host_ed25519_key      /etc/ssh/ssh_host_ed25519_key
ADD podfiles/ssh_host_ed25519_key.pub  /etc/ssh/ssh_host_ed25519_key.pub
ADD podfiles/ssh_host_rsa_key          /etc/ssh/ssh_host_rsa_key
ADD podfiles/ssh_host_rsa_key.pub      /etc/ssh/ssh_host_rsa_key.pub
RUN chown root:ssh_keys   /etc/ssh/ssh_host_ecdsa_key        ; \
    chown root:root       /etc/ssh/ssh_host_ecdsa_key.pub    ; \
    chown root:ssh_keys   /etc/ssh/ssh_host_ed25519_key      ; \
    chown root:root       /etc/ssh/ssh_host_ed25519_key.pub  ; \
    chown root:ssh_keys   /etc/ssh/ssh_host_rsa_key          ; \
    chown root:root       /etc/ssh/ssh_host_rsa_key.pub      ; \
    chmod 640             /etc/ssh/ssh_host_ecdsa_key        ; \
    chmod 644             /etc/ssh/ssh_host_ecdsa_key.pub    ; \
    chmod 640             /etc/ssh/ssh_host_ed25519_key      ; \
    chmod 644             /etc/ssh/ssh_host_ed25519_key.pub  ; \
    chmod 640             /etc/ssh/ssh_host_rsa_key          ; \
    chmod 644             /etc/ssh/ssh_host_rsa_key.pub

# Fix PAM for SSH login
RUN sed -i 's/required/optional/g' /etc/pam.d/sshd

# Open SSH port
EXPOSE 22

# And make sure systemd is running
CMD ["/usr/sbin/init"]

The name of the container image is ansinode and this will used to create a managed node node1 per attendee.

Starting the containers

Once the container image has been built, this image is used to start a container per attendee with specific settings per attendee. This immediately calls for a template.

# vi: set sw=4 ts=4 ai:
#
# Ansible information:
#     Filename : {{ template_path | regex_replace("/.*/ansible[^/]*/", ".../") }}
#     Hostname : {{ template_host }}
#
# podman container service file for {{ gnum }}'s node1
#

[Unit]
Description=Ansible Lab Container-{{ gnum }}-node1.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
RequiresMountsFor=/var/lib/containers/storage /run/containers/storage

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
TimeoutStopSec=70
Restart=always
RestartSec=3
ExecStartPre=/bin/rm -f %t/container-{{ gnum }}-node1.pid %t/container-{{ gnum }}-node1.ctr-id
ExecStart=/usr/bin/podman                                \
    run                                                  \
    --conmon-pidfile %t/container-{{ gnum }}-node1.pid   \
    --cidfile %t/container-{{ gnum }}-node1.ctr-id       \
    --cgroups=no-conmon                                  \
     --replace                                           \
    --detach                                             \
    --rm                                                 \
    --tz UTC                                             \
    --systemd true                                       \
    --tmpfs /run                                         \
    --security-opt seccomp=unconfined                    \
    --security-opt label=disable                         \
    --cap-add SYS_RESOURCE                               \
    --cap-add AUDIT_WRITE                                \
    --volume /sys/fs/cgroup:/sys/fs/cgroup:ro            \
    --hostname=node1.example.net                         \
    --ip={{ ip }}                                        \
    --name=guru{{ gnum }}-node1                          \
    ansinode
ExecStop=/usr/bin/podman stop --ignore --cidfile %t/container-{{ gnum }}-node1.ctr-id -t 10
ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile %t/container-{{ gnum }}-node1.ctr-id
PIDFile=%t/container-{{ gnum }}-node1.pid
Type=simple

[Install]
WantedBy=multi-user.target default.target

Automate it

So far so good, now we have everything in place to automate the heck out of it.

The first playbook is the easiest one, it is the playbook that runs the tasklist per guru in a loop.

---
- name: Create Guru environment in AWX
  hosts: localhost
  connection: local
  become: false
  gather_facts: false

  vars:
    gitea_api_url: "http://localhost:3000/api/v1"
    gitea_token: "<my_very_secret_api_token>"
    gurus: 30

  tasks:
    - name: Create Guru admins role {{ gurunum }}
      ansible.builtin.include_tasks: guru.yml
      loop: "{{ range(0, gurus+1) | list }}"
      loop_control:
        loop_var: gurunum

This creates gurus + 1 organizations in AWX, with an organization admin, a project, inventory, templates, credentials, vault password, the whole caboodle.

---
- name: Ensure the Guru number is the correct format
  ansible.builtin.set_fact:
    gnum: "{{ '%03d' | format(gurunum) }}"

- name: Create Git user through API
  ansible.builtin.uri:
    url: '{{ gitea_api_url }}/admin/users'
    method: POST
    return_content: true
    status_code:
      - 201
      - 204
      - 422
    body_format: json
    headers:
      Authorization: token {{ gitea_token }}
    body: |-
      {
        "email": "guru{{ gnum }}@ansiblelab.nl",
        "full_name": "Slartibartfast Magrathean {{ gnum }}",
        "login_name": "guru{{ gnum }}",
        "must_change_password": false,
        "password": "PlanetBuilder{{ gnum }}",
        "restricted": true,
        "send_notify": false,
        "source_id": 0,
        "username": "guru{{ gnum }}",
        "visibility": "private"
      }      

- name: Add the ping repository to Git user through API
  ansible.builtin.uri:
    url: '{{ gitea_api_url }}/repos/templater/template_ping/generate'
    method: POST
    return_content: true
    status_code:
      - 201
      - 204
      - 409
      - 422
    body_format: json
    headers:
      Authorization: token {{ gitea_token }}
    body: |-
      {
        "owner": "guru{{ gnum }}",
        "name": "ping",
        "avatar": true,
        "description": "The Ping Project",
        "default_branch": "main",
        "git_content": true,
        "git_hooks": true,
        "labels": true,
        "topics": true,
        "webhooks": true,
        "private": true
      }      

- name: Create Guru organization {{ gnum }}
  awx.awx.organization:
    name: ORG | Guru{{ gnum }} Universe Building LTD
    description: ORG | Ansible Universe for guru{{ gnum }}
    galaxy_credentials:
      - Ansible Galaxy
    state: present

- name: Create Guru user {{ gnum }}
  awx.awx.user:
    username: guru{{ gnum }}
    password: PlanetBuilder{{ gnum }}
    first_name: Slartibartfast {{ gurunum }}
    last_name: Magrathean {{ gnum }}
    email: guru{{ gnum }}@ansiblelab.nl
    organization: ORG | Guru{{ gnum }} Universe Building LTD

- name: Create Guru admins {{ gnum }}
  awx.awx.team:
    name: TEAM | Planet Building Team {{ gnum }}
    description: TEAM | Ansible Administrator Team for Universe Building {{ gnum }}
    organization: ORG | Guru{{ gnum }} Universe Building LTD
    state: present

- name: Create Guru admins role {{ gnum }}
  awx.awx.role:
    user: guru{{ gnum }}
    role: admin
    target_teams: TEAM | Planet Building Team {{ gnum }}
    organization: ORG | Guru{{ gnum }} Universe Building LTD

- name: Create Guru Host credential {{ gnum }}
  awx.awx.credential:
    name: CRED | Guru{{ gnum }} Machine Credential
    description: CRED | Ansible Machine SSH key Guru{{ gnum }}
    organization: ORG | Guru{{ gnum }} Universe Building LTD
    state: present
    credential_type: Machine
    inputs:
      username: ansible
      ssh_key_data: "{{ lookup('file', 'podfiles/id_rsa') }}"

- name: Create Guru Source Control credential {{ gnum }}
  awx.awx.credential:
    name: CRED | Guru{{ gnum }} Source Control Credential
    description: CRED | Ansible Source Control SSH key Guru{{ gnum }}
    organization: ORG | Guru{{ gnum }} Universe Building LTD
    state: present
    credential_type: Source Control
    inputs:
      username: guru{{ gnum }}
      password: PlanetBuilder{{ gnum }}

- name: Create Guru vault password {{ gnum }}
  awx.awx.credential:
    name: CRED | Guru{{ gnum }} Vault Password
    description: CRED | Ansible Vault password for Guru{{ gnum }}
    organization: ORG | Guru{{ gnum }} Universe Building LTD
    state: present
    credential_type: Vault
    inputs:
      vault_password: VaultPassword

- name: Create inventory for Guru {{ gnum }}
  awx.awx.inventory:
    name: INV | Guru{{ gnum }} Static Inventory
    description: INV | Static Inventory for Managed Hosts for Guru{{ gnum }}
    organization: ORG | Guru{{ gnum }} Universe Building LTD
    state: present

- name: Create node1 {{ gnum }}
  awx.awx.host:
    name: node1
    description: HOST | Guru{{ gnum }} managed host
    inventory: INV | Guru{{ gnum }} Static Inventory
    state: present
    enabled: true
    variables:
      ansible_user: ansible
      ansible_host: 10.88.{{ gurunum }}.10

- name: Create project {{ gnum }}
  awx.awx.project:
    name: PRJ | Guru{{ gnum }} Ping Project
    description: PRJ | Ping project for Guru{{ gnum }}
    organization: ORG | Guru{{ gnum }} Universe Building LTD
    wait: true
    scm_type: git
    scm_url: "http://awx.ansiblelab.nl:3000/guru{{ gnum }}/ping.git"
    scm_clean: true
    scm_update_on_launch: true
    scm_update_cache_timeout: 30
    state: present
    credential: CRED | Guru{{ gnum }} Source Control Credential

- name: Create ping job template {{ gnum }}
  awx.awx.job_template:
    name: TMPL | Guru{{ gnum }} Ping
    description: TMPL | Guru{{ gnum }} Ping Template
    job_type: run
    organization: ORG | Guru{{ gnum }} Universe Building LTD
    project: PRJ | Guru{{ gnum }} Ping Project
    inventory: INV | Guru{{ gnum }} Static Inventory
    playbook: ping.yml
    credentials:
      - CRED | Guru{{ gnum }} Machine Credential
      - CRED | Guru{{ gnum }} Vault Password
    state: present

- name: Create ping job template with survey {{ gnum }}
  awx.awx.job_template:
    name: TMPL | Guru{{ gnum }} Ping Survey
    description: TMPL | Guru{{ gnum }} Ping Template with Survey
    job_type: run
    organization: ORG | Guru{{ gnum }} Universe Building LTD
    project: PRJ | Guru{{ gnum }} Ping Project
    inventory: INV | Guru{{ gnum }} Static Inventory
    playbook: ping.yml
    credentials:
      - CRED | Guru{{ gnum }} Machine Credential
      - CRED | Guru{{ gnum }} Vault Password
    state: present
    survey_enabled: true
    survey_spec: "{{ lookup('template', 'survey.json.j2') }}"

- name: Do things as root
  block:
    - name: Create Guru{{ gnum }} node1 service file
      ansible.builtin.template:
        src: container-node1.service.j2
        dest: /etc/systemd/system/guru{{ gnum }}-node1.service
        owner: root
        group: root
        mode: 0644
      vars:
        ip: 10.88.{{ gurunum }}.10

    - name: Reload systemd Guru{{ gnum }}
      ansible.builtin.systemd:
        daemon_reload: true

    - name: Start node container for Guru{{ gnum }}
      ansible.builtin.service:
        name: guru{{ gnum }}-node1.service
        state: started
        enabled: true
  become: true

ansible automation sysadm

Ansible AWX

A Git server

Managed node

Starting the containers

Automate it

See also