Traveler

Ansible Virtual Environment

Date: March 15, 2020 Author: Brian Hooper

TLDR

A simple approach for setting up Ansible to run in a Python Virtual Environment. Installing Ansible in a Python virtual environment enables us to maintain the Ansible dependencies which are basically python packages, independent of the ones used by the Operating System.

The source code for this post is available via a snippets repo hosted on GitLab.

If you’d like to clone the snippets for this post along with others from my blog:


    git clone git@gitlab.com:KnownTraveler/snippets.git

Configuration

Here are the steps we will be walking through today:

Install pyenv
Install pyenv dependencies
Install Python 3.7.7 using pyenv
Create Ansible project directory
Create virtual environment (venv)
Install Ansible in virtual environment
Run Ansible in virtual environment
Bash Script to run Ansible in Virtual Environment

Install pyenv

pyenv lets you easily switch between multiple versions of Python. Check out pyenv where you want it installed. A good place to choose is $HOME/.pyenv (but you can install it somewhere else).


    # COMMAND
    git clone https://github.com/pyenv/pyenv.git ~/.pyenv

Define environment variable PYENV_ROOT to point to the path where pyenv repo is cloned and add $PYENV_ROOT/bin to your $PATH for access to the pyenv command-line utility.


    # COMMANDS
    echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
    echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc

Add pyenv init to your shell to enable shims and autocompletion. Please make sure eval “$(pyenv init -)” is placed toward the end of the shell configuration file since it manipulates PATH during the initialization.


    # COMMAND
    echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bashrc

What you should now see at the end of the ~/.bashrc file:


    # EXAMPLE
    export PYENV_ROOT="$HOME/.pyenv"
    export PATH="$PYENV_ROOT/bin:$PATH"
    if command -v pyenv 1>/dev/null 2>&1; then
      eval "$(pyenv init -)"
    fi

Restart your shell so the path changes take effect. You can now begin using pyenv.


    # COMMANDS
    exec "$SHELL"
    which pyenv
    /home/<user>/.pyenv/bin/pyenv

Install pyenv System Dependencies

pyenv will try its best to download and compile the wanted Python version, but sometimes compilation fails because of unmet system dependencies, or compilation succeeds but the new Python version exhibits weird failures at runtime. The following instructions are recommendations for a sane Python Build Environment


    # COMMANDS
    sudo apt-get update 
    sudo apt-get install --no-install-recommends make \
    build-essential \
    libssl-dev \
    zlib1g-dev \
    libbz2-dev \
    libreadline-dev \
    libsqlite3-dev \
    wget \
    curl \
    llvm \
    libncurses5-dev \
    xz-utils \
    tk-dev \
    libxml2-dev \
    libxmlsec1-dev \
    libffi-dev \
    liblzma-dev

Install Python 3.7 using pyenv

Using pyenv we will install python 3.7.7


    # COMMANDS w/OUTPUT
    pyenv install 3.7.7

    Downloading Python-3.7.7.tar.xz...
    -> https://www.python.org/ftp/python/3.7.7/Python-3.7.7.tar.xz
    Installing Python-3.7.7...
    Installed Python-3.7.7 to /home/<user>/.pyenv/versions/3.7.7

    which python
    /home/<user>/.pyenv/shims/python
    
    python --version
    pyenv: python: command not found

    The python command exists in these Python versions:
        3.7.7

    Note: See pyenv "help" global for tips on allowing both
    python2 and python3 to be found.

Verify available python versions


    # COMMAND w/OUTPUT
    pyenv versions
    3.7.7

Set Python Global version to 3.7.7


    # COMMANDS w/OUTPUT
    pyenv global 3.7.7
    python --version
    Python 3.7.7
    
    python
    Python 3.7.7 (default, Mar 11 2020, 08:10:34) 
    [GCC 9.3.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>>

Ok, great! Now we have python 3.7.7 setup using pyenv

Create Ansible Project Directory


    # COMMANDS
    mkdir ansible-venv
    cd ansible-venv

Create Python Virtual Environment


    # COMMAND
    python -m virtualenv venv

Install Ansible in Virtual Environment

Ok, next we are going to activate the virtual environment and will install ansible


    # COMMANDS w/OUTPUT
    source ./venv/bin/activate
    (venv)   <--- Notice you are now in the Virtual Environment

    which python
    /home/<user>/Desktop/blog/ansible-venv/venv/bin/python

    which pip
    /home/<user>/Desktop/blog/ansible-venv/venv/bin/pip

    pip install ansible

    which ansible
    /home/<user>/Desktop/blog/ansible-venv/venv/bin/ansible

    ansible --version
    ansible 2.10.4
        config file = /etc/ansible/ansible.cfg
        configured module search path = ['/home/<user>/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
        ansible python module location = /home/<user>/Desktop/blog/ansible-venv/venv/lib/python3.7/site-packages/ansible
        executable location = /home/<user>/Desktop/blog/ansible-venv/venv/bin/ansible
        python version = 3.7.7 (default, Mar 11 2020, 08:10:34) [GCC 9.3.0]

Take a look at venv/lib/python3.7/site-packages/ and you will see where ansible and its dependencies were installed in the virtual environment.

Run Ansible in Virtual Environment

Next, we are going to create a sample playbook in the project directory, ./playbook_sample.yaml


    ---
    # SAMPLE PLAYBOOK
    - name: playbook_sample
      hosts: localhost
      connection: local

      tasks:

        - debug:
            msg: "Default Message Running for Ansible Playbook"

        - name: Custom Linux Task
          shell: |
            echo "Custom Linux Shell Command for Ansible Distribution {{ ansible_distribution }}"            
          when: ansible_distribution == "Ubuntu"

        - name: Custom MacOS Task
          shell: |
            echo "Custom MacOS Shell Command for Ansible Distribution {{ ansible_distribution }}"            
          when: ansible_distribution == "Darwin"

Now that we have a sample playbook, while we are working in the activated virtual environment (venv) lets use ansible to run our new playbook


    # COMMAND
    ansible-playbook -i "localhost," -c local playbook_sample.yaml -v

When running the above command, you may notice a WARNING:


    # OUTPUT
    TASK [Gathering Facts] **********************************
    [WARNING]: Unhandled error in Python interpreter discovery for host localhost: '<' not supported between instances of 'str' and 'int'
    [WARNING]: Platform linux on host localhost is using the discovered Python interpreter at .../ansible-venv/venv/bin/python3.7, \
    but future installation of another Python interpreter could change the meaning of that path. \
    See https://docs.ansible.com/ansible/2.10/reference_appendices/interpreter_discovery.html for more information.

Ansible will automatically detect and use Python 3 on many platforms that ship with it. However, because we are using a Python Virtual Environment (venv) we can explicitly configure a Python 3 interpreter by setting the ansible_python_interpreter passing in the python path for the venv/bin/python


    # COMMAND
    ansible-playbook -i "localhost," -c local playbook_sample.yaml -e "ansible_python_interpreter=$HOME/Desktop/ansible-venv/venv/bin/python" 

    # OUTPUT
    PLAY [playbook_sample] **********************************

    TASK [Gathering Facts] **********************************
    ok: [localhost]

    TASK [debug] ********************************************
    ok: [localhost] => {
        "msg": "Default Message Running for Ansible Playbook"
    }

    TASK [Custom Linux Task] ********************************
    changed: [localhost]

    TASK [Custom MacOS Task] ********************************
    skipping: [localhost]

    PLAY RECAP **********************************************
    localhost                  
    : ok=3    changed=1    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

Prior to deactivating the virtual environment, if you echo $PATH you’ll notice the venv/bin path ansible-venv/venv/bin is present.

Deactivate the Virtual Environment


    # COMMAND
    deactivate

After deactivating the virtual environment, if you echo $PATH you’ll notice the venv/bin path ansible-venv/venv/bin is removed.

Bash Script

Ok, let’s get this scripted. We’ll create a couple of bash scripts, first let’s create a new scripts directory


    # COMMAND
    mkdir ./scripts

Next, let’s create ./scripts/_lib.sh


    function banner(){
        message=$1
        echo ""
        echo "========================================"
        echo "  $message"
        echo "========================================"
        echo ""
    }

    function header(){
        message=$1
        echo ""
        echo "  $message"
        echo "----------------------------------------"
        echo ""
    }

Create /ansible.sh

Ok, next let’s create our ansible.sh script the root of the project directory:


    #!/usr/bin/bash

    # SOURCE LIBRARY FUNCTIONS
    . ./scripts/_lib.sh

    banner "Ansible Bootstrap Script"

    # ACTIVATE VIRTUAL ENVIRONMENT
    if [ ! -f "./venv/bin/activate" ]
    then
        header "Creating Virtual Environment"
        python -m virtualenv venv

        header "Activating Virtual Environment"
        source ./venv/bin/activate
    else 
        header "Activating Virtual Environment"
        source ./venv/bin/activate
    fi

    # INSTALL REQUIREMENTS
    header "Installing Requirements"
    pip install -r requirements.txt

    # RUN ANSIBLE
    header "Running Ansible"
    ansible-playbook -i "localhost," -c local playbook_bootstrap.yaml -e "ansible_python_interpreter=$HOME/Desktop/ansible-venv/venv/bin/python"

    # DEACTIVATE VIRTUAL ENVIRONMENT
    header "Deactivating Virtual Environment"
    deactivate

Let’s update the permissions for ansible.sh


    # COMMAND
    chmod 0755 ./ansible.sh

Create /requirements.txt

Lastly, let’s create a requirements.txt and list ansible as a requirement


    ansible

Project Structure

Our project structure should look like the following:


    /ansible-venv               <-- Project Directory
    |
    |-- /scripts                <-- Scripts Sub-Directory
    |   |-- _lib.sh
    |
    |-- /venv                   <-- Virtual Environment
    |
    |-- ansible.sh              <-- Bash Script to Run Ansible in VENV
    |-- playbook_sample.sh      <-- Sample Playbook
    |-- requirements.txt        <-- Python VENV Requirements

Running the ansible.sh script


    # COMMAND
    ./ansible.sh

    # OUTPUT
    ========================================
    Ansible Bootstrap Script
    ========================================

    Activating Virtual Environment
    ----------------------------------------

    Installing Requirements
    ----------------------------------------

    ...
    < pip install output here >
    ...

    Running Ansible
    ----------------------------------------

    ...
    < ansible output here >
    ---

    Deactivating Virtual Environment
    ----------------------------------------

There you go, by installing Ansible in a Python virtual environment enables us to maintain the Ansible dependencies which are basically python packages, independent of the ones used by the Operating System. This is helpful if you need to support multiple projects with various versions of Python.

Resources

Updates

2020-12-12

Updated to use latest version release of Ansible 2.10.4

2020-05-01

In an upcoming post I will be using the above approach to create a bootstrapping process for a new workspace using Ubuntu 20.04 Focal Fossa that was recently released. More to come!