Building VM Inventory #2: Ansible

~/blog$ render faiz.blog.building-vm-inventory-#2:-ansible

Building VM Inventory #2: Ansible

Sunday, 29 September 2024 17:02:12 WIB | tags: tips, ansible, vultr, vm-inventory | 219 hits | 1 comment(s)

Hey there! Let's dive back into the Building VM Inventory series. In this episode, we'll use Ansible to gather VM inventory. Just like the last time, we'll also do a quick compliance check. But this time, instead of using GCP, we'll try out Vultr, a great option for smaller projects. Let's see how it goes!

Project Requirements

We’ll use similar requirements for our VM inventory problem:

The solution should be able to do all requirements remotely.
The solution should be able to gather hostname.
The solution should be able to gather OS name and version.
The solution should be able to do simple compliance check.

Define Ubuntu 24 as comply, where other OS do not comply.

Vultr Environment

In this section, I’ll prepare a VPC network and deploy 3 VMs that consists of 1 remote manager and 2 remote clients in Vultr. You can skip this section if you already have your own environment and want to use Ansible directly. To start, you can register yourself on the Vultr website.

VPC Network

Once you have registered to Vultr, you can directly create a new VPC network by accessing the sidebar: Products > Network > VPC Networks.

On the VPC Network menu, you’ll find + Add VPC Network button on the top right. On my Vultr account, I already have a VPC, but if your account is newly created, there might be a slight differences on the display. In this exercise, I use Singapore region but you’re free to choose any location you like. For the IP range, I chose Auto-Assign IP Range for the subnet range, this is the private subnet for the VPC Network. I name this VPC Network as rahiemy-vm-inventory. Once created, the VPC will appear on your Virtual Private Clouds menu.

Now that we have the VPC network, let’s continue to the next step which is deploying VMs.

Deploy Main VM

Same like the first episode, I’ll deploy 1 main VM and 2 client VMs. For the main VM, the specifications are like the listed below:

remote-server

Type: Cloud Compute - Shared CPU
Machine: Regular Cloud Compute
Image: Ubuntu 22.04 LTS
Specification: 1vCPU, 25 GB SSD, 1GB memory, 1 TB bandwidth
VPC: rahiemy-vm-intenvory
Firewall group: rahiemy-vm-inventory-server

To deploy the VM, you can easily access Deploy + button on the top right menu and click on Deploy New Server, which will redirect you to the Deploy New Instance menu. In this menu, again, you can select the region where you want the VM to reside, the image, machine type, and storage size. Furthermore, in Vultr there is also additional features. By default, Auto Backups and IPv6 is enabled, but I disabled these features on this exercise.

Also, unlike GCP, in Vultr, VM is not assigned to VPC by default and you need to explicitly chose the option. I also checked Limited User Login feature to generate a random password for non-default user (linuxuser). Ideally, this user should also be disabled once the VM created and we should use SSH key instead, but this is based on your risk-appetite, since we’ll whitelist our IP as well.

On the bottom, there is another two sections: Server Settings and Server Hostname & Label. The earlier one is to assign SSH Keys for the created user and to assign the VM to a pre-created Firewall Group. Since we don’t have any Firewall Group for now, let’s keep it as is. While for the hostname field, let’s put remote-server to identify the VM. Then, click on the Deploy Now button. You’ll be redirected to Cloud Compute menu and see your newly created VM with status of Installing.

Create Main Firewall Group

By default, if the VM is not assigned to any firewall group, it will accept any traffic from anywhere. To implement security control, we should create a firewall group and define from which source IP and source port only the traffic will be accepted. For the beginning, let’s create a firewall group for the main VM:

Source IP: My IP - Vultr will automatically check your current public IP and put on the IP field
Source Port: 22

To add the rule, you need to click on + button on the right hand side of the row. By adding this firewall group, only port 22 from my current IP is allowed to access the VM, and as you can see, Vultr will create an implicit deny if we have at least 1 active inbound rule. After we create the rule, we need to link our instance to this firewall group. To do this, click on the Linked Instance sidebar and select the remote-server instance. Click on the + button on the right side to apply the changes.

Create Client Firewall Group

Now, before we create the two remaining VMs let’s prepare the firewall group so only our main VM that can access these VMs. To create a new firewall group, the steps are similar with before, the only difference is the whitelisted IP address. We need to put remote-server’s private IP and to access this, you need to access remote-server’s Settings menu under Cloud Compute. You’ll find the IP address under VPC network table. Now, add this IP address to the firewall group. I name this firewall group as rahiemy-vm-inventory-client.

Accessing the Main VM

To access the VM, you can directly access the command below. For the public IP, you can get it from remote-server’s instance detail in Vultr Cloud Compute menu. While for the ssh key path, you can set it to your private key. By default, it is generated under ~/.ssh/ directory. You can remove -i [ssh key path] command if you didn’t create the SSH key and use the generated password instead (provided in Vultr menu).

ssh linuxuser@[public ip address] -i [ssh key path]

* Documentation:  https://help.ubuntu.com
* Management:     https://landscape.canonical.com
* Support:        https://ubuntu.com/pro

System information as of Sun Sep 29 06:50:25 AM UTC 2024

    System load:  0.0                Processes:               122
    Usage of /:   28.5% of 22.88GB   Users logged in:         1
    Memory usage: 24%                IPv4 address for enp1s0: 000.000.000.000
    Swap usage:   0%


Expanded Security Maintenance for Applications is not enabled.

0 updates can be applied immediately.

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status


The list of available updates is more than a week old.
To check for new updates run: sudo apt update
New release '24.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


Last login: Sun Sep 29 06:45:42 2024 from 000.000.000.000
~

Generate Key to Access Clients

As we have already accessed the main VM, our next step is to generate an SSH key pair to access the VM. Not having a password login method and using SSH key instead is technically more secure.

ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:xxx root@remote-server
The key's randomart image is:
+---[RSA 3072]----+
|                 |
|      xxx        |
|                 |
+----[SHA256]-----+

Once the key pair is ready, you can add new SSH Key from VM creation mode. For now, just keep the public key of the SSH key and we’ll get back to this during client VM creation.

Deploy Client VMs

Now the remaining components of the infrastructure: client VMs. The specifications are as defined below:

remote-client-1

Type: Cloud Compute - Shared CPU
Machine: Regular Cloud Compute
Image: Ubuntu 22.04 LTS
Specification: 1vCPU, 25 GB SSD, 1GB memory, 1 TB bandwidth
VPC: rahiemy-vm-intenvory
Firewall group: rahiemy-vm-inventory-client

remote-client-2

Type: Cloud Compute - Shared CPU
Machine: Regular Cloud Compute
Image: Ubuntu 24.04 LTS
Specification: 1vCPU, 25 GB SSD, 1GB memory, 1 TB bandwidth
VPC: rahiemy-vm-intenvory
Firewall group: rahiemy-vm-inventory-client

Several first steps of the client VMs creation are similar with the main VM creation. After chosing the VPC, on the next section you can add new SSH Key, which is generated from the previous step. Once the key is added, select it and don’t forget to select the right firewall group as well. After put the hostname, you can click on Deploy Now button. Repeat the similar step for the second remote client, but use Ubuntu 24.04 LTS image for the second one. Once this step is done, you should have three VMs on your Cloud Compute menu.

To verify the access, let’s try ssh remote-client-1 from remote-server:

[email protected] -i ~/.ssh/id_rsa


    The authenticity of host '10.40.112.4 (10.40.112.4)' can't be established.
    ED25519 key fingerprint is SHA256:ADyoQDnY5gPqq5/9KDNqMHJT3ugo4hGwlUyUyt/pdXw.
    This key is not known by any other names
    Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
    Warning: Permanently added '10.40.112.4' (ED25519) to the list of known hosts.
    Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-119-generic x86_64)
    
     * Documentation:  https://help.ubuntu.com
     * Management:     https://landscape.canonical.com
     * Support:        https://ubuntu.com/pro
    
     System information as of Sun Sep 29 07:38:57 AM UTC 2024
    
      System load:  0.0                Processes:               127
      Usage of /:   28.0% of 22.88GB   Users logged in:         0
      Memory usage: 22%                IPv4 address for enp1s0: 000.000.000.000
      Swap usage:   0%
    
    
    Expanded Security Maintenance for Applications is not enabled.
    
    0 updates can be applied immediately.
    
    Enable ESM Apps to receive additional future security updates.
    See https://ubuntu.com/esm or run: sudo pro status
    
    
    The list of available updates is more than a week old.
    To check for new updates run: sudo apt update
    
    
    The programs included with the Ubuntu system are free software;
    the exact distribution terms for each program are described in the
    individual files in /usr/share/doc/*/copyright.
    
    Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
    applicable law.    
    ~

Now that’s settled, let’s explore the Ansible.

Ansible

Ansible is an automation tools that has multiple built in plugins & modules as well as community developed to run an automation task. You can use the Ansible by directly using command line or create a playboook consisting of multiple tasks. In the exercise, I created a playbook to tackle the defined requirements.

Install Ansible on the Main VM

Ansible is a Python based library and you can install it by using pip command:

pip install ansible
Collecting ansible
  Downloading ansible-10.4.0-py3-none-any.whl (48.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 MB 20.2 MB/s eta 0:00:00
Collecting ansible-core~=2.17.4
  Downloading ansible_core-2.17.4-py3-none-any.whl (2.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 22.0 MB/s eta 0:00:00
Collecting packaging
  Downloading packaging-24.1-py3-none-any.whl (53 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.0/54.0 KB 11.1 MB/s eta 0:00:00
Requirement already satisfied: jinja2>=3.0.0 in /usr/lib/python3/dist-packages (from ansible-core~=2.17.4->ansible) (3.0.3)
Requirement already satisfied: cryptography in /usr/lib/python3/dist-packages (from ansible-core~=2.17.4->ansible) (3.4.8)
Requirement already satisfied: PyYAML>=5.1 in /usr/lib/python3/dist-packages (from ansible-core~=2.17.4->ansible) (5.4.1)
Collecting resolvelib<1.1.0,>=0.5.3
  Downloading resolvelib-1.0.1-py2.py3-none-any.whl (17 kB)
Installing collected packages: resolvelib, packaging, ansible-core, ansible
Successfully installed ansible-10.4.0 ansible-core-2.17.4 packaging-24.1 resolvelib-1.0.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

Since we will also gathering VM’s private IP address, additional Python library is required: netaddr. Once all of the requirements are installed. Let’s prepare the playbook.

pip install netaddr
    Collecting netaddr
    Downloading netaddr-1.3.0-py3-none-any.whl (2.3 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 26.1 MB/s eta 0:00:00
  Installing collected packages: netaddr
  Successfully installed netaddr-1.3.0
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

Ansible Playbook

For this exercise, I’ve prepared a simple playbook with the structure as below:

    ansible-gather-vm/
    ├── tasks/
    |   └── main.yml
    ├── hosts
    └── site.yml

site.yml

This is the root file of the Playbook. In this file inventory, credential, and roles are determined. By putting “./” on the roles, it will run tasks/main.yml file. As you see, on remote_user I use linuxuser as it is the default Vultr user, and I also use ansible_ssh_private_key_file to let the Ansible use the private key when accessing the remote clients. On the hosts, :localhost added implicit localhost to the inventory so it will do the tasks on itself as well.

---
# This playbook contains steps to gather OS name and version and verify whether it is comply with the requirements or not

- name: Run the Main Tasks
  hosts: all:localhost
  remote_user: linuxuser
  vars:
    ansible_ssh_private_key_file: "/root/.ssh/id_rsa"
  roles:
    - role: "./"

hosts

This is an inventory file to determine the VM to be gathered.

[all]
put remote IP address here

tasks/main.yml

This is the actual tasks executed by the playbook. In this file, the playbook gathers facts and saves it to a variable. And in the end, it will print the output.

---
# This playbook contains steps to gather OS name and version and verify whether it is comply with the requirements or not

- name: Get Hostname
  set_fact:
    hostname: "{{ ansible_hostname  }}"

- name: Get OS Name
  set_fact:
    os_distribution: "{{ ansible_distribution }}"

- name: Get OS Major Version
  set_fact:
    os_major_version: "{{ ansible_distribution_major_version }}"

- name: Get OS Major Version
  set_fact:
    os_version: "{{ ansible_distribution_version }}"

- name: Set not comply by default
  set_fact:
    is_comply: false

- name: Check if comply with OS Version
  set_fact:
    is_comply: true
  when: ansible_distribution|lower == "ubuntu" and "24" in ansible_distribution_major_version

- debug:
    msg:
    - "hostname: {{ hostname }}"
    - "ip address: {{ ansible_all_ipv4_addresses | ipaddr('10.0.0.0/8') | first }}"
    - "os name: {{ os_distribution }}"
    - "os version: {{ os_version }}"
    - "os comply: {{ is_comply }}"

Running the Playbook

Now the main course of this episode, let’s run the playbook! The playbook can be run by executing the command below:

ansible-playbook -i hosts site.yml

The -i flag determines inventory file, while the site.yml determines the playbook that we want to run. If you’ve never accessed the remote clients from the main VM, you may encounter interactive stdin like the below, you can just put yes and it won’t interrupting the playbook on the next run.

TASK [Gathering Facts] *************************************************************************************************
    ED25519 key fingerprint is SHA256:digest.
    This key is not known by any other names
    Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

Now, let’s see the actual result of the playbook below.

ansible-playbook -i hosts site.yml
    PLAY [Run the Main Tasks] **********************************************************************************************
    
    TASK [Gathering Facts] *************************************************************************************************
    ok: [localhost]
    [WARNING]: Platform linux on host 10.40.112.4 is using the discovered Python interpreter at /usr/bin/python3.10, but
    future installation of another Python interpreter could change the meaning of that path. See
    https://docs.ansible.com/ansible-core/2.17/reference_appendices/interpreter_discovery.html for more information.
    ok: [10.40.112.4]
    [WARNING]: Platform linux on host 10.40.112.5 is using the discovered Python interpreter at /usr/bin/python3.12, but
    future installation of another Python interpreter could change the meaning of that path. See
    https://docs.ansible.com/ansible-core/2.17/reference_appendices/interpreter_discovery.html for more information.
    ok: [10.40.112.5]
    
    TASK [./ : Get Hostname] ***********************************************************************************************
    ok: [10.40.112.4]
    ok: [10.40.112.5]
    ok: [localhost]
    
    TASK [./ : Get OS Name] ************************************************************************************************
    ok: [10.40.112.4]
    ok: [10.40.112.5]
    ok: [localhost]
    
    TASK [./ : Get OS Major Version] ***************************************************************************************
    ok: [10.40.112.4]
    ok: [10.40.112.5]
    ok: [localhost]
    
    TASK [./ : Get OS Major Version] ***************************************************************************************
    ok: [10.40.112.4]
    ok: [10.40.112.5]
    ok: [localhost]
    
    TASK [./ : Set not comply by default] **********************************************************************************
    ok: [10.40.112.4]
    ok: [10.40.112.5]
    ok: [localhost]
    
    TASK [./ : Check if comply with OS Version] ****************************************************************************
    skipping: [10.40.112.4]
    ok: [10.40.112.5]
    skipping: [localhost]
    
    TASK [./ : debug] ******************************************************************************************************
    ok: [10.40.112.4] => {
        "msg": [
            "hostname: remote-client-1",
            "ip address: 10.40.112.4",
            "os name: Ubuntu",
            "os version: 22.04",
            "os comply: False"
        ]
    }
    ok: [10.40.112.5] => {
        "msg": [
            "hostname: remove-client-2",
            "ip address: 10.40.112.5",
            "os name: Ubuntu",
            "os version: 24.04",
            "os comply: True"
        ]
    }
    ok: [localhost] => {
        "msg": [
            "hostname: remote-server",
            "ip address: 10.40.112.3",
            "os name: Ubuntu",
            "os version: 22.04",
            "os comply: False"
        ]
    }
    
    PLAY RECAP *************************************************************************************************************
    10.40.112.4                : ok=7    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
    10.40.112.5                : ok=8    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
    localhost                  : ok=7    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

The playbook generates output as we expected. It returns all required information:

VM hostname
VM private IP address
VM OS name
VM OS version
VM compliance status

And it also checks the compliance status in the right manner, where it only returns True on Ubuntu 24.04 VM.

Summary

Ansible can answer all defined requirements to gather VM information remotely.
Ansible has a lot of built-in plugins & modules, it also has community-developed plugins & modules that available by default
Writing the task only requires simple scripting skill as it is on YAML format, with rich official documentation provided
There is a way to not use uniform credential, however it is not demonstrated in this exercise.
Client VMs should be accessible from main VM.