Identifying orphoned virtual machines and hard disks in vSphere

Connect to the shell of the ESXi host system and browse to location of the datastore (/vmfs/volumes/{datastore name}) to which you want to identity orphoned virtual machine hard disks and invoke the following command. This will return all virtual machine hard disks to which the modified date is seven days in the past (-mtime +7) and return the filename pattern match (find -name “*flat.vmdk*) with the modified date timestamp (-exec stat -c “%n %y” {} \;).

find -iname "*flat.vmdk" -mtime +7 -exec stat -c "%n %y" {} \;

This approach assumes that if the virtual machine hard disk has not been touched in the specified timespan the virtual machine hard disk is orphoned. This may not be accurate, as the virtual machine may be in a powered off state for a period that exceeds the timespan and the virtual machine may be valid. The output should be similar to the below:

./vmfs/volumes/424b26ec-4294ea41/ubuntu-01/ubuntu-01-flat.vmdk 2017-01-29 10:52:03.000000000
./vmfs/volumes/424b26ec-4294ea41/ubuntu-02/ubuntu-02-flat.vmdk 2017-01-29 10:17:46.000000000
./vmfs/volumes/424b26ec-4294ea41/ubuntu03/ubuntu03-flat.vmdk 2017-01-29 11:04:18.000000000
./vmfs/volumes/424b26ec-4294ea41/ubuntu04/ubuntu04-flat.vmdk 2017-01-29 10:17:49.000000000
./vmfs/volumes/424b26ec-4294ea41/ubuntu05/ubuntu05-flat.vmdk 2017-01-29 10:52:03.000000000

An alternative method, is using RVTools which can identify orphoned virtual machines and hard disks (categorised as ‘zombie’) from the inventory scan of a vCenter Server System or ESXi Host System. On completion of the inventory scan, the vHealth tab will display any virtual machines and hard disks identified and may be exported for reference.

Creating new address in a subnet using {php}IPAM REST API

In previous posts I have discussed phpIPAM which is an open-source IP address management application (IPAM). In this blog, I will discuss interacting with the REST API (version 2.0) to programatically create new IP addresses for a specific subnet.

For the purpose of this article, I will be using the Postman REST client to submit requests and interact with the API and the following variables reference values you will need to replace for your environment if you are replicating the process:

Variable Description Example
{appId} API application identifier dean
{subnet} Subnet in CIDR format 10.10.1.0/24
{subnetId} Id of subnet address 3

In order to use the API you will need to authenticate (required for none and SSL) with the username and password and use the HTTP authentication header.

If authentication is successful, you will receive the token to include in the token header for subsequent requests to the API. Also, you will receive the expiry of the token which is set to six hours by default. For each subsequent request to the API the expiration time for the token is reset.

POST api/{appId}/user/

By specifying the authorization type as ‘Basic Authentication’ and specifying a username and password this will generate the required Authorization header for the request by encoding the values to a base64 encoded string.

postman-phpipam-post-user

A status code of ‘200’ will be returned if the authentication attempt is successful, the required token value will also be returned in the response body with the expiration date.

{
  "code": 200,
  "success": true,
  "data": {
    "token": ".Cy9kljLjG=WIt9i.gmA%tUY",
    "expires": "2016-12-03 13:13:06"
  },
  "time": 5.036
}

Once the token has been received the cool stuff can now begin and we can start to interact with the API. In the next example, I am going to use a common use case for phpIPAM. I need to request the next available free IP address for a specified subnet and allocate this to a host.

So, lets look at the method we need to use to create a new address from a specified subnet from the addresses section of the documentation

POST  /api/{appId}/addresses/first_free/{subnetId}/

An alternative GET method exists, however whilst this will return the next available IP address it will not create the object for the new IP address in the database.

GET /api/{appId}/addresses/first_free/subnet

From the POST method we need to provide the Id of the subnet we require to create an IP address, therefore prior to invoking the request we need to return this value, by referencing the subnets section of the documentation, the following can be used by providing the subnet in CIDR format.

GET /api/{appId}/subnets/cidr/{subnet}/

In this example I am searching for the subnet containing the CIDR format 10.10.1.0/24. From the below response body we can retrieve the subnet Id from the data[0].id element and in this example, the subnet Id for the subnet 10.10.1.0/24 is ‘3’. As you can see from the headers section, I am now specifying the ‘token’ header previously returned from the successful authorization.

postman-phpipam-post-freeip

{
  "code": 200,
  "success": true,
  "data": [
    {
      "id": "3",
      "subnet": "10.10.1.0",
      "mask": "24",
      "sectionId": "1",
      "description": "Customer 1",
      "linked_subnet": null,
      "firewallAddressObject": null,
      "vrfId": "0",
      "masterSubnetId": "2",
      "allowRequests": "1",
      "vlanId": "0",
      "showName": "1",
      "device": "0",
      "permissions": "{\"3\":\"1\",\"2\":\"2\"}",
      "pingSubnet": "0",
      "discoverSubnet": "0",
      "DNSrecursive": "0",
      "DNSrecords": "0",
      "nameserverId": "0",
      "scanAgent": null,
      "isFolder": "0",
      "isFull": "0",
      "tag": "2",
      "threshold": "0",
      "location": null,
      "editDate": null,
      "links": [
        {
          "rel": "self",
          "href": "/api/dean/subnets/3/"
        }
      ]
    }
  ],
  "time": 0.002
}

Now, lets return to creating a new address in the specified subnet with the below request to create a new address in the subnet from the POST method. From the response body, the data element contains the IP address ‘10.10.1.26’ created for the subnet.

POST  /api/{appId}/addresses/first_free/3/

phpipam-post-first_free

{
  "code": 201,
  "success": true,
  "message": "Address created",
  "id": "34",
  "data": "10.10.1.26",
  "time": 0.015
}

We can also include a request body in the POST method to include information for creating the new IP address in the subnet. The supported parameters are listed for the address controller in the addresses section. In this example, I have included the hostname, description and owner values for the request.

phpipam-post-first_free-body

As you can see if you have read the provided API documentation, I am only touching the surface of what is possible by programmatically interacting with the API and how you can leverage this in your automation and orchestration practices.

Getting started with DevStack as OpenStack playground environment on Ubuntu

I was recently looking to deploy a minimal installation of OpenStack to use as a development environment and to minimise the complexity of installing a complete OpenStack environment where I can build and tear down the environment with minimal effort. This is where DevStack comes into play, as to quote:

DevStack is a series of extensible scripts used to quickly bring up a complete OpenStack environment based on the latest versions of everything from git master. It is used interactively as a development environment and as the basis for much of the OpenStack project’s functional testing.

For my minimal installation of DevStack I am using new installation of Ubuntu Server 16.10 (clean installation and isolated instance recommended due to significant changes to the system). In terms of hardware configuration, the following recommendations are provided. However, this will all be dependant on your use case for DevStack and what you require to achieve and DevStack may also be installed without satisfying the below criteria but you may experience performance/stability issues.

  • Processor – at least 2 cores
  • Memory – at least 8GB
  • Hard Drive – at least 60GB

First of all we will create a user account named ‘stack’ to install DevStack and grant the user sudo privileges with the no password parameter. It is important in this step that we do not complete the installation as the root user.

sudo adduser stack 
echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

Now log out as the current user and reconnect as the ‘stack’ user created in the previous step and clone the repository to the current directory, in this example i am changing the directory to ‘/var’ to create my local copy.

cd /var 
sudo git clone git://github.com/openstack-dev/devstack.git

We will now need to create a configuration file (/var/local.conf) to specify user configuration variables to use when the installation script (/var/devstack/stack.sh) is executed in a subsequent step. The installation will complete without

The below example, contains only password variables so that you are not required to input the values at installation. For a more detailed configuration file containing additional parameters, check out the sample from the respoistory.

cd /var/devstack
sudo vi local.conf 
[[local|localrc]]
ADMIN_PASSWORD=openstack
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD

Once the configuration file has been created, we can now execute the installation script (stack.sh). The installation process will take approximately 15-20 minutes to complete to which will receive console output. Once the installation has completed you will receive an installation summary, URLs, accounts and passwords.

./stack.sh

Once the installation has completed successfully you should be able to browse to the Horizon dashboard and authenticate with the admin credentials configured during installation, at http://{ip address}/dashboard.

horizon-login

If you need to remove the installation of DevStack there is a script included in the repository (./clean.sh) which will remove the installation of DevStack and dependancies and then cleanup the directories touched by the installation. For a detailed list of impacted files and directories during the installation refer to this link.

cd /var/devstack
./clean.sh
rm -rf /opt/stack
rm -rf /usr/local/bin

From my initial attempt at installation, I encountered a number of issues which appear to be permission related I believe this was due to not cloning the repository as the ‘stack’ user account used for the installation. In this case, to resolve you could run the below or alternatively clean-up your installation process and repeat the installation.

sudo chown -R stack:stack /var/devstack 
sudo chmod 770 /var/devstack

Regular Expression Cheat Sheet

Below is a quick reference guide for regular expressions that I frequently use and find useful.

Online RegEx debugger and library – https://regex101.com/

Windows PowerShell cmdlets to secure PSCredential Objects

I have previously discussed securing credentials using Windows Powershell atPowershell: Securing credentials using the PSCredential class. In this article, I will discuss a number of cmdlets I have created to secure credentials using a Advanced Encryption Standard (AES) encryption key to retrieve the content from a encrypted standard string.

As I am using an encryption key and storing the information in a content file, I will be using ACLs on the NTFS filesytem to control access. Alternative methods could be to store the encryption key in a database or use a certificate to control access to the item. Also, in practice I will store the encryption key on a remote file server.

Firstly, we need to create the encryption key using the ‘New-EncryptionKey’ cmdlet to which we use the RNGCryptoServiceProvider class to generate a random byte array for the encryption length. By default, the cmdlet using a 32-byte array to support the AES 256-bit encryption length. The cmdlet also supports using 128-bit and 192-bit encryption lengths which a 16-byte and 24-byte array. The random byte array for the specified encryption length is generated and sends the output to a file which will be the encryption key content.

Once the content has sent to the output file, the content of random byte array is then removed from the current session.

# Creates an AES 256-bit encryption key at the location D:\Output\Keys\mykey.key
New-EncryptionKey -Output D:\Output\Keys\mykey.key 

# Creates an AES 192-bit encryption key at the location D:\Output\Keys\mykey.key
New-EncryptionKey -Bytes 24 -Output D:\Output\Keys\mykey.key

We have now created an encryption key so that we may now convert the secure string for a credential object password using the specified encryption key and sends the output to a password file using the ‘New-EncryptedString’ cmdlet. The cmdlet will retrieve the content of the specified file containing the encryption key and from the stored credential objects convert the secure string of the password to an encrypted standard string and send the output to a file and clear the content of the stored encryption key from the current session.

New-EncryptedString -KeyFile D:\Output\Keys\mykey.key -PasswordFile D:\Output\Passwords\mypassword.txt 

Finally we want to retrieve the credential object to use as a variable to pass to a subsequent cmdlet which will require authentication. The content of the password file is retrieved and converted to a secure string using the content of the encryption key and stored as a password variable and passed to the PSCredential class to represent a set of security credentials and return the object. For subsequent cmdlets I can use the ‘$Password.GetNetworkCredential().Password’ property value for authentication from the PSCredential object.

$Password = Get-PSCredentialObject -Username administrator -KeyFile D:\Output\Keys\mykey.key -PasswordFile D:\Output\Passwords\mypassword.txt 

The cmdlets are available from the below:

New-EncryptionKey – https://github.com/dean1609/PowerShell/blob/master/Functions/New-EncryptionKey.ps1
New-EncryptedString – https://github.com/dean1609/PowerShell/blob/master/Functions/New-EncryptedString.ps1
Get-PSCredentialObject – https://github.com/dean1609/PowerShell/blob/master/Functions/Get-PSCredentialObject.ps1

Creating JetBrains YouTrack issues with Windows PowerShell

YouTrack is a proprietary, commercial browser-based bug tracker, issue tracking system and project management software developed by JetBrains. There is also a REST API provided which allows for various actions to be performed programmatically. In this article I will describe the cmdlets I have created to create issues for projects which leverage the Invoke-WebRequest cmdlet to interact with the REST API using Windows PowerShell.

The cmdlets require a minimum of Windows PowerShell 3.0 and uses the response object for HTML content without Document Object Model (DOM) parsing, this is required when Internet Explorer is not installed on the local instance invoking the cmdlets. An example of a use case for the cmdlets, is to be configured as an event handler from an infrastructure monitoring solution which will create issues for alarms raised.

The REST API is enabled by default and you can confirm connection and access permissions by browsing to ‘http://{baseURL}/rest/admin/project’ which should return an XML file with a list of all the projects. A more detailed description of the REST API can be found at https://confluence.jetbrains.com/display/YTD65/YouTrack+REST+API+Reference.

In this example, we are using cookie-based authorization. However this process can be adapted to also used Hub OAuth 2.0 authentication as well, which is described at https://www.jetbrains.com/hub/help/1.0/OAuth-2.0-Authorization.html.

Firstly, we need to establish a connection to the REST API using specified credentials and store the web request session object using the session variable. This will allow for cookie information to be re-used for use in subsequent web requests. The specified parameters of login and password are required to be used in the POST method as below:

POST /rest/user/login?{login}&{password}

The cmdlet Connect-YouTrack establishes a connection by specifying the YouTrack Uri, Username and Password parameters and returns the web request session object, which in this example I am storing in the Connection variable.

$Connection = Connect-YouTrack -YouTrackUri http://server1 -Username administrator -Password P@55Word! 

Once a connection has been established and the web session object has been returned, we can now create an issue in a project and specify a summary and description. The cmdlet ‘New-YouTrackItem’ will invoke a PUT request to the REST API with the specified parameters of project, summary and description as below.

PUT /rest/issue?{project}&{summary}&{description}

Once the item has been created, the URI of the new issue is retrieved from the response header and using a regular expression pattern match we retrieve the item number so that we can use a subsequent web request to update the item with additional information as a variable, in this example ‘NewIssue’. The Connection variable returned when establishing a connection to the REST API is passed to the session variable for authentication.

$NewIssue = New-YouTrackItem -YouTrackUri http://server1 -Project YT -Summary "Summary API Rest Method" -Description "Description API Rest Method" -SessionState $Connection

Finally, we want to apply a command to the issue created to update items with specified paramaters, as below.

POST /rest/issue/{issue}/execute?{command}

The cmdlet ‘Update-YouTrackItem’ specifies a mandatory parameter named ‘ExecuteCommand’ to specify all the items you wish to update in the issue. In this example, I will be invoking a command to set the priority as ‘High’, the type as ‘ Service Desk’ and category as ‘Support’ to update the issue previously created from the stored ‘NewIssue’ variable.

Update-YouTrackItem -YouTrackUri http://server1 -Item $NewIssue -ExecuteCommand "priority High type Service Desk category Support" -SessionState $Connection

The cmdlets are available from the below:

Connect-YouTrack – https://github.com/dean1609/PowerShell/blob/master/Functions/Connect-YouTrack.ps1
New-YouTrackItem – https://github.com/dean1609/PowerShell/blob/master/Functions/New-YouTrackItem.ps1
Update-YouTrackItem – https://github.com/dean1609/PowerShell/blob/master/Functions/Update-YouTrackItem.ps1