Storage – Configure Software iSCSI port binding on an ESXi Host

In order to configure multipathing for an iSCSI storage device we can configure software iSCSI port binding on an ESXi Host in order to load-balance between paths when all paths are present and to handle failures of a path at any point between the server and the storage. Without port binding, all iSCSI LUNs will be detected using a single path per target. By default, ESX will use only one vmknic as an egress port to connect to each target, and you will be unable to use path failover or to load balance I/O between different paths to the iSCSI LUNs

To enable vmknic-based software iSCSI multipathing, we must perform the following from the vSphere Web Client:

1) Select the ESXi host system we wish to configure the Software iSCSI port binding.

2) Select Manage > Networking.

3) Select Add Host Networking and VMkernel Network Adapter.

4) Create a new Standard Switch (vSS).

5) Assign two physical network adapters, in this example vmnic1 and vmnic2.

6) Specify a network label, in the first instance I will name this vmk-iscsi-1.

7) Assign an IPv4 address to the the VMkernel Network Adapter.

8) Select Finish to complete the configuration.

Now, I will repeat Steps 3-8, but I will use the existing Standard Switch (vSS) I previously created assign the network label vmk-iscsi-2 and configure a unique IPv4 address. Once you have created two VMkernel Network Adapters on the Standard Switch (vSS) we will need to configure the failover order of each one to assign only a single active unique physical network adapter.

1) Highlight the VMkernal Network Adapter vmk-iscsi-1 and select Edit.

2) Select Teaming and Failover, enable the Override option and specify vmnic1 as the active adapter and vmnic2 as the unused adapter.

Again, repeat the above steps but for vmk-iscsi-2 select the active adpater as vmnic2 and the vmnic1 as the unused adapter.  Finally, we need to configure the network configuration of our Software iSCSI adapter to bind to port VMkernel Network Adapters created in the above steps.

1) Select Manage > Storage > Storage Adapters.

2) Highlight the Software iSCSI Adapter and select Network Port Binding.

3) Add both vmk-iscsi-1 and vmk-iscsi2 and rescan the adapter to apply the configuration.

http://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-binding.pdf

Storage – Identity and tag SSD and local devices on an ESXi Host

In my lab environment I was looking to tag a local hard drive as SSD in order to configure and manage vSphere Flash Read Cache, this activity actually ticks two boxes in the VCAP5-DCA blueprint as in order to achieve this I was required to use the Pluggable Storage Architecture (PSA) related commands from the esxcli storage namespace.

Firstly, as my ESXi host systems are nested I created a virtual machine hard disk to be used as a local SSD device and then I connected to the ESXi host system using an SSH client. By using the esxcli storage namespace I will list the devices available to determine the device name to which I require to create a rule to tag the local device as an SSD.

esxli storage nmp device list
Device Display Name: Local VMware Disk (mpx.vmhba1:C0:T1:L0)

Now, we want to create and add a PSA  rule to the device we have discovered and tag this as a SSD device. Once the rule has been created we will be required to restart the ESXi host system for the changes to apply to the local device.

esxcli storage nmp satp rule add –-satp=VMW_SATP_LOCAL –-device mpx.vmhba1:C0:T1:L0 --option "enable_local enable_ssd"

On restart, we can now confirm that the local device is now being discovered as an SSD drive type.

ssd_device


http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2013188

Security – Part Five: Security Profile, Services and Firewall

When an ESXi host system is installed by default the firewall is enabled. All incoming and outgoing ports are blocked except the default TCP and UDP ports using for Management Services. The ESXi firewall protects the management interface of the ESXi host system, but provides no protection to virtual machines. By default, the following management services are open:

Management Service Description
lbtd (/sbin/net-lbtd) Load balancing teaming for distributed switches is an operation that evaluates the uplink load
vpxa (/usr/lib/vmware/vpxa/bin/vpxa) Virtual Center Agent to enable communication from the vCenter Server System to hostd service on the ESXi host system.
NTP Daemon (/sbin/ntpd) Network Time Protocol Daemon to synchronise the time between the ESXi host system and either a stratum or an atomic clock over the network
Direct Console UI (/sbin/dcui) Direct Console User Interface to enable the starting and stopping of the system and to perform limited setup, maintenance and troubleshooting tasks.
CIM Server (/bin/cimslp) Common Interface Model interface to monitor and manage health of the manager server hardware

In addition to the management services, by default the following ports are open for management access on the ESXi host system:

Port Description
22 SSH Server for incoming TCP
53 DNS client for incoming and outgoing UDP
68 DHCP client for incoming and outgoing UDP
161 SNMP server for incoming UDP
80 Fault tolerance for incoming TCP and outgoing TCP and UDP
427 CIM client SLPv2 to discover server for incoming and outgoing UDP
443 HTTPS access for incoming TCP
902 Host access and heartbeat for incoming and outgoing TCP and outgoing UDP
1234, 1235 vSphere replication for outgoing TCP
5988 CIM transactions over HTTP for incoming TCP
5989 CIM XML transactions over HTTPS for incoming and outgoing TCP
8000 vMotion requests for incoming and outgoing TCP
8100, 8200 Fault tolerance traffic for incoming and outgoing TCP and UDP

In order to protect the management interface the ESXi host system uses firewall rulesets to allow and disallow access to the ESXi host system. The default rulesets are defined initially in the read-only ‘/etc/vmware/firewall/service.xml’ file. To retrieve a list of firewall rule sets invoke the esxcli network firewall namespace.

esxli network firewall ruleset list

In addition to the firewall rulesets, more detailed information maybe retrieved:

esxli network firewall ruleset rule list

The ESX host system firewall ports and services can be made either using the vSphere Web Client, using the esxcli command line tool or using PowerCLI.

Configuring Firewall Service Properties Using vSphere Web Client

1) Select the ESXi host system to configure the Firewall service properties.

2) Browse to Manage > Settings > Security Profile > Services and select Edit.

Configure the ESXi Firewall Properties Using vSphere Web Client

1) Select the ESXi host system to configure the Firewall properties.

2) Browse to Manage > Settings > Security Profile > Firewall and Edit.

Configure the ESXi Firewall Using PowerCLI 

To retrieve information regarding the ESXi Firewall we can invoke both the Get-VMHostService and Get-VMHostFirewallException cmdlets.

1) For retrieving the status of services running on the ESXi host system, invoke the following:

Get-VMHost esxi1.dean.local | Get-VMHostService

2) To retrieve more detailed information for the incoming and outgoing service ports for each service invoke the following:

Get-VMHost esxi1.dean.local | Get-VMHostFirewallException | Where-Object {$_.Enabled}

By default, management services on the ESXi host system are enabled to provide local and remote client access and allowed through the firewall if the service ports are open. For Example, the syslog service is enabled on the ESXi host system but the firewall service ports are not allowing outbound connections. To modify the configuration using the esxcli network firewall namespace, perform the following steps.

1) Connect to the ESXi host system using an SSH client.

2) Retrieve the status of the syslog ruleset status.

esxcli network firewall ruleset list | grep syslog
syslog false

3) Enable the syslog firewall service.

esxcli network firewall ruleset set -e true -r syslog

We may also disable the syslog firewall service by invoking the following using the esxcli network firewall namespace.

esxcli network firewall ruleset set -e false -r syslog

There may be certain scenarios where the you may need to add more services than those preconfigured on the ESXi host system, in order to perform this action we must create a configuration file in the directory ‘/etc/vmware/firewall’. In this example, I will configure a service named ‘DeanTestApplication’ to enable inbound connectivity on TCP service port 3210 where the default state will be disabled.

1) Connect to the ESXi host system using an SSH client.

2) Create the following file ‘/etc/vmware/firewall/deantestapplication.xml’, with the following configuration.

<! -- Firewall information for Dean Test Application -->
<ConfigRoot>
  <service>
    <id>DeanTestApplication</id>
    <rule id='0000'>
      <direction>inbound</direction>
      <protocol>tcp</protocol>
      <porttype>dst</porttype>
      <port>3210</port>
    </rule>
  </service>
</Config>

3) Save the file and reload the firewall, which will restart the services listed for the firewall.

esxcli network firewall refresh

Once the firewall has been reloaded we can use the vSphere Web Client to confirm if the firewall service has been loaded into memory and further configure the service. Alternatively, we can confirm this form the esxcli network firewall namespace:

esxcli network firewall ruleset list | grep DeanTestApplication
DeanTestApplication false

Security – Part Four: Enabling ESXi Lockdown Mode

To increase the security of an ESXi host system which is being managed by a vCenter Server system you can enable Lockdown Mode to restrict users from performing actions directly on an ESXi host using SSH or the ESXi shell. Also, users without the DCUI Access privelage will be restricted from accessing the DCUI. As Lockdown Mode restricts access all actions on managed ESXi host systems must be performed using a connection to the vCenter Server system using an account with necessary permissions.

Lockdown Mode is only available when an ESXi host system is managed by a vCenter Server and can be enabled and disabled using the following methods.

Configuring Lockdown Mode using the vSphere Web Client 

1) Select the ESXi host system you wish to configure Lockdown Mode.

2) Browse to Manage  > Security Profile > Lockdown Mode and select Edit

3) Enable or Disable the Lockdown Mode checkbox and select OK.

Configuring Lockdown Mode using the ESXi Shell Command Line

1) Connect to the ESXi host system using an SSH client.

2) To retrieve the status of Lockdown mode, invoke the following:

~ # vim-cmd -U dcui vimsvc/auth/lockdown_is_enabled
false

3) To enable Lockdown Mode, invoke the following:

~ # vim-cmd -U dcui vimsvc/auth/lockdown_mode_enter

4) To disable Lockdown Mode, invoke the following:

~ # vim-cmd -U dcui vimsvc/auth/lockdown_mode_exit

Configure Lockdown Mode Using the Direct Console User Interface

1) Connect to the ESXi host system using the DCUI.

2) Press F2 to customise the system.

3) Select Configure Lockdown Mode

4) Enable or Disable Lockdown Mode using the spacebar to toggle the configuration state.

5) Select OK.

Configure Lockdown Mode using PowerCLI

1) Connect to the ESXi host system using the Connect-VIServer cmdlet.

2) To retrieve the status of the Lockdown Mode, invoke the following using the Get-VMHost cmdlet.

Get-VMHost esxi1.dean.local | Select Name, @{N="Lockdown Mode";E={$_.ExtensionData.Config.adminDisabled}}

3) Invoke the following to enable Lockdown Mode using the Get-View cmdlet.

(Get-VMHost esxi1.dean.local | Get-View).EnterLockdownMode() | Get-VMHost | Select Name, @{N="Lockdown Mode";E=$_ExtensionData.Config.AdminDisabled}} 

4) Invoke the following to disable Lockdown Mode using the Get-View cmdlet

(Get-VMHost esxi1.dean.local | Get-View).ExitLockdownMode()

 

 

Security – Part Three: Generating ESXi Host Certificates

In order to secure connections between clients, ESXi host systems and the vCenter Server system SSL is used. When an ESXi host system or vCenter Server system is installed, the installation will include SSL certificates by default to establish an initial connection. In order to connect ESXi host systems to a managed vCenter Server system and to connect to those managed objects the SSL certificate is used.

For SSL certificates generated in VMware products these use standard X.509 version 3 (x.509v3) certificates to encrypt session information over the SSL protocol connections between the client and the server.  When replacing the default ESXi host system and vCenter Server system you are required to generate your SSL certificates to conform to the Privacy Enhanced Mail (PEM) key format and to be signed. The key used to sign certificates must be a standard RSA key with an encryption length which ranges between 512 and 4.096 bits, where the VMware recommendation is 2,048.

If you replace a certificate with one signed by your own local root CA or plan to use the default certificates, you must pre-trust the certificate by importing into the local certificate store for each vSphere Client instance. For certificates signed by a local root CA you must pre trust any valid default certificates you will continue to use on the vCenter Server system. For ESXi host systems that are exposed to the internet, you should use a trusted commercial security authority.

On an ESXi host system, the two default certificate files are located at:

  • Private Key – /etc/vmware/ssl/rui.key
  • Certification File – /etc/vmware/ssl/rui.crt

In order to replace the default SSL certificates perform the following on an ESXi host system:

1) Connect to the ESXi host system using a SSH client as root user.

2) Create a backup of the existing certificates if they exist on the ESXi host system.

/etc/vmware/ssl # mv rui.crt rui.crt.backup
/etc/vmware/ssl # mv rui.key rui.key.backup

3) Generate the new certificates on the ESXi host system or replace the default certificate with CA signed certificate.

/bin # generate-certificates

4) Restart the services or alternatively restart the ESXi host system.

services.sh restart

We may also configure timeout values that can affect the SSL connection for when a connection becomes idle that can be configured for an ESXi host system. By default, SSL connections between the server and the client do not timeout. There are two timeout settings which can be configured on an ESXi host system, these being:

  • Read Timeout – connections that have completed the SSL handshake process using TCP service port 443 of the ESXi host system.
  • Handshake Timeout – connections that have not completed the SSL handshake process using TCP service port 443 of the ESXi host system.

The timeout values can be configured by connecting to an ESXi host system using a SSH client and modifying the file ‘/etc/vmware/hostd/config.xml’ file and adding the below which requires a  restart of the vmware-hostd process to apply the configuration change.

<vmacore>
  <http>
     <readTimeoutMs>20000</readTimeoutMs>
  </http>
  <ssl>
    <handshakeTimeoutMs>20000</handshakeTimeoutMs>
  </ssl>
</vmacore>
services.sh restart

Security – Part Two: Configure ESXi Host SSH Settings

In order to access an ESXi host system using a SSH client there is a requirement to connect to the remote host. By default, the ESXi host system does not enable SSH connections and therefore there is a requirement to enable access to an ESXi host system to use SSH. This can be performed from the vSphere Web Client by performing the following:

1) Select the ESXi host system you require to enable SSH.

2) Select Manage > Security Profile > Services and select Edit.

3) Select the SSH service and select Start.

I have previously written about enabling the SSH service on multiple ESXi host systems using PowerCLI at https://deangrant.wordpress.com/2014/08/19/powercli-starting-and-stopping-the-ssh-service-on-multiple-esxi-hosts/.

By default the startup policy will be configured to be stop and start manually, on restart of the ESXi host system the service will not be started on startup. To secure the SSH connectivity further you may configure a timeout value for each connection. In order to configure a timeout value we will need to modify the advanced settings of the ESXi host system for the value ‘UserVars.ESXiShellTimeout’ which can be performed in the vSphere Web Client as follows:

1) Select the ESXi host system you require to configure a SSH timeout value

2) Select Manage > Advanced System Settings

3) Browse to the value ‘UserVars.ESXiShellTimeout’ and select Edit.

The value in seconds can be set from 0 (no timeout value) to 86,400 seconds.  Alternatively, you can use PowerCLI and the Set-AdvancedSetting cmdlet to configure this value as below to configure a timeout value of one hour.

Get-VMHost esxi1.dean.local | Get-AdvancedSetting -Name "UserVars.ESXiShellTimeOut" | Set-AdvancedSetting -Value "3600" -Confirm:$False

 

Security – Part One: Enabling strong passwords and configuring password policies

By default, there are no restrictions set on the local root user account on an ESXi host system. However, the local non-root users must satisfy the requirements of the password compliance policy defined by the Pluggable Authentication Module (PAM). By default, the ESXi host system checks for password compliance using the pas_passwdqc.so PAM module.

The pam_passwdqc plug-in is inserted into the PAM stack so when a user creates a password rules are enforced on the password chosen for the local non–root user on the ESXi host system. The plug-in ensures that password requirements must be satisfied for all local non–root users.

In order to modify the password complexity you will be required to edit the ‘/etc/pam.d/passwd’ file, which by default contains the following:

#%PAM-1.0

password requisite /lib/security/$ISA/pam_passwdqc.so retry=3 min=8,8,8,7,6
password sufficient /lib/security/$ISA/pam_unix.so use_authtok nullok shadow sha512
password required /lib/security/$ISA/pam_deny.so

The syntax of the password file is as follows, where the last five numbers control the complexity of the password and refer to the four character classes (numbers, lowercase letters, uppercase letters and specical characters.

password requisite /lib/security/$ISA/pam_passwdqc.so retry=n1 min=n2,n3,n4,n5,n6
  • n1 – The number of attempts the users is able to enter a strong password
  • n2 – Passwords containing at least one character classes must be a minimum of eight characters in length.
  • n3 – Passwords containing at least two character classes must be a minimum of eight characters in length.
  • n4 – Each word in the passphrase must be a minimum of eight characters in length.
  • n5 – Passwords containing at least three character classes must be a minimum of seven characters in length.
  • n6 – Passwords containing at all  character classes must be a minimum of six characters in length.

For Example, if we wanted to ensure that passwords containing at least one character classes must be a minimum of ten characters in length, I would modify the above line as follows:

password requisite /lib/security/$ISA/pam_passwdqc.so retry=3 min=10,8,8,7,6

Following the above change, if I was to connect to an ESXi host system using the vSphere Client and create a user account where the password is eight characters in length and only contains one character class, I would receive the following error message:

PasswordComplex

 

 

 

 

The pam_passwdqc plug-in does not count uppercase letters used as the first character in the password and number used as the last character of a password when the number of character classes is being counted. If you were to configure any of the above requirements with the value ‘-1’ password complexity requirements will be ignored. The above change takes effect on saving the configuration changes to the ‘/etc/pam.d/passwd’ file to which local non–root users can change their passwords using the passwd utility.