Enabling logging for Likewise agents in ESXi

In order to join ESXi hosts to the domain Likewise agents are used to join to the Active Directory domain and for user authentication requests. In order to troubleshoot Active Directory integration you will need to enable logging of the agent as by default they do not generate a log file.

This can be performed by enabling logging for the netlogond, lwoid and lsassd daemons.

1) Modify the file ‘/etc/init.d/netlogond’ file to change the line ‘PROG_ARGS=”–start-as-daemon –syslog‘ to one of the below, if you are using a scratch partition use the second option:

PROG_ARGS="--start-as-daemon --logfile /var/log/netlogond.log --loglevel debug" 

PROG_ARGS="--start-as-daemon --logfile /scratch/log/netlogond.log --loglevel debug"

2) Modify the file ‘/etc/init.d/lwoid‘ file to change the line ‘PROG_ARGS=”–start-as-daemon –syslog to one of the below, if you are using a scratch partition use the second option:

PROG_ARGS="--start-as-daemon --logfile /var/log/lwiod.log --loglevel trace" 

PROG_ARGS="--start-as-daemon --logfile /scratch/log/lwiod.log --loglevel trace"

3) Modify the file ‘/etc/init.d/lsassd‘ file to change the line ‘PROG_ARGS=”–start-as-daemon –syslog”‘ to one of the below, if you are using a scratch partition use the second option:

PROG_ARGS="--start-as-daemon --logfile /var/log/lsassd.log --loglevel trace"

PROG_ARGS="--start-as-daemon --logfile /scratch/log/lsassd.log --loglevel trace"

4) Restart each of the services to apply the changes:

/etc/init.d/netlogond restart

/etc/init.d/lwiod  restart

/etc/init.d/lsassd restart

Unable to join ESXi host to the domain – Error when handling SMB socket

I was recently joining ESXi (5.5.0, 1892794) hosts to a domain to which the task would  fail with the status ‘Errors in Active Directory operations’.  On further investigation of the Likewise agent log on the impacted ESXi host, the following was being written to the log file:


ERROR:[SMBSocketReaderMain() /build/mts/release/bora-1471401/likewise/esxi-esxi/src/linux/lwio/server/rdr/socket.c:660] Error when handling SMB socket


This issue is due to  the size of the Kerberos Ticket Granting Service (TGS) being very high. From the network capture for SMB errors in the Likewise agent logs where the ‘Security Blob Length’ and ‘Byte Count’ values are greater than the  Max Buffer Size on the domain controller to which the ESXi host is setting up a SMB session which by default is 16644 bytes or 4356 bytes if total memory is less than or equal to 512 MB on the host.

Below is an example of the above values in an SMB network capture:

Security Blob Length: 19314
Byte Count (BCC): 19371


In order to resolve this issue, I was required to add a DWORD value name ‘SizeReqBuf’ to the registry key ‘HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters’ where the the value data (Decimal) is greater than the values being returned from the network capture and then restart the domain controller(s).



Enable strict replication consistency on domain controllers within Active Directory domain

I recently enabled strict replication consistency on my domain controllers in order to follow best practices, where this is not enabled there can be a risk that lingering objects could be replicated to a domain controller. This can occur  when a domain controller in your Active Directory environment is disconnected from the replication topology for an extended period of time, this can cause problems when these lingering objects on the source domain controller are updated and these updates are sent by replication to the destination domain controllers.

You can identity existing lingering objects on your domain controllers by filtering for Event ID 1388 or Event ID 1988 in the domain controller logs, to which can be described as the below:

  • Event ID 1388: Inbound replication of the lingering object has occurred on the destination domain controller. 
  • Event ID 1988: Inbound replication of the directory partition of the lingering object has been blocked on the destination domain controller. 

If you have any lingering objects identified, you will need to remove this prior to enabling strict replication consistency, by performing the below:

1) Identity the DSA object GUID of the domain controller reporting lingering objects or partitions.

repadmin /showrepl %ServerName%

2)  Review the lingering objects reported on the domain controller for the directory parition reported in the event log, the adivsory_mode switch will log the lingering objects but not remove them. 

repadmin /removelingeringobjects %ServerName% %ServerGUID% %DirectoryPartition% /advisory_mode

3) Once reviewed, remove the lingering objects.

repadmin /removelingeringobjects %ServerName% %ServerGUID% %DirectoryPartition%

So once the lingering objects have been removed we  can enable stict replication consistency on each domain controller or for all domain controllers in the forest (*).

repadmin /regkey <DC_LIST> +strict

This will modify the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters. to set the value for Strict Replication Consistency to 0 (enable). You can disable strict replication consistency by running the command but with the disable switch

repadmin /regkey <DC_LIST> -strict

Enable protection from accidental deletion on all organizational units in Active Directory domain

As part of running the best practice analyser for Active Directory, I wanted to protect all organizational units from accidental deletion, this is achievable by using both the Get-ADOrganizationalUnit and Set-ADOrganizationalUnit cmdlets.

First of all I wanted to see which organizational units were not protected from accidental deletion, we can do this by invoking the Get-ADOrganizationalUnit against the domain to filter all organizational units where the ProtectedFromAccidentalDeletion value is set to ‘false’.

Get-ADOrganizationalUnit -filter * -Properties ProtectedFromAccidentalDeletion | where {$_.ProtectedFromAccidentalDeletion -eq $false} | Select-Object DistinguishedName

Now, I wanted to edit the value to be true and enable accidental protection on those organizational units returned, therefore using the original command and piping the output to the Set-ADOrganizationalUnit cmdlet to set the value to ‘true’.

Get-ADOrganizationalUnit -filter * -Properties ProtectedFromAccidentalDeletion | where {$_.ProtectedFromAccidentalDeletion -eq $false} | Set-ADOrganizationalUnit -ProtectedFromAccidentalDeletion $true

By invoking  the initial command I can now verify that all organizational units are now protected agaisnt accidental deletion as no results are returned.

However, I would have one issue with the above is that I required one organizational unit not to be protected against accidental deletion due to an automated process which created and removed child organizational units from a particular search base in the domain. Therefore, I used the cmdlets above to return all organizational units that have the ProtectedFromAccidentalDeletion value set to true by filtering the query to a particular organizational unit and recursively setting the value on each child organizational unit.

Get-ADOrganizationalUnit -filter * -SearchBase "OU=Application,OU=ServiceAccounts,DC=domain,DC=local" -Properties ProtectedFromAccidentalDeletion | where {$_.ProtectedFromAccidentalDeletion -eq $true} | Set-ADOrganizationalUnit -ProtectedFromAccidentalDeletion $false

It is possible to set the ProtectedFromAccidentalDeletion value as true as the default behavior when creating new organisational units, as per this article.

I have not performed this step yet as I would like to do some further testing and also modify the automated workflow as described above that for any organizational units that are created as part of my process, the ProtectedFromAccidentalDeletion value is set to ‘false’.

Enabling Active Directory authentication for Puppet Console Authentication

As part of my Puppet Master server installation I wanted to enable Active Directory authentication based on the membership of security group to restrict access to the console rather than using the default local console authentication mechanism.

Firstly, you are required to disable the console authentication by running the following with elevated root privelages:

sudo /opt/puppet/bin/rake -f /opt/puppet/share/console-auth/Rakefile console:auth:disable

Once the console authentication mechanism has been disabled you will be required to edit the following file to use Active Directory authentication by running the following with elevated root privelages:

sudo vi /etc/puppetlabs/httpd/auth.d/puppetconsole_auth.ad 

In the below example I will use the following configuration settings in order to build the configuration file:

Username: SVCreadad
Password: P@55word!
Domain Controller: DC1
LDAP Search Parameter: OU=Users,DC=dean,DC=local
Security Group: PuppetConsole-Allow

<Location />
AuthType Basic
AuthBasicProvider ldap
AuthName “Puppet Enterprise Console”

# Binding credentials, most AD doesn’t allow anon binding.
AuthLDAPBindDN “CN=SVCreadad,OU=Users,DC=dean,DC=local”
AuthLAPBindPassword “P@55word!”

#User Specification
AuthLDAPURL “ldap://dc1.dean.local/OU=Users,DC=dean,DC=local?SAMAccountName?sub?(objectClass=*)

# Requires that the user is a member of the specified security groups
AuthLDAPGroupAttributeIsDN on
require ldap-group CN=PuppetConsole-Allow,DC=dean,DC=local

</Location >

Save the updated file and run the following command with elevated root privelages:

sudo vi /etc/puppetlabs/httpd/conf.d/puppetdashboard.conf

Uncomment the below line which references the file we have just modified and save the configuration file

# Include /etc/puppetlabs/httpd/auth.d/puppetconsole_auth.ad 

Restart the pe-httd service by running the following command with elevated root privelages

sudo /etc/init.d/pe-httpd restart