Unable to join ESXi host to the domain – Error when handling SMB socket

I was recently joining ESXi (5.5.0, 1892794) hosts to a domain to which the task would  fail with the status ‘Errors in Active Directory operations’.  On further investigation of the Likewise agent log on the impacted ESXi host, the following was being written to the log file:

 

ERROR:[SMBSocketReaderMain() /build/mts/release/bora-1471401/likewise/esxi-esxi/src/linux/lwio/server/rdr/socket.c:660] Error when handling SMB socket

 

This issue is due to  the size of the Kerberos Ticket Granting Service (TGS) being very high. From the network capture for SMB errors in the Likewise agent logs where the ‘Security Blob Length’ and ‘Byte Count’ values are greater than the  Max Buffer Size on the domain controller to which the ESXi host is setting up a SMB session which by default is 16644 bytes or 4356 bytes if total memory is less than or equal to 512 MB on the host.

Below is an example of the above values in an SMB network capture:


Security Blob Length: 19314
Byte Count (BCC): 19371

 

In order to resolve this issue, I was required to add a DWORD value name ‘SizeReqBuf’ to the registry key ‘HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters’ where the the value data (Decimal) is greater than the values being returned from the network capture and then restart the domain controller(s).