It’s been a long time since my last post.
Here I come with a new issue in vCloud Director Service Provider (8.10 but I think it’s applicable to more recent and previous versions).
Our platform consists of vCenter 6.0U2 (external PSC , tiny model), NSX 6.2.5 and vCloud Director SP 8.10.1.
A few days ago I faced a weird issue while I was trying to import a vApp template in my organization catalog :
“[xxxxx] Folder xxxx does not exist in our inventory, but vCenter Server claims that is does.”
Meanwhile, I was able to upload media files in the same catalog.
I observed the same results after running the same tests on all our catalogs and organization.
How the problem has been solved ?
I first thought it was due to a vCloud inventory issue. I forced a synchronization with vCenter without any improvement. I even cleared the INV tables in the vCloud database. Same result, still unable to upload vApp or OVF in the catalogs.
VMware support pointed out the issue without even requiring the support bundles.
This was actually a vCenter issue and not a vCloud one. Our vCenter was suffering of low memory.
vCenter did what vCloud asked but took too much time to inform to update the cell. This is the explanation given by VMware. A simple reboot should solve the issue.
To confirm the RAM problem, i simply ran the command “free -m” on our vCenter appliance, the output showed that the swap partition was heavily used, almost entirely, more than 20 GB. I do not mention the RAM on purpose because it almost always consumes around 8 GB.
In this case, swapping very likely means that the vCenter has memory leaks…
A simple reboot could have freed the memory and flushed the swap partition. I think so, however I decided to add some more RAM and adjust the VM to the small model. This, because our platform also backs vRA and a lot of other components that interact with vCenter.
After the reboot my upload issue was solved !
Good to know !
Notice that as of the version 6 of vcsa, it is no more required to manually adjust the RAM dedicated to the JVM. The JVM memory is dynamically adjusted.
The famous William Lam (blog VirtuallyGhettto) talks about that in this post.
Disk sizing upgrade
Moreover, no need to manually resize all the file systems. A script checks the disks and volumes and resizes them automatically at the boot of the appliance. If the resizing occurs while the appliance is already running a simple command line does the job. W. Lam explains that too here.
If you want to get more information about the VCSA partitioning you can check this KB.
Here, a reminder for the different VCSA sizing models.
This post describes how to replace the machine certificate for vCenter installed in External mode.
Replacing the machine certificate only allow vCenter web sites to be trusted by user browsers.
To complete this operation you will need to make sure that your PKI has a VMware certificate template (check KB2112009) and that the root certificate of your PKI is installed on the computer you will run the checks from.
We’ll follow the steps below :
Replacing PSC machine certificate
From your PSC node :
Go to /usr/lib/vmware-vmca/bin folder
As usual, be cautious and take a snapshot of your PSC and vCenter nodes and of course, run a backup of your vCenter DB (if external).
Once all has been checked and done, let’s go through the replacement steps.
Start VMware Certificate Management tool
Select option 1 “Replace Machine SSL certificate…”
Provider the SSO administrator account and his password.
If like me you did not install your PSC node with the default SSO domain, don’t forget to modify the default account.
Then, select option 1 to generate the certificate signing resquest.
Provide the folder where the CSR and the related keys will be created.
Answer all questions
Notice that Name and Hostname are the FQDN of the PSC node.
VMCA Name is the last value to provide. After the script will execute certool commands to create the private key of the and the CSR of the PSC node.
You can leave the wizard at this step and provide the CSR to the security team so they can issue the certificate.
Once received, you can check differents properties like :
The common name must be the FQDN of your PSC
Must display the FQDN and the IP address at least (depends on the information you provided when you prepared the CSR
It must match with the key usage of the vSphere certificate template (remember the KB, “Non-repudiation”)
Once you’ve copied the certificate in a folder of your PSC, restart the certificate management tool and select option 1 to proceed with the replacement.
Then select option 2 to import the certificate.
Provide the full path of the certificate, its key and of the root certificate (of the Certificate Authority).
Accept to continue the operation
The replacement process is starting and will check all PSC services
9 services have been updated. PSC services are restarted.
Next action : Restart the services of each vCenter node(s).
Before going further, I usually check the certificate update and the SSO service by going to the PSC web portal and try a connection on the certificate management portal of the PSC : https://FDQN/psc.
No security warning displayed and SSO service running as expected
Now, as required, restart the vCenter node thanks to service-control –all –stop and service-control –all –start
Of course, validate that everything is running correctly by opening a session on the vSphere web client.
We can now proceed with the replacement of the self-signed certificate of the vCenter.
From your vCenter node :
As we did for the PSC, let’s go to /usr/lib/vmware-vmca/bin folder
One more time, prevention is better than cure, take a snapshot of your psc and vCenter nodes and of course, run a new backup of your vCenter DB (if external).
Once done, let’s start Certifficate Manager tool.
Select the first option to replace the machine SSL certificate
Provide the SSO administrator account and password
Select option 1 to generate the CSR
If you have an external PSC, its IP address will be required
Give the certificate details
Remember that VMCA Name is the last propertie you will set so do not validate it too fast (because the script will run right after, without any confirmation)
The script has just created the CSR and the private key.
Leave the tool (actually, it’s up to you 🙂 )
Once received, have a quick look at some properties of the delivered certificate.
When you have copied the certificate on your vCenter node, restart the certificate manager tool.
Again, select the first option
Like the last time, provide the SSO administrator account credentials
Select option 2 to import the signed certificate
Give the path of the certificate, its private key and the root certificate of Certificate Authority
Validate to achieve the replacement
The message indicates that 21 services have been updated and the services of the vCenter have been rebooted.
To check that vCenter functions properly, just open a vSphere web client session.
As you can see, the certificate of the vCenter is now trusted.
As you have just seen, replacing SSL machine certificates is pretty easy and can be done quickly.
In a previous lab I also changed the user solution certificates directly thanks to the certool commands but the problems I experienced after with NSX (I had to cheat and apply the old fingerprint to the new PSC certificate to reconnect to lookup services. VMware referenced this issue here) and vROM (I lost the connection to vCenter) led me to give up the idea of changing all vCenter certificates. Too paintful for administrators teams.
Depending on the customer security policies you might have to replace them all, then be cautious and don’t forget to double check the connection of your other components to vCenter nodes.
As I hate loosing, I will try to update you soon with a new post for the “FULL” replacement 🙂