vCloud Director : Unable to upload a vApp template in catalog

Hi folks,

It’s been a long time since my last post.
Here I come with a new issue in vCloud Director Service Provider (8.10 but I think it’s applicable to more recent and previous versions).

Our platform consists of vCenter 6.0U2 (external PSC , tiny model), NSX 6.2.5 and vCloud Director SP 8.10.1.

A few days ago I faced a weird issue while I was trying to import a vApp template in my organization catalog :

“[xxxxx] Folder xxxx does not exist in our inventory, but vCenter Server claims that is does.”

Meanwhile, I was able to upload media files in the same catalog.

I observed the same results after running the same tests on all our catalogs and organization.

 

How the problem has been solved ?

I first thought it was due to a vCloud inventory issue. I forced a synchronization with vCenter without any improvement. I even cleared the INV tables in the vCloud database. Same result, still unable to upload vApp or OVF in the catalogs.

VMware support pointed out the issue without even requiring the support bundles.

This was actually a vCenter issue and not a vCloud one. Our vCenter was suffering of low memory.

vCenter did what vCloud asked but took too much time to inform to update the cell. This is the explanation given by VMware. A simple reboot should solve the issue.

To confirm the RAM problem, i simply ran the command “free -m” on our vCenter appliance, the output showed that the swap partition was heavily used, almost entirely, more than 20 GB. I do not mention the RAM  on purpose because it almost always consumes around 8 GB.

In this case, swapping very likely means that the vCenter has memory leaks…

A simple reboot could have freed the memory and flushed the swap partition.  I think so, however I decided to add some more RAM and adjust the VM to the small model. This, because our platform also backs vRA and a lot of other components that interact with vCenter.

After the reboot my upload issue was solved !

 

Good to know !

RAM Upgrade

Notice that as of the version 6 of vcsa, it is no more required to manually adjust the RAM dedicated to the JVM. The JVM memory is dynamically adjusted.

The famous William Lam (blog VirtuallyGhettto) talks about that in this post.

Disk sizing upgrade

Moreover, no need to manually resize all the file systems. A script checks the disks and volumes and resizes them automatically at the boot of the appliance. If the resizing occurs while the appliance is already running a simple command line does the job. W. Lam explains that too here.

 

If you want to get more information about the VCSA partitioning you can check this KB. 

Here, a reminder for the different VCSA sizing models.

 

 

 

 

Posted in Cloud computing, Virtualization, vSphere Tagged with: vCenter, vCloud Director, VMware

Replacing VCSA web certificates

This post describes how to replace the machine certificate for vCenter installed in External mode.

Replacing the machine certificate only allow vCenter web sites to be trusted by user browsers.

To complete this operation you will need to make sure that your PKI has a VMware certificate template (check KB2112009) and that the root certificate of your PKI is installed on the computer you will run the checks from.

We’ll follow the steps below :

1. Create PSC CSR
2. Verify the signed certificate of the PSC
3. Import the signed certificate on PSC
4. Create vCenter CSR
5. Verify the signed certificate of the vCenter
6. Import the signed certificate of the vCenter

 

Replacing PSC machine certificate

From your PSC node :

Go to /usr/lib/vmware-vmca/bin folder

As usual, be cautious and take a snapshot of your PSC and vCenter nodes and of course, run a backup of your vCenter DB (if external).

Once all has been checked and done, let’s go through the replacement steps.

Creating PSC CSR

Start VMware Certificate Management tool

./certificate-manager.sh

Select option 1 “Replace Machine SSL certificate…”

Provider the SSO administrator account and his password.

If like me you did not install your PSC node with the default SSO domain, don’t forget to modify the default account.

Then, select option 1 to generate the certificate signing resquest.

Provide the folder where the CSR and the related keys will be created.

Answer all questions

Notice that Name and Hostname are the FQDN of the PSC node.

VMCA Name is the last value to provide. After the script will execute certool commands to create the private key of the and the CSR of the PSC node.

You can leave the wizard at this step and provide the CSR to the security team so they can issue the certificate.

Verifying PSC certificate

Once received, you can check differents properties like :

• The subject of the certificate (can also be called the distinguished name)

The common name must be the FQDN of your PSC

.

• The Subject Alternate Name

Must display the FQDN and the IP address at least (depends on the information you provided when you prepared the CSR

• The Key Usage

It must match with the key usage of the vSphere certificate template (remember the KB, “Non-repudiation”)

Importing PSC CSR

Once you’ve copied the certificate in a folder of your PSC, restart the certificate management tool and select option 1 to proceed with the replacement.

Then select option 2 to import the certificate.

Provide the full path of the certificate, its key and of the root certificate (of the Certificate Authority).

Accept to continue the operation

The replacement process is starting and will check all PSC services

9 services have been updated. PSC services are restarted.

Next action : Restart the services of each vCenter node(s).

Before going further, I usually check the certificate update and the SSO service by going to the PSC web portal and try a connection on the certificate management portal of the PSC : https://FDQN/psc.

No security warning displayed and SSO service running as expected

Now, as required, restart the vCenter node thanks to service-control –all –stop and service-control –all –start

Of course, validate that everything is running correctly by opening a session on the vSphere web client.

We can now proceed with the replacement of the self-signed certificate of the vCenter.

The operation is almost the same, all steps done for the PSC will be repeted for the vCenter.
CREATING VCENTER CSR

From your vCenter node :

As we did for the PSC, let’s go to /usr/lib/vmware-vmca/bin folder

One more time, prevention is better than cure, take a snapshot of your psc and vCenter nodes and of course, run a new backup of your vCenter DB (if external).

Once done, let’s start Certifficate Manager tool.

Select the first option to replace the machine SSL certificate

Provide the SSO administrator account and password

Select option 1 to generate the CSR

If you have an external PSC, its IP address will be required

 

Give the certificate details

Remember that VMCA Name is the last propertie you will set so do not validate it too fast (because the script will run right after, without any confirmation)

The script has just created the CSR and the private key.

Leave the tool (actually, it’s up to you 🙂 )

CHECKING VCENTER SIGNED CERTIFICATE

Once received, have a quick look at some properties of the delivered certificate.

• The Subject

• The Subject Alternate Name

IMPORT VCENTER SIGNED CERTIFICATE

When you have copied the certificate on your vCenter node, restart the certificate manager tool.

Again, select the first option

Like the last time, provide the SSO administrator account credentials

Select option 2 to import the signed certificate

 

Give the path of the certificate, its private key and the root certificate of Certificate Authority

Validate to achieve the replacement

The message indicates that 21 services have been updated and the services of the vCenter have been rebooted.

To check that vCenter functions properly, just open a vSphere web client session.

As you can see, the certificate of the vCenter is now trusted.

As you have just seen, replacing SSL machine certificates is pretty easy and can  be done quickly.

In a previous lab I also changed the user solution certificates directly thanks to the certool commands but the problems I experienced after with NSX  (I had to cheat and apply the old fingerprint to the new PSC certificate to reconnect to lookup services. VMware referenced  this issue here) and vROM (I lost the connection to vCenter) led me to give up the idea of changing all vCenter certificates. Too paintful for administrators teams.

Depending on the customer security policies you might have to replace them all, then be cautious and don’t forget to double check the connection of your other components to vCenter nodes.

As I hate loosing, I will try to update you soon with a new post for the “FULL” replacement 🙂

 

Posted in Virtualization, vSphere Tagged with: Certificates, vCenter