Log in to vCenter GUI is not possible.
If Web GUI is still available, any log in attempt with correct credentials fails.
Restart of VCSA services fails.
Restart of services does not bring up all services.
Errors observed:
/var/log/vmware/vpxd-svcs/vpxd-svcs.log:
2020-06-03T09:31:04.523Z [pool-8-thread-1 INFO com.vmware.identity.token.impl.X509TrustChainKeySelector opId=905f6864-c067-4db6-828c-1d59c4b43bf8] Failed to find trusted path to signing certificate <CN=ssoserverSign>
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
Cause
vCenter certificates are expired.
VxRail which was initially built prior to 4.7, may have certificates issued with a lifespan of two years from the date of installation. At the time of writing this article, a VxRail build on 4.7.410 has all certificates with a 10 year lifespan.
Minor version upgrades will not touch the certificates!
For a VxRail which was initially built on 4.5.210 and later versions, the certificates have a two-year validity period. Check VMware KB article 79248 to confirm the detailed description.
Use the view certificate in the browser of the log in page of the VCSA to confirm the certificate has expired or list the certificates in the CLI of the PSC/VCSA with the command from VMware KB article 76719:
for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
Resolution
This procedure will generate new self-signed certificates on PSC and VCSA.
IMPORTANT: This procedure is intended for single PSC/VCSA VMs which are maintained through VxRail LCM. For HA / ELM / Customer deployed VCs – please open a VMware ticket!
IMPORTANT: Take OFFLINE snapshots of VRM, PSC, and VCSA!
IMPORTANT: Check if the snapshot creating process has finished without errors! Do NOT continue without valid snapshots!
IMPORTANT: If issues encountered, do not retry without reverting to snapshots!
- Fix PSC:
- Reset all Certificates – Option 8 – (This will fail but it is expected to)
- Start certificate manager:
/usr/lib/vmware-vmca/bin/certificate-manager
- Select option “8. Reset all Certificates”
- Confirm “Do you wish to generate all certificates using configuration file : Option[Y/N] ?”
- Enter credentials
- Enter values
- leave “IPAddress” field empty
- Enter FQDN of PSC into “Hostname“
- VMCA “Name” field is name of new Root CA being created (e.g. “VxRail CA“)
- Confirm “Continue operation : Option[Y/N] ?“
- Confirm “Continue operation : Option[Y/N] ?“
- This operation will fail with:
Get site nameCompleted [Reset Machine SSL Cert…]
g3node-site
Lookup all services
Get service g3node-site:9ebfb9b2-d202-4d8f-9282-b08f3a317b8f
Update service g3node-site:9ebfb9b2-d202-4d8f-9282-b08f3a317b8f; spec: /tmp/svcspec_a1hipoqq
Status : 0% Completed [Reset operation failed]
please see /var/log/vmware/vmcad/certificate-manager.log for more information.
root@g3psc [ ~ ]#
2. Fix the STS issue – Download and run fixsts.sh from VMware KB 76719
- Stop all services
- service-control –all –stop
- Start all services – will fail – expected…
- service-control –all –start
- Either wait for the process to time-out or you can stop it when it gets to the “vmware-vmon” service
/usr/lib/vmware-vmca/bin/certificate-manager
- Select option “6. Replace Solution user certificates with VMCA certificates“
- Confirm “Do you wish to generate all certificates using configuration file : Option[Y/N] ?“
- Enter credentials
- Deny (enter “N” ) for “certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ?” – as all options were configured above
- Confirm “You are going to regenerate Solution User Certificates using VMCA, Continue operation : Option[Y/N] ?“
- Wait until procedure exits, it will:
- generate all certificates
- stop the services
- start the services
- Confirm if all services are running
service-control –all –status
3. Fix certificates on VCSA
- Stop & start all services – must be done AFTER all PSC services are running – IMPORTANT!!!
- service-control –all –stop
- service-control –all –start
- Either wait for the process to time-out or you can stop it when it gets to the “vmware-vmon” service
- Reset all Certificates – option 8
- Start certificate manager
/usr/lib/vmware-vmca/bin/certificate-manager
- Select option “8. Reset all Certificates“
- Confirm “Do you wish to generate all certificates using configuration file : Option[Y/N] ?“
- Enter credentials
- Enter PSC IP
- Enter values
- leave “IPAddress” field empty
- Enter FQDN of VCSA into “Hostname“
- VMCA “Name” field is name of new Root CA being created (e.g. “VxRail CA“)
- Confirm “Continue operation : Option[Y/N] ?“
- Confirm “Continue operation : Option[Y/N] ?“
- Wait until all certificates are generated and the message “Reset status : 100% Completed [Reset completed successfully]” appears
- Check if all services are running
service-control –all –status
- Access vCenter GUI
- Access by DNS will fail in Chrome due to HSTS – open the VCSA IP or use another supported browser (e.g. FireFox)
Additional Information
Notes:
- ALWAYS take snapshots of System VMs (PSC/VCSA/VRM) before following this KB.
- This procedure is intended for PSC/VCSA VMs which are maintained through VxRail LCM.
- If user has certificates from their own infrastructure, they can replace them at this point again.
- After fix for VxRail version 4.7.100+, follow KB article 486959: VxRail: How to manually import vCenter SSL certificate on VxRail Manager to import new root certificate into VRM (plugin will not work).