------------------------------------------------------------------------- HOTFIX ID: VM-V6.2.0.9063-RL.EA LVL09 DATE: 8 March 2017 TITLE: Stratus everRun VM Hotfix LVL09 -- V6.2.0.9209-HF.EA ------------------------------------------------------------------------- ISSUES RESOLVED Issues addressed by Hotfix Level 09 are itemized here. All released changes are included in this update kit. 1) Hotfix LVL01 -- a) Add Ivy Bridge Server support. b) everRun pvm-repair command fails to unpair and pair PVM in host replacement operation. c) A halted PVM may auto boot regardless of the auto-power-on setting when everRun service restarts. d) XenServer 6.0.2 running on the Intel 55x0 chipset may crash. Disable IOMMU interrupt remapping. Refer to http://support.citrix.com/article/CTX136517. e) Guest OS crashes with MEMX_REPLICATION_ERROR after first synch and a fresh protect. f) COMX fault can result in a guest OS loss because all AM vCPUs are running at 100% CPU usage. g) PVMs may be lost when a segmented management LAN is repaired, if one of the XenServers had been restarted during the time the management LAN was not functional (between pool members). L2 PVMs or L3 PVMs running without redundancy at the time may be lost. 2) Hotfix LVL02-- a) A network failover can result in up to two minutes of multicast traffic interruption. b) A slave PDM detected DOM0 disk failure can cause the guest OS to pause for over two minutes. c) The everRun repair command fails when the pool has Q-links configured. d) The log harvest utility can fill up the everRun disk partition due to large amount of collected log data. e) The kernel_events log shows UTC time when the system timezone is GMT+ hours. f) A Hotfix rollback fails to restore all everRun components symbolic links g) A failure in xenstore access can cause the everRun disk partition to fill up with everrun log. h) Integrate dmidecode V2.12 into everRun Install kit. i) A PVM created from a snapshot will generate a reference VM with wrong memory size after the PVM is unprotected. j) Update install guide to include Window 2003 R2 support. 3) Hotfix LVL03-- a) Guest OS clock may gain a few seconds after a transition. b) Guest OS may experience performance degradation after upgrade from VM6.1.0. c) Running the mtc_log_collector with option -e takes too long to complete. 4) Hotfix LVL04-- a) MMS prevents use of ALINK IP addresses with 255 in 2nd and/or 3rd octet. b) Guest PVM Bios UUID may acquire a different value after each boot. c) Xenstore may grow by 11 entries for every CI creation. d) Update SSH to support sha2 256, and sha2 512 encryptions. e) Update OpenSSL to resolve CVE-2014-0224 (a.k.a. CSS Injection). f) Include Citrix XS602E037 bash Shellshock security update. g) Update Xen Hypervisor to resolve CVE-2014-7188 titled "Improper MSR range used for x2APIC emulation" h) A PVM may not unevacuate after a host reboot, and MMS log shows a Quorum exception. i) MMS may display a Quorum link adapter as Faulted when the adapter is good. 5) Hotfix LVL05-- a) Integrate with Citrix XenServer Hotfix XS602E038; a rollup of existing Citrix Hotfixes and also addresses CVE-2014-8595 (NVD), CVE-2014-8866, CVE-2014-8867, and CVE-2014-1666. b) Include Citrix XenServer Hotfix XS602E039 with remediation for glibc GHOST Vulnerability (CVE-2015-0235). c) Convert check health messages (AM Console) from error to informational. d) Update NTP version to ntp-4.2.2p1-18.el5.centos to resolve CVE-2014-9293, CVE-2014-9294, CVE-2014-9295 and CVE-2014-9296. e) Add support to persist site local parameters in protect_override.py across hotfix installations. NOTES: It is not possible to roll-back the XenServer host installation of HF5 to an earlier hotfix or version of v6.2 (see comments in the roll back section of this document). A new ixgbe Intel 10Gb NIC driver is being included in a supplemental driver disk compatible with HF5. Support additional SFP+ modules is possible with the newer driver. NOTES: Optional use of SSL security with 3rd party signed certificates for eAC access is not supported in this release. Customers can use the everRun SSL security with self-signed certificates option. 6) Hotfix LVL06-- a) L3 PVM Performance may degrade after 5 months. Checkpoint cycles become "Hard Paused". b) Fuse may not install correctly. c) AM may crash during snapshot when there is also an ALINK failure. d) Host may not unevacuate after a host reboot. e) A PVM may not join if ALINKS are not initialized correctly after a host reboot. f) Some PVMs may not sync (L3) or migrated (L2) after HF5 installation. g) xenstore-access logs may show large number of ENOENT entries. h) Add two MMS commands, "pool-query-timers", and "pool-adjust-timers" to query and control time offsets of various timers. i) Include CVE-2015-3456 (VENOM vulnerability) resolution in Hotfix installation. 7) Hotfix LVL07-- a) Include all CVE updates from Citrix XenServer 6.0.2 Hotfixes from XS602E041 to XS602E050. b) A cache issue in XenGuestAgent on the PVM can result in timejumps after a transition, or during steady state. c) Performance may degrade in VM6.2 when ALINK compression is on. d) DOM0 may crash after running a period of time. e) PVM may degrade with a MEMX_APPLICATION_ERROR with RESUME_ERR as the cause. f) Add capability to restore log rotation policy to default. g) The ev pool-query_timers and pool-adjust-timers command can fail when the host timezone is greater than +7 hours or less than -7 hours. 8) Hotfix LVL08-- a) An AM may experience high CPU usage after upgrade to HF6 or HF7 in a multi redirected network interface configuration. b) At the beginning of a new year, the PVM clock may roll-back for a day. c) The pool query timers and adjust timers commands can present erroneous data for time-zone equals to GMT+9:30 (e.g. Darwin, and Adelaide). 9) Hotfix LVL09-- a) Include Citrix XenServer 6.0.2 Hotfixes XS602E057. Update everRun MX components to include the following CVE updates: CVE-2016-7092, CVE-2016-7093, CVE-2016-7094, CVE-2016-7154, CVE-2016-6258, CVE-2016-6259, CVE-2016-3710, CVE-2016-3712, CVE-2016-2107, CVE-2016-2108, CVE-2016-2271, CVE-2016-3158, CVE-2016-3159, CVE-2016-3960, CVE-2016-1571, CVE-2016-0800, CVE-2016-1571, CVE-2016-6258, CVE-2016-6259. b) Fix a grant table copy memory problem which can cause a MXL_LOST_VS_CONNECTION failure. c) Add detection of syslog failure to messages file. d) Remove linkwatch daemon hang during a host shutdown. e) Fix an ISCSI mirrored disk corruption scenario when one member disk fails to come online between two startups. f) Update tzdata database. g) Add additional isolation for rare checkpoint aborting failures. h) MX management is unable to resolve dual Master XenHosts when there is only one FT VM running. i) xapissl Service may hang during normal operation. IMAGES WITH CHANGED FUNCTIONALITY SINCE THE PREVIOUS RELEASE everRun VM: am-6.2HF9.iso everRun VM SOFTWARE REQUIREMENTS Oldest compatible everRun software Version: 6.2.0.9063-RL.EA Newest compatible everRun software Version: 6.2.0.9209-HF.EA STRATUS CONTACT If you need additional information regarding this update release, contact your service provider, or contact Stratus customer support at Phone: +1-602.852.3094 +1-866.763.1813 (USA and Canada) Email: support@stratus.com Web Portal: https://everrun-support.stratus.com INSTALLATION INSTRUCTIONS WARNING: A HOST THAT IS UPGRADED TO HF9 CANNOT BE ROLLED BACK TO HF8 OR ANY VERSION BEFORE HF8. The installation process is a two-step process. The first step is to install HF9 on each host; the next step installs HF9 on each of the protected VMs (PVMs). These instructions explain how to prepare for the installation, how to install the hotfix on the pool master host and the pool member host(s), and how to install the hotfix on a PVM. NOTE: If you have questions about upgrading a more complex multihost pool or other supported configuration, please consult Stratus Customer Support. NOTE: If the Supplemental Driver Disk was used during the original installation of V6.2.0 you may be required to update drivers. During the installation detailed instructions will be provided in the installer screen. When asked to install the current Supplemental Driver Disk follow these steps: 1. Load the CD into the CD-ROM drive of the host 2. On DOM0 console: mount -t iso9660 -o ro /dev/cdrom /mnt 3. cd /mnt 4. ./install.sh 5. This will install all drivers that are on the CD 6. cd 7. umount /mnt PREPARING FOR THE HOTFIX INSTALLATION 1. Download the self-extracting zip file (everRun-6.2HF9.exe) to a shared folder that can be accessed from both the XenServer hosts and Windows PVMs that will be updated to HF9. 2. Extract four files and a folder into the shared folder: Readme_Patch.txt,everRun-6.2HF9-installer.bin, Update_PVM_6.2HF9.exe Setup_QSVC_6.2HF9.exe CitrixHotfixes folder 3. To verify that the software version is correct, issue: ev pool-param-get param-name=everrun-version The everRun version indicates "V6.2.0.9063-RL.EA" OR The everRun version indicates "V6.2.0.9084-HF.EA" OR The everRun version indicates "V6.2.0.9100-HF.EA" OR The everRun version indicates "V6.2.0.9107-HF.EA" OR The everRun version indicates "V6.2.0.9125-HF.EA" OR The everRun version indicates "V6.2.0.9142-HF.EA" OR The everRun version indicates "V6.2.0.9164-HF.EA" OR The everRun version indicates "V6.2.0.9185-HF.EA" OR The everRun version indicates "V6.2.0.9190-HF.EA" 4. Determine possible sources of automatic failover of the pool master, and use the appropriate commands to disable it: ev pool-param-get param-name=master-failover-via If master failover indicates "manual", proceed to the next section If master failover indicates "platform", disable it as follows: xe pool-ha-disable If master failover indicates "everRun", disable it as follows: ev pool-master-failover-disable --force Repeat Step 4 to verify automatic failover is disabled. After you disable the automatic pool management failover mechanism(s), master failover will not occur during the upgrade. INSTALLING THE HOTFIX ON THE MASTER HOST 1. In a XenServer (master) Console window, verify that you are operating on the pool master host by issuing the following command: ev host-list --localhost The host-role indicates "Master". 2. Evacuate PVMs from the master host: ev host-evacuate host-role=Master This may take a minute; when evacuation is complete, you'll see a message, "PVMs have been disabled on host: ." NOTE: If any PVM failed to evacuate, make repairs or shutdown the PVM and issue the host-evacuate command again. To verify evacuation completed successfully: ev host-list host-role=Master params=host-name ev pvm-list host1-name= params=state-host1 ev pvm-list host2-name= params=state-host2 All PVMs listed should indicate "evacuated" for state-host parameters. 3. Migrate or shutdown non-protected VMs that may be running on the master: xe host-evacuate uuid= 4. Log in on each VM that cannot be migrated, and shut it down manually. 5. In the XenServer console window, verify that you are operating on the pool master host by issuing: ev host-list --localhost The indicated host-role should still be "Master". 6. Still in the XenCenter Console window, mount the network share where the hotfix files were located in step 1 of PREPARING FOR THE HOTFIX INSTALLATION: mount -t cifs /// /mnt -o user= At the screen prompt, enter the password for your Windows computer. 7. To execute the hotfix installer file on the master host, enter the following command at the # prompt: bash /mnt/everRun-6.2HF9-installer.bin --install 8. Follow the onscreen instructions to complete the installation. At the end of the installation, a message states that a reboot is required. 9. Unmount the network share. umount /mnt 10. Reboot the host you've just upgraded: ev host-reboot --localhost 11. To confirm that no components remain evacuated or disabled on the master host, issue the following commands on the master host after recovering from the reboot: ev host-list host-role=Master params=host-name ev pvm-list host1-name= params=state-host1 ev pvm-list host2-name= params=state-host2 All PVMs listed should indicate "good" or "standby" for state-host parameters. Note: PVMs that were shutdown before the upgrade indicate "offline" and MUST NOT be started until after the final member host has been upgraded. 12. If necessary unevacuate the host to permit recovery of PVM redundancy: ev host-unevacuate host-role=Master 13. Use the eAC to restart any VM (not PVMs) you may have manually shutdown in step 4. 14. Use the eAC to confirm that all PVM redundancy is recovered, and mirror copies finish before you proceed. All PVMs will be degraded until their mirror copies are complete. NOTE: eAC indicates one warning for the pool because the master is operating at a newer (updated) version of everRun software than all member hosts. INSTALLING THE HOTFIX ON THE MEMBER HOST(S) 1. In a XenServer (member server) Console window, verify that you are operating on a pool member host by issuing the following command: ev host-list --localhost The host-role indicates "Slave". 2. Evacuate PVMs from the member host: ev host-evacuate --localhost This may take a minute; when evacuation is complete, you'll see a message, "PVMs have been disabled on host: ." NOTE: If any PVM failed to evacuate, make repairs or shutdown the PVM and issue the host-evacuate command again. To verify evacuation completed successfully: ev host-list --localhost params=host-name ev pvm-list host1-name= params=state-host1 ev pvm-list host2-name= params=state-host2 All PVMs listed should indicate "evacuated" for state-host parameters. 3. Migrate non-protected VMs that may be running on the member host: ev host-list --localhost xe host-evacuate uuid= 4. Log in on any VM that cannot be migrated and shut it down manually. 5. In a Console window, verify that you are operating on a pool member host by issuing: ev host-list --localhost The indicated host-role should still be "Slave". 6. Still in the XenCenter Console window, mount the network share where the hotfix files were located in step 1 of PREPARING FOR THE HOTFIX INSTALLATION: mount -t cifs /// /mnt -o user= At the screen prompt, enter the password for your Windows computer. 7. To run the hotfix installer file from the member host, enter the following command at the # prompt: bash /mnt/everRun-6.2HF9-installer.bin --install 8. Follow the onscreen instructions to complete the installation. At the end of the installation, a message states that a reboot is required. 9. Unmount the network share. umount /mnt 10. Reboot the host you've just upgraded: ev host-reboot --localhost 11. To confirm that no components remain evacuated or disabled on the member host, issue the following commands on the member host after recovering from the reboot. ev host-list --localhost params=host-name ev pvm-list host1-name= params=state-host1 ev pvm-list host2-name= params=state-host2 All PVMs listed should indicate "good" or "standby" for state-host parameters. Note: PVMs that were shutdown before the upgrade indicate "offline" and MUST NOT be started until after the final member host has been upgraded. 12. If necessary unevacuate the host to permit recovery of PVM redundancy: ev host-unevacuate --localhost 13. Use the eAC to restart any VM (not PVMs) you may have manually shutdown in step 4. 14. Use the eAC to confirm that all PVM redundancy is recovered, and mirror copies finish before you proceed. All PVMs will be degraded until their mirror copies are complete. NOTE: eAC indicates one warning for the pool if additional member hosts need to be upgraded. After all member hosts are operating at the updated version of everRun software, the warning is removed from the pool. Repeat this section for each additional member host in the pool AFTER UPDATING THE FINAL MEMBER HOST 1. After the hotfix has been installed on the final member host, re-enable automatic failover of the pool master: If you disabled everRun master failover in a previous step, re-enable it now: ev pool-master-failover-enable OR If you are using XenServer HA re-enable it: xe pool-ha-enable (or use XenCenter). This completes your hotfix update of the everRun software. Continue with UPDATING INDIVIDUAL PVMs, below. UPDATING INDIVIDUAL PVMs When installing the hotfix update, you can choose whether or not to update existing protected VMs. This section describes the PVM update procedure. For all fixes to be applied, updating each PVM is required. NOTE: You are encouraged to update all PVMs. Perform the update only when it is appropriate to reboot the PVM to facilitate the update. To upgrade individual PVMs: 1. Ensure that the Update_PVM_6.2HF9.exe file is available on a network share and is visible to Windows (PVM). In most cases, the network share is where the hotfix files were located in step 1 of PREPARING FOR THE HOTFIX INSTALLATION. 2. Log in to the Windows Administrator account on the PVM. 3. Execute the PVM update program on the protected Windows environment: Establish a network connection from the PVM to the shared location, then navigate to the shared location and double-click the Update_PVM_6.2HF9.exe file. OR Copy the update file Update_PVM_6.2HF9.exe to a local folder on the PVM and run it locally. 4. When the file opens, a pop-up screen warns that the PVM will restart immediately after the installation. 5. Click OK to open the Setup wizard, then click Next if you want to continue. A progress bar displays during installation, and the installer creates an un-installation capability that can be used to restore (roll back to) the previous version of the PVM. At the end of the installation, the installer communicates with the Marathon management service (MMS) to request a reboot of the PVM. The MMS configures the PVM to use the updated everRun availability manager(AM) and restarts the PVM. The everRun PVM update is now complete. 6. Perform steps 2-5 as needed to install the hotfix on additional PVMs. UPDATING THE QUORUM SERVICE NOTE: If the Quorum Service has been updated using the Setup_QSVC file from V6.1RL or later hotfix, it is not necessary to update again. If the Quorum Service requires updating then use the Setup_QSVC file from this hotfix release. 1. On the master console window verify that all PVMs states are good by issuing: ev pvm-list params=state Repair any PVM that is not in a good state. 2. Issue the following command: ev pool-list params=all | grep quorum If the "quorum-services-mode" shows enabled, continue to the next step. Otherwise ignore the rest of the QSS update steps. 3. The results from step 2 should show both "quorum-server-preferred-status" and "quorum-server-alternate-status" are valid. If the "quorum-server-alternate-status" is not valid install an alternate quorum server before continuing. Then restart the Quorum Service update procedure from step 1. Note both the quorum-server-preferred and quorum-server-alternate IP addresses. 4. Validate the quorum services are also reachable from the PVMs by issuing the command: ev pvm-list params=all | grep quorum The command results should show "quorum-server-alternate-host1-state", "quorum-server-alternate-host2-state", "quorum-server-preferred-host1-state", and "quorum-server-preferred-host2-state" are all reachable. 5. Copy Setup_QSVC_6.2HF9.exe to the alternate quorum server, whose IP address was noted in step 3. 6. Run Setup_QSVC_6.2HF9.exe to update the Quorum service. 7. Back on the master console window, issue the commands: ev pvm-list params=state ev pool-list params=all | grep quorum ev pvm-list params=all | grep quorum The results show that all PVMs state are good, both "quorum-server-preferred-status" and "quorum-server-alternate-status" are valid, and all quorum server host states are reachable. 8. Repeat steps 6, 7 and 8 to update QSS on the preferred quorum server whose IP address matches the preferred IP address noted in step 3. This completes the installation of the hotfix kit. Should it be necessary to reverse the update -- refer to ROLLING BACK THE HOTFIX INSTALLATION, below. NOTE: ROLLING BACK THE QUORUM SERVICE Should it become necessary to rollback the Quorum Service, please consult Stratus Customer Support. ROLLING BACK THE HOTFIX INSTALLATION Rollback removes the latest installed version of the everRun software. WARNING: A HOST THAT IS UPGRADED TO HF9 CANNOT BE ROLLED BACK TO HF8 OR ANY VERSION BEFORE HF8. NOTE: PVMs that were not protected until after the hotfix update cannot be rolled back. PVMs initially protected at the hotfix level would need to be unprotected instead. NOTE: PVMs that are operating with a Marathon, OEM (free) XenServer license and have the "cores-per-socket" function enabled must be restarted once more after completing the rollback on all XenServer hosts. ROLLING BACK PROTECTED VMs To roll back a protected VM: 1. Log in to the Windows Administrator account on the PVM. 2. Open the Windows Control Panel for the protected VM. Choose "Add or Remove Programs" for a Windows 2003 PVM. Choose "Programs and Features" for a Windows 2008 PVM. 3. When the list populates, you will only be able to remove the most recent version of the software. In this case, you will remove version 6.2.0.9209 Hotfix 9. 4. Click the "Remove" button for a Windows 2003 PVM or the "Uninstall" button for a Windows 2008 PVM to proceed. The rollback procedure and pop-up screens are very similar to the update installer. Follow the instructions displayed in the dialog boxes to uninstall the hotfix. When the wizard finishes, the software restarts the PVM. 5. If you plan to also rollback to the previous version of everRun on the XenServer hosts, repeat steps 1-4 on every PVM that has the HF9 kit installed. NOTE: Remember that some PVMs cannot be rolled back. If a PVM has been protected only after the hotfix was installed, it must be unprotected, instead of rolled back. Note: PVMs can be rolled back to earlier version or unprotected, but hosts cannot be rolled back to HF8 or any version before HF8. ROLLING BACK THE INSTALLED everRun VERSION ON THE MASTER HOST Hosts running HF9 cannot be rolled back to HF8 or any version before HF8.