Author: Carl Stalhood

Navigation

• Changelog
• Overview
• Prepare New Appliances
• Export/Import Files
• Export Configuration
• Cutover

Changelog

• 2024 May 22 – added link to NetScaler Docs Migrate the configuration of a NetScaler instance across SDX appliances
• 2022 Sep 11 – added a note about question marks ( ? ) in the config
• 2022 Sep 9 – added /var/download
• 2022 Sep 1 – added /var/netscaler/locdb
• 2022 Aug 10 – check management services to make sure TLS12 is not disabled.
• 2021 Sept 8 – added KEK key files

Overview

This article details one method of migrating a Citrix NetScaler ADC configuration from an old appliance pair to a new appliance pair while preserving the IP addresses. It requires a brief outage to move the IP addresses from the old pair to the new pair. Keeping the existing IP addresses avoids needing to make any changes to DNS, NAT, firewall, and routing.

• NetScaler Docs Migrate the configuration of a NetScaler instance across SDX appliances details using HA sync to migrate a NetScaler instance.
• Another method is to migrate one VIP at a time. However, you won’t be able to move the SNIP, so you might need new firewall rules for the new appliance pair.

Notes:

• The new appliances must be connected to the same networks as the old appliances. If not, then the IP addresses need to change.
• If you migrate the entire configuration at once, then the SNIPs can also be migrated.
  • The new appliances won’t have a SNIP until cutover time, so DNS will not work on the new appliances until cutover.
• It’s much easier and quicker if you don’t migrate the NSIPs, so the new appliances will need new NSIPs.
• You can use temporary SNIPs to validate connectivity on the new appliances.
• Firmware versions can usually be different on the new appliances as compared to the old appliances.
  • If the old appliances are 10.5 or older, and if you have NetScaler Gateway with a customized logon page, then you’ll have to redo the customization as a Portal Theme on the new appliances.
  • GSLB Sites (MEP) can usually have different firmware versions.

Here’s a summary of the migration process. See the rest of this article for details.

1. Prepare new appliances by connecting them to the network and pairing them together.
2. Export files (e.g., SSL certificates) from old appliances and import to new appliances.
3. Export configuration from old appliance and modify the configuration for later import to the new appliances.
4. During a brief outage window, disable the old appliances (power off or disconnect), then run the config on the new appliances.
5. Test the new appliances. If a problem is discovered, then you can roll back by disabling the new appliances and re-enable the old appliances.

Prepare New appliances

If VPX (not on SDX):

1. Import new VPX appliances into your hypervisor.
   • If vSphere, change NIC to VMXNET3.
2. Configure anti-affinity for VPX appliances in the same hypervisor cluster.
3. Configure new management IP Addresses in the virtual machine console.
   • Default gateway should be on the NSIP subnet. The default gateway will be changed to a data VLAN during the cutover.
4. In the Welcome Wizard, don’t configure any SNIPs. And DNS won’t work without SNIP.
5. Install licenses on the new VPX appliances.
   • If pooled licensing, enter ADM IP address instead of DNS name to avoid needing DNS/SNIP configuration.
6. Optional: configure VPX CPU Yield.

If VPX on SDX:

1. Configure switch ports for LACP port channel and VLAN trunking.
2. Connect the physical SDX hardware to port channels or individual interfaces.
3. In SDX SVMs, create Channels and confirm that the channels are distributing.
4. In SDX SVMs, upload desired VPX XVA firmware virtual appliance.
5. In SDX SVMs, create new instance and connect the instance to channels.
   • New management IP address for each instance.
   • Default gateway should be on the NSIP subnet. The default gateway will be changed to a data VLAN during the cutover.
   • Enable NSVLAN if the VLAN trunk doesn’t have a native untagged VLAN.
   • For HA pair, create each instance on separate SDX appliances.

If MPX:

1. Configure switch ports for LACP port channel and VLAN trunking.
2. Connect the physical MPX hardware to port channels or individual interfaces.
3. Use LCD, LOM, serial cable, or 0/1 Ethernet crossover to configure new NSIP for each MPX appliance.
   • Default gateway should be on the NSIP subnet. The default gateway will be changed to a data VLAN during the cutover.
   • Enable NSVLAN if the VLAN trunk doesn’t have a native untagged VLAN.
4. In the Welcome Wizard, don’t configure any SNIPs. And DNS won’t work without SNIP.
5. Upgrade firmware to desired version.
6. At System > Network > Interfaces, configure LACP and confirm that the channels are distributing.
7. At System > Network > Interfaces, disable all interfaces that are not plugged in.
8. Install licenses on the new MPX appliances.
   • If pooled licensing, enter ADM (or ADM Agent) IP address instead of DNS name to avoid needing DNS/SNIP configuration.

For all ADC models:

1. Create new DNS names for the new NSIP IP Addresses.
2. Ensure you can SSH (e.g., Putty) and SCP (e.g., WinSCP) to the new NSIPs and old NSIPs.
3. In the Welcome Wizard, don’t configure any SNIPs. And DNS won’t work without SNIP.
4. Configure NTP and Time Zone on each node before you pair them together.
   • Enter NTP IP address instead of DNS name to avoid needing DNS/SNIP configuration.
   • From CLI, run shell, then run ntpdate command with NTP Server IP address to sync the local time with NTP server. NTP Sync needs to be disabled before you can run this command.
5. Pair the two new appliances together.
6. Configure a new password on the RPC Nodes at System > Network > RPC. Do it on both nodes.
7. Set NSIP to Secure Access Only at System > Network > IPs, edit NSIP, checkbox at bottom of page. Do it on both nodes.
8. Configure VLANs without SNIPs.
9. For dedicated management subnet, configure PBRs.
10. If you have channels that are not configured with a native, untagged VLAN, then configure tagged High Availability Heartbeats for non-NSIP interfaces by marking one VLAN as untagged and enabling the tagall option on the channel.
    • From CLI, run show ha node to confirm that there are no interfaces on which heartbeats are not seen.
11. Use new temporary SNIP addresses to test VLAN connectivity
    • Bind the temporary SNIP to VLAN and channel and try to ping it.
    • Remove temporary SNIP when done testing.
12. Install custom Management certificates that match new NSIP DNS names.
13. If Default SSL Profile is enabled on old appliance, then enable it on new appliance pair.
14. Add new pair to Citrix ADM.
15. Add new pair to SNMP Manager.
16. Add new pair to ControlUp Monitoring or any other Citrix NetScaler ADC monitoring tool.
17. If any RADIUS authentication, and if RADIUS is not load balanced, then add the new NSIPs as RADIUS Client IP Addresses on the RADIUS servers.
    • If RADIUS is load balanced, then SNIP is the RADIUS Client IP, and the SNIP will be moved over during cutover.

Export/Import Files

Enact a change freeze to prevent any changes to the old appliances so that the files and Running Configuration don’t have to be downloaded again.

A full backup of the ADC will contain every file you need to upload to the new appliances.

• On Windows, you’ll need a tool like 7-zip to extract the .tgz file.
  
• Inside the backup folder are the files that you’ll need to upload to the new appliances.
  

Use WinSCP to transfer the following files from the old appliances to the same place on the new appliances.

• Certificate files and DH files from /nsconfig/ssl
  • UNIX is case sensitive but Windows is not. While downloading, if you see a message about overwriting files, that’s probably because you have two files with the same name but different cases.
  • Don’t upload the files that start with ns- (e.g. ns-server.cert) since those are management certificate files.
    
• LDAP certificate verification files from /nsconfig/truststore, if exists.
• KEK files from /flash/nsconfig/keys. This is more likely to be needed when moving a 13.0 and newer configuration.
  
• Custom monitors (e.g. nsldaps.pl) from /nsconfig/monitors, if exists. Grant execute permission to the uploaded .pl files.
• Login Schemas from /nsconfig/loginschema.
• /var/download has Bot signatures, WAF signatures, and Responder HTML Page imports. Signatures have their own method of export/import. Import these objects manually using the ADC GUI.
  
• Location database from /var/netscaler/locdb.
• Custom Portal Themes from /var/netscaler/logon/themes, if exists.
  • There’s no need to download the built-in themes. Only download the non-built-in themes.
  • On the new appliances, create Portal Themes with the same names and same template as the old appliances.
  • Then upload the Portal Theme files to the new appliances and overwrite the default files.
• VPN bookmarks from /var/vpn/bookmark, if exists.
• Gateway logon page customizations from /var/vpn/vpn, if any.
  • Try to redo these customizations as Portal Themes instead of uploading to the new appliances. The .js files in NetScaler 11.0 and newer are quite different from NetScaler 10.5 and older.
• /nsconfig/rc.netscaler if it contains anything other than ntpd.
  • If this file contains customizations for the Gateway logon page, then try to redo them as a Portal Theme.
  • After uploading this file to the new appliances, reboot the new appliances so the commands are executed. Or you can manually run rc.netscaler after you chmod +x to the file.
• /nsconfig/nsafter.sh if it exists.
  • If this file contains customizations for the Gateway logon page, then try to redo them as a Portal Theme.
  • After uploading this file to the new appliances, reboot the new appliances so the commands are executed. Or you can manually run nsafter.sh after you chmod +x to the file.
• SSH Banners, like /nsconfig/issue.net, /nsconfig/issue, and /nsconfig/motd.

To test the Portal Themes, you can create a temporary Gateway Virtual Server with a temporary VIP. Remove the temporary configuration when done testing.

Export Configuration

Export running configuration from the old appliances by going to System > Diagnostics > Running Configuration and then click the link to Save text to a File.

Edit the downloaded nsrunning.conf file to prepare it to be executed on the new appliances:

• Remove all NSIP and hostname lines, including IPv6 and MAC addresses. These lines start with the following:
  • set ns config -IPAddress
  • set lacp -sysPriority 32768 -mac
  • set ns hostName
  • add ns ip -type NSIP – leave the other “add ns ip” commands, especially the SNIP (-novserver) commands
  • add ns ip6 -type NSIP
• Remove the nsroot line that starts with:
  • set system user nsroot
• Remove all interface commands that start with:
  • set interface
• Remove all channel commands that start with:
  • add channel
  • set channel
• Fix VLAN binding commands for new channels/interfaces.
• Edit set snmp mib command, if it exists.
  • set snmp mib
• Remove PBR commands for NSIP that start with. Or you can adjust the commands for the new NSIPs.
  • add ns pbr NSIP
• Remove High Availability / Cluster commands that start with:
  • add HA node 1
• Remove RPC Node commands that start with:
  • set ns rpcNode
• Review the add route commands to make sure they are correct for the new appliances.
• Search for set ssl service and look for the ones that start with ns (e.g., nshttps-127.0.0.1-443). If any of them have the parameter -tls12 DISABLED, then remove that parameter.
• Search the entire config file for question marks ( ? ). Question marks do not import correctly from the CLI so you’ll have to fix them in the GUI after the import is complete.

Make sure the name of the modified configuration file is all lower case and doesn’t have any spaces. The batch command doesn’t work if there are upper case letters or spaces.

Upload the modified config file to /tmp on the new primary appliance.

Optional: review the SSL commands and look for any bindings of custom ciphers to SSL vServers. After the cutover, the new appliances will have your custom ciphers but unfortunately will also have the DEFAULT ciphers. To speed up the post-cutover process, you can pre-create the CLI commands that unbind the DEFAULT cipher groups. Here’s an example command:

unbind ssl vserver MySSLvServer -cipherName DEFAULT

Cutover

Decide how you will disable the old appliances. Here are some options:

• Shutdown MPX Ethernet interfaces on network switch – if rollback, it’s easy to unshut the interfaces
• Power off old VPX appliances, including on SDX – if rollback, it’s easy to power them on again
• If old appliances have dedicated management network (no SNIP on management network), then you can to go to System > Network > Interfaces and disable them – if rollback, easy to re-enable
• Power off old MPX appliances – if rollback, is somebody available in the data center to power them on again?

Before outage – Before the outage window, do the following on the old appliances:

• Backup the old appliances and download the backup files.
• Review the old appliances and record what’s up or down – After cutover, you’ll confirm up/down is the same on the new appliances.
  • Look in Traffic Management > Load Balancing > Virtual Servers for up/down status, including the number of UP services.
  • Go to Authentication > Dashboard to see which authentication servers are reachable

Outage – During outage window, do the following:

• Disable the old appliances.
• SSH to the new primary appliance and run the batch command to run the commands in your modified configuration file. For example:

  batch -fileName /tmp/nsrunningmodified.conf -outfile /tmp/outfile.txt

• Review the outfile for errors. Some warnings can be ignored. You can use WinSCP to access the out file. Or cat it.

After running the batch command, do the following to fix the imported configuration.

• Default Route – Remove extra Default Route, if any.
  • System > Network > Routes. If there are two 0.0.0.0 routes, then remove the one for the NSIP subnet. Check it on both nodes.
• Question Marks – for any line that had a question mark ( ? ), fix the config in the GUI. The question marks are probably missing.
• VIPs are UP? – Ensure VIPs (Load Balancing, Content Switching, Gateway) are up or down as noted on the old appliance.
  • Traffic Management > Load Balancing > Virtual Servers.
• Go to Authentication > Dashboard and ensure the status matches the old appliance.
  • If needed, fix LDAP Password in load balancing monitor and in LDAP Policies/Servers.
  • If needed, fix RADIUS Secret in load balancing monitor and in RADIUS Policies/Servers.
• Portal Themes – Point your browser to the Citrix Gateway logon pages and verify Portal Themes customizations are correct.
• GSLB MEP – Set GSLB RPC nodes with Secure checked.
  • System > Network > RPCs. Check the Secure box. Change RPC password.
• DEFAULT ciphers – If Default SSL Profile is enabled, then remove DEFAULT cipher group from SSL Profiles that have custom Ciphers.
  • System > Profiles > SSL tab. Edit each profile. In SSL Ciphers section, remove DEFAULT if it’s not the only cipher.
• If Default SSL Profile is not enabled, then remove DEFAULT cipher group from all SSL Virtual Servers that have custom ciphers.
  • E.g. Traffic Management > Load Balancing > Virtual Servers. Edit an SSL Virtual Server. Scroll down to the SSL Ciphers section. If you see both DEFAULT and something else, then remove DEFAULT.
• SSL Labs – Use SSL Labs to test externally reachable VIPs.
• HA failover – Do a High Availability failover and verify that the same VIPs and authentication servers are UP on new primary appliance.