VMware Horizon Load Balancing – NetScaler / Citrix ADC

Last Modified: Jul 25, 2024 @ 12:06 pm

135 Comments

Navigation

Use this procedure to load balance VMware Unified Access Gateway.

Change Log

Overview

To simplify this post, this post is focused on Unified Access Gateway, which is the replacement for Horizon Security Servers.

For load balancing of other Horizon components:

  • Internal Horizon Connection Servers – This is standard load balancing on SSL_BRIDGE protocol, port 443, and Source IP persistence. See the CLI commands for a sample configuration.
    • If you enabled the Secure Gateways (PCoIP, Blast) on the Connection Servers, then load balance the Connection Servers using the same procedure as load balancing UAGs.

UAG appliances vs Horizon Security Servers

There are two VMware-provided remote access solutions for Horizon View:

Unified Access Gateway appliances are preferred over Horizon Security Servers for the following reasons:

  • No need to pair with internal Connection Servers, which simplifies the configuration.
  • Linux appliance instead of Windows server.
  • Authentication can be offloaded to the Unified Access Gateway. This includes: Smart Cards, RSA, and RADIUS.
  • Blast Extreme Adaptive Transport (BEAT) in Horizon 7.1 and newer only works with Unified Access Gateway 2.9 and newer. Security Server and older Access Points don’t work.
    • VMware Blog Post Deep Dive into VMware Horizon Blast Extreme Adaptive Transport – Blast Extreme Adaptive Transport is enabled by default in VMware Horizon View 7.1 and Horizon Client 4.4. If the clients are connecting from outside the demilitarized zone (DMZ), you would also need to have VMware Unified Access Gateway (not Security Server) to take full advantage of the new transport. The adaptive transport will automatically sense the network for UDP availability and will fallback to legacy Blast TCP if UDP is not available.

Here is a typical Unified Access Gateway architecture:

  • Two Internal Connection Servers – these need to be load balanced on an internal VIP on TCP 443. Internal users connect to the internal VIP.
    • Instructions for load balancing the internal Connection Servers are not detailed in this post. Instead, see the CLI Commands.
  • Two DMZ Unified Access Gateway (Access Point) appliances – these need to be load balanced on a DMZ VIP on several ports. External users connect to the DMZ VIP.
    • Unified Access Gateway appliances connect to the internal Load Balancing VIP for the internal Connection Servers using HTTPS protocol
    • Unified Access Gateway appliances connect directly to Horizon Agents using Blast or PCoIP protocol.

During UAG configuration, you specify the address of the internal Connection Servers. The address you enter should be a DNS name that resolves to an internal load balancing VIP for the Connection Servers.

If you have split DNS, you can use the same DNS name for both external UAG access, and for internal Connection Server load balancing. Externally, configure the DNS name to resolve to the UAG Load Balancing VIP. Internally, configure the DNS name to resolve to the internal VIP that load balances the Connection Servers.

For Cloud Pod Architecture, configure separate VIPs in each datacenter. Then configure NetScaler Citrix ADC GSLB to resolve a single DNS name to multiple VIPs.

Protocols/Ports

To support Blast Extreme, PCoIP, and HTML Blast connectivity, the following ports must be load balanced to the UAGs:

  • TCP 443
  • UDP 443
  • TCP 4172
  • UDP 4172
  • TCP 8443
  • UDP 8443

The initial connection to UAG is always TCP 443 (HTTPS). If a user is load balanced on port 443 to a particular UAG, then the connection on UDP 4172 must go the same UAG. Normally load balancing persistence only applies to a single port number, so whatever UAG was selected for port 443, won’t be considered for the 4172 connection. But in NetScaler Citrix ADC, you can configure a Persistency Group to use a single persistency across multiple load balancing Virtual Servers with different port numbers. In F5, you configure Match Across Services, as detailed by Aresh Sarkari at Persistence Profile – F5 LTM Load Balancing for VMware Unified Access Gateway Appliance.

This topic primarily focuses on NetScaler Citrix ADC GUI configuration. Alternatively, you can skip directly to the CLI commands.

Load Balancing Monitors

Users connect to Unified Access Gateway appliances on multiple ports: TCP 443, UDP 443, TCP 8443, UDP 8443, TCP 4172, and UDP 4172. Create Load Balancing Monitors for each port number. Since UDP can’t be easily monitored, use TCP monitors as substitutes for UDP. That means you only need four monitors:

  • TCP 443 – HTTPS
    • HEAD /favicon.ico or GET /favicon.ico. This string can detect if UAG is in Quiesce mode, or if Connection Server is disabled in Horizon Console. See VMware 56636 for supported monitoring strings.
  • TCP 4172
  • TCP 8443

The procedure for configuring monitors changed in NetScaler 12.0 build 56 and newer.

SSL (443) Monitor for Connection Server Health

  1. On the left, expand Traffic Management, expand Load Balancing, and click Monitors.
  2. On the right, click Add.
  3. Name it Horizon-SSL or similar.
  4. In the Type field, click where it says Click to select.
  5. In the Monitor Types list, click the circle next to HTTP.
  6. Scroll up and click the blue Select button.
  7. To reduce UAG CPU, VMware recommends setting the Interval to 30 seconds.
  8. In the HTTP Request field, enter GET /favicon.ico. See VMware 56636 for supported health monitoring requests.
  9. Check the box next to Secure.
  10. Scroll down and click Advanced Parameters to expand it.
  11. In the Advanced Parameters section, in the Destination Port field, enter 443.
  12. Scroll down to the TROFS Code field and enter 503. When UAG is in Quiesce mode, TROFS will cause the service to transition to out of service allowing existing connections instead of directly going down and killing existing sessions. (h/t Henry Heres)
  13. Scroll down, and click Create.

PCoIP (4172) Monitor

  1. On the left, expand Traffic Management, expand Load Balancing, and click Monitors.
  2. On the right, click Add.
  3. Name it Horizon-PCoIP or similar.
  4. For the Type field, click where it says Click to select and select TCP from the Monitor Types list.
  5. Scroll down to Advanced Parameters, click Advanced Parameters to expand it, and then enter 4172 in the Destination Port field. This forces the monitor to connect to port 4172 even if the monitor is bound to a service group that is configured for a different port number.
  6. Scroll down, and click Create.

Blast (8443) Monitor

  1. On the left, expand Traffic Management, expand Load Balancing, and click Monitors.
  2. On the right, click Add.
  3. Name it Horizon-Blast or similar.
  4. For the Type field, click where it says Click to select and select TCP from the Monitor Types list.
  5. Scroll down to Advanced Parameters, click Advanced Parameters to expand it, and then enter 8443 in the Destination Port field. This forces the monitor to connect to port 8443 even if the monitor is bound to a service group that is configured for a different port number.
  6. Scroll down, and click Create.

UAG DDoS

NetScaler Citrix ADC Monitors might trigger UAG’s DDoS protection. To stop this: (source = UAG breaks after a few days. They break 100% of the time. at VMware Discussions)

  1. Point your browser to the UAG appliance’s admin interface using https, port 9443 and path /admin.
  2. Login to the admin interface.
  3. On the right, under Configure Manually, click Select.
  4. In the Advanced Settings section, click the gear icon for System Configuration.
  5. Scroll down.
  6. Change Request Timeout to 0.
  7. Change Body Receive Timeout to 0.
  8. Click Save.

Load Balancing Servers

Create Load Balancing Server Objects for the DMZ Unified Access Gateway appliances.

  1. On the left, expand Traffic Management, expand Load Balancing, and click Servers.
  2. On the right, click Add.
  3. Enter a descriptive server name, usually it matches the actual appliance name.
  4. Enter the IP address of a Unified Access Gateway appliance.
  5. Enter comments to describe the server. Click Create.
  6. Continue adding Unified Access Gateway appliances.

Load Balancing Service Groups

Overview

Since there are six protocol/ports to UAG, there will be six service groups – one for each protocol/port:

  • TCP 443 – SSL_BRIDGE
  • UDP 443
  • UDP 4172
  • TCP 4172
  • TCP 8443 – SSL_BRIDGE
  • UDP 8443

Users will initially connect to TCP port 443, and then must be redirected to one of the other ports on the same UAG appliance that was initially used for the TCP 443 connection. If TCP 443 is up, but UDP 4172 is down on the same appliance, then you probably TCP 443 to go down too. To facilitate this, bind all three port number monitors to the TCP 443 service. If any of the bound monitors goes down, then TCP 443 is also taken down.

  • Only the TCP 443 service group needs to monitor all port numbers.
  • Other port number service groups only need to monitor that specific port number. For example, the TCP 8443 Service Group should monitor port TCP 8443.
  • Since UDP is difficult to monitor, the UDP Service Groups will monitor the equivalent TCP port. For example, the UDP 4172 Service Group will monitor TCP 4172. This isn’t the best option, but it’s better than ping.

TCP 443 Load Balancing Service Group

  1. On the left, expand Traffic Management, expand Load Balancing, and click Service Groups.
  2. On the right, click Add.
  3. Give the Service Group a descriptive name (e.g. svcgrp-Horizon-SSL).
  4. Change the Protocol to SSL_BRIDGE.
  5. Click OK to close the Basic Settings section.
  6. On the left, in the Service Group Members section, click where it says No Service Group Member.

    1. Change the selection to Server Based.
    2. In the Select Server field, click where it says Click to select.
    3. Select the Unified Access Gateway appliances you created earlier, and then at the top of the page click the blue Select button.
    4. In the Port field, enter 443, and click Create.
  7. Click OK to close the Service Group Members section.
  8. On the right, in the Advanced Settings column, click Monitors to move it to the left.
  9. On the left, at the bottom of the page, in the Monitors section, click where it says No Service Group to Monitor Binding.

    1. In the Select Monitor field, click where it says Click to select.
    2. Click the circle next to the Horizon-SSL monitor, and then at the top of the page click the blue Select button.
    3. Click Bind.
  10. For load balancing UAGs, this Service Group should monitor all port numbers so that if any of the port numbers are down then the entire server should no longer receive connections. If load balancing internal Connection Servers, then you don’t need to bind any more monitors since the PCoIP Gateway and Blast Gateway are usually disabled. To bind more monitors, on the left, click where it says 1 Service Group to Monitor Binding.

    1. Click Add Binding.
    2. In the Select Monitor field, click where it says Click to select.
    3. Click the circle next to the Horizon-PCoIP monitor, and then at the top of the page click the blue Select button.
    4. Then click Bind.
    5. Repeat these steps to bind the Horizon-Blast monitor. Unfortunately you can only bind one monitor at a time.
      • If any of these monitors goes down, then the UAG is taken offline.
    6. Click Close.
  11. To verify the monitors, on the left, higher up the page in the Service Group Members section, click the line that says # Service Group Members.

    1. Right-click one of the members, and click Monitor Details.
    2. The Last Response should indicate Success. If you bound multiple monitors to the Service, then the member will only be UP if all monitors succeed.

      • UAG has a Quiesce mode, which tells the load balancer to stop sending it connections. You can enable Quiesce mode in the UAG’s System Configuration menu.
      • Connection Servers can be disabled.
      • If Quiesce mode is enabled on the UAG, or if the Connection Server is disabled, then the member of the service group will show Going Out of Service. You can see this by right-clicking on the Server Group and clicking Manage Members.

    3. Click Close when done.
  12. Then click Done to finish creating the Service Group.

Other Ports Load Balancing Service Groups

Here are general instructions for the other Horizon UAG load balancing service groups.

  1. On the left, go to Traffic Management > Load Balancing > Service Groups.
  2. On the right, click Add.
  3. Name it svcgrp-Horizon-UDP443 or similar.
  4. Change the Protocol to UDP. Click OK to close the Basic Settings section.
  5. On the left, click where it says No Service Group Member.

    1. Change the selection to Server Based, and then click Click to select.
    2. Select your multiple Unified Access Gateway appliances, and then at the top of the page click the blue Select button.
    3. Enter 443 as the Port. Click Create.
  6. Click OK to close the Service Group Members section.
  7. On the right, in the Advanced Settings column, click Monitors to move it to the left.

    1. On the left, in the Monitors section, click where it says No Service Group to Monitor Binding.
    2. For the Select Monitor field, click where it says Click to select.
    3. Select the Horizon-SSL monitor, click Select, and then click Bind. Since we don’t have a UDP monitor, we’re binding the TCP monitor instead.
  8. Click Done to finish creating the Service Group for UDP 443.
  9. Add another Service Group for PCoIP on TCP 4172.
    1. Name = svcgrp-Horizon-PCoIPTCP or similar.
    2. Protocol = TCP
    3. Members = multiple Unified Access Gateway appliances.
    4. Port = 4172.
    5. Monitors = Horizon-PCoIP. You can add the other monitors if desired.
  10. Add another Service Group for PCoIP on UDP 4172.
    1. Name = svcgrp-Horizon-PCoIPUDP or similar.
    2. Protocol = UDP
    3. Members = multiple Unified Access Gateway appliances
    4. Port = 4172.
    5. Monitors = Horizon-PCoIP. You can add the other monitors if desired.
  11. Add another Service Group for SSL_BRIDGE 8443.
    1. Name = svcgrp-Horizon-TCP8443 or similar.
    2. Protocol = SSL_BRIDGE
    3. Members = multiple Unified Access Gateway appliances
    4. Port = 8443.
    5. Monitors = Horizon-Blast. You can add the other monitors if desired.
  12. Add another Service Group for UDP 8443 (Blast Extreme in Horizon 7).
    1. Name = svcgrp-Horizon-UDP8443 or similar.
    2. Protocol = UDP
    3. Members = multiple Unified Access Gateway appliances
    4. Port = 8443.
    5. Monitors = Horizon-Blast. You can add the other monitors if desired.
  13. The six service groups should look something like this:

Load Balancing Virtual Servers

Unified Access Gateway appliances listen on multiple ports so you will need separate load balancers for each port number. Here is a summary of their Virtual Servers, all listening on the same Virtual IP address:

  • Virtual Server on SSL_BRIDGE 443 – bind the SSL_BRIDGE 443 service group.
  • Virtual Server on UDP 443 (Horizon 7) – bind the UDP 443 service group.
  • Virtual Server on UDP 4172 – bind the PCoIP UDP service group.
  • Virtual Server on TCP 4172 – bind the PCoIP TCP service group.
  • Virtual Server on SSL_BRIDGE 8443 – bind the SSL_BRIDGE 8443 service group.
  • Virtual Server on UDP 8443 (Horizon 7) – bind the UDP 8443 service group.

Do the following to create the Virtual Servers:

  1. On the left, under Traffic Management > Load Balancing, click Virtual Servers.
  2. On the right, click Add.
  3. In the Basic Settings section:
    1. Name it lbvip-Horizon-SSL or similar.
    2. Change the Protocol to SSL_BRIDGE.
    3. Specify a new VIP. This one VIP will be used for all of the Virtual Servers.
    4. Enter 443 as the Port.
  4. Click OK to close the Basic Settings section.
  5. On the left, in the Services and Service Groups section, click where it says No Load Balancing Virtual Server ServiceGroup Binding.

    1. Click where it says Click to select.
    2. Click the circle next to the Horizon-SSL Service Group, and then at the top of the page click the blue Select button.
    3. Click Bind.
  6. Click Continue to close the Services and Service Groups section.
  7. Then click Done to finish creating the Load Balancing Virtual Server. Persistence will be configured later.
  8. Create another Load Balancing Virtual Server for UDP 443. You can right-click the existing Load Balancing Virtual Server and click Add to copy some settings.

    1. Name = lbvip-Horizon-UDP443
    2. Same VIP as the TCP 443 Load Balancer.
    3. Protocol = UDP, Port = 443
    4. Service Group Binding = the UDP 443 Service Group

  9. Create another Load Balancing Virtual Server for PCoIP UDP 4172:
    1. Name = lbvip-Horizon-PCoIPUDP
    2. Same VIP as the 443 Load Balancer.
    3. Protocol = UDP, Port = 4172
    4. Service Group Binding = the PCoIP UDP Service Group.

  10. Create another Load Balancing Virtual Server for PCoIP TCP 4172:
    1. Name = lbvip-Horizon-PCoIPTCP
    2. Same VIP as the 443 Load Balancer.
    3. Protocol = TCP, Port = 4172
    4. Service Group Binding = the PCoIP TCP Service Group

  11. Create another Load Balancing Virtual Server for SSL_BRIDGE 8443:
    1. Name = lbvip-Horizon-8443SSL
    2. Same VIP as the 443 Load Balancer.
    3. Protocol = SSL_BRIDGE, Port = 8443
    4. Service Group Binding = the TCP 8443 SSL_BRIDGE Service Group

  12. Create another Load Balancing Virtual Server for UDP 8443:
    1. Name = lbvip-Horizon-8443UDP
    2. Same VIP as the 443 Load Balancer.
    3. Protocol = UDP, Port = 8443
    4. Service Group Binding = the UDP 8443 Service Group

  13. This gives you six Load Balancing Virtual Servers on the same VIP, but different protocols and port numbers.

Persistency Group

Users will first connect to SSL_BRIDGE 443 and be load balanced. Subsequent connections to the other port numbers must go to the same load balanced appliance. Create a Persistency Group to facilitate this.

  1. On the left, under Traffic Management, expand Load Balancing, and click Persistency Groups.
  2. On the right, click Add.
  3. Give the Persistency Group a name (e.g. Horizon).
  4. Change the Persistence drop-down to SOURCEIP.
  5. Enter a Time-out that is equal to, or greater than the timeout in Horizon View Administrator, which defaults to 10 hours (600 minutes).
  6. In the Virtual Server Name section, click Add.
  7. Move all six Horizon Load Balancing Virtual Servers to the right. Click Create.

Horizon Connection Server Locked.properties File

Horizon Connection Server’s default security settings might prevent you from connecting to load balanced Connection Servers and/or Unified Access Gateways. On the Connection Servers, go to C:\Program Files\VMware\VMware View\Server\sslgateway\conf, edit or create locked.properties file, and enter the following:

allowUnexpectedHost=true
checkOrigin=false
enableCORS=false

More details at VMware 2144768 Accessing the Horizon View Administrator page displays a blank error window in Horizon and 85801 Cross-Origin Resource Sharing (CORS) with Horizon 8 and loadbalanced HTML5 access. allowUnexpectedHost defaults to false in Horizon 2306 and Horizon 2212.1 and newer. Another option is to add portalHost entries as detailed at Allow HTML Access Through a Gateway at VMware Docs.

Load Balancing CLI Commands

Internal Connection Server Load Balancing

add server VCS01 10.2.2.19
add server VCS02 10.2.2.20
add serviceGroup svcgrp-VCS-SSL SSL_BRIDGE
add lb vserver lbvip-Horizon-SSL SSL_BRIDGE 10.2.5.203 443 -persistenceType SOURCEIP -timeout 600
bind lb vserver lbvip-Horizon-SSL svcgrp-VCS-SSL
add lb monitor Horizon-SSL HTTP -respCode 200 -httpRequest "GET /favicon.ico" -destPort 443 -secure YES
bind serviceGroup svcgrp-VCS-SSL VCS01 443
bind serviceGroup svcgrp-VCS-SSL VCS02 443
bind serviceGroup svcgrp-VCS-SSL -monitorName Horizon-SSL

Unified Access Gateway load balancing with Blast and PCoIP

add server UAG01 10.2.2.187
add server UAG02 10.2.2.188
add serviceGroup svcgrp-Horizon-SSL SSL_BRIDGE
add serviceGroup svcgrp-Horizon-UDP443 UDP
add serviceGroup svcgrp-Horizon-PCoIPTCP TCP
add serviceGroup svcgrp-Horizon-PCoIPUDP UDP
add serviceGroup svcgrp-Horizon-TCP8443 SSL_BRIDGE
add serviceGroup svcgrp-Horizon-UDP8443 UDP
add lb vserver lbvip-Horizon-SSL SSL_BRIDGE 10.2.5.204 443
add lb vserver lbvip-Horizon-UDP443 UDP 10.2.5.204 443
add lb vserver lbvip-Horizon-PCoIPUDP UDP 10.2.5.204 4172
add lb vserver lbvip-Horizon-PCoIPTCP TCP 10.2.5.204 4172
add lb vserver lbvip-Horizon-8443SSL SSL_BRIDGE 10.2.5.204 8443
add lb vserver lbvip-Horizon-8443UDP UDP 10.2.5.204 8443
bind lb vserver lbvip-Horizon-SSL svcgrp-Horizon-SSL
bind lb vserver lbvip-Horizon-UDP443 svcgrp-Horizon-UDP443
bind lb vserver lbvip-Horizon-PCoIPTCP svcgrp-Horizon-PCoIPTCP
bind lb vserver lbvip-Horizon-PCoIPUDP svcgrp-Horizon-PCoIPUDP
bind lb vserver lbvip-Horizon-8443SSL svcgrp-Horizon-TCP8443
bind lb vserver lbvip-Horizon-8443UDP svcgrp-Horizon-UDP8443
add lb group Horizon -persistenceType SOURCEIP -timeout 600
bind lb group Horizon lbvip-Horizon-SSL
bind lb group Horizon lbvip-Horizon-UDP443
bind lb group Horizon lbvip-Horizon-PCoIPUDP
bind lb group Horizon lbvip-Horizon-PCoIPTCP
bind lb group Horizon lbvip-Horizon-8443SSL
bind lb group Horizon lbvip-Horizon-8443UDP
set lb group Horizon -persistenceType SOURCEIP -timeout 600
add lb monitor Horizon-SSL HTTP -respCode 200 -httpRequest "GET /favicon.ico" -destPort 443 -secure YES -trofscode 503
add lb monitor Horizon-PCoIP TCP -LRTM DISABLED -destPort 4172 -secure YES
add lb monitor Horizon-Blast TCP -LRTM DISABLED -destPort 8443 -secure YES
bind serviceGroup svcgrp-Horizon-SSL UAG01 443
bind serviceGroup svcgrp-Horizon-SSL UAG02 443
bind serviceGroup svcgrp-Horizon-SSL -monitorName Horizon-SSL
bind serviceGroup svcgrp-Horizon-SSL -monitorName Horizon-PCoIP
bind serviceGroup svcgrp-Horizon-SSL -monitorName Horizon-Blast
bind serviceGroup svcgrp-Horizon-UDP443 UAG01 443
bind serviceGroup svcgrp-Horizon-UDP443 UAG02 443
bind serviceGroup svcgrp-Horizon-UDP443 -monitorName Horizon-SSL
bind serviceGroup svcgrp-Horizon-PCoIPTCP UAG01 4172
bind serviceGroup svcgrp-Horizon-PCoIPTCP UAG02 4172
bind serviceGroup svcgrp-Horizon-PCoIPTCP -monitorName Horizon-PCoIP
bind serviceGroup svcgrp-Horizon-PCoIPUDP UAG01 4172
bind serviceGroup svcgrp-Horizon-PCoIPUDP UAG02 4172
bind serviceGroup svcgrp-Horizon-PCoIPUDP -monitorName Horizon-PCoIP
bind serviceGroup svcgrp-Horizon-TCP8443 UAG01 8443
bind serviceGroup svcgrp-Horizon-TCP8443 UAG02 8443
bind serviceGroup svcgrp-Horizon-TCP8443 -monitorName Horizon-Blast
bind serviceGroup svcgrp-Horizon-UDP8443 UAG01 8443
bind serviceGroup svcgrp-Horizon-UDP8443 UAG02 8443
bind serviceGroup svcgrp-Horizon-UDP8443 -monitorName Horizon-Blast

135 thoughts on “VMware Horizon Load Balancing – NetScaler / Citrix ADC”

  1. Hi Carl

    I have implemented Load Balancing for UAG and Connection Server as per your guide here. However, I’m seeing that my connection servers aren’t being properly load balanced. I’m quite sure this is because of the 600 second persistence.

    We have 9 UAGs and 5 Connection Servers. Each UAG is configured to point to the Connection Sever Load Balancer. The CS LB has 600s Source IP persistence, all connections are coming from the 9 UAGs so the persistence never gets a chance to time out and Load Balance again.

    Is there anything we can do to fix this? Thanks

    Reply

    1. Another option is to configure each UAG to connect to a single Connection Server. However, if that Connection Server goes down then the UAG will drop its existing connections.

      Reply

      1. Thanks for the reply Carl. Yes, I just noticed here – https://kb.omnissa.com/s/article/56636, that this is a recommended configuration.

        “One-to-one mapping of UAG with Connection Broker is recommended to have a fair distribution of load”

        This is definitely not ideal as we are patching our Connection Servers monthly. It would take significant manual intervention to do this without severing UAG connections.

        I’m thinking changing the Load Balancing method to Round Robin might get us around this. That way all 5 connection servers will be selected and then if Persistence kicks in it doesn’t matter.

        Reply

        1. I’m observing the same behavior but on a much smaller scale with 2x CS and 2x UAG. The UAGs are configured to use the CS LBVS but only one of the CS is getting ALL of the connections. I’m not as skilled as I’d like to be in the NetScaler area so I’d greatly appreciate hearing what you have done.

          Reply

  2. Recently upgraded to 2312.1 from 7.x and instantly annoyed by the Dashboard reporting issues with internal Connection Servers with increasing “XML API Unrecognized Requests”. It seems the only thing on the entire Internet that discusses such a thing is a Reddit post. In it, someone suggested adjusting a pae-NameValuePair “enable-xml-api-warning=0” but fails to accurately specify the object in which to set. Since this is occurring only with the Connection Servers that are NOT using UAG (only load balanced by Citrix ADC) then I suspect a persistency issue but the guide here doesn’t set persistency for non-UAG Connection Servers.
    Does anyone have any insight into correcting this behavior or otherwise disabling the reporting of “XML API Unrecognized Requests”?

    Reply

    1. …aaaand I believe I have already found my answer. Just goes to show that putting stock in a single, very popular search engine isn’t always the best idea.

      DuckDuckGo found me this: https://wbsdv95928.lithium.com/t5/Horizon-Desktops-and-Apps/Connection-server-console-message-shows-quot-DETECTED/td-p/2960521/page/3

      After adding “enable-xml-api-warning=0” to the pae-NameValuePair attribute of CN=Common,OU=Global,OU=Properties,DC=vdi,DC=vmware,DC=int then rolling reboot of the Connection Server reporting the “XML API Unrecognized Requests”, I’m no longer being annoyed by the “issues”.

      Reply

      1. The link above no longer works. Pretend like I’m 5, and explain the steps you took to add the attribute in ADSI Edit. Thanks.

        Reply

        2. Apologies, I did not intend to ignore this conversation. I hope this helps:

          1. Sign into Windows as an administrator on any of the Horizon Connection Servers
          2. Launch “ADSI Edit” from the Start menu
          3. In the ADSI Edit window that appears, right-click the “ADSI Edit” object in the tree located in the left pane, then click the “Connect to…” item
          4. In the new “Connection Settings” form:
          4a. Select the “Select or type a Distinguished Name or Naming Context” option
          4b. Input “dc=vdi,dc=vmware,dc=int” (without the quotes) into the “Select or type a Distinguished Name or Naming Context” field
          4c. Select the “Select or type a domain or server: (Server | Domain [:port])” option
          4d. Input “localhost” (without the quotes) into the “Select or type a domain or server: (Server | Domain [:port])” field
          4e. Click the “OK” button
          5. Using the tree in the left pane, navigate to ADSI Edit > Default naming context [localhost] > DC=vdi,dc=vmware,dc=int > OU=Properties > OU=Global
          6. In the right pane, double-click the “CN=Common” object
          7. In the new “CN=Common Properties” form:
          7a. Double-click the “pae-NameValuePair” attribute
          7b. In the new “Multi-valued String Editor” form:
          7bi. Input “enable-xml-api-warnings=0” (without the quotes) into the “Value to add:” field
          7bii. Click the “Add” button
          7biii. Click the “OK” button to save the data and close the “Multi-valued String Editor” form
          7c. Click the “OK” button to commit the change and close the “CN=Common Properties” form
          8. Close the ADSI Edit window
          9. Reboot each of the Connection Servers, one at a time

          Reply

  3. We recently finished our migration to Horizon 8. Upon moving the majority of our user based to Horizon 8 we started getting reports of users intermittently getting disconnected from Horizon while working. In the Horizon client logs we have noticed messages like

    WARN (01) [libcdk] : CdkSetLastUserActivityTask_Transition: Ignore error(57,429,Error: HTTP error 429) by re-marking SetLastUserActivityTask as DONE from FAIL.

    Then shortly after that

    INFO (01) [libcdk] : CdkTunnelClient_DisconnectCb: TUNNEL INTERRUPTED: Error: Client disconnected due to lack of activity.

    We opened a support case with Omnissa and they referenced this KB https://kb.omnissa.com/s/article/87822

    What is your opinion on “Disable multiplexing or any applicable feature on the load balancer appliance that reuses sessions that are presented to the UAG appliances.”

    Reply

    1. If you configure SSL_BRIDGE (SSL Passthrough) instead of terminating SSL on the VIP, then it’s not an issue.

      Reply

      1. Thanks Carl. Just to confirm that would be configuring TCP 443 with SSL Bridge instead of SSL?

        Thanks again

        Reply

  4. Hi Carl, great guide, a real life saver.
    I’ve noticed a copy/paste issue in the cli section.
    Where is:
    bind serviceGroup svcgrp-Horizon-SSL UAG01 443
    bind serviceGroup svcgrp-Horizon-SSL UAG01 443
    bind serviceGroup svcgrp-Horizon-SSL -monitorName Horizon-SSL
    bind serviceGroup svcgrp-Horizon-SSL -monitorName Horizon-PCoIP
    bind serviceGroup svcgrp-Horizon-SSL -monitorName Horizon-Blast
    bind serviceGroup svcgrp-Horizon-UDP443 UAG01 443
    bind serviceGroup svcgrp-Horizon-UDP443 UAG01 443

    It should be:
    bind serviceGroup svcgrp-Horizon-SSL UAG01 443
    bind serviceGroup svcgrp-Horizon-SSL UAG02 443
    bind serviceGroup svcgrp-Horizon-SSL -monitorName Horizon-SSL
    bind serviceGroup svcgrp-Horizon-SSL -monitorName Horizon-PCoIP
    bind serviceGroup svcgrp-Horizon-SSL -monitorName Horizon-Blast
    bind serviceGroup svcgrp-Horizon-UDP443 UAG01 443
    bind serviceGroup svcgrp-Horizon-UDP443 UAG02 443

    Reply

  5. Hi Carl,

    I just upgraded 2 Connection servers and 2 UAG’s from version v21.03 to v23.12. UAG’s updated using the .ini files and the PowerShell method. All seems to be working ok when we connect to our external blast URL from the Horizon client. (vdi.company.com).

    We are now doing some MFA testing and and want to connect direct to one of the UAG’s using its DNS name (uag1.company.com). Im getting an error saying Connection Failed error: HTTP error 400 as soon as I try to connect.

    Before the upgrade this was possible and we received no errors. Do you know what could have possibly changed?

    Reply

    1. The hostname in your Horizon Client must match the hostname in the UAG Edge configuration. In the System Configuration page, you can see the Allowed Host Headers values.

      Reply

  6. Hi Carl!
    Your guides are amazing and ever my go to when we have issues. I had missed the boat last year on that allowUnexpectedHost setting now defaulting to false. I think this will fix some issues for us, thank you SO MUCH!

    One thing that has plagued us from day one (we’ve used your guide 2 years ago once we converted to Horizon 8 and Security Servers to a pair of UAGs) has been external users getting dropped when we reboot Connection servers or restart the Connection Server service. It baffles me and Omissa support recently blamed the Netscaler monitoring but Netscaler only ever shows partial up since we NEVER do both of our Connection servers at the same time. It’s like it’s tunneling and we can’t figure out why. We’ve all that turned off and fully expect the UAG client traffic to go right the VM clones in the pools. Our UAGs are “2 armed bandits” with one IP in the DMZ that’s loadbalanced and one IP in the server LAN. We do have a separate load balancer for the connection servers as suggested by some CLI commands you recommended too so it seeeeeeeems like all SHOULD be configured correctly.
    Any clue as to why external users (and ONLY external users) get dropped when we reboot one of our Connection servers? Any clue where to look?

    Reply

    1. In Edge Service settings, does the Connection Server address point to a load balanced address? Or is it a single Connection Server?

      Reply

      1. It points to (for us) https://vdi.southcountyhealth.org:443 which resolves via DNS to either our public IP which of course goes to the UAG load balancer VIP, or if on site resolves to the loadbalanced VIP for the Connection servers locally (10.5.0.25).

        So in short, yes, the URL should always resolve to something loadbalanced by the Netscaler.

        Reply

        1. Is this our problem?
          We have these settings configured in the UAGs:
          For Enable UDP Tunnel Server, enable the setting.
          For Tunnel External URL, enter https://:443 (e.g., https://view.corp.com:443). This FQDN should resolve to your external load balancer that’s load balancing TCP 443 to multiple Unified Access Gateways.

          Here’s a snipit of the UAG config:
          [Horizon]
          proxyDestinationUrl=https://vdi.southcountyhealth.org:443
          gatewayLocation=External
          tunnelExternalUrl=https://vdi.southcountyhealth.org:443
          blastExternalUrl=https://vdi.southcountyhealth.org:8443
          pcoipExternalUrl=198.7.231.208:4172

          Reply

          1. Any idea? VMware/Broadcom support is at a loss and Citrix couldn’t care less lol.

          2. If the proxyDestinationUrl is set to a load balanced FQDN, then the UAG should stay up if one Connection Server is still reachable. If all Connection Servers are not reachable, then UAG will disable itself.

          3. So I did a little testing and found that if I unbind all the monitors for the service groups made to load balance Horizon in the Netscalers external connections are NOT dropped when we reboot one Connection Server at a time (waiting for it to come back up before doing the next one). They ARE when we have all the monitors enabled.

            I did notice something.. I’m using the same Horizon-SSL monitor for the Connection Servers as I am for the UAGs that use SSL. Your commands ARE different but use the same monitor name.
            For the Connection Servers it’s:
            add lb monitor Horizon-SSL HTTP -respCode 200 -httpRequest “GET /favicon.ico” -destPort 443 -secure YES

            But of course the UAG monitor sports the Trofs code:
            add lb monitor Horizon-SSL HTTP -respCode 200 -httpRequest “GET /favicon.ico” -destPort 443 -secure YES -trofscode 503

            Is THAT the problem? Should I have essentially the same monitor without the Trofs bit for the Connection Servers? Is that why Netscaler is dropping connections on us?

          4. So it is in either case, that url is loadbalanced by the netscaler but if you’re getting the IP from AD’s DNS it’s the local loadbalanced one but if you’re not it’s the one from our public DNS and thus routed through the firewall to the UAGs (we only use the UAGs externally). The IP would be 198.7.231.208 if resolving with public DNS and making it ultimately to the UAGs or 10.5.0.25 which is the single simple load balancer of our 2 connection servers.
            When I ping with the Netscaler diagnostics and choose either of our SNIPs they both resolve from the local 10.5.0.25 IP which again is the simple load balancer for the connection servers.

  7. We are load balancing both our UAGs and Connection servers on different NetScalers. Do we need to setup Persistency Group on both NetScalers for the UAGs and the Connection servers? We currently only have a Persistency Group setup on the DMZ NetScaler for our UAGs. We are having some users reporting issues connecting while testing Horizon 8. It looks like we are seeing sessions occasionally route through different UAGs. I’m not sure if not having a Persistency Group on the internal NetScaler for the Connection servers could cause misrouting to the UAGs? On the UAGs we are using the Internal VIP for the Connection Server address on the UAGs.

    Reply

    1. Usually only the UAGs listen on multiple port numbers. Connection Servers usually only need port 443.

      Reply

      1. Thanks Carl. Do you have any recommendations on troubleshooting traffic being sent to multiple UAGs?

        Reply

  8. Any ideas on how to load balance from Workspace One Access? You put in the FQDN of the primary connection server for your virtual app collection and then what? If you attempt to put a second one in it yells at you and tells you just to put the primary.

    When it goes down however, clients can’t connect and are instead just sent to the dead primary.

    Reply

    1. Are your Connection Servers load balanced? If so, enter the FQDN that resolves to the Load Balancing VIP.

      Reply

  9. Hi Carl,

    do you have information about the new setting called “Enable Host Redirection” under Connection server settings. The little info I find on the internet is a bit confusing. Thank you

    Reply

  10. Carl,

    Is setting persistence on the virtual server the same as setting up a persistency group? We are trying to load balance 4 UAGs and are having random issues connecting to a VM. When trying to configure a persistency group we are getting an error “The binding entities have incompatible traffic domain identifiers” The traffic domain is not 0. Can you configure a persistency group on a traffic domain that is not 0? Our NetScaler engineer recommended setting the persistence on the virtual server.

    Reply

    1. The purpose of a Persistency Group is to extend a single persistence across multiple vServers (multiple port numbers). For example, a user connects on port 443 and a UAG is chosen. The user next connects on port 8443. The single persistency group containing both vServers ensures that the same UAG is chosen for both connections. If you instead do persistence on each vServer, then each port number connection will end up on different UAGs, which will cause problems in Horizon Client.

      Reply

  11. Hi Carl,

    Great step by step. We updated from 7.10 to 7.13.3 and since then we noticed that from 2 connection servers brokering connections, it’s now down to just one. Granted the server seems to be able to handle it but we’d prefer it being the way it was before where the brokering was being done between the 2. No changes were made during the upgrade, we simply hit next throughout the updates. Any info would be great, thank you.

    Reply

  12. Hey Carl, is there still a typo in the CLI commands below? Or what is “Horizon01”? I saw there was another comment in 2019 about it, and some where changed to UAG01 and UAG02 but not all.

    Cheers!

    *Unified Access Gateway load balancing with Blast and PCoIP*
    add server UAG01 10.2.2.187
    add server UAG02 10.2.2.188
    add serviceGroup svcgrp-Horizon-SSL SSL_BRIDGE
    add serviceGroup svcgrp-Horizon-UDP443 UDP
    add serviceGroup svcgrp-Horizon-PCoIPTCP TCP
    add serviceGroup svcgrp-Horizon-PCoIPUDP UDP
    add serviceGroup svcgrp-Horizon-TCP8443 SSL_BRIDGE
    add serviceGroup svcgrp-Horizon-UDP8443 UDP
    add lb vserver lbvip-Horizon-SSL SSL_BRIDGE 10.2.5.204 443
    add lb vserver lbvip-Horizon-UDP443 UDP 10.2.5.204 443
    add lb vserver lbvip-Horizon-PCoIPUDP UDP 10.2.5.204 4172
    add lb vserver lbvip-Horizon-PCoIPTCP TCP 10.2.5.204 4172
    add lb vserver lbvip-Horizon-8443SSL SSL_BRIDGE 10.2.5.204 8443
    add lb vserver lbvip-Horizon-8443UDP UDP 10.2.5.204 8443
    bind lb vserver lbvip-Horizon-SSL svcgrp-Horizon-SSL
    bind lb vserver lbvip-Horizon-UDP443 svcgrp-Horizon-UDP443
    bind lb vserver lbvip-Horizon-PCoIPTCP svcgrp-Horizon-PCoIPTCP
    bind lb vserver lbvip-Horizon-PCoIPUDP svcgrp-Horizon-PCoIPUDP
    bind lb vserver lbvip-Horizon-8443SSL svcgrp-Horizon-TCP8443
    bind lb vserver lbvip-Horizon-8443UDP svcgrp-Horizon-UDP8443
    add lb group Horizon -persistenceType SOURCEIP -timeout 600
    bind lb group Horizon lbvip-Horizon-SSL
    bind lb group Horizon lbvip-Horizon-UDP443
    bind lb group Horizon lbvip-Horizon-PCoIPUDP
    bind lb group Horizon lbvip-Horizon-PCoIPTCP
    bind lb group Horizon lbvip-Horizon-8443SSL
    bind lb group Horizon lbvip-Horizon-8443UDP
    set lb group Horizon -persistenceType SOURCEIP -timeout 600
    add lb monitor Horizon-SSL HTTP -respCode 200 -httpRequest “GET /favicon.ico” -destPort 443 -secure YES -trofscode 503
    add lb monitor Horizon-PCoIP TCP -LRTM DISABLED -destPort 4172 -secure YES
    add lb monitor Horizon-Blast TCP -LRTM DISABLED -destPort 8443 -secure YES
    bind serviceGroup svcgrp-Horizon-SSL Horizon01 443
    bind serviceGroup svcgrp-Horizon-SSL Horizon02 443
    bind serviceGroup svcgrp-Horizon-SSL -monitorName Horizon-SSL
    bind serviceGroup svcgrp-Horizon-SSL -monitorName Horizon-PCoIP
    bind serviceGroup svcgrp-Horizon-SSL -monitorName Horizon-Blast
    bind serviceGroup svcgrp-Horizon-UDP443 Horizon01 443
    bind serviceGroup svcgrp-Horizon-UDP443 Horizon02 443
    bind serviceGroup svcgrp-Horizon-UDP443 -monitorName Horizon-SSL
    bind serviceGroup svcgrp-Horizon-PCoIPTCP UAG01 4172
    bind serviceGroup svcgrp-Horizon-PCoIPTCP UAG02 4172
    bind serviceGroup svcgrp-Horizon-PCoIPTCP -monitorName Horizon-PCoIP
    bind serviceGroup svcgrp-Horizon-PCoIPUDP UAG01 4172
    bind serviceGroup svcgrp-Horizon-PCoIPUDP UAG02 4172
    bind serviceGroup svcgrp-Horizon-PCoIPUDP -monitorName Horizon-PCoIP
    bind serviceGroup svcgrp-Horizon-TCP8443 UAG01 8443
    bind serviceGroup svcgrp-Horizon-TCP8443 UAG02 8443
    bind serviceGroup svcgrp-Horizon-TCP8443 -monitorName Horizon-Blast
    bind serviceGroup svcgrp-Horizon-UDP8443 UAG01 8443
    bind serviceGroup svcgrp-Horizon-UDP8443 UAG02 8443
    bind serviceGroup svcgrp-Horizon-UDP8443 -monitorName Horizon-Blast

    Reply

  13. Can this be done via a content switch? I am trying to make my horizon 8 accessible externally as well as my Citrix environment. Citrix works fine, but horizon doesn’t fully tunnel because it needs ports 8443 and 4172.

    I am using SSL with content switching, so you can get to the horizon UAG site, but the portal for horizon login doesn’t work and pointing horizon client doesn’t work because it needs different ports other than 443.

    Reply

    1. You can add LB vServers on those additional port numbers. Set the LB vServers with the same VIP as the Content Switch.

      Reply

  14. Hello, We need to update to Horizon 8 but are still running Netscaler version 11.1. Are there any known issues with this older version of Netscaler in front of Horizon 8?

    Reply

      1. Thanks. We have also noticed that our LB Vservers seem to be stuck in the Slow Start RR and never transition to LeastConnection. Services are up and working.

        Any suggestions to troubleshoot that?

        Output below:
        95) LB-HORIZON-SSL (X.X.X.X:443) – SSL_BRIDGE Type: ADDRESS
        State: UP
        Last state change was at Wed May 31 03:36:38 2023
        Time since last state change: 0 days, 15:28:19.870
        Effective State: UP
        Client Idle Timeout: 180 sec
        Down state flush: ENABLED
        Disable Primary Vserver On Down : DISABLED
        Appflow logging: ENABLED
        No. of Bound Services : 2 (Total) 2 (Active)
        Configured Method: LEASTCONNECTION
        Current Method: Round Robin, Reason: Bound service’s state changed to UP BackupMethod: ROUNDROBIN
        Group: PG-HORIZON
        Mode: IP
        Persistence: SOURCEIP Persistence Mask: 255.255.255.255 Persistence Timeout: 600 min
        Connection Failover: DISABLED
        L2Conn: OFF
        Skip Persistency: None
        Listen Policy: NONE
        IcmpResponse: PASSIVE
        RHIstate: PASSIVE
        New Service Startup Request Rate: 0 PER_SECOND, Increment Interval: 0
        Mac mode Retain Vlan: DISABLED
        DBS_LB: DISABLED
        Process Local: DISABLED
        Traffic Domain: 25
        TROFS Persistence honored: ENABLED
        Retain Connections on Cluster: NO
        96) LB-HORIZON-UAG-SSL (X.X.X.X:443) – SSL_BRIDGE Type: ADDRESS
        State: UP
        Last state change was at Wed May 31 03:36:38 2023
        Time since last state change: 0 days, 15:28:19.880
        Effective State: UP
        Client Idle Timeout: 180 sec
        Down state flush: ENABLED
        Disable Primary Vserver On Down : DISABLED
        Appflow logging: ENABLED
        No. of Bound Services : 2 (Total) 2 (Active)
        Configured Method: LEASTCONNECTION
        Current Method: Round Robin, Reason: Bound service’s state changed to UP BackupMethod: ROUNDROBIN
        Group: PG-HORIZON-UAG
        Mode: IP
        Persistence: SOURCEIP Persistence Mask: 255.255.255.255 Persistence Timeout: 600 min
        Connection Failover: DISABLED
        L2Conn: OFF
        Skip Persistency: None
        Listen Policy: NONE
        IcmpResponse: PASSIVE
        RHIstate: PASSIVE
        New Service Startup Request Rate: 0 PER_SECOND, Increment Interval: 0
        Mac mode Retain Vlan: DISABLED
        DBS_LB: DISABLED
        Process Local: DISABLED
        Traffic Domain: 13
        TROFS Persistence honored: ENABLED
        Retain Connections on Cluster: NO

        Reply

          1. Yes we are sending traffic to the VIP. StartupRR Factor is set to 0. All of our LB VServers using SSL_Bridge are having the same problem. Its very odd.

  15. Carl,
    Great site. Finally upgrading from Security Servers to UAGs in my environment. I have 2 Horizon Pods in the site and each pod as 2 UAGs. Each UAG connects to its own broker for simplicity. The UAGs are in the DMZ (uaga101, uaga103, uaga202, uaga204)
    I can currently connect via each of the listed UAGs to a virtual desktop through our NetScaler. I need to provide the users with a single URL which will balance across all the UAGs and not sure how to configure the final piece.

    Reply

    1. For on-prem Horizon, you use a Global load balancer (e.g., NetScaler GSLB) to load balance a single DNS name across multiple IP addresses. This is used along with Cloud Pod Architecture.

      Horizon Cloud has a Universal Broker that can send traffic to UAGs that have different FQDNs.

      Reply

  16. Hi Carl,
    I am not sure whether this question that had ever been inquired before. Have been trying through the ways to integrate NetScaler OTP (AAA) before the Horizon Connection Server: 1. PCoIP Proxy — success but only supports PCoIP rather than Blast (BEAT) protocol. 2. UAG via ADC LB vServer — supports both PCoIP and Blast by following this article but unable to be integrated with Citrix Gateway to function NetScaler OTP (AAA). 3. Wondering if SAML can help, but have no idea how the configuration should be done on UAG as a SP to be accompanied with ADC as an IDP. My goal is to accomplish the task of having full features with PCoIP, Blast (BEAT) and NetScaler OTP on the platform built with Horizon, may I have your advices? Thank you in advance.

    Reply

      1. Hi Carl, I am not sure if my question makes any sense. I’d done a wonderful SAML model built with Sharefile (SP) to NetScaler AAA (IDP), then like to do replacement on SP from Sharefile to UAG, whereas there exists no identical form field filled with Sharefile on UAG. The only way I speculate would be doing import/export SAML metadata (XML?) between each other. May I have your advice addressing this issue in detail? Thank you in advance.

        Reply

  17. I think I’m getting confused with the internal Connection Server versus the UAG configurations. I tried to clear up my understanding through the provided CLI commands but I’m just not getting it.

    For both sets of CLI commands there is a monitor named Horizon-SSL.

    If an implementation has both internal Connection Servers that *do not* have UAGs and external Connection Servers that *do* have UAGs, then I have two assumptions that I’d like to clear up:
    1) Unique monitors would need to be created since the UAG monitor has the additional “-trofscode 503” option.
    2) The same monitors can be used for internal and external Connection Servers when each also have their own VIPs for load balancing (one VIP for internal Connection Servers and a different VIP for external Connection Servers used by UAGs).

    Reply

  18. Hi, Carl!

    Thanks for your blog again!

    Can you please give an advise, why in Horizon Agent ONLY in balanced connection after 20-30 minutes it’s asking password again and client seems initiates new connection with message “The connection to remote applications has been disconnected. The connection to the remote computer has been closed due to anew connection request.”? When I connecting through CS I don’t have such this a problem. In which way I should search?

    Reply

    1. It usually means a persistence problem. What is your persistence timeout?

      Or maybe your UAG or Connection Server has a short timeout configured in Global Settings.

      Reply

  19. Hello Carl!

    Excellent article as always. I’ve followed it very successfully for a VDI deployment I’m working on but am at the step of integrating multifactor. RSA Authenticate (SecurID) is supported right within the UAG so it seems to be the place of choice to require the MFA — all external users would normally be the ones who’d need to do it. However, there’s always a but, and in this case it’s a list of client source IP addresses that, despite being external, are to be exempt from the MFA. So I’m forced with deciding where best to put the logic and the SecurID.
    Today the ADC in question already has successful integration with SecurID working for both a vpn gateway and for a Citrix storefront environment with great success however neither includes and kind of content switching policies nor any advanced authentication policies to conditionally skip or require that 2nd factor.

    So my question is this, do we integrate SecurID into the UAGs and make a content switching load balancer of type SSL and bind the cert to it and conditionally point MFA needed traffic to the 443 TCP virtual server load balancer in front of the UAGs, and another directing to the connection server load balancer? I’m leery of this as what about the BLAST and PCoIP traffic? I bet there are nuances to ensure the content switching load balancer doesn’t undo the successes of setting all those virtual servers and monitors up in front of the UAGs.

    Or do we forego the option of SecurID on the UAGs and instead do everything authentication wise on the ADC like we do for the VPN gateway and if so the users who do authenticate would have to at least single factor again once the connection server finally loads so we’d want that to SSO.

    I’m guessing it’s the former and the right home for SecurID is the UAGs and we just need a way to magically skip them when the source client IP is among a white list. This isn’t an issue for folks already on site or VPN as DNS already points them to the connection server load balancers — unless we have a need to REQUIRE MFA for some folks who are internal (something that I think did get mentioned, in that case it’s essentially the same question as I’d just adjust internal DNS to the switcher making the decision.

    I had thought about RSA Authenticate having some semblance of Conditional Access like Azure AD does and I believe they do but the end users would still need to have registered with RSA SecurID at least once or it wouldn’t work. The use case above is for some trusted offices that for whatever reason don’t have a site to site VPN appliance in place. Most of these users never access from outside that trusted source IP ever so they’d never need to have registered with SecurID etc.

    Thoughts?? Anyone else please feel free to chime in. It’s been a head scratcher for me as I’m new to the wonderful world of the ADC and this requirement threw me.

    Reply

    1. F5 might be able to do it but I don’t think UAG has that capability. Citrix ADC can’t proxy Blast without UAG, so that’s probably not an option. One option is to build a separate pair of non-MFA UAGs for the offices. Then configure your firewall to NAT your offices to the non-MFA pair but everything else is NAT’d to the MFA pair.

      Reply

      1. So basically do the conditional traffic routing on the firewall but use UAGs without MFA for the externals who don’t need it and UAGs with MFA for the ones that do. Less than stellar. We’d like to replace Kemp altogether if we could and not require handling such lists of IPs on the firewall vs within the application specific thing like the load balancing settings for the service in question.
        Hm.
        So no way to do this purely with the ADC huh.. Like not even a content switching load balancer in front of the double pairs of UAGs? I don’t mind making 2 more IPs, one for the non MFA UAG load balancer (and some more UAGs of course) and one for the thing in front of it to route my external ip to. To be clear even this method won’t work as the content switching load balancer in front of the dual pair of UAGs due to its incapability of proxying blast etc?

        And there’s no way to add some source ip content switching to the 6 load balancers detailed in your guide to route traffic conditionally do a different pair of UAGs I don’t set up with MFA? Or would that be possible somehow? In that example external traffic would (as it does now) go to the UAG load balancer already set up in the ADC now (and I could add MFA to the UAGs we have now) but have some kind of conditional content switch policy to route traffic to the non MFA UAGs that I’d have to set up?

        Reply

        1. You would still need separate pairs of UAGs for MFA and non-MFA. Yes, Content Switching would probably also work based on Source IP.

          Reply

        2. I don’t see SSLBRIDGE as an option for content Switching, so you might need TCP instead if you are not terminating SSL at the ADC.

          Reply

          1. Yea I saw that bridge wasn’t and option and I’d need our wildcard cert added to it with ssl. Just wasn’t sure if the re-encryption would muck up anything. Will test and get back to you! Thanks so so much for your advice.

  20. Hi Carl,

    We are setting up Load Balancing for two Connection Servers, we are not using UAG.
    Do we need to install any certificate in the Netscaler? We have just installed a certificate in each Connection Server, with CN=VIP_FQDN.

    Reply

    1. If your ADC is set to SSL_BRIDGE then you don’t need certificate on ADC. SSL load balancing does need a certificate.

      Reply

  21. We are trying to upgrade our UAGs to version 2.11.1 or 2.11.2. We are using Radius, configured on the UAGs and Citrix Netscaler VPX (persistience set to sessionip).

    It works just fine with v20.09, but with either of the new version, the login hangs after the push from the client.

    using the htlm access, the login and authenication happens, the apps are displayed and the error is “failed to resolve proxy route for request”

    Thanks for any suggestions.

    Ellen

    Reply

  22. Hello, sir! Why after adding Citri ADC load balancing to the front end of the two connection servers, I can’t connect through the WEB, and can only use the Horizon client to connect. Of course, all I use are VIP addresses. thanks

    Reply

    1. What version of Horizon? You might have to set balancedHost in locked.properties to allow your load balanced FQDN.

      Reply

      1. Hello Carl, does this setting for balancedHost in locked.properties help to let the horizon agent recognize the correct connection type?
        Our problem: Horizon agen always declares connection type as “EXTERNAL” even if not connected via UAG but ADC load balanced Connection server URL: ViewClient_Broker_GatewayLocation=External
        As we plan to implement DEM and SmartPolicies this behaviour does mess all up.
        Thanks for any help, Stefan

        Reply

          1. Well yes, I know that article and I wonder what would happen if I manually set the gateway location to INTERNAL on the connection servers. As we use UAG we have not activated any Security Gateway.
            We’ve already opened a Vmware call on that – waiting for an answer.
            Need to get this fixed.
            Thanks Carl

  25. Due to external facing NAT etc on the DMZ, can’t we really use a difference LB Algorithm on NetScaler like some sort of Hash ? SrcIP hash, SrcDstIP Hash etc ?

    I think VMware’s AVI suggests we do this way ?

    Reply

    1. If it’s SSL instead of SSL_BRIDGE, then probably. However, I suspect that Horizon Client won’t know what to do with the logon page or other ADC AAA responses.

      Reply

  28. Carl,

    Should I be able to use ssl instead of ssl_bridge? I want to attach multiple certificates with SNI? I can get to the login page but after logging in I get an error not being able to connect to connection server. Since I am getting a login page, I assume I’m getting to the UAG.

    Reply

    1. SSL should work assuming you’re not doing smart cards or client certificate authentication.

      In UAG, in Horizon Settings, for the SSL Gateway you enter a URL. Users need to be able to reach the same UAG on that URL.

      Reply

      1. Thank you for the quick response! I’m not doing smart cards or client cert auth.

        However I am attempting multi-tenet, so this may be the wrong direction.

        Thanks again for your help!!

        Chris

        Reply

  29. Hi Carl,

    First of all thanks for wonderful Article !!
    I am working on a Horizon deployment , which have external users from different offices . Their source IP gets NAT at firewall before its move out , because of this we can’t use source IP affinity method . My humbIe request , if you can help with setting details to be made in Citrix ADC and in UAG for opting the “Multiple Port Number Groups” method .

    Reply

    2. Hemant, I was also doing this at one point and decided to go with the internal HA that is integrated with the UAG appliances. The overall management of the Multiple Port Number Groups is a nightmare. So many VIPS and only 1 server behind it. Probably the biggest waste of your time.

      I would suggest since they are being NAT’d, to just use the HA feature as the UAGs would be NAT’d to a public IP anyways.

      Reply

  30. Carl,

    Do i need an internal VIP? can i just go from my external VIP to the UAG’s directly to the connection servers? I notice i get a gray page when i try to load a desktop from the html client, i can see the pool and after selecting it, its a gray screen.

    Chris

    Reply

    1. Internal LB of Connection Servers is not required, but it does increase availability.

      Is port 22443 opened from UAG to the Horizon Agents?

      Reply

  31. Carl,

    This is a great article thank you for this! Do you have any additional info when it comes to multi-tenant with horizon view? I think i would need to use SSL instead of SSL_BRIDGE so i could attach multiple domain certs with SNI? Any direction you can offer would be great!

    Thanks

    Chris

    Reply

  32. Hi Carl
    Im having issues with my Horizon PCoIP clients facing a lag while I have both my unified access gateway enabled in netscaler. when I have just one UAG enabled I do not have this issue.

    Reply

    1. Does it matter which UAG is enabled?

      Did you create the persistency group and add all of the Horizon vServers?

      Reply

      1. Uag01 is the one when enabled having lag issues. I’m running on uag02. No I did not try the other way yet ( with Uag01 enabled and uag02 disabled).

        Yes all the virtual servers are in the persistency group

        Reply

  33. Hello Carl,

    We have deployed UAG 3.10 behind ADC. We have ran into some connectivity issues. Working through the issues with VMware, they noticed that the UAG timeout was longer than the Netscaler timeout in the Persistency Group. They showed us an article in the VMware Communities that shows that the affinity and UAG need to match. So if users do extend the timeout on the UAG, they need to make sure that it matches on the Netscaler side.

    https://communities.vmware.com/docs/DOC-32792#jive_content_id_Method_1__Source_IP_Affinity

    Reply

    1. Hello Carl,

      I am wondering if you have any experience or documentation with setting up Citrix ADC and VMware UAG w/o using Source IP Affinity? My issue above continues. My corporate overlords have proxies dishing out 443 traffic but the Source IP can change between the 443 authentication to the 4172/8443 display protocol.

      VMware is recommending using Multiple Port Number Groups, method 2 in the link above, but unable to provide any sort of documentation on exactly how to do that. They keep pointing me here, to this blog, yet they are not understanding that I need a different method for Session Persistence.

      Any assistance would be greatly helpful.

      Thanks!

      Reply

      1. I think we’re limited by Layer 4 packet fields like Source IP. For Blast/PCoIP, I don’t think we can see anything Layer 7.

        Reply

        1. Thank you for your reply!

          Would you be able to dumb that down just a bit for me please? I, unfortunately, do not know network layering.

          Thank you!

          Kevin

          Reply

          1. Layer 4 is TCP information, like source IP and source port. https://en.wikipedia.org/wiki/Transmission_Control_Protocol

            Layer 7 is HTTP information, like Cookies. PCoIP is definitely not HTTP. I’m not sure if Horizon Client can share a cookie provided on port 443 to the Blast connection.

            UAG supports TCP 443 port sharing for Blast. I wonder if that would help for persistence since HTTP and Blast would be the same destination port number.

  34. One of the restrictions of SourceIP persistence is the 24 hour max. To go above that, it appears we’d need to use CookiePersistence which isn’t supported in SSL_BRIDGE. Thus we would have to switch to a TLS terminated VIP on the NetScaler.

    Why is this important – if you close Horizon after your persistence has timed out, the close connection may not reach the same UAG and therefore leave your connection open until you eventually hit the UAG session timeout limit.

    VMware article: https://communities.vmware.com/docs/DOC-32792
    This article further explains that below UAG3.5, the cert on the VIP must match the cert on the UAGs. They can be different at UAG3.5+.

    Have you tried this config, any gotchas?
    What would you do with the persistencegroup – e.g. pcoip is a TCP/4172 VIP which won’t support cookies.

    Thanks

    Reply

    1. Hi Carl,
      We have implemented our horizon system based on your instructions and the whole system works (thanks a lot for that) but for some reason all the sessions go to one of the UAG’s. If we bring the working UAG down the sessions start going to the other UAG only and continue to go only there, so as it is it offers HA but not load balancing (HA on the ADC, we haven’t implemented HA through the UAG setup).

      Internally the load balancer VIP for the connection servers is behaving normally, internal sessions connect to both connection servers as expected.

      We are using only SSL and Blast, no pcoip.

      Is there anything we should check? persistency groups are there for 600 mins., we are using least connection as LB method.

      Thanks

      Reply

        1. Hi Carl, Thanks a lot for your reply, it pointed to the problem straight away. The traffic was being nat’d in the firewall ans the netscaler was seeing it as coming from same source ip. Once I disabled NAT, it all started working as expected.

          regards,
          Juan

          Reply

  36. Hello Carl,

    We are having a small dispute with VMware about this. We have UAG 3.5 deployed with Horizon 7.8. When we have multiple UAGs in our load balancers, we have infrequent connections. They are stating that sticky sessions is not enabled. I setup our UAGs exactly like you have documented.

    Here is a snippet of the logs that they had me gather. Any assistance would be helpful.

    2020-02-21T13:20:02.088-08:00> LVL:2 RC: 0 PRI :tera_pri_client_set_tag: Client Session ID Tag: ‘PYN/vl1qTasA’ Value:3d837fbe5d6a4dab

    I see a start message for the tag PYN/vl1qTasA in the PCoIP log in the UAG2:

    2020-02-21T21:20:01.214761+00:00> LVL:info : Header received by the XMLTCP control server (remote endpoint: 127.0.0.1:10760)
    2020-02-21T21:20:01.214907+00:00> LVL:info : Received command: add-connection
    2020-02-21T21:20:01.214936+00:00> LVL:info : Parameter: ip-address Value: 172.29.147.25
    2020-02-21T21:20:01.214945+00:00> LVL:info : Parameter: tcp-port Value: 4172
    2020-02-21T21:20:01.214952+00:00> LVL:info : Parameter: ctag Value(session_id portion): PYN/vl1qTasA
    2020-02-21T21:20:01.214960+00:00> LVL:info : Parameter: connection-id Value: 515a28a5-ff74-4eb7-b4dd-b03dc89dff4e
    2020-02-21T21:20:01.214999+00:00> LVL:info : parse_janus_session_tag: session tag flags = 0x5A81
    2020-02-21T21:20:01.215176+00:00> LVL:info : [Sess: PYN/vl1qTasA] parse_janus_session_tag: session tag flags = 0x5A81
    2020-02-21T21:20:01.216472+00:00> LVL:info : [S: 172.29.147.25:4172] Connected to PCoIP server (session ID: PYN/vl1qTasA)

    I then see the session 3d837fbe5d6a4dab being sent to the UAG1 per its PCoIP logs:

    2020-02-21T21:20:02.220450+00:00> LVL:info : [C: 192.168.203.5:29606] failed to get ctag – no matching server connection found for given session ID=PYN/vl1qTasA
    2020-02-21T21:20:02.220488+00:00> LVL:info : [C: 192.168.203.5:29606] failed to find matching server connection for PCOIP_HELLO XML, xml=5fbe0bb8426ac1eb681f8bb4de337668f2947eea7d7033a98b95830c3071131a

    This tells me that sticky sessions are not setup for PCoIP. I would advise setting up sticky sessions for PCoIP as well per the documentation below:

    https://communities.vmware.com/docs/DOC-32792

    Reply

    1. You configured the Persistency Group on NetScaler?

      On your Connection Servers, did you disable the PCoIP Gateway?

      On UAG, the PCoIP address is set to the load balancing VIP?

      Reply

  37. Hello Carl,

    What configuration would need to be changed if we enable “Use Source IP” in order to log the real source IP of clients? Our security officer requires logs to show this information for external clients accessing our network via Horizon View.

    All help is greatly appreciated!

    Reply

    1. You’d have to change the Default Gateway of the Connection Servers to point back to the ADC SNIP so ADC can see both sides of the conversation.

      Reply

  38. Hi Carl,

    Excuse my ignorance but can the netscaler detect whether you have disabled the connection server in the admin console?
    I am pretty sure this worked at one stage but now it seems like I have to take the manual step of taking a server out of the netscaler pool. Noticed this first when upgrading to View 7.5. Also, what benefit does monitoring for /broker/xml give me over just monitoring the state of port 443, assuming disabling the connection server has no affect on either

    Reply

  39. Thanks for a great write up, these instructions got my setup working great. One call out is the section “Create another Load Balancing Virtual Server for PCoIP UDP 4172” shows a picture for using Service group for UDP 8443, while text instructs to use PCOIP 4172.

    Reply

  40. Thanks for the response Carl. Question about the certificate chains. If using SSL bridging we would need a SSL cert on the netscaler and then the same on the UAG correct? Also, if I wanted to point the UAG to use a connection server vip instead of a singular connection server would the connection server vip and the UAG need the same internally issued cert? Thanks again your articles are truly amazing!

    Reply

    1. On NetScaler, SSL bridging means no decrypt.

      I assume you mean decrypt, which means you need a valid, trusted cert on the load balancer.

      NetScaler does not verify the back-end certificate so it can be anything, but you’d want a Connection Server certificate that avoids the red error in the Horizon Administrator dashboard.

      Reply

  41. Hi Carl,

    I wanted to verify that on the load balancing portion with Blast and PCoIP where you have listed at the end:
    bind serviceGroup svcgrp-Horizon-UDP443 Horizon01 443
    bind serviceGroup svcgrp-Horizon-UDP443 Horizon02 443
    bind serviceGroup svcgrp-Horizon-UDP443 -monitorName Horizon-SSL
    bind serviceGroup svcgrp-Horizon-PCoIPTCP Horizon01 4172
    bind serviceGroup svcgrp-Horizon-PCoIPTCP Horizon02 4172
    bind serviceGroup svcgrp-Horizon-PCoIPTCP -monitorName Horizon-PCoIP
    bind serviceGroup svcgrp-Horizon-PCoIPUDP Horizon01 4172
    bind serviceGroup svcgrp-Horizon-PCoIPUDP Horizon02 4172
    bind serviceGroup svcgrp-Horizon-PCoIPUDP -monitorName Horizon-PCoIP
    bind serviceGroup svcgrp-Horizon-TCP8443 Horizon01 8443
    bind serviceGroup svcgrp-Horizon-TCP8443 Horizon02 8443
    bind serviceGroup svcgrp-Horizon-TCP8443 -monitorName Horizon-Blast
    bind serviceGroup svcgrp-Horizon-UDP8443 Horizon01 8443
    bind serviceGroup svcgrp-Horizon-UDP8443 Horizon02 8443
    bind serviceGroup svcgrp-Horizon-UDP8443 -monitorName Horizon-Blast

    Should this be instead UAG01 and UAG02 as you had mentioned at the top when creating the servers or am I missing something.

    Reply

    1. To be more specific should UAG01 and UAG02 replace what is current cited as Horizon01 and Horizon02

      Reply

  42. Hello

    i used the UAG Quiesce Mode an getting an 503 Error as described in this Article.
    This prevents that Horizon-SSL-External Virtual Server stays in the Down State because of the 503 Error on the UAG Quiesce Monitor.
    What i am doing wrong ?

    Br

    Reply

  43. Are you 100% sure that the monitor for the SSL 443 is GET /broker/xml? In the UAG the default health check URL is /favicon.ico not /broker/xml.

    Reply

      1. Ahh so they do. OK I’ll have to test that. Just going over the configuration and making sure all i’s are dotted and t’s crossed. Just trying to be cautious. By the way your page kicks ass and I think most of us couldn’t have got this far without your guides and informative content.

        Reply

        1. UAG should be forwarding the HTTP request to the Connection Server. Maybe an issue with your Proxy Pattern?

          Reply

      2. For connection servers should use the /favicon.ico with the 200 for the health monitor (same as the UAG) and not /broker/xml. The reason is, if you disabe the server in the admin console for maintenance which prevents users from logging into it, the load balancer will not detect the server is down and still direct users to it. /favicon.ico will change the response to a 503 when the server is disabled thus bringing it down and users will not be directed to it.

        Reply

      1. Just upgraded UAGs from 3.0 to 3.2.1 and getting a lot of “Response 403 Forbidden” in the Connection Server logs, and lots of timeout connection issues – presume they are linked, just working through now. Notice that the default proxy pattern is different in 3.2.1 UAG – could this be the issue? Have temporarily reverted back to 3.0 and resolved issue straight away, even though they are not supported, go figure.

        (Netscaler load balances the UAGs that are pointed to an F5 VIP in-front of the Connection Servers).

        Reply

  44. Hello, First what a great write up unbelievable about of information in your posts. Thank you for that. My question is Netscaler and HA. We were planning two Netscaler load balances in a active\active or active\passive state so we have redundancy in there as well. Looking at the diagram in the start of this article you show two load balanacers one before the UAG and one after. This is not two load balancers just two monitor points, correct? I dont see anything anything in your documentation about HA for netscaler any info would be great. Thanks again.

    Reply

  45. Not sure if this is normal or not but i keep seeing the in Securitygateway logs SSL handshake failure errors coming from the NetScaler data SNIP. I see this on both the internal connection servers and the UAGs. connections are still working but this error floods the logs so wondered if there was a way to clean it up? Below I have listed a example of the error as it is seen in the log.

    2017-10-30T06:50:02.411-05:00> LVL:error : [C: xxx.xxx.xxx.xxx:15325] *** SSIGServer::SSL handshake failure: End of file (2) error:00000002:lib(0):func(0):system lib

    Reply

        1. By the way I did some testing it looks like Monitoring connection close bit “Reset” fixed the errors in the securitygateway logs

          Reply

  46. Does the certificate installed on each UAG (behind the DMZ VIP) need to the same certificate installed on the DMZ VIP?

    Can you use a CA signed certificate (eg From DigiCert) on the DMZ VIP and internal (ie company CA signed) certificates on each UAG?

    Reply

      1. Actually, if we are using SSL_BRIDGING on the Netscaler, we do need to have the same certificate installed on the UAG as is on the Netscaler (See https://communities.vmware.com/docs/DOC-32792).

        This requirement is due to the fact that the UAG includes the thumbprint of it’s own certificate in its communications with the Horizon View client, and it will cause errors if this thumbprint does not match what is being presented by the Netscaler.

        Reply

        1. Hello Carl:

          If we integrate VMware View with NetScaler 12 UAG, is it possible to get historical data on the VMWare View for external users just like we get HDX Insight with Citrix VDI, we want to get info on user WAN latency, DC latency etc. when they are VMWARE View

          Reply

        2. Hi Carl,

          Jeremy is right in this, external certificate on the UAG and internal certificate on the connection server load balancer won’t work. You will see behaviour like horizon destination server up/down in the UAG. Resolution is changing the load balancer vip for horizon connection server with the the same certificate as on the UAG and do SSL encrypt again to the back-end. And of course split-dns with external/internal url options.

          Reply

Leave a Reply

Your email address will not be published. Required fields are marked *