Author: Carl Stalhood

ADC Fundamental Concepts: Part 1 – Request-Response, HTTP Basics, and Networking

Last Modified: Apr 21, 2021 @ 11:32 am

52 Comments

Navigation

Change Log

Introduction

Citrix renamed their NetScaler product to Citrix ADC (aka Application Delivery Controller), which is a fancy Gartner term for a load balancing device that does more than just simple load balancing.

Many ADC appliances are managed by server admins and/or security people that do not have extensive networking experience. This article will introduce you to important networking concepts to aid you in successful deployment and configuration of ADC appliances. Most of the following concepts apply to all networks, but this article will take an ADC perspective.

This content is intended to be read from top to bottom with later topics building on earlier topics.

This content is intended to be introductory only. Search Google for more details on each topic.

Request-Response

Request-Response Overview

Request/Response – fundamentally, a Client sends a Request to a Server Service. The Server Service processes the Request, and sends back a successful Response, or sends back an Error. Request-Response describes almost all client-server networking. (Image from Wikimedia)

Clients send Requests – Clients are machines running software that generate network Requests that are sent to a Server.

  • For ADC, Clients are usually web browsers or any other client software that generates server Requests using the HTTP protocol.

Servers Respond to Requests – Servers receive the client’s Request, do some processing, and then send a Response back to the client.

  • For ADC, Servers are usually web servers that receive HTTP requests from clients, perform the HTTP Method (i.e. command) contained in the request, and then send back the response.

Machines are both Clients and Servers – when a machine or program sends out a Request, that machine/program is a Client. When a machine or program receives a Request from another machine, then this machine/program is a Server. Many machines, especially ADC machines, perform both client and server functions.

  • ADCs receive HTTP Requests from clients, which is a Server function. ADCs then forward the original request to a web server, which is a Client function.

What’s in a Request?

Requests are sent to Web Servers using the HTTP protocol – Web Browsers use the HTTP protocol to send Requests to Web Servers. Web Servers use the HTTP protocol to send Responses back to Web Browsers. (Image from Wikimedia)

  • Protocol – A protocol defines a vocabulary for how machines communicate with each other. Since web browsers and web servers use the same protocol, they can understand each other.

HTTP is an OSI Layer 7 protocol – HTTP is defined by the OSI Model as a Layer 7, or application layer, protocol. Layer 7 protocols run on top of (encapsulated in) other lower layer protocols, as detailed later. (image from Wikimedia)

HTTP Request Commands – HTTP Requests contain commands that the web server is intended to carry out. In the HTTP Protocol, Request Commands are also known as Request Methods.

  • HTTP GET Method – The most common Command in an HTTP Request is GET, which asks the web server to send back a file. In other words, web servers are file servers.
  • Shown below is an HTTP Request with the first line being the GET Method. Right after the GET command is the path to the file that the browser wants to download.
  • Other HTTP Request Methods are used by clients to upload files or data to the web server and will be detailed in Part 2.

HTTP Path – Web servers can host thousands of files but Web Servers will only download one file at a time. Inside the GET Request is the path to a specific file. In HTTP, the path format is something like /directory/directory/file.html.

  • The file path uses forward slashes, not backslashes, because HTTP was originally written for UNIX servers, which uses forward slashes in its file paths.
  • If you enter a directory path but you don’t provide a filename, then the web server will give you the configured default file for that directory instead of giving you every file in the directory.

The Path is one component of the HTTP Request URL, which looks something like https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol.

  • In a Citrix ADC policy expression, you can extract the HTTP path from the HTTP Request URL by entering HTTP.REQ.URL.PATH.
  • More info on URLs will be provided later in Part 2

Addresses Overview

Unique addresses – Every machine on a network has at least one address. Addresses are unique across the whole Internet; only one machine can own a particular address. If you have two machines with the same address, which machine receives the Request or Response?

Requests are sent to a Destination Address – when the client sends a request to a web server, it sends the request to the destination server’s address. This is similar to email when you enter the address of the recipient. The server’s address is put in the Destination Address field of the Request Packet.

  • Shown below is an IP Packet (Layer 3 Packet) that contains a field for the Destination Address. (image from wikimedia)

The network forwards packets to the Destination Address – Request packets are placed on a Network and the Network forwards the request to the destination.

  • Multiple network hops – there are usually multiple network hops between the client and the server. Each hop reads the Destination Address in the packet to know where to send the packet next. This routing process is detailed later.

Web Servers reply to the Source Address – when the Request Packet is put on the network, the client machine inserts its own address as the Source Address. The web server receives the Request and performs its processing and uses the following process to send the Response back to the client:

  1. The Web Server extracts the client’s Source Address from the Request Packet.
  2. The Web Server creates a Response Packet and puts the original Source Address in the Destination Address of the Response Packet.
  3. The Response packet is placed on the network, which forwards the Response packet to the Response’s Destination Address, which formerly was the Source Address of the Request.

Two network paths: Request path, and Response path – If you don’t receive a Response to your Request, then either the Request didn’t make it to the Server, or the Response never made it from the Server back to the Client. The key point is that there are two communication paths: the first is from Client to Server, and the second is from Server to Client. Either one of those paths could fail. Many ADC networking problems are in the reply/response path and not necessarily in the request path.

  • Wrong Source Address – If the original Source Address in the request packet is wrong or missing, then the response will never make it back to the client. This is especially important for client devices, like ADC, that have multiple source addresses that it can choose from. If a non-reachable address is placed in the Source Address field, then the Response will never come back.

Numeric-based addresses – All network addresses are ultimately numeric, because that’s the language that machines understand. Network packets contain Source address and Destination address in numeric form. Routers and other networking equipment read the numeric addresses, perform a table lookup to find the next hop to reach the destination, and quickly place the packet on the next interface to reach the destination. This operation is much quicker if addresses are numbers instead of words.

Layer-specific addresses – Different OSI layers have different layer-specific addresses, each of which is detailed later in this article:

  • MAC Addresses are Layer 2 addresses.
  • IP Addresses are Layer 3 addresses.
  • Port numbers are Layer 4 addresses.

IP Addresses – every Client and Server on the Internet has a unique IP address. Requests are sent to a Destination IP Address. Responses are sent to the original Source IP Address. How the packets get from Source IP address to Destination IP address and back again will be detailed later.

  • IP Address format – Each IP address is four numbers separated by three periods (e.g. 216.58.194.132). Each of the four numbers must be in the range between 0 and 255. Most network training guides cover IP addressing in excruciating detail so it won’t repeated here. IP addressing design is inextricably linked with overall network routing design.
  • Shown below is the format of IP v4 addresses. (image from wikimedia)

Human-readable addresses – When a human enters the destination address of a Web Server, humans much prefer to enter words instead of numbers, but machines only understand numbers, so there needs to be a method to convert the word-based addresses into numeric-based addresses. This conversion process is called DNS (Domain Name System), which will be detailed later. Essentially there’s a database that maps every word-based address into a number-based address. (image from wikimedia)

Web Servers and File Transfer

Web Servers are File Servers – essentially, Web Servers are not much more than file servers. A Web Client requests the Web Server to send it a file and the Web Server sends back a file.

Web Clients use the HTTP Protocol to request files from a Web Server. Web Servers use the HTTP Protocol to send back the requested file.

  • Web Clients can be called HTTP Clients.
  • Web Clients are sometimes called User Agents.

Web Clients are responsible for doing something meaningful with the files downloaded from Web Servers – Clients can do one of several things with the files downloaded from a Web Server:

  • Display the file’s contents to the user
    • HTML files and image files are usually rendered by a browser and displayed to the user.
    • Client-side apps or scripts can extract data from downloaded data files (e.g. JSON files) and then display the data in a user-friendly manner.
  • Launch a program to process the file
    • e.g. launch Citrix Workspace app to initiate a Citrix ICA session based on the contents of the downloaded .ica file.
  • Store the file on the file system
    • e.g. save the file to the Downloads folder.

Web Browsers – Web Browsers are a type of Web Client that usually want to display the files that are downloaded from Web Servers.

  • HTML and CSS – HTML files are simple text files that tell a browser how to structure content that is shown to the user. Raw HTML files are not designed for human consumption. Instead, HTML files should be processed by a web rendering engine (e.g. web browser), which converts the HTML file into a human viewable format. When a non-browser, client-side program downloads a HTML file, the client-side program is able to call an API (e.g. Webview API) to render the HTML file and display it to the user. Here are the contents of a sample HTML file downloaded from a web server. Notice the <> tags. (image from Wikimedia)
  • CSS – CSS (Cascading Style Sheet) files tell Browsers how to render HTML files using a particular theme (e.g. fonts, colors).
  • Images – Images are rendered by the Browser and displayed to the user. There are many different image formats and browsers need to be able to convert the image formats into graphics.
  • Scripts – .js files (JavaScript) files are downloaded by the browser and executed by the browser. JavaScripts usually modify the the HTML that is shown to the user.
  • Data files – Data files (e.g. JSON) are processed by JavaScript running inside the web browser.

HTML vs HTTP – HTTP is a network file transfer protocol (request/response). HTML files are just one of the types of files transferred by HTTP. You’ll find that most HTTP file transfers are not HTML files.

  • Any program that wants to download files from a server can use the HTTP protocol. HTTP is used by far more than just web browsers.
  • HTTP is the language of the cloud. Almost every communication between cloud components uses HTTP as the transfer protocol.

Other Web Client Types – other types of client programs use HTTP Protocol to download files from Web Servers:

  • API Web Clients – API Web Clients (programs/scripts) can use an HTTP-based API to download data files from a web server. These data files are typically processed by a client-side script or program.
  • Downloaders – some Web Clients are simply Downloaders, meaning all they do is use HTTP to download files and store them on the hard drive. Later, the user can do something with those downloaded files.

Web Server Scripting

Web Server Scripting – web servers can do more than just file serving: they can also run server-side scripts that dynamically modify the files before the files are downloaded (sent back) to the requesting web client. This allows a single web server file to provide different content to different clients. The file’s content can even be retrieved from a database.

Web Server Script Languages – different web server programs support different server-side script languages. These server-side script languages include: Java, ASP.NET, Ruby, PHP, Node.js, etc.

  • The Web Server runs a script interpreter for specific file types. (image from wikimedia)
  • FIle Extensions and Server-side Script Processing – When a Web Client requests a file with a specific extension (e.g. .php), the server-side PHP script engine processes the file (runs the script). Output from the script is sent as an HTTP Response. Files without the .php extension are returned as raw files. It is not possible to download the raw .php file without the script engine first processing it.

Web Client Data Upload

Web Clients can include data and attachments in the HTTP Request –  A Web Client sometimes needs to send data to a Web Server so the Web Server can send back a client-specific response. This client-side data is included in the HTTP request and can take one of several forms:

  • URL Query String – a GET request includes a path to the file that the Web Client wants. At the end of the path, the Web Client can append arguments in the form of a URL Query String, which looks like ?arg1=value1&arg2=value2. A script on the Web Server reads this query string and adjusts the response accordingly. (image from wikipedia)
  • Cookies – Web Servers frequently attach small pieces of data called Cookies to HTTP Responses. Web Clients are expected to attach these Cookies to every HTTP Request sent to the same Web Server. Cookies are used to manage Web Sessions and Web Authentication, as described later.
  • JSON or XML document – client-side scripts or programs generate JSON or XML documents and send them to a Web Server. The JSON and XML documents typically contain arguments or other data that a Web Server reads to dynamically generate a response that should be sent back to the client. The response can also be a JSON or XML document and doesn’t have to be an HTML document. For JSON/XML responses, the client-side program reads the response and does something programmatically with it. For example, a JavaScript Web Client can read the contents of a JSON file response and show it to the user as an HTML Table. Here’s a sample JSON from Wikipedia.
  • HTML Form Data – Many HTML pages contain a form. Users enter data into the form’s fields. The data that the user entered is attached to an HTTP Request and sent to the web server for processing. A very common HTML form is authentication (username and password). HTML Form Data can also be uploaded by JavaScript as JSON/XML instead of as raw HTML Form Data.
  • Raw File – Sometimes a Web Client needs to upload a raw file and store it on the Web Server. The raw file is included in the HTTP Request. Users are typically prompted to select a file that the user wants to upload to the Web Server.

Web Client Scripting

Web Browser Scripting – Client-side scripts (JavaScript) add animations and other dynamic features to HTML web pages.

  • Reload Web Page vs Dynamic HTML – in simple HTML pages, when the user clicks a link, the entire web page is downloaded from the Web Server and reloaded in the user’s browser. In more modern HTML pages (aka Single Page Applications), when a user clicks a link or submits a form, JavaScript intercepts the request and sends the request or form data as JSON to the Web Server. The Web Server replies with a JSON document, which JavaScript uses to dynamically change the HTML shown in the user’s browser. This is usually much quicker and more dynamic than fully reloading a browser’s webpage.
  • Web Browser Plug-ins – Additional client-side scripting languages can be added to web browsers by installing plug-ins, like Flash and Java. But today these plug-ins are becoming more rare because JavaScript can do almost everything that Flash and Java can do.

HTTP is the Core of Modern IT

Everybody working in IT today must know HTTP. Here are some IT roles where HTTP knowledge is essential:

  • Server Administrators that build Web Servers or any other server-side software that use HTTP to communicate with clients.
  • Database Administrators where their primary client is HTTP Web Servers that deliver HTML web pages.
  • Desktop Administrators that manage and support HTTP-based client-side programs, including Web Browsers. Almost every modern client-side application uses HTTP, even if that program is not a Web Browser.
  • Network Administrators that forward HTTP Requests to Web Servers and then forward the HTTP Responses back to the Web Clients.
  • Security Teams that inspect the HTTP Requests and HTTP Responses for malicious content.
  • Developers that write code that uses HTTP as the application’s network communication protocol. This is especially true for modern API-based micro-services.
  • Scripters and DevOps teams that write scripts to control other machines and applications. The scripting languages ultimately use HTTP Protocol to send commands. REST API is based on the HTTP protocol.
  • Cloud Administrators since Cloud = HTTP. When an application is “written for the cloud”, that means the app is based on the HTTP protocol.
    • Cloud Storage is accessed using HTTP protocol instead of other storage protocols like SMB, NFS, iSCSI, and Fibre Channel.
  • Mobile Device Administrators since mobile apps are nothing more than Web Clients that use HTTP protocol to communicate with web-server hosted applications.

Web Server Services and Web Server Port Numbers

Web Server Software – there are many web server programs like Microsoft IIS, Apache, NGINX, Ember (Node.js), WebLogic, etc. Some are built into the operating system (e.g. IIS is built into Windows Server), and others must be downloaded and installed.

Web Server Software runs as a Service – The Web Server Software installation process creates a Service (or UNIX/Linux Daemon) that launches automatically every time the Web Server reboots. Services can be stopped and restarted. Server admins should already be familiar with Server Services.

Servers can run multiple Services at the same time – A single Server can run many Services at the same time: an Email Server Service, an FTP Server Service, an SSH Server Service, a Web Server Service, etc. There needs to be some way for the Client to tell the Server that the Request is to be sent to the Web Server Service and not to the SSH Service.

Services listen on a Port Number – when the Web Server Service starts, it begins listening for requests on a particular Port Number (typically port 80 for unencrypted HTTP traffic, and port 443 for encrypted SSL/TLS traffic). Other Server Services listen on different port numbers. It’s not possible for two Services to listen on the same port number on the same IP address.

Clients send packets to a Destination Port Number – When a Client sends an HTTP Request to a Web Server Service, the Client adds a Destination Port Number to the packet. If you open a browser and type a HTTP URL into the browser’s address bar, by default, the browser will send the packet to Port 80, which is usually the port number that Web Server Services are listening on.

  • Shown below is a TCP Packet Header with a field for Destination Port. (image from wikimedia)

Client Programs and Client Ports

Multiple Client Programs – multiple client programs can be running concurrently on a single Client machine; for example: Outlook, Internet Explorer, Chrome, Slack, etc. When the Response is sent from the Server back to the Client, which client-side program should receive the Response?

Client Ephemeral Ports – whenever a client program sends a request to a Server, the operating system assigns a random port number between 1024 and 65535 to the client process. This range of ephemeral client port numbers varies for different client operating systems.

  • Windows ephemeral (dynamic) ports are usually between 49152 and 65535. For details, see Microsoft 929851.

Servers send the Response to the Client’s Ephemeral Port – The Client’s Ephemeral Port is included in the Source Port field of the Request packet. The Server extracts the Source Port from the Request Packet and puts the original source port in the Response packet’s Destination Port field.

  • If you view Requests and Responses in a network trace, you’ll see that the port numbers are swapped for responses vs requests.

Each Client Request can use a different Ephemeral Port – A client program can send multiple requests to multiple server machines, and each of these outstanding requests can have a unique Client Ephemeral port.

Summary of the Network Packet Fields discussed so far – In order for Packets to reach the Server Service and return to the Client Program, every network packet must contain the following fields:

  • Destination IP Address – the Server’s IP address
  • Destination Port – port 80 or port 443 for Web Server Services
  • Source IP Address – the Client’s IP address
  • Source Port – the ephemeral port assigned by the operating system to the client program

Sessions

Sessions Overview

Single Request vs Session – One HTTP Request is accompanied by one HTTP Response. But you usually need to combine multiple HTTP Requests and Responses into something called a Session. For example, if you authenticated to a web server, you shouldn’t have to authenticate every HTTP Request that you send to the Web Server. After authentication is complete, an HTTP Session is established which allows multiple HTTP Requests and Responses to use the same authentication context.

Session Context – Context is the data that defines a session. The first HTTP Response includes a session identifier, which is usually in a cookie. This session identifier (e.g. cookie) is included with the next HTTP Request so the Web Server knows which session is being used. The Web Server looks up the existing context information so it doesn’t have to start over (e.g. re-authenticate the user).

Sessions and the OSI network model – each layer of the OSI network model has its own conception of sessions, but the idea is the same: a session = multiple requests/responses in the same context.

Here are the Session constructs for each layer of the OSI network model:

  • Layer 4 – TCP Connection
  • Layer 6 – SSL/TLS Session
  • Layer 7 – HTTP Session
  • Web Server Session – doesn’t map to the OSI Model. Think “shopping carts”

The OSI model is detailed in every introductory network training book/video/class. In essence, application traffic starts at Layer 7 (HTTP Protocol), which is embedded down the stack inside the Layer 6 packet (typically SSL/TLS encrypted session), then embedded down the stack again into a Layer 4 TCP packet. When the packet is received at the destination, Layer 4 is removed revealing Layer 6. Then Layer 6 is removed revealing Layer 7, which is finally given to the application or service.

Higher layer sessions require lower layer sessions – Sessions at higher layers require Sessions (or Connections) at lower layers to be established first. For example, HTTP Requests can’t be sent unless a TCP Connection is established first.

Session management offloading – Some of the session handling tasks performed by servers can be offloaded to an ADC appliance. For example, Citrix ADC can handle all of the client-side TCP Connections and client-side TLS/SSL encryption sessions so the Web Servers can focus on serving content instead of managing sessions.

Network Sessions

Transport Protocol – TCP (Layer 4) is a Transport Protocol. TCP does a number of things, including the following:

  • Reorder packets that arrive out of order.
    • When a Request or Response is broken into multiple packets, the destination needs to reassemble the packets in the correct order. Each packet contains a Sequence Number. The first packet might have Sequence Number 1, while the second packet might have Sequence Number 2, etc. These Sequence Numbers are used to reassemble the packet in the correct order.
  • The destination acknowledges every packet that it receives so the source knows that the packet arrived at the destination. If the source does not receive an acknowledgement, then the source will retransmit.
    • Acknowledging every packet introduces delay (latency) so TCP uses a dynamically-sized Window to allow the source to send multiple packets before it receives an acknowledgement.
  • Error checking
    • Every TCP packet includes a checksum that was computed at the source. If the computed checksum on the received packet does not match the checksum inside the packet, then the packet is dropped and no acknowledgement is sent to the sender. Eventually the sender will retransmit the missing/corrupted packet.
  • If congestion, the source needs to back off.
    • TCP has several algorithms for handling congestion. These algorithms control how the Windows size grows and shrinks, and also controls how often acknowledgements are sent.

TCP and UDP Overview – there are two Layer 4 Session protocols – TCP, and UDP. TCP handles many of the Transport services mentioned above. UDP does not do any of these services, and instead requires higher layer protocols to handle them.

  • Citrix EDT (Enlightened Data Transport) is an example higher layer protocol that runs on top of UDP because EDT doesn’t want TCP to interfere with the EDT’s Transport services (reassembly, retransmission, etc.). In other words, Citrix EDT replaces the transport capabilities of TCP.
  • HTTP/3 (version 3) also replaces TCP with its own transport layer and thus HTTP/3 runs on top of UDP instead of TCP.
  • The problem with TCP is that it is a generic transport protocol that isn’t specifically designed for specific higher-level protocols like HTTP and Citrix ICA.

TCP Port Numbers and UDP Port Numbers are different – Each Layer 4 protocol has its own set of port numbers. TCP port numbers are different from UDP port numbers. A Server Service listening on TCP 80 does not mean it is listening on UDP 80. When talking about port numbers, you must indicate if the port number is TCP or UDP, especially when asking firewall teams to open ports. Most of the common ports are TCP, but some (e.g. Voice) are UDP.

  • Increasingly, you will need to open both UDP and TCP versions of each port number. For example, Citrix HDX/ICA traffic can run on both TCP 1494 and UDP 1494 (EDT protocol). ICA traffic proxied through Citrix Gateway can use both TCP 443 and UDP 443.

TCP Protocol (Layer 4)

HTTP requires a TCP Connection to be established first – When an HTTP Client wants to send an HTTP Request to a web server, a TCP Connection must be established first. The Client and the Server do the three-way TCP handshake on TCP Port 80. Then the HTTP Request and HTTP Response is sent over this TCP connection. HTTP is a Layer 7 protocol, while TCP is a Layer 4 protocol. Higher layer protocols run on top of lower layer protocols. It is impossible to send a Layer 7 Request (HTTP Request) without first establishing a Layer 4 session/connection.

TCP Three-way handshake – Before two machines can communicate using TCP, a three-way handshake must be performed:

  1. The TCP Client initiates the TCP connection by sending a TCP SYN packet (connection request) to the Server Service Port Number.
  2. The Server creates a TCP Session in its memory, and sends a SYN+ACK packet (acknowledgement) back to the TCP client.
  3. The TCP Client receives the SYN+ACK packet and then sends an ACK back to the TCP Server, which finishes the establishment of the TCP connection. HTTP Requests and Responses can now be sent across this established TCP connection.

TCP Connections are established between Port Numbers – The TCP Connection is established between the Client’s TCP Port (ephemeral port), and the Server’s TCP Port (e.g. port 80 for web servers). Different port numbers on either side means different TCP Connections.

  • TCP Connections run on top of Layer 3 IP traffic. Layer 3 traffic uses an IP Address for the source and an IP address for the destination. If either of these addresses are different, then that’s a different TCP Connection.
  • In summary, each combination of Source IP, Destination IP, Source Port, and Destination Port is a different TCP Connection.

Multiple Clients to one Server Port – A single Server TCP Port can have many TCP Connections with many clients. Each combination of Client Port/Client IP with the Server Port is considered a separate TCP Connection. You can view these TCP Connections by running netstat on the server.

  • Netstat shows Layer 4 only – netstat command shows the TCP connection table (Layer 4) only. Netstat does not show HTTP Requests (Layer 7).
  • Netstat show TCP connections but not UDP – There’s no such thing as a UDP Connection, so Netstat will not show them. However, Netstat can show you any services that are listening on UDP port numbers.

TCP Connection Termination – clients that are done communicating with a server can send a TCP Finish packet that tells the server to delete the TCP connection information from the server’s memory (TCP connection table).

  • TCP Connection time limit – If the client never sends a Finish packet, then servers have a default time limit for inactive TCP connections. Once that time limit expires. the server will send a Finish packet to the client and then remove the TCP connection from its memory. If the client later wants to send an HTTP Request, then the client must first perform the three-way TCP handshake again.

TCP and Firewalls – Firewalls watch for TCP handshake packets. For every SYN packet allowed through the firewall, the firewall opens a port in the opposite direction so the SYN+ACK packet can be sent back to the client. This means firewall administrators do not have to explicitly permit reply traffic since the firewall will do it automatically.

Use Telnet to verify that a Service is listening on a TCP Port number – when you telnet to a server machine on a particular port number, you are essentially completing the three-way TCP handshake with a particular Server Service. This is an easy method to determine if a Server machine has a Service listening on a particular port number and that you’re able to communicate with that port number.

  • Telnet doesn’t work with UDP services because there’s no three-way handshake for UDP.

TCP Three-way handshake can be expensive – TCP handshakes and TCP session management require CPU, memory, and bandwidth on the web servers. For web servers that serve content to thousands of clients, TCP session management can consume a significant portion of the web server’s hardware resources.

  • Citrix ADC can offload TCP session management – Citrix ADC handles all of the TCP sessions between the thousands of clients and the one Load Balancing VIP. Citrix ADC then opens a single TCP session between the ADC and the Web Server. All HTTP requests received by the ADC’s clients are sent across the single web server TCP session, even if the HTTP Requests came from multiple client TCP connections. This means that an ADC might have thousands of client-side TCP connections but only one server-side TCP connection, thus offloading TCP duties from the web server.
  • HTTP/2 Multiplexing – In HTTP 1.1, when an HTTP Request is sent across a TCP session, no other HTTP Request can be sent until the web server sends back a response for the original HTTP Request. In HTTP/2, multiple HTTP Requests can be multiplexed onto the same TCP connection and the web server will reply to all of them when it can.
    • In HTTP 1.1, ADC will open as many TCP connections to the web server as needed to support multiple concurrent HTTP requests. HTTP/2 should only need a single TCP connection for all HTTP requests.

UDP Protocol (Layer 4)

UDP is Sessionless – UDP is a much simpler protocol than TCP. For example, there’s no three-way handshake like TCP. Since there’s no handshake, there’s no UDP session.

  • Notice in the UDP header that there are no Sequence number fields, no Acknowledgement number fields, and no Window size field like there is in TCP. (image from wikipedia)

No Sequence Numbers – UDP packets do not contain sequence numbers. If packets arrive out of order, UDP cannot determine this, and cannot reassemble them in the correct order.

No acknowledgements – When UDP packets are received, the receiver does not send an acknowledgement back to the sender. Since there are no acknowledgements, if a UDP packet is lost, there’s no way for the UDP sender to detect this and thus no way for UDP to resend the packet like TCP does.

  • Audio uses UDP – For audio traffic, there’s usually no point in resending lost packets. Getting rid of retransmissions makes UDP more efficient (less bandwidth, less latency) than TCP for audio.

Why UDP over TCP? – TCP session information is contained in every TCP packet, thus making every TCP packet (20 byte header) bigger than a UDP packet (8 byte header). The smaller header size and the lack of three-way handshake means UDP is a lightweight protocol that performs better on high latency and low bandwidth links.

  • However, some of the transport services provided by TCP are needed by Layer 7 protocols. New transport protocols are being developed to replace TCP. These new transport protocols need to do their work without TCP interfering with them so the new transport protocols run on top of UDP instead of TCP.
  • Citrix EDT (Enlightened Data Transport) is an example UDP-based transport protocol. HTTP/3 has its own transport protocol that will also run on top of UDP instead of TCP.

HTTP Basics

HTTP Protocol Overview

URLs – when a user wants to visit a website, the user enters a URL into a web browser’s address bar. An example URL is https://en.wikipedia.org/wiki/URL. Each URL can be broken up into three sections:

  • URL Scheme = https:// or http:// – the first part of the URL, also known as the scheme, specifies the Layer 7 and Layer 6 protocols that the browser will use to connect to the web server.
    • HTTP Protocol can be transmitted unencrypted, which is the http:// scheme. The http:// scheme defaults to TCP Port 80.
    • Encrypted HTTP traffic means regular unencrypted Layer 7 HTTP packets embedded in a Layer 6 SSL/TLS session. HTTP Encryption is detailed in Part 2. Users enter https:// scheme to indicate that the user wants the HTTP traffic to be encrypted, assuming the Web Server is configured to accept encrypted traffic. The https:// scheme defaults to TCP Port 443.
  • URL Hostname = en.wikipedia.org – the second part of the URL is the human-readable DNS host name that translates to the web server’s IP address. The hostname comes after the scheme but before the URL Path.
    • The Web Browser uses DNS to convert the URL host name into an IP Address for the web server.
    • The Web Browser creates a TCP connection to the web server’s IP Address on port 80 or port 443.
  • URL Path = /wiki/URL – the remaining part of the URL is the Path and Query. The Path indicates the path to the file that you want to download.
    • The Web Browser sends an HTTP GET Request for the specified path to the web server across the already established TCP connection.

Browser handling of URLs – The Browser performs different actions for each portion of the URL:

  • Browser uses the URL Scheme to define how the browser connects to the URL Host name.
  • Browser uses the URL Host name as the destination for the TCP Connection and the HTTP Requests.
    • The Web Browser includes the entered URL Host name in the Host header of each HTTP Request Packet sent to the web server so that the web server knows which web site the user is trying to access. The HTTP Request Host header is detailed in Part 2.
  • Browser includes the URL Path in the GET method of the initial HTTP Request Packet.
    • For the example URL shown above, the first line of the HTTP Request packet is this: GET /wiki/URL

Why forward slashes in URLs? – Web Server programs were originally developed for UNIX and Linux, and thus URLs share some of the characteristics of Linux/UNIX.

  • Some URLs are case sensitive – Since UNIX/Linux is case sensitive, file paths in URLs are sometimes case sensitive. This is more likely to be true for Web Server software running on UNIX/Linux machines than Windows machines.

HTTP Packet

HTTP Request Method – at the top of every HTTP Request packet is the HTTP Method or Request Command. This Method might be something like this: GET /Citrix/StoreWeb/login.aspx HTTP/1.1

  • GET is a simple request to fetch a file.
  • If no URL path is specified, then the first line of the HTTP Request is simply GET / HTTP/1.1 where the / means that it’s requesting the default file in the root folder of the website. Web Servers are configured to return a default file if no specific file is requested.
  • There are other HTTP request methods, like POST, PUT, DELETE, etc. These other HTTP Request Methods are used in REST APIs, JavaScript, and HTML Form Uploads and are partially detailed in Part 2.

HTTP Response Code – at the top of every HTTP Response is a code like this: HTTP/1.1 200 OK. Different codes mean success or error. Code 200 means success. You’ll need to memorize many of these codes.

Header and Body – HTTP Packets are split into two sections: header, and body.

  • HTTP Headers – Below the Request Method or below the HTTP Response Code, are a series of Headers. Web Browsers insert Headers into requests. Web Servers insert Headers into responses. Request Headers and Response Headers are totally different. You’ll need to memorize most of these Headers.
  • HTTP Body – Below the Headers is the Body. Not every HTTP Packet has a Body.
    • In a HTTP Response, the HTTP Body contains the actual downloaded file (e.g. HTML file).
    • HTTP GET Requests do not have a Body.
    • HTTP POST Requests have a Body. The HTTP Body can contain data (HTML Form Data, JSON, XML, raw file, etc.) that is uploaded with the Request.

Raw HTTP packets – To view a raw HTTP packet, use your browser’s developer tools (F12 key), or use a proxy program like Fiddler. In a Browser’s Developer Tools, switch to the Network tab to see the HTTP Requests and HTTP Responses.

Multiple HTTP requests – A single webpage requires multiple HTTP requests. When an HTML file is rendered by a Web Browser, the HTML file contains links to supporting files (CSS files, script files, image files) that must be downloaded from the web server. Each of these file downloads is a separate HTTP Request.

  • HTTP and TCP Connections – Every HTTP Request requires a TCP Connection to be established first. Older Web Servers tear down the TCP Connection after every single HTTP Request. This means that if a web page needs 20 downloads, then 20 TCP Connections, including 20 three-way handshakes, are required. Newer Web Servers keep the TCP Connection established for a period of time, allowing each of the 20 HTTP Requests to be sent across the existing TCP Connection.

HTTP Redirects – one HTTP Response Code that you must understand is the HTTP Redirect.

  • Redirect Behavior – When a HTTP Client sends an HTTP Request to a Web Server, that Web Server might send back a 301/302 Redirect and include a new location (new URL). The HTTP Client is then expected to send a HTTP Request to the new Location (URL) instead of the old URL.
  • Redirect HTTP Response – HTTP Response packets for a Redirect have response code 301 or 302 instead of response code 200.
    • The HTTP Response Header named Location identifies the new URL that the browser is expected to navigate to.
  • Redirect usage – Redirects are used extensively by web applications. Most web-based applications would not function without redirects. A common usage of Redirects is in authenticated websites where an unauthenticated user is redirected to a login page, and after login, the user is redirected back to the original web page. Or a redirect can redirect you to a default page (e.g. redirect to “/Director”)
  • Not all HTTP Clients support HTTP Redirects – Web Browsers certainly can perform a redirect. However, other HTTP Clients (e.g. Citrix Receiver) do not follow Redirects.

Additional HTTP concepts will be detailed in Part 2.

Networking

Layer 2 (Ethernet) and Layer 3 (Routing) Networking

Subnet – all machines connected to a single “wire” are on the same subnet. Machines on the subnet can communicate directly with other machines on the same subnet. When one machine puts electrical signals on the wire, every other machine on the same wire sees the electrical signals. A network packet (or frame) is a collection of electrical signals.

Routers – If two machines are on different subnets, then those two machines can only communicate with each other through an intermediary device that is connected to both subnets. This intermediary device is called a router. The router is connected to both subnets (wires) and can take packets from one subnet and put them on the other subnet.

Layer 2 – When machines on the same subnet want to communicate with each other, they use a Layer 2 protocol, like Ethernet.

Layer 3 – When machines on different subnets want to communicate with each other, they use a Layer 3 protocol, like IP (Internet Protocol).

Local IP address vs remote IP address

Destination is either local or remote – since different protocols are used for intra-subnet (Layer 2) and inter-subnet (Layer 3), the machines need to know which machines on are on the local subnet, and which machines are on a remote subnet.

IP Address Subnet Mask – all machines have an IP address. All machines are also configured with a subnet mask. The subnet mask defines which bits of the machine’s IP address are the subnet ID and which bits of the IP address are the machine’s host ID. If two machines have the same subnet ID, then those two machines are on the same subnet. If two machines have different subnet IDs, then those two machines are on different subnets.

  • IPv4 addresses are 32-bits in length. The left part of the address is the subnet ID. The right part of the address is the machine’s host ID. The subnet mask designates where the left part (subnet ID) ends and the right part (host ID) begins. If the subnet mask is 24 bits, then the first 24 bits of the IP address are the subnet ID, and the last 8 bits are the machine’s host ID.
    • The subnet mask is usually between 8 bits and 30 bits. The variability of subnet mask length allows a router to aggregate many smaller subnet IDs into one larger subnet ID and thus improve router efficiency.
    • IPv6 changes how subnet ID and host ID are determined. IPv6 addresses are 128-bits in length. The subnet ID is the first 64 bits, and the host ID is the last 64 bits, which means the subnet mask is 64 bits. Routers still have the flexibility of reducing the subnet mask to less than 64 bits so multiple small subnets can be aggregated into one larger subnet. However, the host ID is always 64 bits.
  • For example, if a machine with address 10.1.0.1 wants to talk to a machine with address 10.1.0.2, and if the subnet mask is 255.255.0.0, when both addresses are compared to the subnet mask, the masked results are the same, and thus both machines are on the same subnet, and Ethernet is used for the communication.
  • If the masked results are different, then the other machine is on a different subnet, and IP Routing is used for the communication.
  • There is a considerable amount of readily available training material on IP Addressing and subnet masks so I won’t repeat that material here.

Wrong Subnet Mask – If either machine is configured with the wrong subnet mask, then one of the machines might think the other machine is on a different subnet, when actually it’s on the same subnet. Or one of the machines might think the other machine is on the same subnet, when actually it’s on a different subnet. Remember, communication in the same same subnet uses a different communication protocol than between subnets and thus it’s important that the subnet mask is configured correctly.

Layer 2 Ethernet communication

Every machine sees every packet – A characteristic of Layer 2 (Ethernet) is that every machine sees electrical signals from every other machine on the same subnet.

MAC addresses – When two machines on the same subnet talk to each other, they use a Layer 2 address. In Ethernet, this is called the MAC address. Every Ethernet NIC (network interface card) in every machine has a unique MAC address. Ipconfig shows it as the NIC’s Physical Address.

NICs Listen for their MAC address – The Ethernet packet (electrical signals) put on the wire contains the MAC address of the destination machine. All machines on the same subnet see the packet. If the listening NIC has a MAC address that matches the packet, then the listening NIC processes the rest of the packet. If the listening NIC doesn’t have a matching MAC address, then the packet is ignored.

  • You can override this ignoring of packets not destined to the NIC’s MAC address by turning on the NIC’s promiscuous mode, which is used by packet capture programs (e.g. Wireshark) to see every packet on the wire.

Source MAC address – When an Ethernet packet reaches a destination machine, the destination machine needs to know where to send back the reply. Thus both the destination MAC address, and the source MAC address, are included in the Ethernet packet.

  • Shown below is an Ethernet Frame with Destination MAC Address and Source MAC Address. (image from wikimedia)

Ethernet Packet Fields – In summary, a typical Ethernet packet/frame with embedded IP packet and embedded Layer 4 packet contains the following fields:

  • Destination MAC address
  • Source MAC address
  • Destination IP address
  • Source IP address
  • Destination TCP/UDP port number
  • Source TCP/UDP port number

Other Layer 2 technologies – another common Layer 2 technology seen in datacenters is Fibre Channel for storage area networking (SAN). Fibre Channel has its own Layer 2 addresses called the World Wide Names (WWN). Fibre Channel does not use IP in Layer 3, and instead has its own Layer 3 protocol, and its own Layer 3 addresses (FCID).

ARP (Address Resolution Protocol)

Users enter IP Addresses, not MAC Addresses – when a user wants to communicate to another machine, the user enters a DNS name, which is translated to an Layer 3 IP address. If the destination IP address is on the same subnet as the source machine, then the destination IP address must first be converted to a Layer 2 MAC address. Remember, same-subnet (same wire) Layer 2 communication uses MAC addresses, not IP addresses.

  • Machines use the IP address Subnet Mask to determine if the destination IP address is local (same subnet) or remote.

Machines use Address Resolution Protocol (ARP) to find the MAC address that’s associated with a destination IP address that’s on the same subnet as the source machine.

ARP Process – The source machine sends out an Ethernet broadcast with the ARP message “who has IP address 10.1.0.2”. Every machine on the same subnet sees the message. If one of the machines is configured with IP address 10.1.0.2, then that machine replies to the source machine and includes its MAC address in the response. The source machine can now send a packet directly to the destination machine’s Ethernet MAC address.

ARP Cache – after the ARP protocol resolves an IP address to a MAC address, the MAC address is cached on the source machine for a period of time (e.g. 30 seconds). If another IP packet needs to be sent to the same destination IP address, then there’s no need to perform ARP again, since the source machine already knows the destination machine’s MAC address. When the ARP cache entry expires, then ARP needs to be performed again.

IP Conflict – a particular IP address can only be assigned to one machine. If two machines have the same IP address, then both machines will respond to the ARP request. Sometimes the ARP response will be one machine’s MAC address, and sometimes it will be the other machine’s MAC address. This behavior is typically logged as a “MAC move” or an “IP conflict“. Since only half the packets are reaching each machine, both machines will stop working.

Layer 3 on top of Layer 2

Routing to other subnets – When a machine wants to talk to a machine on a different subnet, the source machine needs to send the packet to a router. The router will then forward the packet to the destination machine on the other subnet.

Default gateway – Every client machine is configured with a default gateway, which is the IP address of a router on the same subnet as the client machine. The client machine assumes that the default gateway (router) can reach every other subnet.

  • On a ADC or UNIX/Linux device, the default route (default gateway) is shown as route 0.0.0.0/0.0.0.0.

Router’s MAC address – Since the router and the source machine are physically connected to the same Ethernet subnet, they use Ethernet MAC addresses to communicate with each other. The source machine first ARP’s the router’s IP address to find the router’s MAC address. The source machine creates a packet with the remote destination IP address and the router’s local MAC address, and then puts the packet on the wire.

  • The Destination IP Address in the packet is the final destination’s (the web server’s) IP address, and not the Router’s IP address. However, the destination MAC Address is the Router’s MAC address, and not the final destination’s MAC Address.
  • ARP across subnet boundaries – It’s not possible for a source machine to find the MAC address of a machine on a remote subnet. If you ping an IP address on a remote subnet, and if you look in the ARP cache, you might see the router’s MAC address instead of the destination machine’s MAC address. Routers do not forward Ethernet broadcasts to other subnets.
  • Router must be on same subnet as client machine – since client machines use Ethernet, ARP, and MAC addresses to talk to routers, the router (default gateway) and the client machine must be on the same subnet. More specifically, the router must have an IP address on the same IP subnet as the client machine. When the client machine’s IP address and the router’s IP address are compared to the subnet mask, the subnet ID results must match. You cannot configure a default gateway that is on a different subnet than the client machine.
  • There can only be one default route on a machine, which impacts multi-NIC machines – Some machines (e.g. ADC appliances) might be configured with multiple IP addresses on multiple subnets and multiple physical connections. Only one router can be specified as the default gateway (default route). This default gateway must be on one of the subnets that the client machine is connected to. Keep reading for details on how to handle the limitation of only a single default route.

Routing table lookup – When the router receives the packet on the MAC address of one of its NICs, and if the destination IP address in the packet is not one of the router’s IP addresses, then the router looks in its memory (routing table) to determine what network interface it needs to put the packet on. The router has a list of which IP subnet is on which router interface.

Router ARP’s the destination machine on other subnet – If the destination IP address is on one of the subnets/interfaces that the router is directly connected to, then the router will perform an ARP on that subnet/interface to get the destination machine’s MAC address. If the router is not directly connected to the subnet that contains the destination IP address, then the router will probably send the packet to another router for additional routing.

The Router makes a couple changes to the packet before it puts the modified packet on the destination interface. Here are the modifications:

  • The destination MAC address is changed to the destination machine’s MAC address instead of the router’s MAC address.
  • The source MAC address in the packet is now the router’s MAC address, thus making it easier for the destination machine to send back a reply.
  • The IP Addresses in the packet do not change. Only the MAC addresses change.

Multiple Routers and Routing Protocols

Router-to-router communication– When a router receives a packet that is destined to a remote IP subnet, the router might not be Layer 2 (Ethernet) connected to the destination IP subnet. In that case, the router needs to send the packet to another router. It does this by changing the destination MAC address of the packet to a different router’s MAC address. Both routers need to be connected to the same Ethernet subnet.

Routing Protocols – Routers communicate with each other to build a topology of the shortest path or quickest path to reach a destination IP subnet. Most of the CCNA/CCNP/CCIE training materials detail how routers perform this topology building and path selection and thus will not be detailed here.

  • ADC appliances can participate in routing protocols like OSPF and BGP. ADC can inject routes for IP destinations that only the ADC appliance might know about.

Ethernet Switches

Ethernet Subnet = Single wire – All machines on the same Ethernet subnet share a single “wire”. Or at least that’s how it used to work.

Switch backplane – Today, each machine connects a cable to a port on a switch. The switch merges the switch ports into a shared backplane. The machines communicate with each other across the backplane instead of a single “wire”.

MAC address learning – The switch learns which MAC addresses are on which switch ports.

Switches switch known MAC addresses to only known switch ports – If the switch knows which switch port connects to the destination MAC address of an Ethernet packet, then the switch only puts the Ethernet packet on the one switch port. This means that Ethernet packets are no longer seen by every machine on the wire. This improves security because NIC promiscuous mode no longer sees every packet on the Ethernet subnet.

  • SPAN port – For troubleshooting or security monitoring, sometimes you need to see every packet. Switches have a feature called SPAN or Port Mirroring that can mirror every packet from one or more source interfaces to a single destination interface where a packet capture tool is running.

Switches flood unknown MAC addresses to all switch ports – If the switch doesn’t know which switch port connects to a destination MAC address, then the switch floods the packet to every switch port on the subnet. If one of the switch ports replies, then the switch learns the MAC address on that switch port.

Switches flood broadcast packets – The switch also floods Ethernet broadcast packets to every switch port in the Ethernet subnet.

Switches and VLANs

VLANs – A single Ethernet Switch can have different switch ports in different Ethernet Subnets. Each Ethernet Subnet is called a VLAN (Virtual Local Area Network). All switch ports in the same Ethernet Subnet are in the same VLAN.

VLAN ID – Each VLAN has an ID, which is a number between 1 and 4095. Thus a Switch can have Switch Ports in up to 4095 different Ethernet Subnets.

Switch Port VLAN configuration – a Switch administrator assigns each switch port to a VLAN ID. By default, Switch Ports are in VLAN 1 and shutdown. The Switch administrator must specify the port’s VLAN ID and enable (unshut) the Switch Port.

Pure Layer 2 Switches don’t route – When a Switch receives a packet for a port in VLAN 10, it only switches the packet to other Switch Ports that are also in VLAN 10. Pure Layer 2 Switches do not route (forward) packets between VLANs.

Some Switches can route – Some Switches have routing functionality (Layer 3). The Layer 3 Switch has IP addresses on multiple Ethernet subnets (one IP address for each subnet). The client machine has the Default Gateway set to the Switch’s IP address that’s in the same subnet as the client. When Ethernet packets are sent to the Switch’s MAC address, the Layer 3 Switch forwards (routes) the packets to a different IP subnet.

DHCP (Dynamic Host Configuration Protocol)

Static IP Addresses or DHCP (Dynamic) IP Addresses – Before a machine can communicate on an IP network, the machine needs an IP address. The IP address can be assigned statically by the administrator, or the machine can get an IP address from a DHCP Server.

DHCP Process – When a DHCP-enabled machine boots, it sends a DHCP Request broadcast packet asking for an IP address. A DHCP server sees the DHCP IP address request and sends back a DHCP reply with an IP address in it.

  • Avoid IP Conflicts – DHCP servers keep track of which IP addresses are available. Before the IP address is returned to a DHCP Request, some DHCP servers will ping the candidate IP address to make sure it doesn’t reply.
  • Multiple DHCP Responses – if a DHCP Client machine receives multiple DHCP Responses from multiple DHCP Servers, then the DHCP Client machine will accept and acknowledge the first response. The other unacknowledged responses will time out and their candidate IP addresses will be returned to the IP Address pools.
  • DHCP Lease Expiration – DHCP-issued addresses are only valid for a period of time (days). Before the expiration time, the DHCP Client asks the DHCP Server to renew the lease expiration time for the issued IP Address. If the timer expires, then the previously issued IP Address is returned to the DHCP pool.
    • Short Lease for Non-persistent machines – If you are building and tearing down many machines (e.g. non-persistent virtual desktops) quickly, then you want the DHCP Lease time to be short, usually one hour. That way IP addresses from deleted machines are quickly returned to the DHCP Pool.

DHCP Requests don’t cross routers – The DHCP Request broadcast is Layer 2 (Ethernet) only, and won’t cross Layer 3 boundaries (routers).

  • DHCP Server on same subnet as DHCP Client – If the DHCP server is on the same subnet as the DHCP client, then the DHCP Server will see the DHCP Request and reply to it. But this is rarely the case.
  • DHCP Server on subnet that’s remote from DHCP Client – If the DHCP server is on a different subnet than the client, then the local router needs to forward the DHCP request to the remote DHCP server. DHCP Request Forwarding is not enabled by default and must be configured by a router administrator. Cisco routers call the feature IP Helper Address or DHCP Proxy/Fowarder.
    • Forward to Multiple DHCP Servers – the router administrator specifies the IP addresses of the remote DHCP Servers. Make sure they configure more than one DHCP Server to forward to.
    • DHCP Request Forwarding in datacenters – DHCP Request Forwarding is usually not enabled in datacenter subnets. However, DHCP is usually required for virtual desktops and non-persistent RDSH servers. In a Citrix or VDI implementation, you typically ask the datacenter network team to create new subnets (new VLANs) so DHCP can be enabled on those new VLANs without affecting existing subnets.
  • Citrix Provisioning Servers can host DHCP Server – if you have Citrix Provisioning Target Devices (VDAs) and Citrix Provisioning Servers on the same datacenter subnet, then you can avoid router DHCP configuration by simply installing DHCP Server on the Citrix Provisioning Servers.

DHCP Scopes – A single DHCP server can hand out IP addresses to multiple subnets. Each subnet is a different DHCP Scope. When routers forward DHCP Requests to a DHCP Server, the forwarded Request contains the client’s IP subnet info so the DHCP Server knows which DHCP Scope the issued IP address should come from.

DHCP Scope Options – DHCP Servers can include additional address configuration data in their IP Address responses. Commonly configured additional address configuration data includes: default gateway IP address, DNS server IP addresses, DNS Search Suffix (see below), etc.

  • Subnet-specific – Some of these items, like Default Gateway, are subnet specific, so each DHCP Scope is configured with different Scope Options.
  • Global – Some data, like DNS Server addresses, apply to all scopes and thus are configured as Global DHCP Scope Options. Scope-specific DHCP Scope Options override Global DHCP Scope Options.

DHCP Database – DHCP scope configuration and the list of DHCP-issued IP addresses are stored in a DHCP database on each DHCP Server.

DHCP Server Redundancy – If the DHCP Server is down, then DHCP Clients cannot get an IP address when they boot, and thus can’t communicate on the network. You typically need at least two DHCP Servers.

  • DHCP Database Replication – If there are multiple DHCP Servers, then the DHCP database must be replicated. Windows Server 2012 and newer have a DHCP Scope (i.e. Database) replication capability. As do other DHCP servers like Infoblox.
  • Split Scope – If the DHCP Database is not replicated, then configure each DHCP Server with different IP Address Pools to ensure that there aren’t two DHCP Servers giving out the same IP addresses. You can take a large pool of addresses and split the pool across the DHCP Servers.

DNS (Domain Name Server)

DNS converts words to numbers – When users use a browser to visit a website, the user enters a human-readable, word-based address. However, machines can’t communicate using words, so these words must first be converted to a numeric address. That’s the role of DNS.

DNS Client – Every client machine has a DNS Client. The DNS Client talks to DNS Servers to convert word-based addresses (DNS names) into number-based addresses (IP addresses).

DNS Query – The DNS Client sends a DNS Query containing the word-based address to a DNS Server. The DNS Server sends back an IP Address. The client machine then connects to the IP Address. (image from Wikimedia)

DNS and Traffic Steering – DNS Servers have much leeway in how they choose an IP address to return in response to a DNS Query.

  • DNS Servers can be simple static databases, where the same IP address response is given every time a Query is received for a particular DNS name.
  • Or DNS Servers can be dynamic where the DNS Servers gives out different IP addresses depending on IP reachability and the client’s proximity to the IP address.

DNS Servers play an important role in Traffic Steering (aka Intelligent Traffic Management) by giving out different IP addresses based on network conditions. On Citrix ADC, the GSLB (Global Server Load Balancing) feature performs DNS Traffic Steering.

Do not overlook the importance of DNS when troubleshooting an HTTP Client connectivity problem.

DNS Servers configured on client machine – On every client machine, you specify which DNS Servers the DNS Client should use to resolve DNS names into IP addresses. You enter the IP addresses of two or more DNS Servers.

  • DHCP can deliver DNS Server addresses – These DNS Server IP addresses can also be delivered by the DHCP Server as Scope Options when the DHCP Client requests an IP address.

Local DNS Servers – DNS Clients do not resolve DNS names themselves. Instead, DNS Clients send the DNS Query to one of their configured DNS Servers, and the DNS Server resolves the DNS Name into an IP address. The DNS Server IP addresses configured on the DNS Client are sometimes called Local DNS Servers and/or Resolvers.

  • Recursive queries – A DNS Server can be configured to perform recursive queries. When a DNS Client sends a DNS Query to a DNS Server, if the DNS Server can’t resolve the address using its local database, then the recursive DNS Server will walk the DNS tree to get the answer. If Recursion was not enabled, then the DNS server would simply send back an error (or a referral) to the DNS client.

DNS scalability – The Internet has billions of IP addresses. Each of these IP addresses has a unique DNS name associated with it. It would be impossible for a single DNS server to have a single database with every DNS name contained within it. To handle this scalability problem, DNS names are split into a hierarchy, with different DNS servers handling different portions of the hierarchy. The DNS hierarchy is a tree structure, with the root on top, and leaves (DNS records) on the bottom. (image from Wikimedia)

DNS names and DNS hierarchy – A typical DNS name has multiple words separated by periods. For example, www.google.com. Each word of the DNS name is handled by a different portion of the DNS hierarchy (different DNS Servers).

Walk the DNS tree – Resolving a DNS name into an IP address follows a process called “Walk the tree”. It’s critical that you understand this process: (image from Wikimedia)

  1. Implicit period (root) – DNS names are read from right to left. At the end of www.google.com is an implicit period. So the last character of every fully qualified DNS name is a period (e.g. www.google.com.), which represents the top (root) of the DNS tree.
  2. Next is .com. The DNS recursive resolver sends a DNS Query to the parent Root DNS Servers asking for the IP Addresses of the DNS Servers that host the .com DNS zone.
    • The DNS recursive resolver is configured with a hard coded list of DNS Root Server IP addresses, also called Root Hints.
    • The root DNS servers are usually owned and operated by government agencies, or large service providers.
    • The root DNS Servers have a link (aka delegation or referral) to the .com DNS Servers.
  3. Next is google.com. The DNS recursive resolver sends a DNS Query to the parent .com DNS Servers asking for the IP Addresses of the DNS Servers that host the google.com DNS zone.
    • The .com DNS Servers are usually owned and maintained by the Internet Domain Registrars.
    • The .com servers have a link (aka delegation or referral) to the google.com DNS Servers.
  4. Finally, the DNS recursive resolver asks the google.com DNS Servers to resolve www.google.com into an IP address.
    • The google.com DNS Servers can resolve www.google.com directly without linking or referring to any other DNS Server.

DNS Caching – Resolved DNS queries are cached for a configurable period of time. This DNS cache exists on both the Resolver/Recursive DNS Server, and on the DNS Client. The caching time is defined by the TTL (Time-to-live) field of the DNS record. When a DNS Client needs to resolve the same DNS name again, it simply looks in its cache for the IP address, and thus doesn’t need to ask the DNS Resolver Server again.

  • If two DNS Clients are configured to use the same Local DNS Servers/Resolvers, when a second DNS Client needs to resolve the same DNS name that the first DNS Client already resolved, the DNS Resolver Server simply looks in its cache and sends back the response and there’s no reason to walk the DNS tree again, at least not until the TTL expires.

DNS is not in the data path – Once a DNS name has been resolved into an IP Address, DNS is done. The traffic is now between the user’s client software (e.g. web browser), and the IP address. DNS is not in the data path. It’s critical that you understand this, because this is the source of much confusion when configuring ADC GSLB.

FQDN – When a DNS name is shown as multiple words separated by periods, this is called a Fully Qualified Domain Name (FQDN). The FQDN defines the exact location of the DNS Name in the DNS tree hierarchy.

DNS Suffixes – But you can also sometimes just enter the left word of a DNS name and leave off the rest. In this case, the DNS Client will append a DNS Suffix to the single word, thus creating a FQDN, and send the FQDN to the DNS Resolver to get an IP address. A DNS Client can be configured with multiple DNS Suffixes, and the DNS Client will try each of the suffixes in order until it finds one that works.

  • When you ping a single word address, ping will show you the FQDN that it used to get an IP address.

Authoritative DNS Servers – Each small portion of the DNS hierarchy/tree is stored on different DNS servers. These DNS servers are considered “authoritative” for one portion of the DNS tree. When you send a DNS Query to a DNS Server that has the actual DNS records in its configuration, the DNS Server will send back the IP Address and flag the DNS response as “authoritative”. But when you send a DNS query to a DNS Resolver that doesn’t have google.com‘s DNS records in its local database, the DNS Recursive Resolver will get the answer from google.com‘s DNS servers, and your local DNS Server flags the DNS Response as “non-authoritative”. The only way to get an “authoritative” response for www.google.com is to ask the google.com‘s DNS servers directly.

  • DNS Zones – The portion of the DNS tree hosted on an authoritative DNS server is called the DNS Zone. A single DNS server can host many DNS Zones. DNS Zones typically contain only a single domain name (e.g. google.com). If DNS records for both company.com and corp.com are hosted on the same DNS server, then these are two separate zones.
  • Zone Files – DNS records need to be stored somewhere. On UNIX/Linux DNS servers, DNS records are stored in text files, which are called Zone Files. Microsoft DNS servers might store DNS records inside of Active Directory instead of in files.

DNS records – Different types of DNS records can be created on authoritative DNS servers:

  • A (host) – this is the most common type of record. It’s simply a mapping of one FQDN to one IP address.
    • DNS Round Robin – If you create multiple Host records (A records) with the same FQDN, but each A record has a different IP address, then the DNS Server will round robin pick one of the IP Addresses for its DNS Response. Different DNS Queries will get different IP addresses in their DNS Response. This is called DNS Round Robin load balancing. One problem is that most DNS Servers do not know if the IP address is reachable or not.
  • CNAME (alias) – this record aliases one FQDN into another FQDN. To get the IP address for a CNAME FQDN, get the IP address for a different FQDN instead. See below for details.
  • NS (name server) – NS records are referrals to other DNS servers that are authoritative for a specified portion of the DNS hierarchy.
    • To delegate a FQDN or DNS sub-zone to a different DNS Server, you create NS records in the parent zone.
    • Create NS records when you need to delegate DNS resolution to Citrix ADC so Citrix ADC’s GSLB feature can intelligently resolve DNS names to multiple IP Addresses.

Resolving a CNAME – While the DNS Resolver is walking the tree, a CNAME might be returned instead of an IP address. The CNAME response contains a new FQDN. The Resolver starts walking the tree again, but this time for the new FQDN. If the DNS Resolver gets another CNAME, then it starts over again until it finally gets an IP Address. The IP Address is returned in the DNS Response to the original DNS Query.

  • Actual FQDN – The DNS Response also contains the actual FQDN that the IP address actually came from, instead of the original FQDN that was in the DNS Query.
    • If you ping the original FQDN, ping shows you the CNAME’d FQDN.
    • However, if a Web Browser performed the DNS Query, then the Web Browser ignores the CNAME’d FQDN and instead leaves the original FQDN in the browser’s address bar. This means that DNS CNAME does not cause a Browser redirect.
  • CNAMEs and Cloud – CNAMEs are used extensively in Cloud Web Hosting services. If you create a website on Azure Web Apps, Azure creates a DNS A record for the website but the A record FQDN is based on Azure’s DNS suffixes, not your DNS suffix. If you want to use your own DNS suffix, then you create a CNAME from a FQDN using your DNS suffix, to the FQDN that Azure created.

Public DNS Zones – Public DNS names are hosted on publicly (Internet) reachable DNS Servers. The same companies that provide public website hosting also provide public DNS zone hosting.

Private DNS zones – Private DNS Zones are hosted on internal DNS Servers only. Private DNS zones are used extensively in Microsoft Active Directory environments to allow domain-joined machines to resolve each other’s IP Addresses. Each Active Directory domain is a different Private DNS Zone. Private DNS zones are not resolvable from Public DNS Servers.

Private DNS Servers – When a private client (behind the firewall) wants to communicate with a private server, the private DNS Clients should send a DNS Query to a Private DNS Server, not to a public DNS Server. In Microsoft Active Directory environments, Domain Controllers are usually the Private DNS Servers.

  • Private DNS Servers can resolve Public DNS names – When a private client wants to communicate with a server on the Internet, the private DNS Client sends the DNS Query for the public DNS name to a Private DNS Server. Private DNS Servers are recursive resolvers so they can walk the Internet DNS servers to resolve public DNS names too.
    • For DNS Clients that need to resolve both Private DNS names and Public DNS names, configure the DNS Clients to only use Private DNS Servers. Do not add any Public DNS Server (e.g. 8.8.8.8) to the NIC’s IP configuration. If you do, then sometimes the DNS Client might send a DNS Query for a Private DNS name to the Public DNS Server and it won’t succeed.
  • DNS Forwarding – if a Private DNS Server does not have access to the Internet, then the Private DNS Server can be configured to forward unresolved DNS Queries to a different Private (or public) DNS Server that is configured as a Recursive Resolver.
    • DNS Tree Short Circuit – DNS Forwarding can short circuit DNS Tree walking. If a corp.com DNS server wants to resolve a company.com FQDN, then you can configure the corp.com DNS servers to forward all company.com FQDNs directly to a DNS Server that hosts the company.com DNS zone. This eliminates needing to walk the tree to find the company.com DNS Servers.
    • DNS Forwarding for Private Zones – Walking the tree usually only works for public FQDNs since the root DNS servers are public DNS servers. If you have multiple private DNS zones on different private DNS Servers, you usually have to configure DNS forwarding for each of the private zones.

Split DNS – Since Public DNS and Private DNS are completely separate DNS Servers, you can host DNS Zones with the same name in both environments. Each environment can be configured to resolve the same FQDNs to different IP Addresses. If a Private DNS Client resolves a particular FQDN through a Private DNS Server, then the IP address response can be an internal IP address. If a Public DNS Client resolves the same FQDN through a Public DNS Server, then the IP address response can be a public IP address. Internal clients resolving FQDNs to internal IP addresses avoids internal clients needing to go through a firewall to reach the servers.

Physical Networking

Layer 1 (Physical cables)

ADC appliances connect to network switches using several types of media (cables).

  • Gigabit cables are usually copper CAT6 twisted pair with 8-wire RJ-45 connectors on both sides.
  • 10 Gigabit or higher bandwidth cables are usually fiber optic cables with SC connectors on both sides.

Transceivers (SFP, SFP+, QSFP+, SFP28)

  • Transceivers convert optical to electrical and vice versa – To connect a fiber optic cable between two network ports, you must first insert a transceiver into the switch ports. The transceiver converts the electrical signals from the switch or ADC into optical (laser) signals. The transceiver then converts the laser signals to electrical signals on the other side.
  • Transceivers are pluggable – just insert them.
    • Switch ports and ADC ports only accept specific types of transceivers.
  • Different types of transceiver –
    • SFP transceivers only work up to gigabit speeds.
    • For 10 Gig, you need SFP+ transceivers.
    • For 40 Gig, you need QSFP+ transceivers.
    • For 25 Gig, you need SFP28 transceivers
    • For 100 Gig, you need QSFP28 transceivers

For cheaper 10 Gig+ connections, Cisco offers Direct Attach Copper (DAC) cables:

  • Transceivers are built into both sides of the cable so you don’t have to buy the transceivers separately.
  • The cables are based on Copper Twinax. Copper means cheaper metal, and cheaper transceivers, than optical fiber.
  • The cables are short distance (e.g. 5 meters). For longer than 10 meters, you must use optical fiber instead.

Port Channels (cable bonds)

Cable Bonding – Two or more cables can be bound together to look like one cable. This increases bandwidth, and increases reliability. If you bond 4 Gigabit cables together, you get 4 Gigabit of bandwidth instead of just 1 Gigabit of bandwidth. If one of those cables stops working for any reason, then traffic can still use the other 3 cables. (image from wikipedia)

Cable Bonding does not impact network functionality – Cable bonding does not affect networking in any way. Ethernet and IP routing don’t care if there’s one cable, or if there are multiple cables bonded into a single link.

  • However, if you connect multiple unbonded cables to the same VLAN, then Ethernet and IP routing see each unbonded cable as a separate connection and this will mess up your switching and routing designs. Don’t connect multiple cables to one VLAN unless you bond those cables.

Various Names for Cable Bonding – On Cisco switches, cable bonding functionality has several names. Probably the most common name is “port channel”. Other names include: “link aggregation”, “port aggregation”, “switch-assisted teaming”, and “Etherchannel”.

Bond Both Sides – to bond cables together, you must configure both sides of the connection identically. You configure the switch to bond cables. And you configure the ADC (or server) to bond cables.

  • On ADC, the bonding feature is called Channel. On ADC, a Channel is represented by a new interface called LA/1 or something like that. LA = Link Aggregate. The 1 indicates the number of the Port Channel. An ADC can have multiple Port Channels (LA/1, LA/2, etc.), with each Channel configured for a different VLAN.

Cable Bonds have one MAC address – Each cable is plugged into a NIC port. Each NIC port has its own MAC Address. An IP Address can only be ARP’d to a single MAC address, which means the incoming traffic only goes to one of the cables. To get around this problem, when a port channel (bond) is configured, a single MAC address is shared by all of the cables in the bond, and both sides of the cable bond know that the single MAC address is reachable on all members of the cable bond.

  • If you connect two unbonded cables from one ADC to the same VLAN, then each port/cable will have its own MAC address. When a router does an ARP for an IP address that is owned by the ADC, the ARP will reach the ADC on both cables, and ADC will respond using the MAC address of each interface. Sometimes the ARP response will be for one of the MAC addresses, and sometimes the ARP response will be for the other MAC address. This results in an error called “MAC Move” and is sometimes interpreted as an IP Conflict. To avoid this problem, always bond cables that are connected to the same VLAN so that both cables will have the same MAC address.

Load Balancing across the bond members – The Ethernet switch and the ADC will essentially load balance traffic across all members of the bond. There are several port channel load balancing algorithms. But the most common algorithm is based on source IP and destination IP; all packets that match the same source IP and destination IP will go down the same cable. Packets with other combinations of source IP and destination IP might go down a different cable. If you are bonding Gigabit cables, since a single source/destination connection only goes down one cable, it can only use up to 1 Gigabit of throughput. Bonds only provide increased bandwidth if there are many source/destination combinations.

LACP – Cables can be bonded together manually, or automatically. LACP is a protocol that allows the two sides (switch and ADC) of the bonded connection to negotiate which cables are in the bond, and which cables aren’t. LACP is not the actual bonding feature; instead, LACP is merely a negotiation protocol to make bonding configuration easier.

Multi-switch Port Channels – Port Channels (bonds) are usually only supported between one ADC and one switch. To bond ports from one ADC to multiple switches, you configure something called Multi-chassis Port Channel. Multi-chassis refers to multiple switches. You almost always want multi-chassis since that lets your Port Channel survive a switch failure.

  • Virtual Port Channel – On Cisco NX-OS switches, the multi-chassis port channel feature is called “virtual port channel”, or vPC for short. When connecting a single ADC Port Channel to multiple Nexus switches, ask the network team to create a “virtual port channel”.
  • Stacked Switches – Other switches support a “stacked” configuration where multiple switches look like one switch. There are usually cables in the back of the switches that connect the switches together.

Port Channel Configuration – first, ask the switch administrator to create a port channel using multiple switch ports. The switch administrator can optionally enable LACP on the Channel. Then the ADC administrator can create a Channel on the ADC appliance.

  • Create Manual Channel On ADC – if LACP is not enabled on the switch’s port channel, then create a “manual” channel (without LACP). On the ADC, go to System > Network > Channels, create a channel, select the channel interface name (e.g. LA/2), and then add the member interfaces.
  • Create LACP Channel on ADC – if LACP is enabled on the switch’s port channel, then on the ADC, go to System > Network > Interfaces, double-click a member interface, scroll down, check the box to enable LACP, and enter a “key” (e.g. 1). All members of the same channel must have the same key. If you enter “1” as the key, then a new interface named LA/1 is created, where the “1” = the LACP key.
    • On ADC, LACP is enabled on the interface, not the Channel. There’s no need to manually create a Channel. The Chanel will be created automatically once you set the LACP Key on the member interfaces. After the Channel is automatically created on the ADC, you can Edit the Channel to see the member interfaces and make sure they are “distributing”.
    • The LACP “key” configured on the ADC does not need to match the port channel number on the switch side. ADC appliances typically have Channels named LA/1, LA/2, etc., while Switches can have port channel interfaces named po281, po282, etc.

VLAN tagging

VLAN review – Earlier I mentioned that switches can support multiple Ethernet subnets and each of these Ethernet subnets is a different VLAN. Each switch port is configured to belong to a particular VLAN. Ports in the same VLAN use Ethernet to communicate with each other. Ports in separate VLANs use routers to communicate with each other.

Multiple VLANs on one port – Switches can also be configured to allow a single switch port to be connected to multiple VLANs. An ADC usually needs to be connected to multiple subnets (VLANs). You can either assign each VLAN to a separate cable/channel, or you can combine multiple VLANs/subnets onto a single cable/channel.

VLAN tagging – If a single switch port supports multiple VLANs, when a packet is received by the switch port, the switch needs some sort of identifier to know which VLAN the packet is on. A VLAN tag is added to the Ethernet packet where the VLAN tag matches the VLAN ID configured on the switch.

  • Tags are added and removed on both sides of the switch cable – The ADC adds the VLAN tag to packets sent to the switch. The switch removes the tag and switches the packet to other switch ports in the same VLAN. When packets are switched to the ADC , the switch adds the VLAN tag so ADC knows which VLAN the packet came from.

Trunk Port vs Access Port – When multiple VLANs are configured on a single switch port (or channel), this is called a Trunk Port. When a switch port (or channel) only allows one VLAN (without tagging), this is called an Access Port. Switch ports default as Access Ports unless a switch administrator specifically configures it as a Trunk Port. Access Ports don’t need VLAN tagging, but Trunk Ports do need VLAN tagging. When you want multiple VLANs on a single switch port, ask the networking team to configure a Trunk Port.

  • Trunk Ports and VLAN ID tagging – when a switch port is configured as a Trunk Port, by default, every VLAN assigned to that Trunk Port requires VLAN ID tagging. The ADC must be configured to apply the same VLAN ID tags that the switch is expecting.
  • Trunk Ports and Native VLAN – One of the VLANs assigned to the Trunk Port can be untagged. This untagged VLAN is called the native VLAN. Only one VLAN can be untagged. Native VLAN is an optional configuration. Some switch administrators, for security reasons, will not configure an untagged VLAN (native VLAN) on a Trunk Port. If untagged VLANs are not allowed on the switch, then you must configure the ADC to tag every packet, including the NSIP. Configuring a Native VLAN on the trunk port simplifies some ADC configuration (especially NSIP and High Availability), as detailed later.

Trunk Ports reduce the number of cables – if you had to connect a different cable (or Port Channel) from ADC for each VLAN, then the number of cables (and switch ports) can quickly get out of hand. The purpose of Trunk Ports is to reduce the number of cables.

Trunk Ports and Port Channels are separate features – If you want to bond multiple cables together, then you configure a Port Channel. If you want multiple VLANs on a single cable or Port Channel, then you configure a Trunk Port. These are two completely separate features. Port Channels can be Access Ports.

Trunk Ports and Routing are separate features – Configuring a Trunk port with multiple VLANs does not automatically enable routing between those VLANs. Each VLAN on the Trunk Port is a separate Layer 2 Ethernet broadcast domain and they can’t communicate with each other without routing. Routing is configured in a separate part of the Layer 3 switch, or on a separate router device. In other words, Trunk Ports are unrelated to routing.

Multiple NICs in one machine

A single machine (e.g. ADC appliance) can have multiple NIC ports with multiple connected cables.

Single VLAN/subnet does not need VLAN configuration on the ADC – if the ADC is only connected to one subnet/VLAN, then no special configuration is needed. Just create the NSIP, SNIP, and VIPs in the one IP subnet.

  • You can optionally bond multiple cables into a Port Channel for redundancy and increased bandwidth.

Two or more NICs to one VLAN requires cable bonding – If two or more NIC ports are connected to the same VLAN, then the NIC ports must be bonded together into a Port Channel. Port Channels require identical configuration on the switch side and on the ADC side. If you don’t bond them together, then you run the risk of bridging loops and/or MAC moves.

Multiple VLANs/subnets requires VLAN configuration on the ADC – if a ADC is connected to multiple IP subnets, then the ADC must be configured to identify which subnet is on which NIC port. Here’s the configuration process:

  1. On the ADC, for each IP subnet, create a Subnet IP address (SNIP) with subnet mask for that subnet. If the ADC is connected to four IP subnets/VLANs, then you should have at least four SNIP addresses, one for each subnet.
  2. On the ADC, create a VLAN object for each IP subnet and bind each VLAN object to one network interface or Port Channel.
    • The ADC’s VLAN object asks for a VLAN ID number, and has a checkbox to indicate if the VLAN ID number is tagged or not.
  3. Bind a subnet IP address (SNIP) with subnet mask to each VLAN object so ADC knows which IP addresses are on which VLAN and by extension which network interface.

ADC VLAN objects are always required on a multi-subnet ADC, even if the switch does not need VLAN tagging – If an ADC is connected to two or more subnets, it doesn’t matter if VLAN tagging is required by the switch or not; VLAN objects (and SNIPs) still must be defined on the ADC so the ADC knows which IP subnets go with which interfaces.

ADC VLAN Tagging – On the ADC, when you create a VLAN object, you specify a VLAN ID number. This VLAN ID number can be tagged or untagged.

  • If the switch port is an access port that does not expect VLAN-tagged traffic, then the VLAN ID number entered on the ADC is only locally significant and doesn’t have to match the switch’s VLAN ID.
    • It’s easier to understand the networking configuration if the ADC’s VLAN ID and the switch port’s VLAN ID match.
  • If the switch port is a Trunk Port, then there’s a checkbox in the ADC’s VLAN object to enable tagging for this VLAN ID number.
    • The tagged VLAN ID number configured on the ADC’s VLAN object must match the tagged VLAN number configured on the switch port.
  • You can bind multiple ADC VLAN objects to a single ADC interface/channel that is connected to a switch trunk port
    • Each ADC VLAN object bound to the same Trunk Interface must be tagged.
    • Only one untagged ADC VLAN object can be bound to a Trunk Interface. The untagged VLAN will only work if the switch Trunk Port is configured with a native VLAN (untagged).

Routing table – When you create a SNIP/VLAN on a ADC, a “direct” connection is added to the routing table. You can view the routing table at System > Network > Routes. “Direct” means the ADC has a Layer 2 (ARP) connection to the IP Subnet.

One Default Route – the routing table usually has a route 0.0.0.0 that points to the Default Gateway/Router. There can only be one default route on a device even if that device is connected to multiple VLANs.. The ADC can send Layer 2 packets out any directly connected interface/VLAN, but Layer 3 packets only go out the one default route, which is on only one VLAN.

  • Add Static Routes to override Default Route – To use a different router than the default router, you add static routes to the routing table. A static route is a combination of the destination subnet you are trying to reach, and the address of the Next Hop router that can reach that destination. The Next Hop router address must be on one of the VLANs that the ADC is connected to, and is usually a different router than the default router.
  • A common use case for static routes is for an ADC appliance that is connected to both a DMZ subnet, and an internal subnet. The default router is usually a router in the DMZ that can route to the Internet. For internal networks, you add static routes for destinations to internal networks through an internal router as the next hop address.

How ADC Source IP is chosen – when a multi-subnet ADC wants to send a packet to a remote routed subnet (not directly connected), it does the following:

  1. The ADC looks in its routing table for the next hop address to reach the destination.
    • The matching routing entry can be a static route, or it can be the default route.
  2. The ADC selects one of its SNIP addresses that is on the same subnet as the next hop router address.
  3. ADC looks in its VLAN configuration to determine the network interface that is connected to the router.
    • The SNIP is bound to a VLAN object, which is bound to an interface or Channel.
  4. ADC does an ARP to get the router’s MAC address.
  5. ADC puts a packet on the wire with the subnet’s SNIP as the Source IP address and the router’s MAC address as the Ethernet destination address.
  6. Router receives the packet and routes it to the destination machine (e.g. web server).
  7. The web server responds to the Source IP in the packet, which is the ADC’s SNIP address.
  8. The web server sends the packet to its local default gateway, which routes the packet to the ADC’s SNIP address.

ADC Networking

Traffic flow through ADC

ADC-owned IP addresses – ADCs have several types of owned IP addresses: Virtual IP (VIP), Subnet IP (SNIP), NetScaler IP (NSIP), and GSLB Site IP. When you create one of these types of IP addresses on an ADC, the ADC owns that IP, which means the ADC replies to ARP requests for those IP addresses. ADC-owned IP addresses cannot be configured on any other networking device.

  • Non-ADC-owned IP addresses – When you configure an ADC to load balance web servers, you enter the IP addresses of the web servers so the ADC knows where to send the load balanced traffic. The web server IP addresses are not owned by the ADC.
  • ADCs receive traffic destined to owned IP addresses. ADCs send traffic to non-owned IP addresses.

NSIP (NetScaler IP) is the ADC’s management IP address. ADC administrators connect to the NSIP to manage the ADC appliance.

VIPs (Virtual IP) – VIPs receive traffic. When you create a Virtual Server (e.g. Load Balancing Virtual Server), you specify a Virtual IP address (VIP) that clients connect to.

  • You also specify a port number for the Virtual Server to listen on. The ADC drops traffic that is not destined to the specified port number.

SNIPs (Subnet IP) – SNIPs are the Source IP addresses that ADC uses when ADC sends traffic to a web server. When ADC appliances need to send a packet, they look in the routing table for the next hop address and select a SNIP on the same subnet as the next hop. The Source IP address of the request packet is set to the SNIP. Web servers reply to the SNIP.

The following image is from the ADC Welcome Wizard when configuring a SNIP.

Load Balancing traffic overview – here’s a simplified description of ADC Load Balancing:

  • VIP/Virtual Server – Clients send traffic to a Load Balancing VIP.
  • Services – Bound to the Load Balancing Virtual Server are one or more Load Balancing Services (or Service Group). These Load Balancing Services define the web server IP addresses and the web server port number. ADC chooses one of the Load Balancing services, and forwards the HTTP request to it.
    • ADC replaces with the source IP with one of its SNIPs. The Web Servers reply to the SNIP.
  • Monitors – ADC should not send traffic to a web server unless that web server is healthy. Monitors periodically send health check probes to web servers.

ADC Source IP and Logging

ADC SNIP replaces original Client IP – When ADC communicates with a back-end web server, the original source IP address is replaced with an ADC SNIP. The web server does not see the original Client IP address. This behavior is sometimes called Source NAT.

If SNIP is the source IP, how can web servers log the original Client IP? – Since web servers behind a ADC only see the ADC’s SNIP, the HTTP request entries in the web server access logs (e.g. IIS log) all have the same ADC SNIP as the client IP. If the web server needs to see the real Client IP, then ADC provides three options:

  • Client IP Header Insertion – when you create a Load Balancing Service on the ADC, there’s a checkbox to insert the real client IP into a user-defined HTTP Header. This HTTP Header is typically named X-Forwarded-For, or Real IP, or Client IP, or something like that. The web server then needs to be configured to extract the custom HTTP header and log it. The packets on the wire still have an ADC SNIP as the Source IP.
  • Use Source IP (USIP) – The default mode for ADC is Use Subnet IP (USNIP), which replaces the original Client IP address with ADC SNIP address. This mode can be changed to Use Source IP (USIP), which leaves the original Source IP (Client IP) in the packets. When web servers respond, they send the reply to the Client IP, and not to the ADC SNIP. If the Response does not go through the ADC , then ADC is only seeing half of the conversation, which breaks many ADC features. If you need USIP mode, then reconfigure the web server’s default gateway to point to a ADC SNIP. When the web server replies to the Client IP, it will send the reply packet to its default gateway, which is a ADC SNIP, thus allowing ADC to see the entire conversation.
    • USIP can be enabled globally for all new Load Balancing Services, or can be enabled on specific Load Balancing Services, so you can use SNIP for some web servers and USIP for others.
    • If a web server’s default gateway is an ADC SNIP, then all outbound traffic from web server, including general Internet traffic, will go through the ADC. The ADC must have a sufficient bandwidth license to handle this traffic.
    • USIP tends to complicate ADC network configurations, so avoid it if you can.
  • Direct Server Return (DSR) – An option similar to USIP is Direct Server Return (DSR). DSR does not need the web server to change its default gateway so reply traffic bypasses the ADC and doesn’t eat up the ADC’s bandwidth license. DSR is typically used for SMTP Load Balancing so SMTP Servers can filter messages based on real client IP addresses. However, DSR is a complicated configuration that requires the addition of loopback adapters to the destination servers.

ADC Forwarding Tables

ADC has at least three tables for choosing how to forward (route) a packet. They are listed below in priority order. MBF overrides PBR, which overrides routing table.

  • Mac Based Forwarding (MBF) – MBF keeps track of which interface/router a client request came in on and replies out the same interface/router. Only works for replies and doesn’t do anything for ADC-initiated connections. Since it overrides routing tables, MBF is usually discouraged.
  • Policy Based Route (PBR) – Normal routing only uses destination IP to choose the next hop router. PBRs can choose a next hop address based on packet fields other than just Destination IP. These other packet fields include: source IP, source port, and destination port. PBRs are difficult to maintain and thus most networking people try to avoid them, but they are sometimes necessary (e.g. dedicated management network).
  • Routing Table – the routes in the routing table come from three sources: SNIPs (directly connected subnets), manually-configured Static Routes (including default route), and Dynamic Routing (OSPF, BGP).

ADC networking configuration vs Server networking configuration

ADC networking is configured completely differently than server networking. ADC is configured like a switch, not like a server.

  • Servers assign IPs to NICs – On servers, you configure an IP address directly on each NIC. Most servers only have one NIC.
  • ADC appliances are configured like a Layer 3 Switch – On ADC appliances, you assign VLANs to interfaces, just like you do on a switch. Then you put ADC-owned IP addresses into each of those VLANs, which again, is just like a Layer 3 switch. More specifically, you create VLAN objects, bind the VLAN to an interface/channel, and then bind a SNIP to the VLAN.

ADC VLAN Design

ADC VLAN Options:

  • VIP VLANs – The ADC must be Layer 2 connected to any VLAN that hosts ADC VIPs (Load Balancing VIPs, Citrix Gateway VIPs, etc.)
    • Security zones – Ideally, an ADC’s VIPs should only be hosted in one security zone (e.g. DMZ only). If you need to host VIPs in multiple security zones (e.g. both DMZ and internal), then you should acquire separate ADC appliances for each security zone.
    • Route Partitioning – Citrix ADC has several technologies that can split the ADC’s single routing table into multiple routing tables. These features include: Traffic Domains, Administration Partitioning, Net Profiles, and ADC SDX multi-tenant appliances. However, each of these features has cons. See the Firewalls section below for details.
  • Dedicated management VLAN? – Dedicated management VLAN means that only the ADC NSIP (management IP) is placed on this VLAN. Don’t put any VIPs on this management VLAN. Don’t put any SNIPs on this management VLAN. If no VIPs or SNIPs, then the management VLAN cannot be used for incoming or outgoing data traffic (e.g. load balancing traffic).
    • Citrix ADC does not have a true management interface – a management interface should have its own default gateway/route that is separate from the data plane’s default route. And there should be no possibility of data traffic ever going out the management interface. Unfortunately, Citrix ADC does not support this. Alternatively, we can configure PBRs or MBF to simulate a dedicated management network, but be aware that this is simply configuration and is not enforced in hardware. For details, see Dedicated Management Subnet (PBRs).
  • Server VLANs – If the ADC can route from a VIP VLAN to the web servers, then there’s usually no need to connect the ADC directly to the web server VLANs. Most core routing is performed at wire speed so there’s no performance benefit to connecting the ADC to the server VLANs.
    • Advanced load balancing configurations like Use Source IP (USIP) and Direct Server Return (DSR) do require the ADC to be connected directly to the server VLANs.
    • Security zones – if the VIPs are hosted in a DMZ VLAN, and if you connect the ADC directly to an internal server VLAN, then there is very high risk of bypassing a firewall. Client traffic reaches the ADC through the VIPs. The ADC uses an internal SNIP to communicate with the web servers. The VIP and the SNIP are in two different security zones.

ADC Physical Connectivity

Interface 0/1 is only for management – Dedicated management VLANs are usually connected to interface 0/1 on the ADC . If you don’t have a dedicated management VLAN, then don’t use interface 0/1 on physical ADC appliances and instead, use interfaces 1/1 and higher. That’s because interface 0/1 is not optimized for high-throughput traffic.

One subnet for everything? – if the ADC is only connected to a single subnet, then the NSIP, VIPs, and SNIP will all be on one VLAN.

  • Connect two or more of the non-0/1 interfaces to two or more switches. Don’t connect anything to the 0/1 interface.
  • Bond the two data interfaces into a port channel (preferably multi-switch port channel). For configuration details, see Port Channels on Physical ADC.
  • Configure the switch’s port channel as an access port with a single VLAN.
  • On the ADC, there’s no need for any VLAN configuration.

Dedicated Management VLAN? – One option is to configure the management VLAN on its own Gigabit cable connected to the 0/1 interface on the ADC. The switch port should be an access port for the management VLAN.

Another option is to add the management VLAN to a trunk port.

  • The trunk port can be 10 Gig or higher instead of limited to the Gigabit of interface 0/1. And the trunk port is usually a port channel.
  • Can the switch trunk port be configured with the management VLAN as the native VLAN (untagged)? If so, this simplifies the ADC NSIP and ADC High Availability configuration.
  • If the management VLAN is tagged, then Citrix ADC requires NSVLAN configuration and tagging of High Availability heartbeat packets.

NSIP is special – NSIP (the management IP) lives in VLAN 1. If you don’t need to tag the management VLAN, then leave NSIP in VLAN 1. VLAN 1 on the ADC does not need to match any VLAN configured on the switch because you’re not tagging the NSIP packets with the VLAN ID. When you bind untagged VLANs to ADC interfaces, those interfaces are removed from VLAN 1 and put in other VLANs. The remaining interface in VLAN 1 is your management interface.

  • NSVLAN – if your management VLAN is tagged, then normal VLAN tagging configuration won’t work for the NSIP VLAN. Instead, you must configure NSVLAN to tag the NSIP/management packets with the VLAN ID. All other VLANs are configured normally.

Link Redundancy – for each VLAN, connect at least two cables, preferably to different switches, and then bond the cables together into a port channel.

  • Trunk Port – You can create a separate port channel for each VLAN. Or you can combine multiple VLANs onto the same port channel by configuring a Trunk Port.
  • LACP – if your port channels have LACP enabled on the switch, then on the ADC go to System > Network > Interfaces, edit two or more member interfaces, check the box for LACP, and enter the same LACP Key. If you enter 1 as the key, then a channel named LA/1 is created.
  • For manual port channels, go to System > Network > Channels, add a channel, select LA/1 or similar, and bind the member interfaces.

Disable unused Interfaces – if a ADC interface (NIC) does not have a connected cable, then disable the interface (System > Network > Interfaces, right-click an interface, and Disable). If you don’t disable the unused interfaces, then High Availability (HA) will think the interfaces are down and thus failover.

ADC VLAN Configuration

SNIPs – every VLAN needs a VLAN-specific SNIP address, except the dedicated management VLAN. If you don’t put a SNIP on the dedicated management VLAN, then the dedicated management VLAN should not be used for outbound traffic.

  • One subnet – If the ADC is only connected to one subnet, then you only need one SNIP for the entire appliance, and there’s no need to perform any ADC VLAN configuration.

NSIP VLAN – if your NSIP VLAN (management VLAN) is untagged, then you do not need to create a VLAN object for the NSIP VLAN since the ADC is already configured with VLAN 1.

  • If your NSIP VLAN is also used for data traffic, then create a SNIP in the NSIP VLAN.
  • If your NSIP VLAN needs to be tagged, then configure NSVLAN instead of the normal VLAN configuration.

VLANs – If the ADC is connected to multiple VLANs:

  1. Create a SNIP for each VLAN (except the dedicated management VLAN).
  2. Create a VLAN object for each VLAN (except the NSIP VLAN), and specify the same VLAN ID as configured on the switch.
    • It doesn’t matter if the VLAN is tagged or not, you still must create a separate ADC VLAN object for each subnet.
  3. Bind a VLAN object to one interface or channel.
  4. If the switch needs the VLAN to be tagged, then check the box to tag the packets with the specified VLAN ID.
  5. Bind the VLAN object to the SNIP for that VLAN.

Static Routes – Add static routes for internal subnets through an internal router on a “data” VLAN (not the dedicated management VLAN).

NSIP Replies – do not adjust the default route of the appliance until you configure the appliance to route NSIP replies properly. When an administrator connects to the NSIP to manage the appliance, the administrator was probably routed through a router on the NSIP VLAN. If you change the appliance’s default router to a DMZ router, then NSIP replies will go out the Internet and will never make it back to the administrator. ADC has a few options for routing these NSIP replies correctly.

  • NSIP is not on a dedicated management VLAN – in this case, the NSIP is on one of the data VLANs and you should already have static routes for internal destinations through a router on the internal data VLAN.
  • NSIP is on a dedicated management VLAN – the static routes for internal IP addresses result in NSIP replies going out the wrong interface, resulting in asymmetric routing. Or the ADC might not be connected to an internal VLAN and thus the NSIP replies go out the default DMZ (Internet-facing) router. To handle this scenario, configure one of the following on the ADC:
    • Policy Based Route – Configure a Policy Based Route (PBR) that looks for NSIP as Source IP and uses a management VLAN router as the next hop. Only packets with NSIP as source will go out the management VLAN router. All other packets will use the appliance’s routing table to look up the next hop address.
      • NSIP as source IP – some ADC features use NSIP as the source IP. These ADC features include: NTP, SNMP, Syslog, etc. PBRs properly route this NSIP-sourced traffic too.
    • Mac Based Forwarding (MBF) – MBF records which interface a packet came in on, and replies out the same interface, bypassing the routing table.
      • MBF is easy to configure, but MBF doesn’t do anything for the ADC features that initiate connections using NSIP as the source IP.

Change Default Route to DMZ router – The Default Router should be on a data VLAN (e.g. DMZ VLAN) and not on the NSIP VLAN. Now that NSIP replies and Static Routes are configured, you can probably safely delete the default route (0.0.0.0) and recreate it without you losing connection to the NSIP.

Layer 2 Troubleshooting – To verify VLAN connectivity, log into another device on the same VLAN (e.g. router/firewall) and ping the ADC SNIP or NSIP. Immediately check the ARP cache to see if a MAC address was retrieved for the IP address. If not, then layer 2 is not configured correctly somewhere (e.g. wrong VLAN configuration), or there’s a hardware failure (e.g. bad switch port).

Layer 3 Troubleshooting – There are many potential causes of Layer 3 routing issues. A common problem is incorrect Source IP chosen by the ADC. To see the Source IP, SSH to the ADC’s NSIP, run shell, and then run nstcpdump.sh host <Destination_IP>. You should see a list of packets with Source IP/Port and Destination IP/Port. Then work with the firewall and routing teams to troubleshoot packet routing.

ADC High Availability (HA)

HA Pair – You can HA pair two identical ADC appliances. The two appliances must be identical hardware, identical firmware version, and identical licensed edition.

HA and ARP – In an HA pair, each node has its own NSIP, but all of the other ADC-owned IP addresses float between the two appliances. More specifically, when there are ARP requests for VIPs and SNIPs, the primary appliance replies with its MAC addresses, but the secondary appliance does not reply to ARP requests. After an HA failover, the former secondary appliance starts responding to VIP and SNIP ARP requests with its MAC addresses, and the former primary appliance stops responding to ARPs.

  • ARP Caching and Gratuitous ARP (GARP) – When an HA pair fails over, the new primary appliance performs a Gratuitous ARP. GARP tells routers to update their ARP cache to map IP addresses to the new appliance’s MAC address instead of the old appliance’s MAC address.
    • Some routing devices (e.g. firewalls) will not accept GARP packets, and instead will wait for the ARP cache entry to time out. Or the router/firewall might not allow the IP address to move to a different MAC address. If HA failover stops all traffic, then work with the router/firewall admin to troubleshoot GARP and ARP cache.

HA monitored interfaces:

  • Disable unused interfaces – All network interfaces on ADC by default have HA monitoring enabled. If any enabled interface is down (e.g. cable not connected), then HA will failover. Disable the unused interfaces so HA won’t monitor them any more.
    • Disabling unused interfaces also stops the flashing of the front LCD.
  • Port Channels and HA failover – A port channel has two or more member interfaces. If one of the member interfaces is down, should the appliance failover? How many member interfaces must fail before HA failover should occur? On ADC, double-click the channel and you can specify a minimum throughput. If bonded throughput falls below this number due to member interface failure, then HA fails over.
  • HA Fail Safe – If at least one interface is down on both HA nodes, then both HA nodes will be unhealthy and both nodes will stop responding. You can enable HA Fail Safe so that at least one of the HA nodes will be up, even if both nodes are unhealthy.

HA heartbeat packets are untagged – Each node in a HA pair sends heartbeat packets out all interfaces. These heartbeat packets are untagged. If the switch does not allow untagged packets (i.e. no native VLAN on a Trunk Port), then some special configuration is required.

  1. On ADC, for each Trunk interface/channel, turn off tagging for one VLAN. Don’t worry about the switch configuration. Just do this on the ADC side.
  2. On ADC, go to System > Network > Interfaces (or Channels), double-click the interface/channel, and enable Tag All VLANs. The VLAN you untagged in step 1 will now be tagged again. As a bonus, HA heartbeat packets will also be tagged with the same VLAN ID you untagged in step 1.
  3. To verify that HA heartbeats are working across all interfaces, SSH to each ADC node, and run show ha node. Look for “interfaces on which HA heartbeat packets are not seen”. There should be nothing in the list.

Firewalls

DMZ – public-facing ADC VIPs should be on a DMZ VLAN that is sandwiched between two firewalls. That means the public-facing ADC must be connected to the DMZ VLAN.

  • Firewalls can route – When you connect a ADC to a DMZ VLAN, the firewall is usually the router.
  • NAT – Most DMZ VLANs use private IPs (10.0.0.0/8, 172.16.0.0/20, 192.168.0.0/16) instead of public IPs. These private IP addresses are not routable across the Internet. To make the VIPs accessible to the Internet, a firewall NATs a company-owned public IP to the private DMZ IP. Ask the firewall administrator to configure the NAT translations for each publicly-accessible DMZ VIP.
  • ADC VIPs listen on specific port numbers – the public-facing firewall only needs to allow specific port numbers to reach the public-facing VIPs. For web server load balancing, these ports are usually TCP port 80 and TCP port 443, and sometimes UDP 443.

Internal VIPs – internal VIPs (accessed by internal users) should be on an internal VLAN (not in the DMZ).

Multiple security zones – If you connect a single ADC to both DMZ and Internal, here’s how the traffic flows:

  1. Clients connect to DMZ VIPs, which goes through the firewall that separates the Internet from the DMZ.
  2. ADC internal SNIP connects to internal server. Since the ADC is connected directly to the Internal network, ADC will use an internal SNIP for this traffic. If you have a firewall between DMZ and internal, that firewall has now been bypassed.

Separate ADC appliances for DMZ and internal – Bypassing the DMZ-to-internal firewall is usually not what security teams want. Ask your Security team for their opinion on this architecture. A more secure approach is to have different ADC appliances for DMZ and internal. The DMZ appliance is connected only to DMZ (except dedicated management VLAN). When the DMZ ADC needs to communicate with an internal server, the DMZ ADC uses a DMZ SNIP to send the packet to the DMZ-to-internal firewall. The DMZ-to-internal firewall inspects the traffic, and forwards it if the firewall rules allow. The firewall rule allows the DMZ SNIP to talk to the web server, but the firewall does not allow client IPs (on the Internet) to talk directly to the web server.

Traffic Isolation – ADC has some features that can isolate/partition traffic so that the ADC traffic stays in its security zone.

  • Net Profiles – Net Profile selects a particular SNIP to be used by a Virtual Server, or Service, for traffic to back-end servers.
    • If your appliance has multiple SNIPs, create a Net Profile for each SNIP. Then bind the Net Profiles to Virtual Servers and/or Services.
    • Firewalls can permit different SNIPs to access different web servers. The firewall can allow one Virtual Server with one Net Profile to access specific internal servers, and the firewall can allow a different Virtual Server with a different Net Profile to access a different set of internal servers. The list of accessible servers for each SNIP is enforced at the firewall. However, it’s the ADC administrator’s responsibility to ensure that each Virtual Server has the correct Net Profile assigned.
    • If your ADC has multiple SNIPs on one VLAN that can reach a web server, then ADC will round robin across those SNIPs, which is very annoying for firewall configuration. It’s not possible to tell ADC to not use a SNIP unless that SNIP is in a Net Profile. To stop the round robin behavior, you must quickly assign Net Profiles (each with one SNIP) to every Virtual Server on the ADC appliance.
  • Traffic Domains – Citrix ADC supports Traffic Domains, which are similar to Cisco VRF (Virtual Route Forwarding). Traffic Domains split an ADC into multiple routing tables, which theoretically can prevent traffic from one security zone from crossing into another security zone.
  • Administrative Partitions – you carve up an MPX/VPX appliance into different administrative partitions, with each partition having access to a subset of the hardware. Each partition is essentially a separate ADC config, which means separate routing tables. VLANs can be dedicated to a partition, or VLANs can be shared across multiple partitions.
  • ADC SDX – SDX appliances carve up physical hardware into multiple virtual machines. Each VM is a full ADC VPX, each with its own configuration and own partitioned hardware (dedicated CPU, Memory, and NICs). There are no ADC feature limitations.

Network firewall (Layer 4) vs ADC Web App Firewall (Layer 7) – Most network firewalls only filter on port numbers and IP addresses, meaning that Layer 4 firewalls usually don’t inspect the traffic and instead simply permit traffic that is destined to specific port numbers.

HTTP Content Inspection – As you read earlier, HTTP is a file transfer protocol. Sometimes file uploads contain malicious content (e.g. viruses), or file downloads contain private content (e.g. credit card numbers). You probably want a firewall that can block malicious HTTP traffic. Layer 4 firewalls do not inspect the HTTP traffic and thus a Layer 4 firewall cannot protect your websites. That leaves website security as a responsibility of the web server administrator and developer.

Web App Firewalls – Layer 7 firewalls, also called Web App Firewalls (WAF), can inspect HTTP traffic and block HTTP packets that match string patterns. These matching string patterns usually come from a signature rules list.

  • Signatures – some string patterns should always cause an HTTP Request to be blocked every time the string pattern is seen in an HTTP Request. These malicious string patterns come from security firms that have identified hackers using these strings to exploit vulnerabilities in web server software. Snort is a popular provider of these string patterns.
  • OWASP Top 10 Security Checks – Web App Firewalls have additional protections for the OWASP Top 10 Most Critical Web Application Security Risks. These include: SQL Injection, Cross Site Scripting, Forceful Browsing, Buffer Overflows, etc. A WAF can block any HTTP Packet that matches these security risks, however, these blocks might break a web application’s functionality. In that case, WAF can either disable a signature rule, or relax a protection category for specific URLs.
    • Learning Mode and Relaxations – WAFs have a Learning mode where they log all HTTP traffic that would have been blocked by protection rules. The learned violations for specific URLs can be deployed as firewall relexations, just like you would permit a port number through a Layer 4 firewall. Any relaxation places the onus of security on the web server instead of on the WAF.

Citrix ADC WAF – Citrix ADC Premium Edition has a WAF feature. Here’s a brief configuration summary:

  1. Configure the web site traffic to flow through an ADC Load Balancing Virtual Server.
  2. Create an ADC WAF Signatures Object based on Snort rules.
  3. Create an ADC WAF Profile and enable various HTTP and XML security checks.
    • Enable security check Learning Mode.
  4. Create an ADC WAF Policy for the WAF Profile and then bind the WAF Policy to the Load Balancing Virtual Server.
  5. Send test traffic through the Load Balancing Virtual Server so the ADC can learn the WAF violations.
  6. Review the list of learned violations and deploy the legitimate ones as WAF relaxations. Test again.
  7. After testing, reconfigure the WAF Profile to enable enforcement. WAF will start blocking malicious HTTP traffic.

Put network firewalls in front of ADC – ADC is not designed as a Layer 4 firewall like a Cisco ASA, Check Point, or Palo Alto. Thus you should always put a network firewall in front of your ADC, even if you enabled the ADC WAF feature.

Next Step

EUC Weekly Digest – June 17, 2017

Last Modified: Nov 7, 2020 @ 6:34 am

Leave a comment

Here are some EUC items I found interesting last week. For more immediate updates, follow me at http://twitter.com/cstalhood.

For a list of updates at carlstalhood.com, see the Detailed Change Log.

 

XenApp/XenDesktop

VDA

App Layering (Unidesk)

Director/Monitoring

Provisioning Services

Receiver

NetScaler

XenMobile

VMware

EUC Weekly Digest – June 10, 2017

Last Modified: Nov 7, 2020 @ 6:34 am

2 Comments

Here are some EUC items I found interesting last week. For more immediate updates, follow me at http://twitter.com/cstalhood.

For a list of updates at carlstalhood.com, see the Detailed Change Log.

 

XenApp/XenDesktop

WEM/Profile Management

Provisioning Services

App Layering / Unidesk

Receiver

NetScaler

NetScaler MAS

XenMobile

VMware

EUC Weekly Digest – June 3, 2017

Last Modified: Nov 7, 2020 @ 6:34 am

Leave a comment

Here are some EUC items I found interesting last week. For more immediate updates, follow me at http://twitter.com/cstalhood.

For a list of updates at carlstalhood.com, see the Detailed Change Log.

 

Citrix

XenApp/XenDesktop

Director/Monitoring

WEM/Profile Management

NetScaler MAS

XenMobile

XenServer

Citrix Cloud

Microsoft

Other

Site Updates – May 2017

Last Modified: Nov 7, 2020 @ 6:35 am

3 Comments

To trigger RSS Feed, Mailing List, etc., here is the May 2017 excerpt from the Detailed Change Log.

Session Recording 2411

Last Modified: Feb 8, 2025 @ 7:52 am

90 Comments

Navigation

This article applies to Session Recording 2411, 2203 LTSR CU6, and 1912 LTSR CU9.

💡 = Recently Updated

Change Log

Planning

Citrix links:

Licensing – Citrix Virtual Apps and Desktops Premium Edition licensing is required.

Farms – There is no relation between Session Recording farms and Citrix Virtual Apps and Desktops farms. You can have Agents from multiple Citrix Virtual Apps and Desktops farms recording to a common Session Recording server. Or you can split a Citrix Virtual Apps and Desktops farm so that different Agents point to different Session Recording servers.

  • Load balancing – Session Recording 7.14 and newer can be load balanced. Build two Session Recording servers pointing to the same SQL database. Configure both of them to store recordings on the same UNC path. More details at Configure Session Recording with load balancing at Citrix Docs.
    • Note: Load Balancing was removed from Session Recording 7.15 LTSR. In Session Recording 7.16 and newer, Load Balancing is fully supported.
  • Scaling – To scale Session Recording to 20,000 users, see Hal Lange and Ryan Revord at Scaling and Load Balancing Session Recording at CUGC.

Disk space – The Session Recording server will need a hard drive to store the recordings. Disk access is primarily writes. You can also store recordings on a UNC path (this is required if load balancing).

Offloaded content (e.g. Teams, Browser Content Redirection) is not recorded.

Certificate – Session Recording server needs a certificate. The certificate must be trusted by Agents and Players. Internal Certificate Authority recommended.

  • If load balancing, on the Citrix ADC, install a certificate that matches the load balanced name.
  • On each Session Recording server, install a certificate that matches the Session Recording server name.

SQL:

  • Supported Versions = SQL 2008 R2 Service Pack 3 through SQL 2019. See Citrix Docs for the list.
    • Azure SQL Database is supported as detailed at Citrix Docs.
  • The SQL database is very small.
  • The database name defaults to CitrixSessionRecording and can be changed.
  • A separate database is created for CitrixSessionRecordingLogging.
  • Temporary sysadmin (or dbcreator and securityadmin) permissions are needed to create the database, and sysadmin can be revoked after installation.
  • SQL Browser Service must be running.
  • SQL Server High Availability (AlwaysOn Availability Groups, Clustering, Mirroring) is supported. See Install Session Recording with database high availability at Citrix Docs. And see Citrix Blog Post Session Recording 7.13 – New HA and Database Options

Session Recording Versions

Session Recording is located on the Citrix Virtual Apps and Desktops (CVAD) or XenApp/XenDesktop ISO.

Session Recording security vulnerability fixed in version 2311, version 2203 LTSR CU4, and version 1912 LTSR CU9 and later.

The most recent Current Release version of Session Recording is 2411.

For LTSR versions of Citrix Virtual Apps and Desktops (CVAD), deploy the version of Session Recording that came with your version of CVAD. The installation instructions for Session Recording 2311, Session Recording 2203, Session Recording 1912 and Session Recording 7.15 are essentially the same.

Session Recording Server Upgrade

You can upgrade from Session Recording 7.6 and newer.

  1. If this is a new installation, skip to Install.
  2. Session Recording 2203 and newer do not support Windows Server 2012 R2.
  3. Go to the downloaded Citrix Virtual Apps and Desktops (CVAD) ISO and run AutoSelect.exe.
  4. On the bottom right, click the Session Recording box.
  5. In the Licensing Agreement page, change the selection to I have read, understand, and accept the terms, and click Next.
  6. In the Summary page, click Upgrade.
  7. Click OK to acknowledge that the upgrade cannot be cancelled.
  8. The machine will probably require a restart.

    1. After the reboot, and after logging in again, you might see a Locate ‘Citrix Virtual Apps and Desktops 7’ installation media window. Don’t click anything yet.
    2. Go to the Citrix_Virtual_Apps_and_Desktops_7_2407_LTSR.iso file and mount it.
    3. Go back to the Locate ‘Citrix Virtual Apps and Desktops 7’ installation media window.
    4. On the left, expand This PC, and click the DVD Drive.
    5. Click Select Folder.
  9. In the Finish page, click Finish.
  10. Also upgrade Broker_PowerShellSnapIn_x64.msi from \x64\Citrix Desktop Delivery Controller on the CVAD ISO.

  11. Upgrade the Session Recording Agents.

Session Recording Server New Installs

Install

  1. Session Recording 2203 and newer no longer support Windows Server 2012.
  2. Go to the downloaded Citrix Virtual Apps and Desktops (CVAD) ISO and run AutoSelect.exe.
  3. On the bottom right, click the Session Recording box.
  4. In the Licensing Agreement page, change the selection to I have read, understand, and accept the terms, and click Next.
  5. In the Core Components page, uncheck the box next to Session Recording Player. The Player is typically installed on physical workstations, but not on the Session Recording server. Click Next.
  6. In the Features page, on the first Session Recording server, install everything.
  7. In the Database and Server page, fill out the fields. Enter the SQL server name. Azure SQL Database is supported as detailed at Citrix Docs. Enter the database name. Click Test connection. Each load balanced Session Recording server must point to the same database.
  8. Session Recording 2402 and newer automatically assign the local machine’s computer account as db_owner to the new database. In older Session Recording, enter the domain\computer_account$ for the Session Recording server. Click Next.
  9. In the Administrator Logging Configuration page, enter the name of the SQL database, click Test connection, and then click Next.
  10. In the Summary page, click Install.
  11. The installer might require a restart. Let it restart, and login again.

    1. After the reboot, and after logging in again, you might see a Locate ‘Citrix Virtual Apps and Desktops 7’ installation media window. Don’t click anything yet.
    2. Go to the Citrix_Virtual_Apps_and_Desktops_7_2407_LTSR.iso file and mount it.
    3. Go back to the Locate ‘Citrix Virtual Apps and Desktops 7’ installation media window.
    4. On the left, expand This PC, and click the DVD Drive.
    5. Click Select Folder.
  12. In the Finish page, click Finish.

Antivirus Exclusions

See Endpoint Security and Antivirus Best Practices at Citrix Tech Zone

IIS Certificate

  1. Use MMC Certificates snap-in (certlm.msc), or IIS, or similar, to request a machine certificate.
  2. In IIS Manager, right-click the Default Web Site, and click Edit Bindings.
  3. On the right, click Add.
  4. Change the Type to https.
  5. Select the certificate, and click OK.

Session Recording Server Configuration

  1. From Start Menu, run Session Recording Server Properties.
  2. In the Storage tab, specify a path that has disk space to hold the recordings. UNC is supported. If load balancing, UNC is required. Azure Files is an option in Session Recording 2103 and newer.

    1. When using a UNC path, all Session Recording servers (AD computer objects) need modify access.
    2. The share must have a subfolder. The recordings will be saved to the subfolder.
    3. In the Session Recording Server Properties tool, add the UNC path with subdirectory to the Storage tab.
  3. In the Signing page, select (Browse) a certificate to sign the recordings.
  4. In the Playback tab, notice that Session Recording files are encrypted before transmit. Also, it’s possible to view live sessions but live sessions are not encrypted.
  5. In the Notifications tab, you can change the message displayed to users before recording begins. Session Recording 2311 and newer have an option to Allow end user to deny recording of their session.

  6. The CEIP tab lets you enable or disable the Customer Experience Improvement Program.
  7. See https://www.carlstalhood.com/delivery-controller-cr-and-licensing/#ceip for additional places where CEIP is enabled.
  8. The Logging tab lets you configure Logging.
  9. The RBAC tab lets you enable Recording Viewing Policies.
  10. The Email tab is for Email Alert Policies.
  11. Session Recording 2206 adds the Report tab, which lets you schedule weekly reports with statistics on screen recordings: Total size of all recordings; Storage Usage per day; and number of sessions recorded per day. Session Recording 2311 can Send event response reports by email.
  12. The Cloud DB tab lets you store the Recording database in Azure SQL.
  13. Web Player tab lets you hide recordings in Web Player unless the user received a URL in an Email Alert Policy.
  14. When you click OK, you’ll be prompted to restart the service.
  15. Session Recording relies on Message Queuing. In busy environments, it might be necessary to increase the Message Queuing storage limits. See CTX209252 Error: “Data lost while recording file…” on Citrix SmartAuditor.


David Ott Session Recording Cleanup Script: You may notice that the session recording entries/files don’t go away on their own. Here is how to clean them up. Just create a scheduled task to run the code below once per day (as system – elevated). See David’s blog post for details.

C:\Program Files\Citrix\SessionRecording\Server\Bin\icldb.exe remove /RETENTION:7 /DELETEFILES /F /S /L

Session Recording 2303 and newer support the RELOCATE parameter to the IMPORT command and the FILTER parameter to the ARCHIVE and REMOVE commands of the ICLDB utility. See CTX134777 How To Remove Old Session Recording Files From File Server And Database and Citrix Docs Manage Recordings.

Load Balancing

Note: Session Recording load balancing was removed from 7.15 LTSR but added back in 7.16 and newer.

  1. In SQL Server Management Studio, make sure each load balanced Session Recording server (AD computer account) is granted db_owner role in the Session Recording databases.
  2. On each Session Recording server, open regedit.
  3. Navigate to HKLM\Software\Citrix\SmartAuditor\Server.
  4. Create a new DWORD value named EnableLB and set it to 1. Repeat on both Session Recording servers.
  5. Configure Citrix ADC load balancing as detailed at Citrix Docs. Load balance ports 80, 1801, 22334, and 443.
    • The 443 vServer Load Balancing Method should be set to LEASTBANDWIDTH (or LEASTPACKETS).
    • The 22334 vServer needs WebSockets enabled in an HTTP profile.
  6. Create a DNS host record that resolves to the Load Balancing VIP and matches the certificate bound to the vServer.
  7. Go to C:\Windows\System32\msmq\Mapping and edit the file sample_map.xml.
  8. Follow the instructions at Configure Session Recording with load balancing at Citrix Docs. Each Session Recording server has a unique configuration for this file since the <to> element points to the local server name.
  9. When saving the file, you might have to save it to a writable folder, and then move it to C:\Windows\System32\msmq\Mapping.
  10. Then restart the Message Queuing service on each Session Recording server.
  11. C:\Program Files\Citrix\SessionRecording\Scripts\SrServerConfigurationSync.ps1 can export configurations from this Server registry and import the registry to the other Session Recording Server registries. You can also use the SrServerConfigurationSync.ps1 script to add redirection mapping for message queuing. See Citrix Docs for details.

Authorization

Authorization is configured separately on each load balanced Session Recording server.

  1. From the Start Menu, run Session Recording Authorization Console.
  2. Right-click the PolicyAdministrator role, expand Assign Users and Groups, click From Windows and Active Directory and then add your Citrix Admins group.
  3. If you use Director to configure Session Recording, add the Director users to the PolicyAdministrator role.
  4. In the Player role, add users that can view the recordings.
  5. By default, nobody can see the Administration Log. Add auditing users to the LoggingReader role.
  6. Repeat the authorization configuration on additional load balanced Session Recording servers.
  7. In Session Recording 2009 and newer, if you open WebPlayer (https://SRFQDN/WebPlayer), you can see the Administrator Logging. Only members of the LoggingReader role can see the data.

    • In older versions of Session Recording, opens the webpage https://SR01.corp.local/SessionRecordingLoggingWebApplication/. Only members of the LoggingReader role can see the data.

Recording Policies

  1. Recording Policies can be configured to apply only specific Delivery Groups. To enumerate the Delivery Groups, on your Session Recording server, install Broker_PowerShellSnapIn_x64.msi, which is located under \x64\Citrix Desktop Delivery Controller on the CVAD ISO (e.g., CVAD 2402).

    • You’ll need to update this snap-in whenever you update CVAD.

  2. From the Start Menu, run Session Recording Policy Console.
  3. Enter the hostname of the Session Recording server, and click OK.
  4. Under Recording Policies, only one policy can be enabled at a time. By default, no recording occurs. To enable recording, right-click one of the built-in policies and click Activate Policy. Session Recording 2308 and newer have built-in policies to record audio.
  5. Or you can create your own policy by right-clicking Recording Policies and clicking Add New Policy.
  6. After the policy is created, right-click it, and click Add Rule.
  7. Decide if you want notification or not. Decide if you want to record only events. Session Recording 2308 and newer have options to record audio and/or lossy screen recording. Session Recording 2311 and newer let you Hide specific applications during screen recording. Click Next. 

    1. Event only recording is available in Session Recording 2106 and newer. In the Web Player, it looks something like this:
  8. Session Recording 2109 adds statistics to the Web Player. See View graphical event statistics at Citrix Docs.
  9. Click OK to acknowledge this message.
  10. Choose the rule criteria. You can select more than one. Session Recording has an IP Address or IP Range rule.
  11. Then click the links on the bottom specify the groups, applications, servers, and/or IP range for the rule.

  12. Session Recording 2402 and newer have a Filter option where you can enter SmartAccess tags.
  13. Click Next.
  14. Give the rule a name and click Finish.
  15.  Continue adding rules.
  16. When done creating rules, right-click the policy, and click Activate Policy.
  17. You can also rename the policy you created.

Recording Viewing Policies

Session Recording 1906 and newer support creating policies to limit whose recordings a viewer can see.

  1. On Session Recording servers 1909 and newer, open Session Recording Server Properties, switch to the tab named RBAC, and check the box next to Allow to configure recording viewing policies.

  2. When you re-open the Citrix Session Recording Policy Console, you’ll see a new node named Recording Viewing Policies.
  3. Right-click Recording Viewing Policies and click Add New Policy.
  4. Right-click the new policy and click Add Rule.
  5. In the Rules Wizard, specify a user group that can view recordings, specify user groups whose recordings can be viewed, and then click Next.

    • Make sure the “view recordings” group is also added to the Player role in the Authorization Console.
  6. Click Finish to close the wizard.
  7. You can right-click the Viewing Policy to rename it.
  8. Recording Viewing Policies do not need to be activated.
  9. You can create more than one Recording Viewing Policy.

Event Response Policies

Session Recording 2009 and newer support creating policies to send emails when a user’s session starts. Session Recording 2012 and newer can send email notifications when an event occurs. Session Recording 2109 and newer can trigger session recording after events occur.

  1. On Session Recording servers 2009 and newer, open Session Recording Server Properties, switch to the tab named Email, and complete the fields. Make sure you check the box to Allow sending email notifications.

  2. In Citrix Session Recording Policy Console, right-click Event Response Policies and click Add New Policy.
  3. Right-click the new policy and click Add Rule.
  4. In the Rules Wizard, check the desired boxes.

    • For “event triggers”, click the button labelled Configure Event Triggers and select the events you want a response for. Session Recording 2203 and newer have more options for Event type.
    • On the far right, check boxes if you want Send email, Start screen recording, or both. Session Recording 2206 adds Log off session, Disconnect session, and Lock session.
  5. In Step 1-2, enter Email recipients and Time span for dynamic screen recording. Click Next.
  6. In Step 2, enter criteria for when this rule should trigger, and then click Next.
  7. In Step 4, you can name the rule. Click Finish to close the wizard.
  8. You can right-click the Event Response Policy to rename it.
  9. Then Activate the new policy. Only one policy can be activated.
  10. The email notification looks something like this. Notice that the Playback URL is not a FQDN.

    • To fix it, go to HKLM\Software\Citrix\SmartAuditor\Server and configure the LinkHost value.

Session Recording Agent

Agent Installation

Install the Agent on the VDAs. Platinum Licensing is required.

  1. Session Recording 2203 and newer no longer support Windows Server 2012.
  2. On the Master VDA, go to the downloaded Citrix Virtual Apps and Desktops (CVAD) ISO, and run AutoSelect.exe.
  3. On the bottom right, click the Session Recording box.
  4. In the Licensing Agreement page, change the selection to I have read, understand, and accept the terms, and click Next.
  5. In the Core Components page, uncheck everything except Session Recording Agent. Click Next.
  6. In the Agent page, enter the FQDN of the Session Recording server (or load balanced FQDN), click Test connection, and click Next.
  7. In the Summary page, click Install.
  8. In the Finish page, click Finish.
  9. Agent Installation can also be automated. See Automating installations at Citrix Docs.
  10. For antivirus exclusions, see Endpoint Security and Antivirus Best Practices at Citrix Tech Zone

Agent Configuration

  1. In the Agent’s Start Menu is Session Recording Agent Properties.
  2. You can enable or disable session recording on this Agent.
  3. For Custom event recording, notice the option for third party applications like Intelligent Session Recording (ISR) Nanobot for regulatory compliance from automai that prevents sensitive fields (e.g., Social Security Numbers and Credit Card numbers) from being captured. (h/t CTP Shane Kleinert)
  4. Compressed recording: Session Recording 2303 and newer let you compress parts of recordings that are not already H.264 compressed.
  5. Session Recording 2308 and newer, on the Player tab, support fast seeking.

    • In Web Player, at Configuration > Preferences, make sure Enable fast seeking is checked.
  6. For MCS and PVS VDAs, see the GenRandomQMID.ps1 script at Install, upgrade, and uninstall Session Recording at Citrix Docs.
  7. Session Recording Agent might cause MCS Image Prep to fail. To work around this, set the Citrix Session Recording Agent service to Automatic (Delayed Start). Source = Todd Dunwoodie at Session Recording causes Image preparation finalization Failed error at Citrix Discussions.

Event Detection Policies

Session Recording can automatically mark events in recordings when certain actions occur inside the session. An example event is when USB client drives are connected. Newer versions of Session Recording can record more events than older versions of Session Recording.

The Citrix Session Recording Policy Console has a node named Event Detection Policies. Just like Recording Policies, you can add an Event Detection Policy, add a rule to the policy, and then activate the policy. Event Detection is disabled by default.

  1. Details at Configure event detection policies at Citrix Docs and Citrix Blog Post Session Recording 1903 available for Citrix Virtual Apps and Desktops.
  2. Session Recording 2407 and newer can Log printing activities. It’s at the bottom of the list.
  3. Session Recording 2203 and newer can Log popup windows events. Notice the scroll bar on the far right. Popup windows in web browsers are not monitored.
  4. Session Recording 2109 and newer can log registry modifications.
  5. Session Recording 2012 and newer can log clipboard activities.
  6. After adding rules to an Event Detection Policy, make sure you Activate Policy.

In older versions of Session Recording, you enable Event recording by modifying the registry of each Session Recording Agent:

  1. Configure the following registry values at HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\SmartAuditor\SessionEvents on the Session Recording Agent machine:
    • EnableAppLaunchEvents (DWORD) = 1
    • EnableCDMUSBDriveEvents (DWORD) = 1
    • EnableFileOperationMonitorEvents (DWORD) = 1
    • EnableGenericUSBDriveEvents (DWORD) = 1
    • EnableSessionEvents (DWORD) = 1
  2. Session Recording 7.18 and newer can mark recording events when specific processes are launched. This feature is disabled by default. To enable, set EnableAppLaunchEvents at the same key. And put the list of processes in AppMonitorList. More details at Log application starts at Citrix Docs and Citrix Blog Post Session Recording 1808 Product Update – Log application activities and more!.
    • EnableAppLaunchEvents (DWORD) = 1
    • AppMonitorList (REG_MULTISZ) = list of process names
  3. Session Recording 1811 and later can mark recording events when files in a folder are renamed, created, deleted, and moved. Add the path strings of target folders into the FileOperationMonitorList registry value. More details at Citrix Blog Post Session Recording 1811 is here for Citrix Virtual Apps and Desktops.

When you later playback the recording, the events are shown on the bottom left.

  • Or you can perform an Advanced Search for recordings with specific event types.

Session Recording Web Player

Session Recording 1912 and newer include a Web Player.

Web Player has some features that the standard Player does not.

Web Player is enabled by default in Session Recording 2003 and newer. To enable it in Session Recording 1912:

  1. Go to C:\Program Files\Citrix\SessionRecording\Server\Bin, right-click TestPolicyAdmin.exe and click Copy as path.
  2. Open Command Prompt as administrator, paste the path, add “-enablewebplayer” to the end, and press Enter.

Bind a certificate to the Web Player:

  1. After upgrading to Session Recording 2103 or newer, check HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\SmartAuditor\Server\WebSocketServerVersion.

    1. If it is set to 1.0, then we can change it to IIS instead of Python by running "C:\Program Files\Citrix\SessionRecording\Server\Bin\SsRecUtils.exe" -enablestreamingservice.
    2. The registry value then changes to 2.0.
  2. In IIS Manager, edit the bindings for the Default Web Site and ensure there’s an https binding.
  3. Go back to C:\Program Files\Citrix\SessionRecording\Server\Bin and edit the file SsRecWebSocketServer.config (2003 and newer) or SsRecWebSocketServer.exe.config (1912 and older).
  4. In line 3, set TLSEnable to 1.
  5. If your Session Recording is 2103 and later and hosts the WebSocket server in IIS (i.e.WebSocketServerVersion = 2.0)
    1. In Administrator Command Prompt, run the following command. 
      "C:\Program Files\Citrix\SessionRecording\Server\Bin\SsRecUtils.exe" -stopwebsocketserver
    2. From any machine, launch a browser and point it to https://myrecordingserver.corp.com/WebPlayer or something like that. Internet Explorer is not supported.
    3. Skip the rest of this section.
  6. For Session Recording 2012 and older, or if your WebSocketServer is not integrated with IIS, continue the steps in this section.
  7. You need the Session Recording certificate in PEM format. One option is to use the openssl commands listed at Citrix Docs. Or you can use an ADC appliance’s Import PKCS#12 feature to convert PFX to PEM.
  8. Make sure you don’t encrypt the private key (don’t select Encoding Format).
  9. After conversion (aka import), go to Traffic Management > SSL > SSL Files, switch to the tab named Certificates, and download the .pem file.
  10. Put the unencrypted .pem file somewhere on the Session Recording server.
  11. In the SsRecWebSocketServer.config file, set line 11 and line 13 to the full path to the .pem file.
  12. In Administrator Command Prompt, run the following command: 
    "C:\Program Files\Citrix\SessionRecording\Server\Bin\SsRecUtils.exe" -stopwebsocketserver

  13. The Web Player WebSockets listens on port 22334 by default. Open the port on the firewall.

    1. Create an Inbound Rule. You can either create a Program rule, or a Port rule.
    2. The program path is %ProgramFiles%\Citrix\SessionRecording\Server\Bin\SsRecWebSocketServer.exe.
  14. From any machine, launch a browser and point it to https://myrecordingserver.corp.com/WebPlayer or something like that. Internet Explorer is not supported.
  15. Web Player lets you share recordings as links. These links can be Restricted, or in Session Recording 2305 or newer can be Unrestricted. More details at Citrix Docs.

Session Recording Player

Install the Player on any Windows 7 through Windows 10 desktop machine. 32-bit color depth is required. Because of the graphics requirements, don’t run the Player as a published application.

  1. Go to the downloaded Citrix Virtual Apps and Desktops (CVAD) ISO and run AutoSelect.exe.
  2. On the bottom right, click the Session Recording box.
  3. In the Licensing Agreement page, change the selection to I have read, understand, and accept the terms, and click Next.
  4. In the Core Components page, uncheck everything except Session Recording Player. Click Next.
  5. In the Summary page, click Install.
  6. The installer might require a restart. Let it restart, and login again.

    1. After the reboot, and after logging in again, you might see a Locate ‘Citrix Virtual Apps and Desktops 7’ installation media window. Don’t click anything yet.
    2. Go to the Citrix_Virtual_Apps_and_Desktops_7_2407_LTSR.iso file and mount it.
    3. Go back to the Locate ‘Citrix Virtual Apps and Desktops 7’ installation media window.
    4. On the left, expand This PC, and click the DVD Drive.
    5. Click Select Folder.
  7. In the Finish page, click Finish.
  8. From the Start Menu, run the Session Recording Player.
  9. Open the Tools menu and click Options.
  10. On the Connections tab, click Add.
  11. Enter the FQDN of the Session Recording server (or load balanced FQDN).
  12. On the Cache tab you can adjust the client-side cache size. Click OK.
  13. Use the Search box to find recordings.
  14. Or you can go to Tools > Advanced Search.

  15. Once you find a recording, double-click it to play it.
  16. If you see a message about Citrix Client version incompatibility, see CTX206145 Error: “The Session Recording Player Cannot Play Back This File” to edit the Player’s SsRecPlayer.exe.config file to accept the newer version. Note: Session Recording 2012 no longer complains about this.
  17. To skip spaces where no action occurred, open the Play menu, and click Fast Review Mode.
  18. You can add bookmarks by right-clicking in the viewer pane. Then you can skip to a bookmark by clicking the bookmark in the Events and Bookmarks pane.

Director Integration

  1. On the Director server, run command prompt elevated (as Administrator).
  2. Run C:\inetpub\wwwroot\Director\tools\DirectorConfig.exe /configsessionrecording
  3. Enter the Session Recording FQDN (or load balanced FQDN) when prompted.
  4. Enter 1 for HTTPS.
  5. Enter 443 as the port.
  6. In Director, when you view users or machines, you can change the Session Recording policy. These policy changes don’t apply until a new session is launched.
  7. If the Session Recording menu says N/A, then the Director user needs to be authorized in the Session Recording Authorization Console.

  8. If you use Director to enable or disable recording for a user or machine, rules are added to the active policy on the Session Recording server. They only take effect at next logon.

Dynamic Session Recording

Session Recording 7.18, along with Delivery Controller 7.18, and VDA 7.18, allows you to start recording, even in the middle of a session.

Unfortunately, Director has not been enabled to take advantage of this feature. Instead, you use PowerShell to start recording. The command is Start-BrokerSessionRecording, and is detailed at Dynamically start or stop recording by using PowerShell commands in the Citrix Broker SDK at Citrix Docs. For example:

Start-BrokerSessionRecording -User DomainA \ UserA -NotifyUser $false

Use the Stop-BrokerSessionRecording command to stop recording when the reported issue has been triaged or resolved. For example:

Stop-BrokerSessionRecording -User DomainA \ UserA

The commands are logged to Citrix Studio Logging.

EUC Weekly Digest – May 27, 2017

Last Modified: Nov 7, 2020 @ 6:34 am

Leave a comment

Here are some EUC items I found interesting last week. For more immediate updates, follow me at http://twitter.com/cstalhood.

For a list of updates at carlstalhood.com, see the Detailed Change Log.

 

Citrix

XenApp/XenDesktop

VDA

App Layering (Unidesk)

Director/Monitoring

WEM/Profile Management

StoreFront

Receiver

XenApp 6.5

NetScaler

NetScaler Gateway

XenServer

Citrix Cloud

VMware

EUC Weekly Digest – May 20, 2017

Last Modified: Nov 7, 2020 @ 6:34 am

1 Comment

Here are some EUC items I found interesting last week. For more immediate updates, follow me at http://twitter.com/cstalhood.

For a list of updates at carlstalhood.com, see the Detailed Change Log.

 

Citrix

XenApp/XenDesktop

App Layering / Unidesk

HDX

Provisioning Services

StoreFront

NetScaler

NetScaler Gateway

XenMobile

VMware

Microsoft

Other

nFactor Authentication – NetScaler Gateway 12 / Citrix Gateway 12.1

Last Modified: Apr 16, 2021 @ 2:32 pm

87 Comments

Navigation

💡 = Recently Updated

Change Log

nFactor Overview

nFactor lets you configure an unlimited number of authentication factors. You are no longer limited to just two factors.

Each authentication factor performs the following tasks:

  1. Collect credentials from the user. These credentials can be anything supported by Citrix ADC, including:
    • SAML assertion
    • Client Certificate
    • OAuth OpenID Connect
    • Kerberos ticket
    • StoreFrontAuth – authentication is delegated to Citrix StoreFront
    • Forms-based authentication (traditional web-based logon page) for LDAP, RADIUS, etc.
      • Multiple passwords can be collected with one form.
      • Or prompt the user multiple times throughout the authentication chain.
      • The logon page can contain a domain drop-down.
    • Endpoint Analysis Scan – either pre-authentication, or post-authentication.
    • EULA
    • Google reCAPTCHA
    • Swivel
    • Use a drop-down to select an authentication method
  2. This factor’s credentials are evaluated. The results can be:
    • Authentication success
    • Authentication failure
    • Group extraction
    • Attribute extraction from SAML, Certificate, etc.
  3. Based on the evaluation results, do one of the following:
    • Allow access
    • Select next factor
    • Deny access
  4. If there’s a Next Factor, repeat these steps, until there are no more Next Factors to evaluate.
  5. The Next Factor can also do one of the following:
    • Prompt the user for more credentials
    • If the Login Schema asked for multiple passwords, evaluate the already entered next set of credentials.
    • Use a policy expression to select another Next Factor (no authentication). This is typically used with group extraction so that groups determine the Next Factor.

Here are some nFactor use cases, but the combinations are almost limitless:

  • Choose Authentication method based on Active Directory group: Logon screen asks for user name only. Extract user’s groups from Active Directory. Based on user’s Active Directory groups, either ask user for client certificate, or ask user for LDAP password. If LDAP, the username doesn’t need to be entered again.
  • Ask for Certificate first:
    • If client certificate is valid, perform LDAP only.
    • If no client certificate, perform LDAP + RADIUS
  • Two-factor with passwords checked in specific order: Display logon screen with two password fields. Check the first password. If the first password succeeds, then check the second password. This lets you check RADIUS before LDAP.
  • Run Endpoint Analysis first:
    • If passes, perform LDAP only.
    • If fails, perform LDAP + RADIUS
  • See Sample Configurations later for many more combinations.

All new authentication methods added to Citrix ADC require nFactor configuration and are not supported on native Citrix Gateway. These new authentication methods include:

nFactor is a AAA feature, which means you need Citrix ADC Advanced Edition (aka NetScaler Enterprise Edition) or Citrix ADC Premium Edition (aka NetScaler Platinum Edition). Citrix ADC Standard Edition and Citrix Gateway VPX are not entitled for nFactor.

Citrix ADC supports two types of authentication policies – Classic, and Advanced (aka Default). You can bind Classic Authentication Policies directly to Citrix Gateway Virtual Servers, but today you cannot bind Advanced Authentication Policies to Citrix Gateway. The only way to use Advanced Authentication Policies with Citrix Gateway is to configure nFactor on a AAA Virtual Server and then link the AAA Virtual Server to the Gateway Virtual Server.

  • Workspace app 1809 and newer with Citrix Gateway (NetScaler) 12.1 build 49 and newer support nFactor authentication. Older Receivers and older NetScalers don’t support nFactor, so you’ll instead have to use a web browser.

nFactor configuration summary (detailed instructions below):

  • Each factor is a combination of Advanced Authentication Policies and Login Schema.
    • Advanced policy means it uses an Advanced (Default Syntax) expression as opposed to the classic syntax expression traditionally used in Citrix Gateway authentication policies.
    • Login Schema is a custom HTML form where users enter credentials.
  • The first factor (Advanced Authentication Policies and Login Schema) is bound directly to a AAA Virtual Server.
  • Next factors are bound to Authentication Policy Labels. These Labels are then chained to Advanced Authentication Policies in prior factors.
    • AAA vServer > Advanced Authentication Policy > Next Factor > Authentication Policy Label > Advanced Authentication Policy > Next Factor > Authentication Policy Label > … until there are no more Next Factors configured.
  • Authentication Profile links AAA vServer with Citrix Gateway.

Also see:

This article will detail how to configure nFactor from top to bottom.:

  1. Create AAA vServer
  2. Create Login Schema Profiles
  3. Create Login Schema Policy – select Login Schema Profile, bind Login Schema Policy to AAA vServer
  4. Create Authentication Actions – LDAP, RADIUS, etc.
  5. Create Advanced Authentication Policies – select Authentication Action, bind Advanced Authentication Policy to AAA vServer
  6. Create Authentication Policy Labels – bind Login Schema, bind Advanced Authentication Policies
  7. Edit Advanced Authentication Policy binding and select Next Factor Authentication Policy Label.
  8. Create Citrix Gateway Traffic Policy for Single Sign-on to StoreFront.
  9. Create Authentication Profile
  10. Edit Citrix Gateway – bind Authentication Profile, bind Traffic Policy

Once you are familiar with nFactor, due to the way the objects are linked together, it’s probably easier to configure it from bottom up:

  1. Create Authentication Actions – LDAP, RADIUS, etc.
  2. Create Advanced Authentication Policies – select Authentication Action
  3. Create Login Schema Profiles
  4. Create Authentication Policy Labels – bind Login Schema, bind Advanced Authentication Policies
  5. Create Login Schema Policy – select Login Schema Profile
  6. Create AAA vServer – bind Login Schema, bind Advanced Authentication Policies, select Next Factor Authentication Policy Label
  7. Create Citrix Gateway Traffic Policy for Single Sign-on to StoreFront.
  8. Create Authentication Profile
  9. Edit Citrix Gateway – bind Authentication Profile, bind Traffic Policy

It can be difficult to visualize a nFactor configuration so my ADC Virtual Server Configuration Extractor script now includes a nFactor visualizer. Here’s an example for a Native OTP configuration.

AAA Virtual Server

Create AAA Virtual Server

To use nFactor with Citrix Gateway, you first configure it on a AAA Virtual Server. Then you later link the AAA Virtual Server to the Citrix Gateway Virtual Server.

  1. If AAA feature is not already enabled, on the left menu, expand Security, right-click AAA – Application Traffic, and click Enable Feature.
  2. Go to Security > AAA > Virtual Servers.
  3. On the right, click Add.
  4. Give the Virtual Server a name.
  5. If you are only using this AAA Virtual Server for Citrix Gateway, then you can change the IP address Type to Non Addressable. It’s also possible to content switch to AAA (Citrix CTX201949 One Public IP for AAA-TM Deployments on NetScaler).
  6. Click OK.
  7. In the Certificates section, click where it says No Server Certificate.

    1. Click where it says Click to select.
    2. Click the radio button next to a certificate for the AAA Virtual Server, and click Select. Since this AAA Virtual Server is not directly addressable, the chosen certificate doesn’t matter.
    3. Click Bind.
  8. Click Continue to close the Certificate section.
  9. You probably don’t have any Advanced Authentication Policies yet, so just click Continue.

Bind Portal Theme to AAA Virtual Server

If this AAA Virtual Server is used not just for Citrix Gateway but also directly addressable for traffic management (Load Balancing, Content Switching), then you might want to change the AAA Portal theme.

  1. Go to Citrix Gateway > Portal Themes, and add a theme. You create the theme under Citrix Gateway, and then later bind it to the AAA Virtual Server.
  2. Create a theme based on the RfWebUI Template Theme.
  3. After adjusting the theme as desired, at the top of the portal theme editing page, Click to Bind and View Configured Theme.
  4. Change the selection to Authentication.
  5. Use the Authentication Virtual Server Name drop-down to select the AAA Virtual Server, and click Bind and Preview. You can close the preview window.

Client Certificate Authentication

If one of your authentication Factors is client certificate, then you must perform some SSL configuration on the AAA Virtual Server:

  1. Go to Traffic Management > SSL > Certificates > CA Certificates, and install the root certificate for the issuer of the client certificates. Root certificates do not have a key file.

  2. Go to Traffic Management >SSL > Change advanced SSL settings.

    1. Scroll down. If you see Default Profile: ENABLED, then you must use an SSL Profile to enable Client Certificate Authentication. Otherwise, you can enable Client Certificate Authentication directly on the AAA Virtual Server in the SSL Parameters section.
  3. If Default SSL Profiles are enabled, then create a new SSL Profile with Client Authentication enabled:
    1. On the left menu, expand System, and click Profiles.
    2. On the top right, switch to the SSL Profile tab.
    3. Right-click the ns_default_ssl_profile_frontend profile, and click Add. This copies settings from the default profile.
    4. Give the Profile a name. The purpose of this profile is to enable Client Certificates.
    5. Scroll down and find the Client Authentication checkbox. Check the box.
    6. Change the Client Certificate drop-down to OPTIONAL.
    7. Copying the default SSL Profile does not copy the SSL Ciphers so you’ll have to redo them.
    8. Click Done when done creating the SSL Profile.
    9. Go to Security > AAA – Application Traffic > Virtual Servers, and edit a AAA vServer.
    10. Scroll down to the SSL Profile section and click the pencil.
    11. Change the SSL Profile drop-down to the profile that has Client Certificates enabled. Click OK.
    12. Scroll down this article until you reach the instructions to bind the CA certificate.
  4. If default SSL Profiles are not enabled:
    1. Go to Security > AAA > Virtual Servers, and edit an existing AAA Virtual Server.
    2. On the left, in the SSL Parameters section, click the pencil icon.
    3. Check the box next to Client Authentication.
    4. Make sure Client Certificate drop-down is set to Optional, and click OK.
  5. On the left, in the Certificates section, click where it says No CA Certificate.
  6. Click to select.
  7. Click the radio button next to the root certificate for the issuer of the client certificates, and click Select.
  8. Click Bind.

Login Schema

Login Schema XML File

Login Schema is an XML file providing the structure of forms-based authentication logon pages.

nFactor implies multiple authentication Factors that are chained together. Each Factor can have different Login Schema pages/files. In some authentication scenarios, users could be presented with multiple logon screens.

Or you can have one Login Schema gather information that can be passed on to multiple Factors, so that the later Factors don’t need to display another Login Schema. This is particularly useful for traditional two-password logon screens (LDAP + RADIUS), since each password is evaluated in a separate Factor:

  • The first password is evaluated in the first factor (e.g. LDAP). If successful, then proceed to the second factor.
  • The second factor (e.g. RADIUS) evaluates the second password. However, the second password has already been entered, so there’s no need to ask the user for it again. To prevent a Login Schema from being shown to the user, select noschema (LSCHEMA_INT) in the Authentication Policy Label.

Several Login Schema .xml files are included with Citrix ADC under /nsconfig/loginschema/LoginSchema.

You can edit the Login Schema labels from within the Citrix ADC management GUI. When the labels are changed, Citrix ADC copies the Login Schema to a new .xml file based on the Schema Name entered in this widow, or based on the original file name.

Or you can use WinSCP to connect to the appliance, duplicate one of the existing .xml files, and edit it as desired. For example, you can configure fields (InitialValue tag) to pre-fill information from previous Factors, as shown below:

The structure of the Login Schema is documented at Citrix Common Authentication Forms Language Citrix Developer Documentation.

CTP Sam Jacobs at SYN229 – nFactor and Login Schemas explains how to customize the .xml file.

The login schema can also contain a domain drop-down. See CTX201760 nFactor – Domain Drop-Down in First Factor then Different Policy Evaluations Based on Groups for a sample configuration.

Login Schema and Authentication Factor can be a EULA. See Citrix CTX226488 How to Configure EULA as an Authentication Factor in NetScaler nFactor.

Citrix CTX219545 Custom Login Labels in NetScaler nFactor Authentication: add a Requirement element with a Label sub-element to the Login Schema .xml file. Then use JavaScript to populate the label with any desired HTML. Another example is Morten Kallesoee nFactor – adding custom links.

Several more samples can be found later.

Login Schema Profile

To configure a Login Schema Profile:

  1. Create or Edit a Login Schema .XML file based on your nFactor design.
  2. Go to Security > AAA > Login Schema.
  3. On the right, switch to the Profiles tab, and click Add.
  4. In the Authentication Schema field, click the pencil icon.
  5. Click the LoginSchema folder to see the files in it.
  6. Select one of the files. You can see a preview on the right. The labels can be changed by clicking the Edit button on the top right.
  7. When you Save the changes, a new file is created under /nsconfig/LoginSchema.
  8. On the top right, click Select.
  9. Give the Login Schema a name, and click More.
  10. You typically need to use the entered credentials elsewhere. For example, you might need to use the username and one of the passwords to later Single Sign-on to StoreFront. Near the bottom of the Login Schema Profile, click More, and enter unique values for the indexes. These values can be between 1 and 16.

    1. Later you reference these index values in a Traffic Policy/Profile by using the expression HTTP.REQ.USER.ATTRIBUTE(#).
  11. Click Create to create the Login Schema profile.

Note: if you later edit the Login Schema .xml file, the changes might not be reflected until you edit the Login Schema Profile, and Select the .xml file again.

Login Schema Policy

To bind a Login Schema Profile to a AAA vServer, you must first create a Login Schema Policy. Login Schema Policies are not required when binding the Login Schema Profile to an Authentication Policy Label, as detailed later.

To create and bind a Login Schema Policy:

  1. On the left, go to Security > AAA > Login Schema.
  2. On the right, switch to the Policies tab, and click Add.
  3. Use the Profile drop-down to select the Login Schema Profile you already created.
  4. Enter a Default Syntax expression (e.g. true) in the Rule box, and click Create.
  5. On the left, go to Security > AAA > Virtual Servers, and edit an existing AAA Virtual Server.
  6. On the right, in the Advanced Settings column, click Login Schemas.
  7. On the left, in the Login Schemas section, click where it says No Login Schema.
  8. Click where it says Click to select.
  9. Click the radio button next to the Login Schema policy, and click Select. Only Login Schema Policies appear in this list. Login Schema Profiles (without a policy) do not appear.
  10. Click Bind.

Advanced Authentication Policies

Authentication policies are a combination of policy expression, and policy action. If the expression is true, then evaluate the authentication action.

The Action is always an authentication server (LDAP, RADIUS, etc.).

The policy expression can be either in classic syntax, or in the newer default syntax.

The policy type is either Basic or Advanced. Basic policies can only use classic syntax. Advanced policies only use the newer default syntax. Both types of policies use the same Actions (authentication servers).

nFactor requires Advanced Authentication Policies; Basic policies won’t work.

Note: Citrix Gateway 12.0 and newer have deprecated Basic Authentication Policies (Classic Syntax). The only way to bind an Advanced Authentication Policy (Default Syntax) to Gateway is through nFactor and AAA.

Create Advanced Authentication Policy

You will need Authentication Actions/Servers (e.g. LDAP, RADIUS, CERT, SAML, etc.)

  • When creating an Advanced Authentication Policy, there’s a plus (Add) icon that lets you create Authentication Actions/Servers.
  • Or you can create Authentication Actions (Servers) prior to creating the Advanced Authentication Policy. The Authentication Servers are located under Authentication > Dashboard. On the right, click Add and select a Server Type. The instructions for creating these Authentication Servers is not detailed here. See the Authentication – NetScaler 12 / Citrix ADC 12.1 procedures.

To create an Advanced Authentication Policy:

  1. Go to Security > AAA > Policies > Authentication > Advanced Policies > Policy.
  2. On the right, click Add. You typically create at least one Authentication Policy for each Factor. When you create multiple Authentication Policies for one Factor, Citrix ADC checks each authentication policy in priority order until one of them succeeds.
  3. Use the Action Type drop-down to select the Action Type (e.g. LDAP). Typically each Factor is a different type of Authentication Action.
  4. If you don’t currently have any Actions configured, or if you want to create a new one, click the plus icon (Add button) next to the Action drop-down. The Actions/Servers are created in the normal fashion (not detailed here).
  5. In the Expression box, enter an expression using the Default Syntax. ns_true won’t work because that’s Classic syntax. There’s an Expression Editor link on the right. Or hit Ctrl+Space to see your options. true is a valid Default expression. Click Create when done.
  6. Create more Advanced Authentication Policies as needed for your nFactor design.

Bind First Factor Advanced Authentication Policy to AAA

Only the Advanced Authentication Policies for the first Factor are bound directly to the AAA Virtual Server. The Advanced Authentication Policies for the Next Factors are bound to Authentication Policy Labels as detailed in the next section.

  1. Go to Security > AAA > Virtual Servers.
  2. Edit an existing AAA Virtual Server.
  3. On the left, in the Advanced Authentication Policies section, click where it says No Authentication Policy.
  4. Click where it says Click to select.
  5. Click the radio button next to the Advanced Authentication Policy, and click Select.
  6. In the Binding Details section, if this Advanced Authentication Policy fails, then the Goto Expression determines what happens next. If it is set to NEXT, then the next Advanced Authentication Policy bound to this AAA Virtual Server is evaluated. If it is set to END, or if there are no more Advanced Authentication Policies bound to this AAA Virtual Server, then authentication is finished and marked as failed.
  7. The Select Next Factor field can optionally point to an Authentication Policy Label as detailed in the next section. The Next Factor is only evaluated if this Advanced Authentication Policy succeeds.
  8. Click Bind.

LDAP Group Extraction

Sometimes you only want to extract a user’s groups from Active Directory, but you don’t actually want to authenticate with LDAP. These extracted groups can then be used to select the next authentication Factor.

To configure an LDAP Action/Server for only group extraction:

  1. When creating or editing an LDAP Server/Action, make sure Authentication is unchecked.
  2. In the Other Settings section, make sure Group Attribute and Sub Attribute Name are filled in.

Authentication Policy Label

When configuring the first Factor, you bind two objects directly to the AAA Virtual Server:

  • Login schema – for forms-based authentication
  • Advanced Authentication Policy

When binding the Advanced Authentication Policy to the AAA Virtual Server, there’s a field to Select Next Factor. If the Advanced Authentication Policy succeeds, then the Next Factor is evaluated.

The Next Factor is actually an Authentication Policy Label.

Authentication Policy Labels contain three objects:

  • Login Schema
  • Advanced Authentication Policies
  • Next Factor – the next Authentication Policy Label

Here’s the flow:

  1. User connects to AAA or Citrix Gateway Virtual Server.
  2. If forms-based authentication, the Login Schema bound to the AAA Virtual Server is displayed.
  3. Advanced Authentication Policies bound to the AAA Virtual Server are evaluated.
    1. If the Advanced Authentication Policy succeeds, go to the configured Next Factor, which is an Authentication Policy Label.
      1. If Next Factor is not configured, then authentication is complete and successful.
    2. If the Advanced Authentication Policy fails, and if Goto Expression is Next, then evaluate the next bound Advanced Authentication Policy.
    3. If none of the Advanced Authentication Policies succeed, then authentication failed.
  4. If the Next Factor Authentication Policy Label has a Login Schema bound to it, display it to the user.
  5. Evaluate the Advanced Authentication Policies bound to the Next Factor Authentication Policy Label.
    1. If the Advanced Authentication Policy succeeds, go to the configured Next Factor, which is an Authentication Policy Label.
      1. If Next Factor is not configured, then authentication is complete and successful.
    2. If the Advanced Authentication Policy fails, and if Goto Expression is Next, then evaluate the next bound Advanced Authentication Policy.
    3. If none of the Advanced Authentication Policies succeeds, then authentication failed.
  6. Continue evaluating the Next Factor Authentication Policy Label until authentication succeeds or fails. You can chain together an unlimited number of Authentication Policy Labels.

If you are binding a Login Schema to an Authentication Policy Label, then you only need the Login Schema Profile. There’s no need to create a Login Schema Policy.

Not every Factor needs a Login Schema (logon web page). It’s possible for a prior Factor to gather all of the credential information, and simply pass it on to the next Factor. If you don’t need a Login Schema for a particular Authentication Policy Label, simply select LSCHEMA_INT, which is mapped to noschema. Or create a new Login Schema Profile based on noschema.

Create Authentication Policy Label

To create an Authentication Policy Label:

  1. Authentication Policy Labels are configured at Security > AAA > Policies > Authentication > Advanced Policies > PolicyLabel.
  2. On the right, click Add.
  3. Give the Policy Label a name.
  4. Select a Login Schema Profile. This can be a Login Schema Profile that is set to noschema (LSCHEMA_INT) if you don’t actually want to display anything to the user. Then click Continue.
  5. In the Policy Binding section, click where it says Click to select.
  6. Click the radio button next to an Advanced Authentication Policy that evaluates this Factor. Click Select.
  7. Use the Goto Expression drop-down to select NEXT or END. If you want to bind more Advanced Authentication Policies to this Factor, then select NEXT.
  8. In the Select Next Factor field, if you want to chain another Factor, click where it says Click to select, and bind the next Authentication Policy Label (Next Factor). You can click the Add button to create another Policy Label.
  9. Or don’t select anything, and if this Advanced Authentication Policy succeeds, then authentication is successful and complete. This ends the chaining.
  10. Click Bind when done.
  11. You can click Add Binding to add more Advanced Authentication Policies to this Policy Label (Factor). Note: each Label is a separate factor. If your intent is multi-factor, then create a separate Label for the next factor.
  12. When done, click Done.

Bind Authentication Policy Label

Once the Policy Label (Factor) is created, you bind it to an existing Advanced Authentication Policy binding. This is how you chain Factors together. You can select a Next Factor (Policy Label) in two places:

  • Edit an existing AAA Virtual Server that has an Advanced Authentication Policy already bound to it and edit the binding to include the Next Factor.
  • Edit a different Policy Label, and edit an Advanced Authentication Policy binding to include the Next Factor.

To add a Policy Label Next Factor to a AAA Virtual Server:

  1. Edit an existing AAA Virtual Server that has an Advanced Authentication Policy already bound to it.
  2. On the left, in the Advanced Authentication Policies section, click the existing Authentication Policy bindings.
  3. Right-click an existing binding, and click Edit Binding.
  4. In the Select Next Factor field, click where it says Click to select.
  5. Click the radio button next to the Policy Label for the Next Factor, and click Select.
  6. Click Bind.
  7. The far right shows the Next Factor.
  8. Click Close.

To add a Policy Label Next Factor to a different Policy Label:

  1. Go to Security > AAA – Application Traffic > Policies > Authentication > Advanced Policies > PolicyLabel.
  2. On the right, edit a different Policy Label.
  3. Highlight an existing Advanced Authentication Policy binding, open the Action menu, and click Edit Binding.
  4. In the Binding Details section, next to Select Next Factor, click Click to select.
  5. Click the radio button next to a Policy Label for the next factor, and then click the blue Select button on the top of the window.
  6. Click Bind.
  7. On the far right, you can see the configured Next Factor.
  8. Click Close to close the PolicyLabel.

nFactor for Citrix Gateway

AAA Authentication Profile

Authentication Profile links a AAA Virtual Server to Citrix Gateway and enables nFactor on Citrix Gateway.

  1. Go to Citrix Gateway > Virtual Servers.
  2. On the right, edit an existing Gateway Virtual Server.
  3. On the right, in the Advanced Settings column, click Authentication Profile.
  4. On the left, click the plus icon (Add button) next to the Authentication Profile drop-down.
  5. Give the Authentication Profile a name.
  6. In the Authentication Virtual Server field, click where it says Click to select.
  7. Click the radio button next to the AAA Virtual Server that has Login Schema, Advanced Authentication Policy, and Authentication Policy Labels configured. The AAA Virtual Server does not need an IP address. Click Select.
  8. Then click Create.
  9. And click OK to close the Authentication Profile section.
  10. If one of your Factors is client certificates, then you’ll need to configure SSL Parameters and CA certificate as detailed in the next section.
  11. When you browse to your Gateway, you’ll see the nFactor authentication screens.
  12. Workspace app 1809 and newer with Gateway/ADC 12.1 build 49 and newer should support nFactor authentication. Older clients with older builds do not support nFactor, so those users will have to use a web browser.
    1. CTX223386 nFactor Authentication with Native Clients from NetScaler 11.1: Although Native Receiver Clients can now leverage advanced authentication policies, they still interact using legacy Gateway protocol. That is, Native Clients do not yet support nFactor wire protocol. Gateway acts intelligently based on the clients that are interfacing with it. That is, for Browser client, Gateway redirects to advanced login page with all customizations. For a Native Receiver client, Gateway responds according to the legacy protocol. Since Native clients do not process LoginSchema (even if it is sent), Gateway processes the configured LoginSchema, learns the configured factors, and sends hint to the clients in response to /vpn/index.html.

Gateway Client Certificate Authentication

If one of your authentication Factors is certificate, then you must perform some SSL configuration on the Citrix Gateway Virtual Server:

  1. Go to Traffic Management > SSL > Certificates > CA Certificates, and install the root certificate for the issuer of the client certificates. Certificate Authority certificates do not need key files.
  2. If default SSL Profiles are enabled, then you should have already created an SSL Profile that has Client Authentication enabled.
  3. Go to Citrix Gateway > Virtual Servers, and edit an existing Citrix Gateway Virtual Server that is enabled for nFactor.
  4. If default SSL Profiles are enabled:
    1. Scroll down to the SSL Profile section, and click the pencil icon.
    2. In the SSL Profile drop-down, select the SSL Profile that has Client Authentication enabled and set to OPTIONAL.
  5. If default SSL Profiles are not enabled:
    1. On the left, in the SSL Parameters section, click the pencil icon.
    2. Check the box next to Client Authentication.
    3. Make sure Client Certificate drop-down is set to Optional, and click OK.
  6. On the left, in the Certificates section, click where it says No CA Certificate.
  7. Click where it says Click to select.
  8. Click the radio button next to the root certificate for the issuer of the client certificates, and click Select.
  9. Click Bind.
  10. You might have to also bind any Intermediate CA Certificates that issued the client certificates.

Traffic Policy for nFactor Single Sign-on to StoreFront

When performing Single Sign-on to StoreFront, nFactor defaults to using the last entered password. If LDAP is not the last entered password, then you need to create a Traffic Policy/Profile to override the default nFactor behavior.

  1. Go to Citrix Gateway > Policies > Traffic.
  2. On the right, switch to the Traffic Profiles tab.
  3. Click Add.
  4. Give the Traffic Profile a name.
  5. In the Protocol section, select HTTP.
  6. Set Single Sign-on to ON. Scroll down.
  7. In the SSO Expression fields, enter an HTTP.REQ.USER.ATTRIBUTE(#) expression that matches the indexes specified in the Login Schema.
  8. Click Create.
  9. On the right, switch to the Traffic Policies tab, and click Add.
  10. Give the policy a name.
  11. Select the previously created Traffic Profile.
  12. Enter an Advanced Expression (e.g. true), and click Create.
  13. Edit an existing Citrix Gateway Virtual Server.
  14. Scroll down to the Policies section and click the plus icon.
  15. Select Traffic > Request, and click Continue.
  16. Select the previously created Traffic Policy, and click Bind.

Sample Configurations

From Citrix Docs: Sample deployments using nFactor authentication:

  • Get two passwords up-front, pass-through in next factor. Read
  • Username and 2 passwords with group extraction in third factor. Read
  • Configure nFactor to process the second password before the first password, Read
  • Modify first factor username for second factor. Read
    • NO_AUTHN authentication policy expression checks first factor POST Body login value for UPN format. If true, Next Factor is noschema Login Schema with User Expression that transforms the HTTP.REQ.USER.NAME to DOMAIN\Username before passing to second factor authentication policy.
  • Group extraction followed by certificate or LDAP authentication, based on group membership. Read

  • SAML followed by LDAP or certificate authentication, based on attributes extracted during SAML. Read
  • SAML in first factor, followed by group extraction, and then LDAP or certificate authentication, based on groups extracted. Read
  • Capture email address in first factor, and then choose one of multiple SAML iDP based on email address suffix. Read (Manuel Kolloff)

  • Prefill user name from certificate. Read
  • Certificate authentication followed by group extraction for 401 enabled traffic management virtual servers. Read
  • Certificate fallback to LDAP in same cascade; one virtual server for both certificate and LDAP authentication. Read
  • LDAP in first factor and WebAuth in second factor. Read
  • WebAuth in first factor, LDAP in second factor. Read
  • Domain drop down in first factor, then different policy evaluations based on selected domain. Read

    • Domain drop-down, then send Domain\Username to RADIUS.  Read
  • Google reCAPTCHA first factor, LDAP second. Read (George Spiers)
  • Supporting reCaptcha with NetScaler nFactor. Read
  • CTX225938 nFactor – Customizing UI to Display Images – e.g. Swivel
  • With 12.0 version of NetScaler, EPA is blended into the authentication framework thereby making EPA a conditional or on-demand feature. This reduces the need for multiple virtual servers by allowing for all the clients to gain access to a single virtual server. See CTX223597 Concepts and Entities Used for EPA in nFactor Authentication Through NetScaler.
  • Configure Post-Authentication EPA (Endpoint Analysis) Scan as a Factor. Read
  • Configure Pre-Authentication EPA (Endpoint Analysis) Scan as a Factor. Read
  • Configure EULA (End User License Agreement) as an Authentication Factor. Read
  • Show a drop-down box in the logon form and automatically hide or show certain fields based on drop-down selection. Read

  • Step-up authentication – i.e. one Unified Gateway website needs single factor, while other website needs multi-factor. Read
  • RADIUS authentication with reversed PIN – if user enters reversed PIN, then user is under duress. This sample configuration has some interesting components:  Read  💡
    • Policy Extension Function using the Lua language
      • Usage = HTTP.REQ.BODY(1000).TYPECAST_NVLIST_T(’=’,’&’).VALUE(”passwd1”).RPIN
    • NetScaler Variable of type Map with Expiration timer
      • Responder to set the variable
      • Variable identifies duress state for four hours
    • Custom syslog message (audit messageaction) triggered by a Responder
    • Default Authentication Group to put duressed user on site/farm with Session Recording enabled
    • nFactor sequence:

Certificate auth: If Successful, LDAP only. If Failure, LDAP+RADIUS

This scenario is described in Citrix Blog Post Configuration Notes on nFactor

The authentication process flows like this:

  1. User connects to NetScaler Gateway.
  2. NetScaler Gateway asks user for certificate.
  3. If user selects a certificate, NetScaler Gateway compares certificate signature to the CA certificate that is bound to the NetScaler Gateway. If it doesn’t match, then user certificate is ignored.
  4. Bound to the NetScaler Gateway Virtual Server is an Authentication Profile, which links NetScaler Gateway to AAA nFactor.
  5. Certificate authentication: The lowest priority number authentication policy on the AAA Virtual Server is Certificate. If there’s a valid user certificate:
    1. Extract the user’s userPrincipalName from the certificate.
    2. Next Factor = policy label that displays a logon screen (Single-factor Login Schema)
    3. The username field is pre-populated with the userPrincipalName attribute extracted from the certificate.
    4. User is prompted to enter the LDAP password only.
    5. LDAP policy/server is configured to use userPrincipalName to login to LDAP.
    6. If successful, NetScaler Gateway authentication is complete. Next step is to Single Sign-on to StoreFront.
    7. If LDAP authentication fails, then NetScaler Gateway authentication fails, and the user is prompted to try LDAP-only authentication again.
  6. LDAP authentication: If certificate authentication fails, try next authentication policy bound to the AAA Virtual Server, which is a different LDAP Policy.
    1. Bound to the AAA Virtual Server is a Dual Factor Login Schema that asks for username, LDAP password, and RADIUS password.
    2. LDAP policy/server is configured to use sAMAccountName to login to LDAP. SAMAccountName means users don’t have to enter full userPrincipalName.
    3. If LDAP authentication is successful:
      1. Put username in Credential Index 1 and put password in Credential Index 2. These will later be used by a Traffic Policy to Single Sign-on to StoreFront.
      2. Proceed to next factor (Policy Label), which is RADIUS.
    4. If LDAP authentication fails, NetScaler Gateway login fails, and the user is prompted to try two-factor authentication again.
  7. RADIUS authentication: the second factor Policy Label is configured with Noschema. This means no additional logon form is displayed because the RADIUS password was already collected in the previous factor.
    1. When multiple passwords are collected, they are tried in order. The first password was used by the previous factor. The second password is tried by this factor (Policy Label).
    2. RADIUS policy/profile attempts authentication.
    3. If RADIUS authentication is successful, NetScaler Gateway authentication is complete. Next step is Single Sign-on to StoreFront.
    4. If RADIUS authentication fails, NetScaler Gateway login fails, and the user is prompted to try two-factor authentication again.
  8. Single Sign-on to StoreFront: NetScaler Gateway uses the last password collected by nFactor to Single Sign-on with StoreFront. If the last password is LDAP, then no additional configuration is needed. If the last password is not LDAP, then a Traffic Policy/Profile is needed.
    1. Bound to the NetScaler Gateway Virtual Server is a Traffic Policy.
    2. The Traffic Policy/Profile users Credential Index 1 for username and Credential Index 2 for Password. These are the same indexes configured in the Dual Factor Login Schema.

The order of configuration doesn’t match the authentication flow because some objects have to be created before others.

# Create Auth vServer, bind server cert, bind CA cert for client certificates
# Enable Optional client certificates
add authentication vserver nFactorAAA SSL 0.0.0.0 443
bind ssl vserver nFactorAAA -certkeyName WildCorpCom
bind ssl vserver nFactorAAA -certkeyName CorpRoot -CA -ocspCheck Optional
set ssl vserver nFactorAAA -clientAuth ENABLED -clientCert Optional -ssl3 DISABLED

# Create auth policy for LDAP-UPN. UPN is extracted from certificate.
add authentication ldapAction Corp-UserPrincipalName -serverIP 10.2.2.220 -serverPort 636 -ldapBase "dc=corp,dc=local" -ldapBindDn "corp\\ctxsvc" -ldapBindDnPassword "MyPassword" -ldapLoginName userPrincipalName -groupAttrName memberOf -subAttributeName CN -secType SSL -passwdChange ENABLED
add authentication Policy Corp-UserPrincipalName -rule true -action Corp-UserPrincipalName

# Create PolicyLabel LDAPPasswordOnly with Single-factor Login Schema
# Login Schema has InitialValue with username from certificate.
add authentication loginSchema SingleAuth -authenticationSchema "/nsconfig/loginschema/LoginSchema/SingleAuth-Corp.xml"
add authentication policylabel LDAPPasswordOnly -loginSchema SingleAuth
bind authentication policylabel LDAPPasswordOnly -policyName Corp-UserPrincipalName -priority 100 -gotoPriorityExpression NEXT

# Create Cert policy and bind to AAA vServer with LDAPPasswordOnly PolicyLabel as Next Factor
# Cert policy must have lower priority number (higher priority) than LDAP-SAM policy
# Cert is evaluated first. If succeed, ask for LDAP password. If fails, ask for two factor.
add authentication certAction Cert_Auth_Profile -userNameField SubjectAltName:PrincipalName
add authentication Policy Cert_Auth_Policy -rule true -action Cert_Auth_Profile
bind authentication vserver nFactorAAA -policy Cert_Auth_Policy -priority 100 -nextFactor LDAPPasswordOnly -gotoPriorityExpression NEXT

# Create LDAP-SAM Auth Policy for two-factor
# Only evaluated if cert auth fails. Login Schema asks for user, password, and passcode.
add authentication ldapAction Corp-Gateway -serverIP 10.2.2.220 -serverPort 636 -ldapBase "dc=corp,dc=local" -ldapBindDn "corp\\ctxsvc" -ldapBindDnPassword "MyPassword" -ldapLoginName samaccountname -groupAttrName memberOf -subAttributeName CN -secType SSL -passwdChange ENABLED
add authentication Policy Corp-SAMAccountName -rule true -action Corp-Gateway

# Create RADIUS Auth Policy for two-factor
add authentication radiusAction RADIUS-Action -serverIP 10.2.2.42 -serverPort 1812 -radKey MyKey
add authentication Policy RADIUS-Policy -rule true -action RADIUS-Action

# Create Dual-factor Login Schema and bind directly to AAA vServer
# This Login Schema is only shown if Cert auth fails
add authentication loginSchema DualAuth -authenticationSchema "/nsconfig/loginschema/LoginSchema/DualAuth.xml" -userCredentialIndex 1 -passwordCredentialIndex 2
add authentication loginSchemaPolicy DualAuth -rule true -action DualAuth
bind authentication vserver nFactorAAA -policy DualAuth -priority 100 -gotoPriorityExpression END

# Create RADIUS Policy Label with noschema and RADIUS Auth Policy
# Already got passcode from previous factor so don't show Login Schema again
add authentication loginSchema Noschema -authenticationSchema noschema
add authentication policylabel NoSchema-RADIUS -loginSchema Noschema
bind authentication policylabel NoSchema-RADIUS -policyName RADIUS-Policy -priority 100 -gotoPriorityExpression NEXT

# Bind LDAP-SAM Auth Policy to AAA vServer with RADIUS as next factor
# LDAP-SAM Auth Policy must have higher priority number (lower priority) than Cert Policy
bind authentication vserver nFactorAAA -policy Corp-SAMAccountName -priority 110 -nextFactor NoSchema-RADIUS -gotoPriorityExpression NEXT

# Create Authentication Profile to link AAA with Gateway. Bind to Gateway.
add authentication authnProfile nFactor -authnVsName nFactorAAA -AuthenticationHost aaa.corp.com
add vpn vserver gateway.corp.com SSL 10.2.2.220 443 -icaOnly ON -dtls ON -Listenpolicy NONE -tcpProfileName nstcp_default_XA_XD_profile -appflowLog ENABLED -authnProfile nFactor

# Enable Optional Client certs on Gateway
set ssl vserver gateway.corp.com -clientAuth ENABLED -clientCert Optional -ssl3 DISABLED
bind ssl vserver gateway.corp.com -certkeyName CorpRoot -CA -ocspCheck Optional

# Create Traffic Policy to SSON to StoreFront. Bind to Gateway.
add vpn trafficAction nFactorSSO http -kcdAccount NONE -userExpression "http.req.user.attribute(1)" -passwdExpression "http.req.user.attribute(2)"
add vpn trafficPolicy nFactorSSO ns_true nFactorSSO
bind vpn vserver gateway.corp.com -policy nFactorSSO -priority 100

Group Extraction, followed by LDAP (Active Directory), or Azure MFA (NPS)

Azure MFA is available as a plug-in for Microsoft Network Policy Server (NPS), which is a Microsoft RADIUS server and built-in Windows Server Role.

NPS performs both AD authentication, and Azure MFA authentication. NetScaler sends the user’s AD password to NPS. NPS verifies AD, and then the NPS Azure MFA plug-in calls the user (or push notification to the user). If both AD and MFA are successful, then NPS sends back RADIUS-Accept.

Summary:

  1. First factor Login Schema asks for Username only.
    1. LDAP Group Extraction (with Authentication disabled) reads the user’s groups from AD.
  2. Second factor checks for group membership and sends to one of two different third factors.
  3. If user is in LDAP Group, or Client IP is on internal network, then perform LDAP-only authentication.
    1. Login schema asks for AD password.
    2. LDAP Policy authenticates with LDAP Server (Active Directory).
  4. Otherwise, perform RADIUS (two-factor) authentication.
    1. Login schema asks for AD password.
      • Note: NPS with MFA plugin only needs the AD password. Alternatively, you could use a Login Schema that asks for both LDAP password and RADIUS password.
    2. RADIUS Policy uses the entered AD password to authenticate to Microsoft NPS and Azure MFA.

CLI Commands. Note, these objects are created in the required order, which is backwards from how you would want to configure them.

  1. Add cert for AAA vServer. Link the cert to Intermediate. 
    add ssl certKey WildcardCorpCom -cert WildcardCorpCom.pfx -key WildcardCorpCom.pfx -inform PFX -passcrypt "myPassword"

link ssl certKey WildcardCorpCom Intermediate
  2. Enable AAA feature if not already enabled. 
    enable ns feature AAA
  3. Create first factor LDAP Action (LDAP Server) and LDAP Policy (expression) for Group Extraction. Authentication is disabled. This is the first factor that is bound directly to the AAA vServer. 
    add authentication ldapAction LDAP-Corp-GroupExtract -serverIP 10.2.2.11 -serverPort 636 -ldapBase "dc=corp,dc=local" -ldapBindDn ctxsvc@corp.local -ldapBindDnPassword MyPassword -ldapLoginName sAMAccountName -groupAttrName memberOf -subAttributeName cn -secType SSL -authentication DISABLED

add authentication Policy LDAP-Corp-GroupExtract -rule true -action LDAP-Corp-GroupExtract
  4. Create a third-factor LDAP Action (LDAP Server) and Authentication Policy (expression) for Active Directory Authentication. This is the authentication factor if user is in the LDAP Users group. 
    add authentication ldapAction LDAP-Corp -serverIP 10.2.2.11 -serverPort 636 -ldapBase "dc=corp,dc=local" -ldapBindDn ctxsvc@corp.local -ldapBindDnPassword MyPassword -ldapLoginName sAMAccountName -groupAttrName memberOf -subAttributeName cn -secType SSL -passwdChange ENABLED

add authentication Policy LDAP-Corp -rule true -action LDAP-Corp
  5. Create a third-factor RADIUS Action (RADIUS Server) and Authentication Policy (expression) for NPS. 
    add authentication radiusAction NPS -serverIP 10.2.2.42 -serverPort 1812 -radKey MySecret

add authentication Policy NPS -rule true -action NPS
  6. Create the second factor NO_AUTHN authentication policies to determine the next factor based on the user’s group membership. NO_AUTHN means don’t authenticate. Instead, these policies will have a Next Factor that points to the Authentication Policies that we created earlier. If the policy expression is true, then go to Next Factor. Next Factor is configured later when binding these policies to the second factor PolicyLabel. Note: the group name is case sensitive and must match the Active Directory group name. 
    add authentication Policy LDAP-Only -rule "http.REQ.USER.IS_MEMBER_OF(\"LDAP\") || client.IP.SRC.IN_SUBNET(10.2.2.0/24)" -action NO_AUTHN

add authentication Policy TwoFactor -rule "client.IP.SRC.IN_SUBNET(10.2.2.0/24).NOT" -action NO_AUTHN
  7. Create first factor Login Schema Profile for username-only group extraction. You can copy the built-in OnlyUsername.xml and modify it with your desired labels. Since this Login Schema Profile is bound to the AAA vServer, it needs a Login Schema Policy (expression). The other two Login Schema Profiles are bound to PolicyLabels and thus don’t need Login Schema Policies. 
    add authentication loginSchema OnlyUsername -authenticationSchema "/nsconfig/loginschema/LoginSchema/OnlyUsername.xml"

add authentication loginSchemaPolicy OnlyUsername -rule true -action OnlyUsername
  8. Create third factor Login Schema Profile for AD Authentication. The .xml file is copied from the built-in PrefillUserFromExpr.xml but with modified labels for AD authentication. The username is pre-filled in from the first factor. 
    add authentication loginSchema LDAPPasswordOnly -authenticationSchema "/nsconfig/loginschema/LDAPPassword.xml"
  9. Create third factor Login Schema Profile for NPS Authentication. The .xml file is copied from the built-in PrefillUserFromExpr.xml but with modified labels for NPS authentication. The username is pre-filled in from the first factor. 
    add authentication loginSchema NPSPasswordOnly -authenticationSchema "/nsconfig/loginschema/NPSPassword.xml"
  10. Create third factor PolicyLabel for Active Directory authentication with Active Directory Login Schema and Active Directory Authentication Policy. 
    add authentication policylabel LDAPPasswordAuth -loginSchema LDAPPasswordOnly

bind authentication policylabel LDAPPasswordAuth -policyName LDAP-Corp -priority 100 -gotoPriorityExpression NEXT
  11. Create third factor PolicyLabel for NPS authentication with NPS Login Schema and NPS Authentication Policy. 
    add authentication policylabel NPSPasswordAuth -loginSchema NPSPasswordOnly

bind authentication policylabel NPSPasswordAuth -policyName NPS -priority 100 -gotoPriorityExpression NEXT
  12. Create second factor PolicyLabel with Policies that choose Next Factor. This PolicyLabel is processed before the two we just created. 
    add authentication policylabel CheckForAuthType -loginSchema LSCHEMA_INT

bind authentication policylabel CheckForAuthType -policyName TwoFactor -priority 90 -gotoPriorityExpression NEXT -nextFactor NPSPasswordAuth

bind authentication policylabel CheckForAuthType -policyName LDAP-Only -priority 100 -gotoPriorityExpression NEXT -nextFactor LDAPPasswordAuth
  13. Create AAA vServer. Bind Login Schema Policy (username only) and Group Extraction Policy. 
    add authentication vserver AAA SSL 10.x.x.218 443
bind authentication vserver AAA -policy OnlyUsername -priority 100 -gotoPriorityExpression END
bind authentication vserver AAA -policy LDAP-Corp-GroupExtract -priority 100 -nextFactor CheckForAuthType -gotoPriorityExpression NEXT
  14. Perform additional steps not detailed here:
    1. For Traffic Management:
      1. Create a Session Policy and bind it to the AAA vServer.
      2. Enable authentication on the Load Balancing or Content Switching vServer.
    2. For NetScaler Gateway, create an Authentication Profile, and bind it to the Gateway vServer.

Virtual Delivery Agent (VDA) 7.14.1

Last Modified: Nov 7, 2020 @ 6:35 am

15 Comments

Navigation

💡 = Recently Updated

Hardware

Hypervisor Host Hardware

  • Citrix Blog Post Citrix Scalability — The Rule of 5 and 10: Simply take the number of physical cores in a hypervisor host, multiply it by 5 or 10, and the result will be your Single Server Scalability. Use 5 if you’re looking for the number of XenDesktop VMs you can host on a box, and use 10 if you’re looking for the number of XenApp user sessions you can host on a box.

Virtual Machine Hardware

  1. For virtual desktops, give the virtual machine: 2+ vCPU and 2+ GB of RAM
  2. For Windows 2008 R2 RDSH, give the virtual machine 4 vCPU and 12-24 GB of RAM
  3. For Windows 2012 R2 RDSH, give the virtual machine 8 vCPU, and 24-48 GB of RAM
  4. If using RAM caching (MCSIO or PvS), add more RAM for the cache
  5. Remove the floppy drive
  6. Remove any serial or LPT ports
  7. If vSphere:
    1. To reduce disk space, reserve memory. Memory reservations reduce or eliminate the virtual machine .vswp file.
    2. The NIC should be VMXNET3.
  8. If this VDA will boot from Provisioning Services:
    1. For vSphere, the NIC must be VMXNET3.
    2. For vSphere, configure the CD-ROM to boot from IDE instead of SATA. SATA comes with VM hardware version 10. SATA won’t work with PvS.
  9. For Windows 10:
    1. CTX224843 Windows 10 compatibility with Citrix XenDesktop: Current Branch (CB) is not supported.
    2. Visual Studio 2017 is not supported on LTSB. See Visual Studio 2017 Product Family System Requirements.
  10. Install the latest version of drivers (e.g. VMware Tools).
    1. If Windows 7 on vSphere, don’t install the VMware SVGA driver. For more details, see Citrix CTX201804 Intermittent Connection Failures/Black Screen Issues When Connecting from Multi-Monitor Client Machines to Windows 7 VDA with VDA 7.x on vSphere/ESXi.
  11. The vSphere Activity Monitoring Feature with NSX Guest Introspection feature uses a TDI driver (vnetflt.sys), which might cause a “Connection Interrupted” message when users log off of Citrix. See VMware 2121307 Windows virtual machines using the vShield Endpoint TDI Manager or NSX Network Introspection Driver (vnetflt.sys) driver fails with a blue diagnostic screen and XenDesktop 7.12 logoff: Connection interrupted at Citrix Discussions.

If vSphere, disable NIC Hotplug

  1. Users could use the systray icon to Eject the Ethernet Controller. Obviously this is bad.
  2. To disable this functionality, power off the virtual machine.
  3. Once powered off, right-click the virtual machine, and click Edit Settings.
  4. On the VM Options tab, expand Advanced, and then click Edit Configuration.
  5. Click Add Row.
  6. On the left, enter devices.hotplug. On the right, enter false.
  7. Then click OK a couple times to close the windows.
  8. The VM can then be powered on.

Windows Preparation

  1. Computer Group Policy – Make sure the Master VM is in the same OU as the Linked Clones so the Master VM will get the computer-level GPO settings in its registry. Run gpupdate on the master after moving the VM to the correct OU. When Clones are created from the Master, the computer-level GPO settings will already be applied, thus eliminating a timing issue.
  2. If RDSH (Server OS), disable IE Enhanced Security Configuration in Server Manager > Local Server.
  3. Optionally, go to Action Center (Windows 8.1 or 2012 R2) or Control Panel > Security and Maintenance (Windows 10/2016) to disable User Account Control, and enable SmartScreen.

    1. In Windows 10 1703 and newer, search the Settings app for Change User Account Control settings.
    2. SmartScreen is configured in Windows Defender Security Center > App & browser control.
  4. Run Windows Update.

  5. Add your Citrix Administrators group to the local Administrators group on the VDA. Computer Management.
  6. The Remote Desktop Services “Prompt for Password” policy prevents Single Sign-on to the Virtual Delivery Agent. Check registry key HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services. If fPromptForPassword = 1 then you need to fix group policy. The following GPO setting will prevent Single Sign-on from working.
    Computer Configuration | Policies | Administrative Templates | Windows Components | Remote Desktop Services | Remote Desktop Session Host | Security | Always prompt for password upon connection
    Or set the registry value HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\PorticaAutoLogon (DWORD) = 0x10.
  7. For Windows 7/2008 R2 VDAs that will use Personal vDisk, or AppDisk, or any other layering technology, install Microsoft hotfix 2614892 A computer stops responding because of a deadlock situation in the Mountmgr.sys driver. This hotfix solved a Personal vDisk Image update issue detailed at Citrix Discussions.
  8. If this VDA is Windows Server 2008 R2, see https://www.carlstalhood.com/windows-server-2008-r2-post-sp1-hotfixes/.
  9. To remove the built-in apps in Windows 10, see Robin Hobo How to remove built-in apps in Windows 10 Enterprise.
  10. For Remote Assistance in Citrix Director, configure the GPO setting Computer Configuration | Policies | Administrative Templates | System | Remote Assistance | Offer Remote Assistance. See Jason Samuel – How to setup Citrix Director Shadowing with Remote Assistance using Group Policy for more details.
  11. If you intend to use Citrix’s SCOM Management Packs for XenApp/XenDesktop, make sure WinRM is enabled on the VDA by running winrm quickconfig. Or you can enable WinRM using Group Policy.

Install Virtual Delivery Agent 7.14.1

  1. For virtual desktops, make sure you are logged into the console. The VDA won’t install if you are connected using RDP.
  2. Make sure .NET Framework 4.5.2 or newer is installed.

CLI Install:

Command Line Install Options are detailed at Install using the command line at Citrix Docs.

The Citrix Telemetry Service seems to cause problems. You can use the Command Line Installer to exclude Telemetry Service as detailed at VDA upgrade cmdlet at Citrix Discussions.

XenDesktopVDASetup.exe /quiet /noreboot /masterimage /Enable_HDX_PORTS /enable_framehawk_port /Enable_REAL_TIME_TRANSPORT /optimize /controllers "xdc01.corp.local xdc02.corp.local" /Exclude "Citrix Telemetry Service"

GUI Install:

  1. Go to the downloaded XenDesktop 7.14.1 iso file and extract it. If Windows 8 or newer, you can instead mount it, but be aware that with mounting, the install won’t resume correctly after a reboot.

  2. Run AutoSelect.exe.
  3. Alternatively, you can download the standalone VDA package and run that instead. Go the main XenDesktop 7.14.1 download page. Expand the section labelled Components that are on the product ISO but also packaged separately to download the Standalone VDA installers. 7.14.1 has a VDA installer called Desktop OS Core Services that is designed for Remote PC deployments.
  4. Click Start next to either XenApp or XenDesktop. The only difference is the product name displayed in the installation wizard.
  5. Click Virtual Delivery Agent for Windows Desktop OS, or Windows Server OS, depending on which type of VDA you are building.

  6. In the Environment page, select Create a Master Image, and click Next.

  7. For virtual desktops, in the HDX 3D Pro page, click Next.
  8. In the Core Components page, if you don’t need Citrix Receiver installed on your VDA, then uncheck the box. Receiver is usually only needed for double-hop connections (connect to first VDA, and then from there, connect to second VDA). Click Next.
  9. In the Additional Components page, uncheck Citrix AppDisk/Personal vDisk. This feature has been deprecated and is being replaced by Citrix App Layering (Unidesk). Click Next.

  10. In the Delivery Controller page, select Do it manually. Enter the FQDN of each Controller. Click Test connection. And then make sure you click Add. Click Next when done.

  11. In the Features page, check boxes. In 7.12 and newer, only the top box is checked by default. If you want to use the other features, check the boxes. If this is a virtual desktop, you can leave Personal vDisk unchecked now and enable it later. Then click Next.
  12. In the Firewall page, click Next.
  13. In the Summary page, click Install.

  14. If RDSH, click Close when you are prompted to restart.
  15. After the machine reboots twice, login and installation should continue.
  16. If you see a Locate ‘XenApp’ installation media window, click Cancel.

    1. Mount the XenApp_and_XenDesktop_7_14_1.iso.
    2. Run AutoSelect.exe.
    3. Click the Virtual Desktop Agent box to resume installation.
  17. Installation will continue automatically.
  18. Note: NT SERVICE\CitrixTelemetryService needs permission to login as a service.
  19. In the Smart Tools page, click Connect, enter your MyCitrix.com credentials, and then click Next.

  20. In the Finish page, click Finish to restart the machine again.
  21. Programs and Features shows Citrix Virtual Delivery Agent 7.14.1 as version 7.14.1.14098.

Citrix Desktop Helper Service  💡

Citrix Blog Post Augment Your XenDesktop Deployment with the Desktop Helper Service: this installable service adds the following functionality to your VDAs:

  • The “Shutdown Inactive Desktops” feature allows Citrix administrators to enable a timer that shuts down a virtual desktop after it has been registered for a configured amount of minutes without a user connection.
  • Delaying the Citrix Desktop Service start by a configurable amount of time allows the desktop to finish performing on-boot tasks before a user is brokered to it.
  • The “Force Group Policy Update” feature give administrators the ability to force a group policy update after a configured amount of time.

If these features are desirable, download the tool from the blog post and install it.

Configurable Registry keys are located at HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopHelper. Each value is detailed in the accompanying Word document.

Customer Experience Improvement Program (CEIP)

VDA 7.12 and newer enable Customer Experience Improvement Program (CEIP) by default. To disable it, create the registry value HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Telemetry\CEIP\Enabled (DWORD) and set it to 0 (zero). Also see CEIP at Citrix Insight Services at Citrix Docs.

See https://www.carlstalhood.com/delivery-controller-7-14-and-licensing/#ceip for additional places where CEIP is enabled.

Connection Quality Indicator

The Connection Quality Indicator tells the user the quality of the connection. For example:

Position of the indicator is configurable by the user. Thresholds are configurable through group policy.

Download it from CTX220774 Connection Quality Indicator and install it. The article is very detailed.

Group Policy templates are located at C:\Program Files (x86)\Citrix\Connection Quality Indicator\Configuration. Copy the files and folder to <Sysvol>\Policies\PolicyDefinitions, or C:\Windows\PolicyDefinitions.

Find the settings under Computer Config | Policies | Administrative Templates | Citrix Components | Virtual Desktop Agent | CQI

Notification display settings lets you customize the user notifications, or disable them.

Connection Threshold Settings lets you set the notification thresholds.

Adaptive Transport

XenApp/XenDesktop 7.13 and newer include Adaptive Transport, which uses EDT protocol, which uses UDP Ports 1494/2598 for HDX connections to the VDA. The UDP ports should already be open in the Windows Firewall.

Adaptive Transport is disabled by default, but can be enabled in the Citrix Policy setting HDX Adaptive Transport.

Slow Logons

Citrix Discussions Xenapp 7.9: Wait for local session manager: “I have a Xenapp 7.9 environment on Windows 2012 R2. When logging in through Citrix I got message “Wait for local session manager” for 20-30 seconds. When logging in to the server with RDS, I do not have to wait for this.”

“Add the following 2 registry keys to your 7.9 VDA server – then try connecting to it using ICA to see if the issue still occurs:

Add reg keys in “HKLM\SOFTWARE\Citrix\GroupPolicy”
Dword: “CacheGpoExpireInHours” – Value = 5-24 (# of Hours) ***start with value of 5***
Dword: “GpoCacheEnabled” – Value = 1

Restart the machine after adding these registry keys and attempt an ICA connection (at least twice) to see if that helps the Login delay.”

 

Mark DePalma at XenApp slow logon times, user get black screen for 20 seconds at Citrix Discussions says that pushing Tile Refresh to a background task speeds up logons.

  1. Regedit: 
    Windows Registry Editor Version 5.00
 
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\DisableUPMResetCache]
@="DisableUPMResetCache"
"Version"="1,1,1,1"
"StubPath"="REG ADD HKCU\\Software\\Microsoft\\Windows\\CurrentVersion\\ImmersiveShell\\StateStore /v ResetCache /t REG_DWORD /d 0 /f"
"Locale"="*"
  2. UPM Exclusions: 
    Directory - '!ctx_localappdata!\Microsoft\Windows\Caches'
Registry - 'SOFTWARE\Microsoft\Active Setup\Installed Components\DisableUPMResetCache'

 

Marvin Neys at XenApp slow logon times, user get black screen for 20 seconds at Citrix Discussions says that deleting HKCU\Software\Microsoft\Windows\CurrentVersion\UFH\SHC at logoff reduces logon times from 40 seconds to 6 seconds.

Remove-Item HKCU:\Software\Microsoft\Windows\CurrentVersion\UFH\SHC

 

For additional logon delay troubleshooting, see Alexander Ollischer XenApp/XenDesktop – “Please Wait For Local Session Manager” message when logging into RDS. He found some Windows Updates that caused a logon delay.

 

XenApp recalculates WMI filters on every reconnect. CTX212610 Session Reconnect 30 sec Delay – DisableGPCalculation – WMI Filters indicates that recalculation can be disabled by setting HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Reconnect\DisableGPCalculation (DWORD) to 1.

 

CTX212439 Desktop Session Stuck in Pre-Logon State with Message “Please wait for the Local Session Manager”:

  • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\Parameters\MaxTokenSize (DWORD) = 48000
  • Delete HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\RCM\GracePeriod\L$RTMTIMEBOMB

Controller Registration Port

Some environments will not accept the default port 80 for Virtual Delivery Agent registration, even though registration is authenticated and encrypted on port 80. To change the port, do the following on the Virtual Delivery Agent:

  1. Open Programs and Features. If Windows 10 1703 or newer, open Apps and Features.
  2. Find Citrix Virtual Delivery Agent, and click Change or Modify (Windows 10 1703 and newer).

  3. Click Customize Virtual Delivery Agent Settings.
  4. Edit the Delivery Controllers, and click Next.
  5. On the Protocol and Port page, change the port number, and click Next.
  6. In the Summary page, click Reconfigure.
  7. In the Finish Reconfiguration page, click Finish to restart the machine.
  8. You must also change the VDA registration port on the Delivery Controllers by running BrokerService.exe /VDAPort.

Controller Registration – Verify

  1. If you restart the Virtual Delivery Agent machine, or restart the Citrix Desktop Service…
  2. In Windows Logs Application log, you should see an event 1012 from Citrix Desktop Service saying that it successfully registered with a controller. If you don’t see this then you’ll need to fix the ListOfDDCs registry key. See VDA registration with Controllers at Citrix Docs.
  3. You can also run Citrix’s Health Assistant on the VDA.
  4. See CTX220772 Technical Primer: VDA Registration for a very detailed explanation of the VDA Registration process.

Citrix PDF Printer 7.11.0 for Receiver for HTML5/Chrome

  1. To allow printing from Receiver for HTML5/Chrome, install Citrix PDF Printer. Get it from the Receiver for HTML5 download page in the Additional Components section. Note: this PDF Printer is only used by Receiver for HTML5 and Receiver for Chrome.
  2. Go to the extracted CitrixPDFPrinter_7.11.0 and run CitrixPDFPrinter64.msi.
  3. In the Please read the Citrix PDF printer License Agreement page, check the box next to I accept the terms, and click Install.
  4. In the Completed the Citrix PDF Universal Driver Setup Wizard page, click Finish.
  5. In Programs and Features, it is shown as version 7.11.0.11.
  6. Configure a Citrix Policy to enable the PDF printer. The setting is called Auto-create PDF Universal Printer in the user half of a Citrix Policy GPO.

Citrix File Access 2.0.3 for Receiver for Chrome

  1. If you support Receiver for Chrome (Chromebook) and want to open files on Google Drive using published applications, install Citrix File Access on the VDAs. Get it from the Receiver for Chrome download page, in the Additional Components section.
  2. Go to the extracted Citrix_File_Access_2.0.3, and run FileAccess.msi.
  3. In the Please read the File Access License Agreement page, check the box next to I accept the terms, and click Install.
  4. In the Completed the File Access Setup Wizard page, click Finish.
  5. File Access is listed in Programs and Features as version 2.0.3.33.
  6. File Access has a default list of supported file extensions. The list can be expanded by editing the registry on the VDA. See CTX219983 Receiver for Chrome Error: Invalid command line arguments: Unable to open the file as it has an unsupported extension.
  7. To open a file from Google Drive, right-click and and open the file using Citrix Receiver.

Framehawk Configuration

To enable Framehawk, see https://www.carlstalhood.com/citrix-policy-settings/#framehawkconfig

Remote Desktop Licensing Configuration

On 2012 R2 and newer RDSH, the only way to configure Remote Desktop Licensing is using group policy (local or domain). This procedure also works for 2008 R2 RDSH. This procedure is not needed on virtual desktops.

  1. For local group policy, run gpedit.msc. Alternatively, you can configure this in a domain GPO.
  2. Go to Computer Configuration > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Licensing.
  3. Double-click Use the specified Remote Desktop license servers. Change it to Enabled, and enter the names of the RDS Licensing Servers (typically installed on XenDesktop Controllers). Click OK.
  4. Double-click Set the Remote Desktop licensing mode. Change it to Enabled and select Per User. Click OK.
  5. Optionally, you can install the Remote Desktop Licensing Diagnoser Tool. In the Server Manager > Add Roles and Features Wizard, on the Features page, expand Remote Server Administration Tools, expand Role Administration Tools, expand Remote Desktop Services Tools, and select Remote Desktop Licensing Diagnoser Tool. Then Finish the wizard.
  6. If it won’t install from Server Manager, you can install it from PowerShell by running Install-WindowsFeature rsat-rds-licensing-diagnosis-ui.
  7. In Server Manager, open the Tools menu, expand Remote Desktop Services (or Terminal Services), and click Remote Desktop Licensing Diagnoser.

  8. The Diagnoser should find the license server, and indicate the licensing mode. If you’re configured for Per User licenses, then it’s OK if there are no licenses installed on the Remote Desktop License Server.

Several people in Citrix Discussions reported the following issue: If you see a message about RD Licensing Grace Period has expired even though RD Licensing is properly configured, see Eric Verdumen No remote Desktop Licence Server availible on RD Session Host server 2012. The solution was to delete the REG_BINARY in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\RCM\GracePeriod only leaving the default. You must take ownership and give admin users full control to be able to delete this value.

C: Drive Permissions

This section is more important for shared VDAs like RDSH (Windows Server 2008 R2, Windows Server 2012 R2, and Windows Server 2016).

The default permissions allow users to store files on the C: drive in places other than their profile.

  1. Open the Properties dialog box for C:.
  2. On the Security tab, click Advanced.
  3. If UAC is enabled, click Change permissions.
  4. Highlight the line containing Users and Create Folders, and click Remove.
  5. Highlight the line containing Users and Create files (or Special), and click Remove. Click OK.
  6. Click Yes to confirm the permissions change.
  7. If you see any of these Error Applying Security windows, click Continue. This window should appear multiple times.
  8. Click OK to close the C: drive properties.

Pagefile

If this image will be converted to a Provisioning Services vDisk, then you must ensure the pagefile is smaller than the cache disk. For example, if you allocate 20 GB of RAM to your Remote Desktop Session Host, and if the cache disk is only 15 GB, then Windows will have a default pagefile size of 20 GB, and Provisioning Services will be unable to move it to the cache disk. This causes Provisioning Services to cache to server instead of caching to your local cache disk (or RAM).

  1. Open System. In 2012 R2 and newer, you can right-click the Start button, and click System. Note: in Windows 10 1703 and newer, this method no longer opens the correct tool.
  2. Another option is to open File Explorer, right-click This PC, and click Properties. This works in Windows 10 1703.
  3. Click Advanced system settings.
  4. On the Advanced tab, click the top Settings button.
  5. On the Advanced tab, click Change.
  6. Uncheck the box next to Automatically manage paging file size for all drives. Then either turn off the pagefile, or set the pagefile to be smaller than the cache disk. Don’t leave it set to System managed size. Click OK several times.

Direct Access Users

When Citrix Virtual Delivery Agent is installed on a machine, non-administrators can no longer RDP to the machine. A new local group called Direct Access Users is created on each Virtual Delivery Agent. Add your non-administrator RDP users to this local group so they can RDP directly to the machine.

Windows Profiles v3/v4/v5/v6

Roaming Profiles are compatible only between the following client and server operating system pairs. The profile version is also listed.

  • v6 = Windows 10 (1607 and 1703) and Windows Server 2016
  • v5 = Windows 10 (1511 and older)
  • v4 = Windows 8.1 and Windows Server 2012 R2
  • v3 = Windows 8 and Windows Server 2012
  • v2 = Windows 7 and Windows Server 2008 R2
  • v2 = Windows Vista and Windows Server 2008

For Windows 2012 R2, install Microsoft hotfix 2890783, and set the UseProfilePathExtensionVersion registry value to 1.

Registry

Published Explorer

From Citrix CTX128009 Explorer.exe Fails to Launch: When publishing the seamless explorer.exe application, the session initially begins to connect as expected. After the loading, the dialog box disappears, and the Explorer application fails to appear. On the VDA, use the following registry change to set the length of time a client session waits before disconnecting the session:

  • Key = HKLM\SYSTEM\CurrentControlSet\Control\Citrix\wfshell\TWI
    • Value = LogoffCheckerStartupDelayInSeconds (DWORD) = 10 (Hexadecimal)

Screen Saver

From Citrix CTX205214 Screensaver Not Working in XenDesktop: By default, Screen Saver doesn’t work on Desktop OS. To enable it, on the VDA, configure the following registry value:

  • Key = HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Graphics
    • Value = SetDisplayRequiredMode (DWORD) = 0

Logon Disclaimer Window Size

From XenApp 7.8 – Session Launch Security/Warning Login Banner at Citrix Discussions: If your logon disclaimer window has scroll bars, set the following registry values:

  • Key = HKEY_LOCAL_MACHINE\Software\Wow6432node\Citrix\CtxHook\AppInit_DLLS\Multiple Monitor Hook
    • Value = LogonUIWidth (DWORD) = 300
    • Value = LogonUIHeight (DWORD) = 200

Login Timeout

From Citrix CTX203760 VDI Session Launches Then Disappears: XenDesktop, by default, only allows 180 seconds to complete a logon operation. The timeout can be increased by setting the following:

  • Key = HKLM\SOFTWARE\Citrix\PortICA
    • Value = AutoLogonTimeout ( DWORD) = decimal 240 or higher (up to 3600).

Also see Citrix Discussions Machines in “Registered” State, but VM closes after “Welcome” screen.

HDX Flash

From Citrix Knowledgebase article CTX139939 – Microsoft Internet Explorer 11 – Citrix Known Issues: The registry key value IEBrowserMaximumMajorVersion is queried by the HDX Flash service to check for maximum Internet Explorer version that HDX Flash supports. For Flash Redirection to work with Internet Explorer 11 set the registry key value IEBrowserMaximumMajorVersion to 11 on the machine where HDX flash service is running. In case of XenDesktop it would be the machine where VDA is installed.

  • Key = HKLM\SOFTWARE\Wow6432Node\Citrix\HdxMediaStreamForFlash\Server\PseudoServer
    • Value = IEBrowserMaximumMajorVersion (DWORD) = 11 (Decimal)

From Citrix Discussions: Add the DWORD FlashPlayerVersionComparisonMask=0 on the VDA under HKLM\Software\Wow6432Node\Citrix\HdxMediaStreamForFlash\Server\PseudoServer.  This disables the Flash major version checking between the VDA and Client Device.

Receiver for HTML5/Chrome Enhanced Clipboard

From About Citrix Receiver for Chrome 1.9 at Citrix Docs: To enable enhanced clipboard support, create a REG_SZ registry value HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Citrix\wfshell\Virtual Clipboard\Additional Formats\HTML Format\Name=”HTML Format”. Create any missing registry keys. This applies to both virtual desktops and Remote Desktop Session Hosts.

Receiver for HTML5/Chrome Upload Folder

The Receiver for HTML5 (or Chrome) lets upload files.

By default, the user is prompted to select a upload location. If you use the Upload feature multiple times, the last selected folder is not remembered.

Citrix CTX217351 How to Customize File Upload and Download Using Receiver for HTML5 and Receiver for Chrome. You can specify a default uploads location by editing HKLM\Software\Citrix\FileTransfer\UploadFolderLocation on the VDA. Environment variables are supported. When this value is configured, users are no longer prompted to select an upload location. The change takes effect at next logon.

Note: HTML5/Chrome Receiver also adds a Save to My Device location to facilitate downloads.

4K Monitors

From Citrix Knowledgebase article CTX218217 Unable to span across multiple monitors after upgrade to 7.11 VDA, Black/Blank screen appears on the monitors while connecting to ICA session: .

  1. For VDA 7.11 and newer, calculate the video memory that is required for monitors using the following formula:
    SumOfAllMons (Width * Height) * 4 / 0.3, where width and height are resolution of the monitor. Note: There is no hard and fast rule that will work for all cases.
    Example: Consider the resolution of monitor 1 is 1920*1200 and monitor 2 is 1366*768. Then SumOfAllMons will be (1920*1200 + 1366*768)
  2. CTX115637 Citrix Session Graphics Memory Reference describes how multi-monitor resolution is determined.
  3. Open the registry (regedit) and navigate to: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\vbdenum
  4. Increase the value of “MaxVideoMemoryBytes” REG_DWORD value to the above calculated memory.
  5. Reboot the VDA.

Citrix Policies also control graphics performance.

COM Port Threads

CTX212090 COM Port Intermittently Inaccessible During ICA Sessions: increase the default value of “MaxThreads” under the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\picaser\Parameters from 20 to a value greater than the number of COM port connections you want to support. For example, if a XenApp server supports 100 sessions and each session opens two COM ports, the value of “MaxThreads” should be greater than 200.

Legacy Client Drive Mapping

Citrix CTX127968 How to Enable Legacy Client Drive Mapping Format on XenApp: Citrix Client Drive Mapping no longer uses drive letters and instead they appear as local disks. This is similar to RDP drive mapping.

The old drive letter method can be enabled by setting the registry value:

  • Key = HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\UncLinks (create the key)
    • Value = UNCEnabled (DWORD) = 0

When you reconnect, the client drives will be mapped as drive letters (starts with V: and goes backwards).

Print Driver for Non-Windows Clients

This section applies to Windows 8.1/2012 and newer VDAs.

From CTX139020 Configuring Virtual Machines for Mac Client Printer Mapping with Windows 8.x. By default, Non-Windows clients cannot map printers due to a missing print driver on the VDA machine.

  1. Requirements:
    • Internet Access
    • Windows Update service enabled
  2. Click Start, and run Devices and Printers.
  3. In Windows 10 1703, open Printers & scanners, then scroll down, and click Devices and printers.

  4. In the Printers section, highlight a local printer (e.g. Microsoft XPS Document Writer). Then in the toolbar, click Print server properties.

  5. Switch to the Drivers tab. Click Change Driver Settings.
  6. Then click Add.
  7. In the Welcome to the Add Printer Driver Wizard page, click Next.
  8. In the Processor Selection page, click Next.
  9. In the Printer Driver Selection page, click Windows Update. The driver we need won’t be in the list until you click this button. Internet access is required.
  10. Once Windows Update is complete, highlight HP on the left, and then select HP Color LaserJet 2800 Series PS (Microsoft) on the right. Click Next.
  11. In the Completing the Add Printer Driver Wizard page, click Finish.
  12. Repeat these instructions to install the following additional drivers:
    • HP LaserJet Series II
    • HP Color LaserJet 4500 PCL 5

SSL for VDA

If you intend to use HTML5 Receiver internally, install certificates on the VDAs so the WebSockets (and ICA) connection will be encrypted. Internal HTML5 Receivers will not accept clear text WebSockets. External users don’t have this problem since they are SSL-proxied through NetScaler Gateway. Notes:

  • Each Virtual Delivery Agent needs a machine certificate that matches the machine name. This is feasible for a small number of persistent VDAs. For non-persistent VDAs, you’ll need some automatic means for creating machine certificates every time they reboot.
  • As detailed in the following procedure, use PowerShell on the Controller to enable SSL for the Delivery Group. This forces SSL for every VDA in the Delivery Group, which means every VDA in the Delivery Group must have SSL certificates installed.

The following instructions for manually enabling SSL on VDA can be found at Configure SSL on a VDA using the PowerShell script at Citrix Docs.

  1. On the VDA machine, run mmc.exe.
  2. Add the Certificates snap-in.
  3. Point it to Local Computer.
  4. Request a certificate from your internal Certificate Authority. You can use either the Computer template or the Web Server template.

    1. You can also use group policy to enable Certificate Auto-Enrollment for the VDA computers.
  5. Browse to the XenApp/XenDesktop 7.14.1 ISO. In the Support\Tools\SslSupport folder, shift+right-click the Enable-VdaSSL.ps1 script and click Copy as path.
  6. Run PowerShell as administrator (elevated).
  7. Run the command Set-ExecutionPolicy unrestricted. Enter Y to approve.
  8. In the PowerShell prompt, type in an ampersand (&), and a space.
  9. Right-click the PowerShell prompt to paste in the path copied earlier.
  10. At the end of the path, type in -Enable
  11. If there’s only one certificate on this machine, press Enter.
  12. If there are multiple certificates, you’ll need to specify the thumbprint of the certificate you want to use. Open the Certificates snap-in, open the properties of the machine certificate you want to use, and copy the Thumbprint from the Details tab.

    In the PowerShell prompt, at the end of the command, enter ‑CertificateThumbPrint, add a space, and type quotes (").
    Right-click the PowerShell prompt to paste the thumbprint.
    Type quotes (") at the end of the thumbprint. Then remove all spaces from the thumbprint. The thumbprint needs to be wrapped in quotes.
  13. If this VDA machine has a different service already listening on 443 (e.g. IIS), then the VDA needs to use a different port for SSL connections. At the end of the command in the PowerShell prompt, enter -SSLPort 444 or any other unused port.
  14. Press <Enter> to run the Enable-VdaSSL.ps1 script.
  15. Press <Y> twice to configure the ACLs and Firewall.
  16. You might have to reboot before the settings take effect.
  17. Login to a Controller, and run PowerShell as Administrator (elevated).
  18. Run the command asnp Citrix.*
  19. Enter the command: 
    Get-BrokerAccessPolicyRule -DesktopGroupName '<delivery-group-name>' | Set-BrokerAccessPolicyRule ‑HdxSslEnabled $true

    where <delivery-group-name> is the name of the Delivery Group containing the VDAs.

  20. You can run Get-BrokerAccessPolicyRule -DesktopGroupName '<delivery-group-name>' to verify that HDX SSL is enabled.
  21. Also run the following command: 
    Set-BrokerSite –DnsResolutionEnabled $true

You should now be able to connect to the VDA using the HTML5 Receiver from internal machines.

The Citrix blog post How To Secure ICA Connections in XenApp and XenDesktop 7.6 using SSL has a method for automatically provisioning certificates for pooled virtual desktops by enabling certificate auto-enrollment and setting up a task that runs after the certificate has been enrolled.

  • From Russ Hargrove at A note on VDA certificates in 7.14 at Citrix Discussions: Citrix installs a new “Citrix XenApp/XenDesktop HDX Service” certificate in the Personal store which breaks the automation of the Enable-VdaSSL.ps1 script. To fix the problem, modify the task scheduler powershell script to:  💡 
    Enable-VdaSSL.ps1 -Enable -CertificateThumbPrint (Get-ChildItem -path cert:\LocalMachine\My | Where-Object -FilterScript {$_.Subject -eq ""} | Select-Object -ExpandProperty Thumbprint) -Confirm:$False
  • For certificate auto-enrollment on non-persistent Remote Desktop Session Hosts (aka Server OS VDAs), see Non-Persistent Server SSL to VDA by Alfredo Magallon Arbizu at CUGC.

Anonymous Accounts

If you intend to publish apps anonymously then follow this section.

  1. Anonymous accounts are created locally on the VDAs. When XenDesktop creates Anon accounts it gives them an idle time as specified at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Citrix\AnonymousUserIdleTime. The default is 10 minutes. Adjust as desired.
  2. You can pre-create the Anon accounts on the VDA by running “C:\Program Files\Citrix\ICAConfigTool\CreateAnonymousUsersApp.exe”. If you don’t run this tool then Virtual Delivery Agent will create them automatically when users log in.
  3. You can see the local Anon accounts by opening Computer Management, expanding System Tools, expanding Local Users and Groups and clicking Users.
  4. If you open one of the accounts, on the Sessions tab, notice that idle timeout defaults to 10 minutes. Feel free to change it.

Group Policy for Anonymous Users

Since Anonymous users are local accounts on each Virtual Delivery Agent, domain-based GPOs will not apply. To work around this limitation, you’ll need to edit the local group policy on each Virtual Delivery Agent.

  1. On the Virtual Delivery Agent, run mmc.exe.
  2. Open the File menu, and click Add/Remove Snap-in.
  3. Highlight Group Policy Object Editor, and click Add to move it to the right.
  4. In the Welcome to the Group Policy Wizard page, click Browse.
  5. On the Users tab, select Non-Administrators.
  6. Click Finish.
  7. Now you can configure group policy to lockdown sessions for anonymous users. Since this is a local group policy, you’ll need to repeat the group policy configuration on every Virtual Delivery Agent image. Also, Group Policy Preferences is not available in local group policy.

Antivirus

Install antivirus using your normal procedure. Instructions vary for each Antivirus product.

Microsoft’s virus scanning recommendations (e.g. exclude group policy files) – http://support.microsoft.com/kb/822158.

Citrix’s Recommended Antivirus Exclusions

Citrix Blog Post Citrix Recommended Antivirus Exclusions: the goal here is to provide you with a consolidated list of recommended antivirus exclusions for your Citrix virtualization environment focused on the key processes, folders, and files that we have seen cause issues in the field:

  • Set real-time scanning to scan local drives only and not network drives
  • Disable scan on boot
  • Remove any unnecessary antivirus related entries from the Run key
  • Exclude the pagefile(s) from being scanned
  • Exclude Windows event logs from being scanned
  • Exclude IIS log files from being scanned

See the Blog Post for exclusions for each Citrix component/product including: StoreFront, VDA, Controller, and Provisioning Services. The Blog Post also has links to additional KB articles on antivirus.

Symantec

Symantec links:

Trend Micro

Trend Micro Slow login on Citrix environment after installing OfficeScan (OSCE): The following registries can be used to troubleshoot the issue. These registries will allow a delay on the startup procedure of OSCE until the system has launched successfully. This avoids deadlock situations during login.

Citrix CTX136680 – Slow Server Performance After Trend Micro Installation. Citrix session hosts experience slow response and performance more noticeable while users try to log in to the servers. At some point the performance of the servers is affected, resulting in issues with users logging on and requiring the server to be restarted. This issue is more noticeable on mid to large session host infrastructures.

Trend Micro has provided a registry fix for this type of issue. Create the following registry on all the affected servers. Add new DWORD Value as:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TmFilterParameters] “DisableCtProcCheck”=dword:00000001

Trend Micro Links:

Sophos

Best Practice for running Sophos on virtual systems: we’ve amassed the following practical information about how you can optimize our software to work with this technology.

Sophos Anti-Virus for Windows XP+: Installation and configuration considerations for Sophos Anti-Virus on a Remote Desktop Services server: It maybe desirable to disable the Sophos AutoUpdate shield icon

Sophos Anti-Virus for Windows 2000+: incorporating current versions in a disk image, including for use with cloned virtual machines: This procedure will make sure that the produced target/cloned computers:

  • Get their distinct identity with Enterprise Console, under which they can be subsequently managed.
  • Have the desired version of Sophos Anti-Virus already installed and configured on the created image.

Windows Defender Antivirus

Deployment guide for Windows Defender Antivirus in a virtual desktop infrastructure (VDI) environment

Optimize Performance

VDA Optimizer

Installation of the VDA might have already done this but there’s no harm in doing it again. This tool is only available if you installed VDA in Master Image mode.

  1. On the master VDA, go to C:\Program Files\Citrix\PvsVm\TargetOSOptimizer, and run TargetOSOptimizer.exe.
  2. Then click OK. Notice that it disables Windows Update.

Windows 10 / Windows 2012 R2 / Windows 2016 and newer

Optimization Notes:

RDSH 2008 R2

Citrix CTX131577 XenApp 6.x (Windows 2008 R2) – Optimization Guide is a document with several registry modifications that are supposed to improve server performance. Ignore the XenApp 6 content and instead focus on the Windows content.

Norskale has Windows 2008 R2 Remote Desktop and XenApp 6 Tuning Tips Update.

Windows 7

Microsoft has compiled a list of links to various optimization guides. It’s a common practice to optimize a Windows 7 virtual machine (VM) template (or image) specifically for VDI use. Usually such customizations include the following.

  • Minimize the footprint, e.g. disable some features and services that are not required when the OS is used in “stateless” or “non-persistent” fashion. This is especially true for disk-intensive workloads since disk I/O is a common bottleneck for VDI deployment. (Especially if there are multiple VMs with the same I/O patterns that are timely aligned).
  • Lock down user interface (e.g. optimize for specific task workers).

With that said the certain practices are quite debatable and vary between actual real-world deployments. Exact choices whether to disable this or that particular component depend on customer requirements and VDI usage patterns. E.g. in personalized virtual desktop scenario there’s much less things to disable since the machine is not completely “stateless”. Some customers rely heavily on particular UI functions and other can relatively easily trade them off for the sake of performance or standardization (thus enhance supportability and potentially security). This is one of the primary reasons why Microsoft doesn’t publish any “VDI Tuning” guide officially.

Though there are a number of such papers and even tools published either by the community or third parties. This Wiki page is aimed to serve as a consolidated and comprehensive list of such resources.

Daniel Ruiz XenDesktop Windows 7 Optimization and GPO’s Settings

Microsoft Whitepaper Performance Optimization Guidelines for Windows 7 Desktop Virtualization

Seal and Shut Down

If this VDA will be a master image in a Machine Creation Services or Provisioning Services catalog, after the master is fully prepared (including applications), do the following:

  1. Go to the properties of the C: drive, and run Disk Cleanup.
  2. If Disk Cleanup is missing, you can run cleanmgr.exe instead.
  3. Windows 10 1703 and newer has a new method for cleaning up temporary files.
    1. Right-click the Start button, and click System.
    2. Click Storage on the left, and click This PC (C:) on the right.
    3. Click Temporary Files.
    4. Check boxes, and click Remove files.
  4. On the Tools tab, click Optimize to defrag the drive.
    `
  5. Run slmgr.vbs /dlv and make sure it is licensed with KMS and has at least one rearm remaining. It is not necessary to manually rearm licensing. XenDesktop will do it automatically.
  6. Run Delprof2 to clean up local profiles. Get it from http://helgeklein.com/download/.
  7. Machine Creation Services and Provisioning Services require DHCP.
  8. Session hosts (RDSH) commonly have DHCP reservations.
  9. Base Image Script Framework (BIS-F) automates many sealing tasks. The script is configurable using Group Policy.
  10. Shut down the master image. You can now use Studio (Machine Creation Services) or Provisioning Services to create a catalog of linked clones.

Troubleshooting – Graphics

If Windows 7 on vSphere, don’t install the VMware SVGA driver. For more details, see CTX201804 Intermittent Connection Failures/Black Screen Issues When Connecting from Multi-Monitor Client Machines to Windows 7 VDA with VDA 7.x on vSphere/ESXi.

For Citrix Policies that control graphics codecs, see https://www.carlstalhood.com/citrix-policy-settings/#graphics

Citrix Blog post – Optimising the performance of HDX 3D Pro – Lessons from the field

From Citrix Knowledgebase article CTX218217 Unable to span across multiple monitors after upgrade to 7.11 VDA, Black/Blank screen appears on the monitors while connecting to ICA session:

  1. For VDA 7.11 and newer, calculate the video memory that is required for monitors using the following formula :
    SumOfAllMons (Width * Height) * 4 / 0.3, where width and height are resolution of the monitor. Note: There is no hard and fast rule that will work for all cases.
    Example: Consider the resolution of monitor 1 is 1920*1200 and monitor 2 is 1366*768. Then SumOfAllMons will be (1920*1200 + 1366*768)
  2. CTX115637 Citrix Session Graphics Memory Reference describes how multi-monitor resolution is determined.
  3. Open the registry (regedit) and navigate to:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\vbdenum
  4. Increase the value of “MaxVideoMemoryBytes” REG_DWORD value to the above calculated memory.
  5. Reboot the VDA

From Citrix Discussions: To exclude applications from Citrix 3D rendering, create a REG_DWORD registry value “app.exe” with value 0 or a registry value “*” with value 0.

  • XD 7.1 and XD 7.5:
    • x86: reg add hklm\software\citrix\vd3d\compatibility /v * /t REG_DWORD /f /d 0
    • x64: reg add hklm\software\Wow6432Node\citrix\vd3d\compatibility /v * /t REG_DWORD /f /d 0
  • XD 7.6/7.7/7.8/7.9/7.11 both x86 and x64:
    • reg add hklm\software\citrix\vd3d\compatibility /v * /t REG_DWORD /f /d 0

Wildcards are not supported. The asterisk * here has a special meaning “all apps” but is not a traditional wildcard. To blacklist multiple apps e.g. both appa.exe and appb.exe must be done by creating a registry value for each app individually.

This is most problematic in Remote PC since most physical PCs have GPUs. I recently had to blacklist Internet Explorer to prevent lockup issues when switching back to physical.

Uninstall VDA

Uninstall the VDA from Programs and Features.

Then see CTX209255 VDA Cleanup Utility.

To run the VDA Cleanup Tool silently:

  1. Execute VDACleanupUtility.exe /silent /noreboot to suppress reboot.
  2. Once the VDACleanupUtility has finished executing, setup Auto logon for the current user.
  3. Reboot.
  4. After reboot, tool will launch automatically to continue Cleanup.

Another option is to delete CitrixVdaCleanup value under HKLM\Software\Microsoft\Windows\CurrentVersion\RunOnce. Then after reboot, run VDACleanupUtility.exe /silent /reboot to indicate that it’s running after the reboot.

Related Pages