FEATURE INTRODUCTION ==================== Feature Name: ------------- Congestion Control Synopsis of Feature: -------------------- The main purpose of this congestion control feature is to keep track of which hosts are congested so that TS will not forward requests to those congested hosts; instead, TS will send back to clients a Retry-After response to tell them to retry congested hosts at a later time. CASE 1. Connection Failures: ------------------------------ (1) For each request to a live (non-congested) server, TS will try at most m times to connect to the server, and the timeout is n seconds for each try. If TS does not succeed with m tries, then one connection failure is counted towards the server. Note that if a client aborts a request before a timeout occurs, it does not count as a connection failure. (2) A server is marked congested if there are more than M connection failures within N seconds. (3) If a server is marked congested, then TS will not forward requests to it until Proxy Retry After Time (PRAT) (which is current time + t) (4) For a request to a congested server before the server's PRAT time, TS sends a Retry-After response to tell the client to retry the request after Client Retry After Time (CRAT) (= PRAT - current time + T + a random interger from 0 to alpha). (5) For a request to a congested server after the server's PRAT time, TS will try at most m' times to connect to the server, and the timeout is n' seconds for each try. (6) A congested server will stay dead if TS cannot make a successful connection; otherwise, the server becomes live again. CASE 2: Maximum Number of Connections ------------------------------------- TS will temporarily mark a server as congested if a "max_connection" number to the server is reached. If a new client request comes in and needs a new connection to the server, the client will get 503 Retry-After back. There is no PRAT on the "max_connection" reached servers. Here a server can be identified by IP address (per_ip) or by host name (per_host). For example, www.inktomi.com has two IP addresses, 209.131.63.206 and 209.131.63.207. If per_ip is used, then each IP address has its own number of connection failures, and each IP address will be marked congested or not by itself. That is, if 206 is marked congested (but 207 is not), requests can still be forwarded to 207. On the other hand, if per_host is used, then one connection failure to either 206 or 207 will be counted to the number if connection failures of host www.inktomi.com. If the host www.inktomi.com is marked congested, then essentially both 206 and 207 are marked congested. We can also use prefix as a secondary specifier to specify the scope of congestion control to sub-host (service) area. For example, dest_host=www.inktomi.com prefix=/cgi/search.exe This rule can detect the stop of the cgi program or database it depends on. Each specification has an independent counter. The error of requests to www.inktomi.com/index.html will count independently to the counter of this line. The prefix=/cgi/ means all requests to the objects under /cgi/ have one common counter with specified parameters. It does not mean each URI under the directory has its own counter. The TS administrator will be able to specify the customizeable error_page, the error_page can be customized to return the reason (for example: "The site is under maintenance") of congestion for '503'. In the error_page, the URL of the page congested, and the Retry-After time can be returned. ENGINEERING DESCRIPTION: ======================== Risk points of feature: ----------------------- The set of "origin server connect attempts" configuration varibles in records.config will be affected by this feature. See the following "Requirement on Server Management" and the above "Synopsis of feature" sections for more information. Also, there are some known problematic/unexpected behaviors of this feature. See the following "Problematic behavior" section. Effect on SDK/API: ------------------ None. Management Implications: ----------------------- There is one new config variable to enable/disable/test congestion control. proxy.config.http.congestion_control.enabled INT 0|1|2 proxy.config.http.congestion_control.filename STRING congestion.config 1. Number of congestions because of connection failures stat name: proxy.process.congestion.congested_on_conn_failures 2. Number of congestions because of max_connection reached stat name: proxy.process.congestion.congested_on_max_connection A new .config file "congestion.config" is used to specify the parameters for different servers. Each rule will have one primary key to identify the servers, the primaries can be dest_host= dest_domain= dest_ip= regex_host= Each rule can also have secondary keys, secondary keys include prefix= // for different directory / service port= // for different server ports The tag=value pairs are used to specify the rules: max_connection_failures= // M fail_window= // N proxy_retry_interval= // t client_wait_interval= // T wait_interval_alpha= //alpha live_os_conn_timeout= // n live_os_conn_retries= // m dead_os_conn_timeout= // n' dead_os_conn_retries= // m' max_connection= // -1 means unlimited error_page= congestion_scheme=per_ip|per_host snmp=on|off The suggested default values are as follows: max_connection_failures=5 fail_window=120 proxy_retry_interval=10 client_wait_interval=300 wait_interval_alpha=30 live_os_conn_timeout=60 live_os_conn_retries=2 dead_os_conn_timeout=15 dead_os_conn_retries=1 max_connection=-1 error_page="congestion#retryAfter" congestion_scheme="per_ip" snmp="on" The above tag values will be used as default if the tag is not specified in the rule. The default values can be overrided by setting the records.config variables CONFIG proxy.config.http.congestion_control.default. The following "origin server connect attempts" configuration variables may be affected by this congestion control feature: proxy.config.http.connect_attempts_max_retries proxy.config.http.connect_attempts_max_retries_dead_server proxy.config.http.connect_attempts_rr_retries proxy.config.http.connect_attempts_timeout proxy.config.http.down_server.cache_time proxy.config.http.down_server.abort_threshold For a request to a server that does not have an applicable rule in congestion.config, the values for these "origins server connect attempts" variables are used by TS. Otherwise, the corresponding values specified in congestion.config will override them. Add two new alarm types to Traffic Manager: 1) MGMT_SIGNAL_HTTP_CONGESTED_SERVER used to indicate a congested server 2) MGMT_SIGNAL_HTTP_ALLEVIATED_SERVER used to indicate a congested server is no longer congested These alarms are not processed like the other Traffic Manager alarms. Whenever these alarms are signalled (even if they are repeat alarms) *only* an SNMP trap will be sent. Note, that this means that potentially, users can be flooded with SNMP traps if a congested server is always signalling an alarm. - when TS detects a congestion server, an alarm is signaled and an SNMP trap is sent (to the configured console). - when a congested server is alleviated, an alarm is signaled and an SNMP trap is sent. - user can disable/enable support SNMP as a whole. SNMP cannot be disabled/enabled per feature basis (e.g.: turn off all congestion control traps only), but SNMP can be enabled/disabled on per rules basis (e.g.: ignore traps generated by congested server that matches this rule) - two traps are defined in the Inktomi vendor MIB tree, they are: httpCongestedServer 1.3.6.1.4.1.1967.3.1.3.3.2 httpAlleviatedServer 1.3.6.1.4.1.1967.3.1.3.3.3 For configuration purposes, we will add a new "Congestion Control" tab to the "Configure -> Networking -> Connection Management" section of the web UI. Within this tab users can: 1. enable/disable the congestion control feature 2. edit the congestion.config file (which will be displayed in a html text box) Use the traffic_line command-line interface to retrieve the congestion statistics and monitoring information. 1. "traffic_line -r " Returns the value of the statistic specified 2. "traffic_line -q" Returns a list of currently congested sites (one site per line); for each congested site, displays the information in the following format: '