diff --git a/doc/close-options.txt b/doc/close-options.txt deleted file mode 100644 index 0554bb859..000000000 --- a/doc/close-options.txt +++ /dev/null @@ -1,39 +0,0 @@ -2011/04/20 - List of keep-alive / close options with associated behaviours. - -PK="http-pretend-keepalive", HC="httpclose", SC="http-server-close", - -0 = option not set -1 = option is set -* = option doesn't matter - -Options can be split between frontend and backend, so some of them might have -a meaning only when combined by associating a frontend to a backend. Some forms -are not the normal ones and provide a behaviour compatible with another normal -form. Those are considered alternate forms and are marked "(alt)". - -SC HC PK Behaviour - 0 0 X tunnel mode - 0 1 0 passive close, only set headers then tunnel - 0 1 1 forced close with keep-alive announce (alt) - 1 0 0 server close - 1 0 1 server close with keep-alive announce - 1 1 0 forced close (alt) - 1 1 1 forced close with keep-alive announce (alt) - -At this point this results in 4 distinct effective modes for a request being -processed : - - tunnel mode : Connection header is left untouched and body is ignored - - passive close : Connection header is changed and body is ignored - - server close : Connection header set, body scanned, client-side keep-alive - is made possible regardless of server-side capabilities - - forced close : Connection header set, body scanned, connection closed. - -The "close" modes may be combined with a fake keep-alive announce to the server -in order to workaround buggy servers that disable chunked encoding and content -length announces when the client does not ask for keep-alive. - -Note: "http-pretend-keepalive" alone has no effect. However, if it is set in a - backend while a frontend is in "http-close" mode, then the combination of - both will result in a forced close with keep-alive announces for requests - passing through both. - diff --git a/doc/design-thoughts/backends-v0.txt b/doc/design-thoughts/backends-v0.txt deleted file mode 100644 index d350e229e..000000000 --- a/doc/design-thoughts/backends-v0.txt +++ /dev/null @@ -1,27 +0,0 @@ -1 type générique "entité", avec les attributs suivants : - - - frontend *f - - l7switch *s - - backend *b - -des types spécifiques sont simplement des entités avec certains -de ces champs remplis et pas forcément tous : - - listen = f [s] b - frontend = f [s] - l7switch = s - backend = [s] b - -Ensuite, les traitements sont évalués dans l'ordre : - - listen -> s'il a des règles de l7, on les évalue, et potentiellement on branche vers d'autres listen, l7 ou back, ou on travaille avec le back local. - - frontend -> s'il a des règles de l7, on les évalue, et potentiellement on branche vers d'autres listen, l7 ou back - - l7switch -> on évalue ses règles, potentiellement on branche vers d'autres listen, l7 ou backends - - backend -> s'il a des règles l7, on les évalue (quitte à changer encore de backend) puis on traite. - -Les requêtes sont traitées dans l'ordre des chaînages f->s*->b, et les réponses doivent être -traitées dans l'ordre inverse b->s*->f. Penser aux réécritures de champs Host à l'aller et -Location en retour. - -D'autre part, prévoir des "profils" plutôt que des blocs de nouveaux paramètres par défaut. -Ca permettra d'avoir plein de jeux de paramètres par défaut à utiliser dans chacun de ces -types. diff --git a/doc/design-thoughts/backends.txt b/doc/design-thoughts/backends.txt deleted file mode 100644 index d2474ce69..000000000 --- a/doc/design-thoughts/backends.txt +++ /dev/null @@ -1,125 +0,0 @@ -There has been a lot of confusion during the development because of the -backends and frontends. - -What we want : - -- being able to still use a listener as it has always been working - -- being able to write a rule stating that we will *change* the backend when we - match some pattern. Only one jump is allowed. - -- being able to write a "use_filters_from XXX" line stating that we will ignore - any filter in the current listener, and that those from XXX will be borrowed - instead. A warning would be welcome for options which will silently get - masked. This is used to factor configuration. - -- being able to write a "use_backend_from XXX" line stating that we will ignore - any server and timeout config in the current listener, and that those from - XXX will be borrowed instead. A warning would be welcome for options which - will silently get masked. This is used to factor configuration. - - - -Example : ---------- - - | # frontend HTTP/80 - | listen fe_http 1.1.1.1:80 - | use_filters_from default_http - | use_backend_from appli1 - | - | # frontend HTTPS/443 - | listen fe_https 1.1.1.1:443 - | use_filters_from default_https - | use_backend_from appli1 - | - | # frontend HTTP/8080 - | listen fe_http-dev 1.1.1.1:8080 - | reqadd "X-proto: http" - | reqisetbe "^Host: www1" appli1 - | reqisetbe "^Host: www2" appli2 - | reqisetbe "^Host: www3" appli-dev - | use_backend_from appli1 - | - | - | # filters default_http - | listen default_http - | reqadd "X-proto: http" - | reqisetbe "^Host: www1" appli1 - | reqisetbe "^Host: www2" appli2 - | - | # filters default_https - | listen default_https - | reqadd "X-proto: https" - | reqisetbe "^Host: www1" appli1 - | reqisetbe "^Host: www2" appli2 - | - | - | # backend appli1 - | listen appli1 - | reqidel "^X-appli1:.*" - | reqadd "Via: appli1" - | balance roundrobin - | cookie app1 - | server srv1 - | server srv2 - | - | # backend appli2 - | listen appli2 - | reqidel "^X-appli2:.*" - | reqadd "Via: appli2" - | balance roundrobin - | cookie app2 - | server srv1 - | server srv2 - | - | # backend appli-dev - | listen appli-dev - | reqadd "Via: appli-dev" - | use_backend_from appli2 - | - | - - -Now we clearly see multiple things : ------------------------------------- - - - a frontend can EITHER have filters OR reference a use_filter - - - a backend can EITHER have servers OR reference a use_backend - - - we want the evaluation to cross TWO levels per request. When a request is - being processed, it keeps track of its "frontend" side (where it came - from), and of its "backend" side (where the server-side parameters have - been found). - - - the use_{filters|backend} have nothing to do with how the request is - decomposed. - - -Conclusion : ------------- - - - a proxy is always its own frontend. It also has 2 parameters : - - "fi_prm" : pointer to the proxy holding the filters (itself by default) - - "be_prm" : pointer to the proxy holding the servers (itself by default) - - - a request has a frontend (fe) and a backend (be). By default, the backend - is initialized to the frontend. Everything related to the client side is - accessed through ->fe. Everything related to the server side is accessed - through ->be. - - - request filters are first called from ->fe then ->be. Since only the - filters can change ->be, it is possible to iterate the filters on ->be - only and stop when ->be does not change anymore. - - - response filters are first called from ->be then ->fe IF (fe != be). - - -When we parse the configuration, we immediately configure ->fi and ->be for -all proxies. - -Upon session creation, s->fe and s->be are initialized to the proxy. Filters -are executed via s->fe->fi_prm and s->be->fi_prm. Servers are found in -s->be->be_prm. - diff --git a/doc/design-thoughts/be-fe-changes.txt b/doc/design-thoughts/be-fe-changes.txt deleted file mode 100644 index f242f8a00..000000000 --- a/doc/design-thoughts/be-fe-changes.txt +++ /dev/null @@ -1,74 +0,0 @@ -- PR_O_TRANSP => FE !!! devra peut-être changer vu que c'est un complément du mode dispatch. -- PR_O_NULLNOLOG => FE -- PR_O_HTTP_CLOSE => FE. !!! mettre BE aussi !!! -- PR_O_TCP_CLI_KA => FE - -- PR_O_FWDFOR => BE. FE aussi ? -- PR_O_FORCE_CLO => BE -- PR_O_PERSIST => BE -- PR_O_COOK_RW, PR_O_COOK_INS, PR_O_COOK_PFX, PR_O_COOK_POST => BE -- PR_O_COOK_NOC, PR_O_COOK_IND => BE -- PR_O_ABRT_CLOSE => BE -- PR_O_REDISP => BE -- PR_O_BALANCE, PR_O_BALANCE_RR, PR_O_BALANCE_SH => BE -- PR_O_CHK_CACHE => BE -- PR_O_TCP_SRV_KA => BE -- PR_O_BIND_SRC => BE -- PR_O_TPXY_MASK => BE - - -- PR_MODE_TCP : BE côté serveur, FE côté client - -- nbconn -> fe->nbconn, be->nbconn. - Pb: rendre impossible le fait que (fe == be) avant de faire ça, - sinon on va compter les connexions en double. Ce ne sera possible - que lorsque les FE et BE seront des entités distinctes. On va donc - commencer par laisser uniquement fe->nbconn (vu que le fe ne change - pas), et modifier ceci plus tard, ne serait-ce que pour prendre en - compte correctement les minconn/maxconn. - => solution : avoir beconn et feconn dans chaque proxy. - -- failed_conns, failed_secu (réponses bloquées), failed_resp... : be - Attention: voir les cas de ERR_SRVCL, il semble que parfois on - indique ça alors qu'il y a un write error côté client (ex: ligne - 2044 dans proto_http). - - => be et pas be->beprm - -- logs du backup : ->be (idem) - -- queue : be - -- logs/debug : srv toujours associé à be (ex: proxy->id:srv->id). Rien - pour le client pour le moment. D'une manière générale, les erreurs - provoquées côté serveur vont sur BE et celles côté client vont sur - FE. -- logswait & LW_BYTES : FE (puisqu'on veut savoir si on logue tout de suite) - -- messages d'erreurs personnalisés (errmsg, ...) -> fe - -- monitor_uri -> fe -- uri_auth -> (fe->firpm puis be->fiprm). Utilisation de ->be - -- req_add, req_exp => fe->fiprm, puis be->fiprm -- req_cap, rsp_cap -> fe->fiprm -- rsp_add, rsp_exp => be->fiprm, devrait être fait ensuite aussi sur fe->fiprm -- capture_name, capture_namelen : fe->fiprm - - Ce n'est pas la solution idéale, mais au moins la capture et configurable - par les filtres du FE et ne bouge pas lorsque le BE est réassigné. Cela - résoud aussi un pb d'allocation mémoire. - - -- persistance (appsessions, cookiename, ...) -> be -- stats:scope "." = fe (celui par lequel on arrive) - !!!ERREUR!!! => utiliser be pour avoir celui qui a été validé par - l'uri_auth. - - ---------- corrections à effectuer --------- - -- remplacement de headers : parser le header et éventuellement le supprimer puis le(les) rajouter. -- session->proto.{l4state,l7state,l7substate} pour CLI et SRV -- errorloc : si définie dans backend, la prendre, sinon dans front. -- logs : faire be sinon fe. diff --git a/doc/design-thoughts/config-language.txt b/doc/design-thoughts/config-language.txt deleted file mode 100644 index 20c4fbd2b..000000000 --- a/doc/design-thoughts/config-language.txt +++ /dev/null @@ -1,262 +0,0 @@ -Prévoir des commandes en plusieurs mots clés. -Par exemple : - - timeout connection XXX - connection scale XXX - -On doit aussi accepter les préfixes : - - tim co XXX - co sca XXX - -Prévoir de ranger les combinaisons dans un tableau. On doit même -pouvoir effectuer un mapping simplifiant le parseur. - - -Pour les filtres : - - - [ * ] - - = [ req | rsp ] - = [ in | out ] - = [ line | LINE | METH | URI | h(hdr) | H(hdr) | c(cookie) | C(cookie) ] - = [ == | =~ | =* | =^ | =/ | != | !~ | !* | !^ | !/ ] - = "" - = [ allow | permit | deny | delete | replace | switch | add | set | redir ] - = optional action args - - examples: - - req in URI =^ "/images" switch images - req in h(host) =* ".mydomain.com" switch mydomain - req in h(host) =~ "localhost(.*)" replace "www\1" - - alternative : - - [not] [ [ * ]] - - req in switch URI =^ "/images" images - req in switch h(host) =* ".mydomain.com" mydomain - req in replace h(host) =~ "localhost(.*)" "www\1" - req in delete h(Connection) - req in deny not line =~ "((GET|HEAD|POST|OPTIONS) /)|(OPTIONS *)" - req out set h(Connection) "close" - req out add line "Server: truc" - - - [not] [ [ * ]] ';' - - req in switch URI =^ "/images/" images ; replace "/" - req in switch h(host) =* ".mydomain.com" mydomain - req in replace h(host) =~ "localhost(.*)" "www\1" - req in delete h(Connection) - req in deny not line =~ "((GET|HEAD|POST|OPTIONS) /)|(OPTIONS *)" - req out set h(Connection) "close" - req out add line == "Server: truc" - - -Extension avec des ACL : - - req in acl(meth_valid) METH =~ "(GET|POST|HEAD|OPTIONS)" - req in acl(meth_options) METH == "OPTIONS" - req in acl(uri_slash) URI =^ "/" - req in acl(uri_star) URI == "*" - - req in deny acl !(meth_options && uri_star || meth_valid && uri_slash) - -Peut-être plus simplement : - - acl meth_valid METH =~ "(GET|POST|HEAD|OPTIONS)" - acl meth_options METH == "OPTIONS" - acl uri_slash URI =^ "/" - acl uri_star URI == "*" - - req in deny not acl(meth_options uri_star, meth_valid uri_slash) - - req in switch URI =^ "/images/" images ; replace "/" - req in switch h(host) =* ".mydomain.com" mydomain - req in replace h(host) =~ "localhost(.*)" "www\1" - req in delete h(Connection) - req in deny not line =~ "((GET|HEAD|POST|OPTIONS) /)|(OPTIONS *)" - req out set h(Connection) "close" - req out add line == "Server: truc" - -Prévoir le cas du "if" pour exécuter plusieurs actions : - - req in if URI =^ "/images/" then replace "/" ; switch images - -Utiliser les noms en majuscules/minuscules pour indiquer si on veut prendre -en compte la casse ou non : - - if uri =^ "/watch/" setbe watch rebase "/watch/" "/" - if uri =* ".jpg" setbe images - if uri =~ ".*dll.*" deny - if HOST =* ".mydomain.com" setbe mydomain - etc... - -Another solution would be to have a dedicated keyword to URI remapping. It -would both rewrite the URI and optionally switch to another backend. - - uriremap "/watch/" "/" watch - uriremap "/chat/" "/" chat - uriremap "/event/" "/event/" event - -Or better : - - uriremap "/watch/" watch "/" - uriremap "/chat/" chat "/" - uriremap "/event/" event - -For the URI, using a regex is sometimes useful (eg: providing a set of possible prefixes. - - -Sinon, peut-être que le "switch" peut prendre un paramètre de mapping pour la partie matchée : - - req in switch URI =^ "/images/" images:"/" - - -2007/03/31 - Besoins plus précis. - -1) aucune extension de branchement ou autre dans les "listen", c'est trop complexe. - -Distinguer les données entrantes (in) et sortantes (out). - -Le frontend ne voit que les requetes entrantes et les réponses sortantes. -Le backend voir les requêtes in/out et les réponses in/out. -Le frontend permet les branchements d'ensembles de filtres de requêtes vers -d'autres. Le frontend et les ensembles de filtres de requêtes peuvent brancher -vers un backend. - ------------+--------+----------+----------+---------+----------+ - \ Where | | | | | | - \______ | Listen | Frontend | ReqRules | Backend | RspRules | - \| | | | | | -Capability | | | | | | ------------+--------+----------+----------+---------+----------+ -Frontend | X | X | | | | ------------+--------+----------+----------+---------+----------+ -FiltReqIn | X | X | X | X | | ------------+--------+----------+----------+---------+----------+ -JumpFiltReq| X | X | X | | | \ ------------+--------+----------+----------+---------+----------+ > = ReqJump -SetBackend | X | X | X | | | / ------------+--------+----------+----------+---------+----------+ -FiltReqOut | | | | X | | ------------+--------+----------+----------+---------+----------+ -FiltRspIn | X | | | X | X | ------------+--------+----------+----------+---------+----------+ -JumpFiltRsp| | | | X | X | ------------+--------+----------+----------+---------+----------+ -FiltRspOut | | X | | X | X | ------------+--------+----------+----------+---------+----------+ -Backend | X | | | X | | ------------+--------+----------+----------+---------+----------+ - -En conclusion -------------- - -Il y a au moins besoin de distinguer 8 fonctionnalités de base : - - capacité à recevoir des connexions (frontend) - - capacité à filtrer les requêtes entrantes - - capacité à brancher vers un backend ou un ensemble de règles de requêtes - - capacité à filtrer les requêtes sortantes - - capacité à filtrer les réponses entrantes - - capacité à brancher vers un autre ensemble de règles de réponses - - capacité à filtrer la réponse sortante - - capacité à gérer des serveurs (backend) - -Remarque --------- - - on a souvent besoin de pouvoir appliquer un petit traitement sur un ensemble - host/uri/autre. Le petit traitement peut consister en quelques filtres ainsi - qu'une réécriture du couple (host,uri). - - -Proposition : ACL - -Syntaxe : ---------- - - acl ... - -Ceci créera une acl référencée sous le nom qui sera validée si -l'application d'au moins une des valeurs avec l'opérateur -sur le sujet est validée. - -Opérateurs : ------------- - -Toujours 2 caractères : - - [=!][~=*^%/.] - -Premier caractère : - '=' : OK si test valide - '!' : OK si test échoué. - -Second caractère : - '~' : compare avec une regex - '=' : compare chaîne à chaîne - '*' : compare la fin de la chaîne (ex: =* ".mydomain.com") - '^' : compare le début de la chaîne (ex: =^ "/images/") - '%' : recherche une sous-chaîne - '/' : compare avec un mot entier en acceptant le '/' comme délimiteur - '.' : compare avec un mot entier en acceptant el '.' comme délimiteur - -Ensuite on exécute une action de manière conditionnelle si l'ensemble des ACLs -mentionnées sont validées (ou invalidées pour celles précédées d'un "!") : - - on [!] ... - - -Exemple : ---------- - - acl www_pub host =. www www01 dev preprod - acl imghost host =. images - acl imgdir uri =/ img - acl imagedir uri =/ images - acl msie h(user-agent) =% "MSIE" - - set_host "images" on www_pub imgdir - remap_uri "/img" "/" on www_pub imgdir - remap_uri "/images" "/" on www_pub imagedir - setbe images on imghost - reqdel "Cookie" on all - - - -Actions possibles : - - req {in|out} {append|delete|rem|add|set|rep|mapuri|rewrite|reqline|deny|allow|setbe|tarpit} - resp {in|out} {append|delete|rem|add|set|rep|maploc|rewrite|stsline|deny|allow} - - req in append - req in delete - req in rem
- req in add
- req in set
- req in rep
- req in mapuri - req in rewrite - req in reqline - req in deny - req in allow - req in tarpit - req in setbe - - resp out maploc - resp out stsline - -Les chaînes doivent être délimitées par un même caractère au début et à la fin, -qui doit être échappé s'il est présent dans la chaîne. Tout ce qui se trouve -entre le caractère de fin et les premiers espace est considéré comme des -options passées au traitement. Par exemple : - - req in rep host /www/i /www/ - req in rep connection /keep-alive/i "close" - -Il serait pratique de pouvoir effectuer un remap en même temps qu'un setbe. - -Captures: les séparer en in/out. Les rendre conditionnelles ? diff --git a/doc/design-thoughts/connection-sharing.txt b/doc/design-thoughts/connection-sharing.txt deleted file mode 100644 index 99be1cdd6..000000000 --- a/doc/design-thoughts/connection-sharing.txt +++ /dev/null @@ -1,31 +0,0 @@ -2014/10/28 - Server connection sharing - -For HTTP/2 we'll have to use multiplexed connections to the servers and to -share them between multiple streams. We'll also have to do this for H/1, but -with some variations since H1 doesn't offer connection status verification. - -In order to validate that an idle connection is still usable, it is desirable -to periodically send health checks over it. Normally, idle connections are -meant to be heavily used, so there is no reason for having them idle for a long -time. Thus we have two possibilities : - - - either we time them out after some inactivity, this saves server resources ; - - or we check them after some inactivity. For this we can send the server- - side HTTP health check (only when the server uses HTTP checks), and avoid - using that to mark the server down, and instead consider the connection as - dead. - -For HTTP/2 we'll have to send pings periodically over these connections, so -it's worth considering a per-connection task to validate that the channel still -works. - -In the current model, a connection necessarily belongs to a session, so it's -not really possible to share them, at best they can be exchanged, but that -doesn't make much sense as it means that it could disturb parallel traffic. - -Thus we need to have a per-server list of idle connections and a max-idle-conn -setting to kill them when there are too many. In the case of H/1 it is also -advisable to consider that if a connection was created to pass a first non- -idempotent request while other idle connections were still existing, then a -connection will have to be killed in order not to exceed the limit. - diff --git a/doc/design-thoughts/dynamic-buffers.txt b/doc/design-thoughts/dynamic-buffers.txt deleted file mode 100644 index 564d868ca..000000000 --- a/doc/design-thoughts/dynamic-buffers.txt +++ /dev/null @@ -1,41 +0,0 @@ -2014/10/30 - dynamic buffer management - -Since HTTP/2 processing will significantly increase the need for buffering, it -becomes mandatory to be able to support dynamic buffer allocation. This also -means that at any moment some buffer allocation will fail and that a task or an -I/O operation will have to be paused for the time needed to allocate a buffer. - -There are 3 places where buffers are needed : - - - receive side of a stream interface. A connection notifies about a pending - recv() and the SI calls the receive function to put the data into a buffer. - Here the buffer will have to be picked from a pool first, and if the - allocation fails, the I/O will have to temporarily be disabled, the - connection will have to subscribe for buffer release notification to be - woken up once a buffer is available again. It's important to keep in mind - that buffer availability doesn't necessarily mean a desire to enable recv - again, just that recv is not paused anymore for resource reasons. - - - receive side of a stream interface when the other end point is an applet. - The applet wants to write into the buffer and for this the buffer needs to - be allocated as well. It is the same as above except that it is the applet - which is put to a pause. Since the applet might be at the core of the task - itself, it could become tricky to handle the situation correctly. Stats and - peers are in this situation. - - - Tx of a task : some tasks perform spontaneous writes to a buffer. Checks - are an example of this. The checks will have to be able to sleep while a - buffer is being awaited. - -One important point is that such pauses must not prevent the task from timing -out. There it becomes difficult because in the case of a time out, we could -want to emit a timeout error message and for this, require a buffer. So it is -important to keep the ability not to send messages upon error processing, and -to be able to give up and stop waiting for buffers. - -The refill mechanism needs to be designed in a thread-safe way because this -will become one of the rare cases of inter-task activity. Thus it is important -to ensure that checking the state of the task and passing of the freshly -released buffer are performed atomically, and that in case the task doesn't -want it anymore, it is responsible for passing it to the next one. - diff --git a/doc/design-thoughts/entities-v2.txt b/doc/design-thoughts/entities-v2.txt deleted file mode 100644 index 91c4fa97e..000000000 --- a/doc/design-thoughts/entities-v2.txt +++ /dev/null @@ -1,276 +0,0 @@ -2012/07/05 - Connection layering and sequencing - - -An FD has a state : - - CLOSED - - READY - - ERROR (?) - - LISTEN (?) - -A connection has a state : - - CLOSED - - ACCEPTED - - CONNECTING - - ESTABLISHED - - ERROR - -A stream interface has a state : - - INI, REQ, QUE, TAR, ASS, CON, CER, EST, DIS, CLO - -Note that CON and CER might be replaced by EST if the connection state is used -instead. CON might even be more suited than EST to indicate that a connection -is known. - - -si_shutw() must do : - - data_shutw() - if (shutr) { - data_close() - ctrl_shutw() - ctrl_close() - } - -si_shutr() must do : - data_shutr() - if (shutw) { - data_close() - ctrl_shutr() - ctrl_close() - } - -Each of these steps may fail, in which case the step must be retained and the -operations postponed in an asynchronous task. - -The first asynchronous data_shut() might already fail so it is mandatory to -save the other side's status with the connection in order to let the async task -know whether the 3 next steps must be performed. - -The connection (or perhaps the FD) needs to know : - - the desired close operations : DSHR, DSHW, CSHR, CSHW - - the completed close operations : DSHR, DSHW, CSHR, CSHW - - -On the accept() side, we probably need to know : - - if a header is expected (eg: accept-proxy) - - if this header is still being waited for - => maybe both info might be combined into one bit - - - if a data-layer accept() is expected - - if a data-layer accept() has been started - - if a data-layer accept() has been performed - => possibly 2 bits, to indicate the need to free() - -On the connect() side, we need to know : - - the desire to send a header (eg: send-proxy) - - if this header has been sent - => maybe both info might be combined - - - if a data-layer connect() is expected - - if a data-layer connect() has been started - - if a data-layer connect() has been completed - => possibly 2 bits, to indicate the need to free() - -On the response side, we also need to know : - - the desire to send a header (eg: health check response for monitor-net) - - if this header was sent - => might be the same as sending a header over a new connection - -Note: monitor-net has precedence over proxy proto and data layers. Same for - health mode. - -For multi-step operations, use 2 bits : - 00 = operation not desired, not performed - 10 = operation desired, not started - 11 = operation desired, started but not completed - 01 = operation desired, started and completed - - => X != 00 ==> operation desired - X & 01 ==> operation at least started - X & 10 ==> operation not completed - -Note: no way to store status information for error reporting. - -Note2: it would be nice if "tcp-request connection" rules could work at the -connection level, just after headers ! This means support for tracking stick -tables, possibly not too much complicated. - - -Proposal for incoming connection sequence : - -- accept() -- if monitor-net matches or if mode health => try to send response -- if accept-proxy, wait for proxy request -- if tcp-request connection, process tcp rules and possibly keep the - pointer to stick-table -- if SSL is enabled, switch to SSL handshake -- then switch to DATA state and instantiate a session - -We just need a map of handshake handlers on the connection. They all manage the -FD status themselves and set the callbacks themselves. If their work succeeds, -they remove themselves from the list. If it fails, they remain subscribed and -enable the required polling until they are woken up again or the timeout strikes. - -Identified handshake handlers for incoming connections : - - HH_HEALTH (tries to send OK and dies) - - HH_MONITOR_IN (matches src IP and adds/removes HH_SEND_OK/HH_SEND_HTTP_OK) - - HH_SEND_OK (tries to send "OK" and dies) - - HH_SEND_HTTP_OK (tries to send "HTTP/1.0 200 OK" and dies) - - HH_ACCEPT_PROXY (waits for PROXY line and parses it) - - HH_TCP_RULES (processes TCP rules) - - HH_SSL_HS (starts SSL handshake) - - HH_ACCEPT_SESSION (instantiates a session) - -Identified handshake handlers for outgoing connections : - - HH_SEND_PROXY (tries to build and send the PROXY line) - - HH_SSL_HS (starts SSL handshake) - -For the pollers, we could check that handshake handlers are not 0 and decide to -call a generic connection handshake handler instead of usual callbacks. Problem -is that pollers don't know connections, they know fds. So entities which manage -handlers should update change the FD callbacks accordingly. - -With a bit of care, we could have : - - HH_SEND_LAST_CHUNK (sends the chunk pointed to by a pointer and dies) - => merges HEALTH, SEND_OK and SEND_HTTP_OK - -It sounds like the ctrl vs data state for the connection are per-direction -(eg: support an async ctrl shutw while still reading data). - -Also support shutr/shutw status at L4/L7. - -In practice, what we really need is : - -shutdown(conn) = - conn.data.shut() - conn.ctrl.shut() - conn.fd.shut() - -close(conn) = - conn.data.close() - conn.ctrl.close() - conn.fd.close() - -With SSL over Remote TCP (RTCP + RSSL) to reach the server, we would have : - - HTTP -> RTCP+RSSL connection <-> RTCP+RRAW connection -> TCP+SSL connection - -The connection has to be closed at 3 places after a successful response : - - DATA (RSSL over RTCP) - - CTRL (RTCP to close connection to server) - - SOCK (FD to close connection to second process) - -Externally, the connection is seen with very few flags : - - SHR - - SHW - - ERR - -We don't need a CLOSED flag as a connection must always be detached when it's closed. - -The internal status doesn't need to be exposed : - - FD allocated (Y/N) - - CTRL initialized (Y/N) - - CTRL connected (Y/N) - - CTRL handlers done (Y/N) - - CTRL failed (Y/N) - - CTRL shutr (Y/N) - - CTRL shutw (Y/N) - - DATA initialized (Y/N) - - DATA connected (Y/N) - - DATA handlers done (Y/N) - - DATA failed (Y/N) - - DATA shutr (Y/N) - - DATA shutw (Y/N) - -(note that having flags for operations needing to be completed might be easier) --------------- - -Maybe we need to be able to call conn->fdset() and conn->fdclr() but it sounds -very unlikely since the only functions manipulating this are in the code of -the data/ctrl handlers. - -FDSET/FDCLR cannot be directly controlled by the stream interface since it also -depends on the DATA layer (WANT_READ/WANT_WRITE). - -But FDSET/FDCLR is probably controlled by who owns the connection (eg: DATA). - -Example: an SSL conn relies on an FD. The buffer is full, and wants the conn to -stop reading. It must not stop the FD itself. It is the read function which -should notice that it has nothing to do with a read wake-up, which needs to -disable reading. - -Conversely, when calling conn->chk_rcv(), the reader might get a WANT_READ or -even WANT_WRITE and adjust the FDs accordingly. - ------------------------- - -OK, the problem is simple : we don't manipulate the FD at the right level. -We should have : - ->connect(), ->chk_snd(), ->chk_rcv(), ->shutw(), ->shutr() which are - called from the upper layer (buffer) - ->recv(), ->send(), called from the lower layer - -Note that the SHR is *reported* by lower layer but can be forced by upper -layer. In this case it's like a delayed abort. The difficulty consists in -knowing the output data were correctly read. Probably we'd need to drain -incoming data past the active shutr(). - -The only four purposes of the top-down shutr() call are : - - acknowledge a shut read report : could probably be done better - - read timeout => disable reading : it's a delayed abort. We want to - report that the buffer is SHR, maybe even the connection, but the - FD clearly isn't. - - read abort due to error on the other side or desire to close (eg: - http-server-close) : delayed abort - - complete abort - -The active shutr() is problematic as we can't disable reading if we expect some -exchanges for data acknowledgement. We probably need to drain data only until -the shutw() has been performed and ACKed. - -A connection shut down for read would behave like this : - - 1) bidir exchanges - - 2) shutr() => read_abort_pending=1 - - 3) drain input, still send output - - 4) shutw() - - 5) drain input, wait for read0 or ack(shutw) - - 6) close() - ---------------------- 2012/07/05 ------------------- - -Communications must be performed this way : - - connection <-> channel <-> connection - -A channel is composed of flags and stats, and may store data in either a buffer -or a pipe. We need low-layer operations between sockets and buffers or pipes. -Right now we only support sockets, but later we might support remote sockets -and maybe pipes or shared memory segments. - -So we need : - - - raw_sock_to_buf() => receive raw data from socket into buffer - - raw_sock_to_pipe => receive raw data from socket into pipe (splice in) - - raw_sock_from_buf() => send raw data from buffer to socket - - raw_sock_from_pipe => send raw data from pipe to socket (splice out) - - - ssl_sock_to_buf() => receive ssl data from socket into buffer - - ssl_sock_to_pipe => receive ssl data from socket into a pipe (NULL) - - ssl_sock_from_buf() => send ssl data from buffer to socket - - ssl_sock_from_pipe => send ssl data from pipe to socket (NULL) - -These functions should set such status flags : - -#define ERR_IN 0x01 -#define ERR_OUT 0x02 -#define SHUT_IN 0x04 -#define SHUT_OUT 0x08 -#define EMPTY_IN 0x10 -#define FULL_OUT 0x20 - diff --git a/doc/design-thoughts/how-it-works.txt b/doc/design-thoughts/how-it-works.txt deleted file mode 100644 index 2d1cb89a0..000000000 --- a/doc/design-thoughts/how-it-works.txt +++ /dev/null @@ -1,60 +0,0 @@ -How it works ? (unfinished and inexact) - -For TCP and HTTP : - -- listeners create listening sockets with a READ callback pointing to the - protocol-specific accept() function. - -- the protocol-specific accept() function then accept()'s the connection and - instantiates a "server TCP socket" (which is dedicated to the client side), - and configures it (non_block, get_original_dst, ...). - -For TCP : -- in case of pure TCP, a request buffer is created, as well as a "client TCP - socket", which tries to connect to the server. - -- once the connection is established, the response buffer is allocated and - connected to both ends. - -- both sockets are set to "autonomous mode" so that they only wake up their - supervising session when they encounter a special condition (error or close). - - -For HTTP : -- in case of HTTP, a request buffer is created with the "HOLD" flag set and - a read limit to support header rewriting (may be this one will be removed - eventually because it's better to limit only to the buffer size and report - an error when rewritten data overflows) - -- a "flow analyzer" is attached to the buffer (or possibly multiple flow - analyzers). For the request, the flow analyzer is "http_lb_req". The flow - analyzer is a function which gets called when new data is present and - blocked. It has a timeout (request timeout). It can also be bypassed on - demand. - -- when the "http_lb_req" has received the whole request, it creates a client - socket with all the parameters needed to try to connect to the server. When - the connection establishes, the response buffer is allocated on the fly, - put to HOLD mode, and a an "http_lb_resp" flow analyzer is attached to the - buffer. - - -For client-side HTTPS : - -- the accept() function must completely instantiate a TCP socket + an SSL - reader. It is when the SSL session is complete that we call the - protocol-specific accept(), and create its buffer. - - - - -Conclusions ------------ - -- we need a generic TCP accept() function with a lot of flags set by the - listener, to tell it what info we need to get at the accept() time, and - what flags will have to be set on the socket. - -- once the TCP accept() function ends, it wakes up the protocol supervisor - which is in charge of creating the buffers, etc, switch states, etc... - diff --git a/doc/design-thoughts/http2.txt b/doc/design-thoughts/http2.txt deleted file mode 100644 index c21ac108a..000000000 --- a/doc/design-thoughts/http2.txt +++ /dev/null @@ -1,277 +0,0 @@ -2014/10/23 - design thoughts for HTTP/2 - -- connections : HTTP/2 depends a lot more on a connection than HTTP/1 because a - connection holds a compression context (headers table, etc...). We probably - need to have an h2_conn struct. - -- multiple transactions will be handled in parallel for a given h2_conn. They - are called streams in HTTP/2 terminology. - -- multiplexing : for a given client-side h2 connection, we can have multiple - server-side h2 connections. And for a server-side h2 connection, we can have - multiple client-side h2 connections. Streams circulate in N-to-N fashion. - -- flow control : flow control will be applied between multiple streams. Special - care must be taken so that an H2 client cannot block some H2 servers by - sending requests spread over multiple servers to the point where one server - response is blocked and prevents other responses from the same server from - reaching their clients. H2 connection buffers must always be empty or nearly - empty. The per-stream flow control needs to be respected as well as the - connection's buffers. It is important to implement some fairness between all - the streams so that it's not always the same which gets the bandwidth when - the connection is congested. - -- some clients can be H1 with an H2 server (is this really needed ?). Most of - the initial use case will be H2 clients to H1 servers. It is important to keep - in mind that H1 servers do not do flow control and that we don't want them to - block transfers (eg: post upload). - -- internal tasks : some H2 clients will be internal tasks (eg: health checks). - Some H2 servers will be internal tasks (eg: stats, cache). The model must be - compatible with this use case. - -- header indexing : headers are transported compressed, with a reference to a - static or a dynamic header, or a literal, possibly huffman-encoded. Indexing - is specific to the H2 connection. This means there is no way any binary data - can flow between both sides, headers will have to be decoded according to the - incoming connection's context and re-encoded according to the outgoing - connection's context, which can significantly differ. In order to avoid the - parsing trouble we currently face, headers will have to be clearly split - between name and value. It is worth noting that neither the incoming nor the - outgoing connections' contexts will be of any use while processing the - headers. At best we can have some shortcuts for well-known names that map - well to the static ones (eg: use the first static entry with same name), and - maybe have a few special cases for static name+value as well. Probably we can - classify headers in such categories : - - - static name + value - - static name + other value - - dynamic name + other value - - This will allow for better processing in some specific cases. Headers - supporting a single value (:method, :status, :path, ...) should probably - be stored in a single location with a direct access. That would allow us - to retrieve a method using hdr[METHOD]. All such indexing must be performed - while parsing. That also means that HTTP/1 will have to be converted to this - representation very early in the parser and possibly converted back to H/1 - after processing. - - Header names/values will have to be placed in a small memory area that will - inevitably get fragmented as headers are rewritten. An automatic packing - mechanism must be implemented so that when there's no more room, headers are - simply defragmented/packet to a new table and the old one is released. Just - like for the static chunks, we need to have a few such tables pre-allocated - and ready to be swapped at any moment. Repacking must not change any index - nor affect the way headers are compressed so that it can happen late after a - retry (send-name-header for example). - -- header processing : can still happen on a (header, value) basis. Reqrep/ - rsprep completely disappear and will have to be replaced with something else - to support renaming headers and rewriting url/path/... - -- push_promise : servers can push dummy requests+responses. They advertise - the stream ID in the push_promise frame indicating the associated stream ID. - This means that it is possible to initiate a client-server stream from the - information coming from the server and make the data flow as if the client - had made it. It's likely that we'll have to support two types of server - connections: those which support push and those which do not. That way client - streams will be distributed to existing server connections based on their - capabilities. It's important to keep in mind that PUSH will not be rewritten - in responses. - -- stream ID mapping : since the stream ID is per H2 connection, stream IDs will - have to be mapped. Thus a given stream is an entity with two IDs (one per - side). Or more precisely a stream has two end points, each one carrying an ID - when it ends on an HTTP2 connection. Also, for each stream ID we need to - quickly find the associated transaction in progress. Using a small quick - unique tree seems indicated considering the wide range of valid values. - -- frame sizes : frame have to be remapped between both sides as multiplexed - connections won't always have the same characteristics. Thus some frames - might be spliced and others will be sliced. - -- error processing : care must be taken to never break a connection unless it - is dead or corrupt at the protocol level. Stats counter must exist to observe - the causes. Timeouts are a great problem because silent connections might - die out of inactivity. Ping frames should probably be scheduled a few seconds - before the connection timeout so that an unused connection is verified before - being killed. Abnormal requests must be dealt with using RST_STREAM. - -- ALPN : ALPN must be observed on the client side, and transmitted to the server - side. - -- proxy protocol : proxy protocol makes little to no sense in a multiplexed - protocol. A per-stream equivalent will surely be needed if implementations - do not quickly generalize the use of Forward. - -- simplified protocol for local devices (eg: haproxy->varnish in clear and - without handshake, and possibly even with splicing if the connection's - settings are shared) - -- logging : logging must report a number of extra information such as the - stream ID, and whether the transaction was initiated by the client or by the - server (which can be deduced from the stream ID's parity). In case of push, - the number of the associated stream must also be reported. - -- memory usage : H2 increases memory usage by mandating use of 16384 bytes - frame size minimum. That means slightly more than 16kB of buffer in each - direction to process any frame. It will definitely have an impact on the - deployed maxconn setting in places using less than this (4..8kB are common). - Also, the header list is persistent per connection, so if we reach the same - size as the request, that's another 16kB in each direction, resulting in - about 48kB of memory where 8 were previously used. A more careful encoder - can work with a much smaller set even if that implies evicting entries - between multiple headers of the same message. - -- HTTP/1.0 should very carefully be transported over H2. Since there's no way - to pass version information in the protocol, the server could use some - features of HTTP/1.1 that are unsafe in HTTP/1.0 (compression, trailers, - ...). - -- host / :authority : ":authority" is the norm, and "host" will be absent when - H2 clients generate :authority. This probably means that a dummy Host header - will have to be produced internally from :authority and removed when passing - to H2 behind. This can cause some trouble when passing H2 requests to H1 - proxies, because there's no way to know if the request should contain scheme - and authority in H1 or not based on the H2 request. Thus a "proxy" option - will have to be explicitly mentioned on HTTP/1 server lines. One of the - problem that it creates is that it's not longer possible to pass H/1 requests - to H/1 proxies without an explicit configuration. Maybe a table of the - various combinations is needed. - - :scheme :authority host - HTTP/2 request present present absent - HTTP/1 server req absent absent present - HTTP/1 proxy req present present present - - So in the end the issue is only with H/2 requests passed to H/1 proxies. - -- ping frames : they don't indicate any stream ID so by definition they cannot - be forwarded to any server. The H2 connection should deal with them only. - -There's a layering problem with H2. The framing layer has to be aware of the -upper layer semantics. We can't simply re-encode HTTP/1 to HTTP/2 then pass -it over a framing layer to mux the streams, the frame type must be passed below -so that frames are properly arranged. Header encoding is connection-based and -all streams using the same connection will interact in the way their headers -are encoded. Thus the encoder *has* to be placed in the h2_conn entity, and -this entity has to know for each stream what its headers are. - -Probably that we should remove *all* headers from transported data and move -them on the fly to a parallel structure that can be shared between H1 and H2 -and consumed at the appropriate level. That means buffers only transport data. -Trailers have to be dealt with differently. - -So if we consider an H1 request being forwarded between a client and a server, -it would look approximately like this : - - - request header + body land into a stream's receive buffer - - headers are indexed and stripped out so that only the body and whatever - follows remain in the buffer - - both the header index and the buffer with the body stay attached to the - stream - - the sender can rebuild the whole headers. Since they're found in a table - supposed to be stable, it can rebuild them as many times as desired and - will always get the same result, so it's safe to build them into the trash - buffer for immediate sending, just as we do for the PROXY protocol. - - the upper protocol should probably provide a build_hdr() callback which - when called by the socket layer, builds this header block based on the - current stream's header list, ready to be sent. - - the socket layer has to know how many bytes from the headers are left to be - forwarded prior to processing the body. - - the socket layer needs to consume only the acceptable part of the body and - must not release the buffer if any data remains in it (eg: pipelining over - H1). This is already handled by channel->o and channel->to_forward. - - we could possibly have another optional callback to send a preamble before - data, that could be used to send chunk sizes in H1. The danger is that it - absolutely needs to be stable if it has to be retried. But it could - considerably simplify de-chunking. - -When the request is sent to an H2 server, an H2 stream request must be made -to the server, we find an existing connection whose settings are compatible -with our needs (eg: tls/clear, push/no-push), and with a spare stream ID. If -none is found, a new connection must be established, unless maxconn is reached. - -Servers must have a maxstream setting just like they have a maxconn. The same -queue may be used for that. - -The "tcp-request content" ruleset must apply to the TCP layer. But with HTTP/2 -that becomes impossible (and useless). We still need something like the -"tcp-request session" hook to apply just after the SSL handshake is done. - -It is impossible to defragment the body on the fly in HTTP/2. Since multiple -messages are interleaved, we cannot wait for all of them and block the head of -line. Thus if body analysis is required, it will have to use the stream's -buffer, which necessarily implies a copy. That means that with each H2 end we -necessarily have at least one copy. Sometimes we might be able to "splice" some -bytes from one side to the other without copying into the stream buffer (same -rules as for TCP splicing). - -In theory, only data should flow through the channel buffer, so each side's -connector is responsible for encoding data (H1: linear/chunks, H2: frames). -Maybe the same mechanism could be extrapolated to tunnels / TCP. - -Since we'd use buffers only for data (and for receipt of headers), we need to -have dynamic buffer allocation. - -Thus : -- Tx buffers do not exist. We allocate a buffer on the fly when we're ready to - send something that we need to build and that needs to be persistent in case - of partial send. H1 headers are built on the fly from the header table to a - temporary buffer that is immediately sent and whose amount of sent bytes is - the only information kept (like for PROXY protocol). H2 headers are more - complex since the encoding depends on what was successfully sent. Thus we - need to build them and put them into a temporary buffer that remains - persistent in case send() fails. It is possible to have a limited pool of - Tx buffers and refrain from sending if there is no more buffer available in - the pool. In that case we need a wake-up mechanism once a buffer is - available. Once the data are sent, the Tx buffer is then immediately recycled - in its pool. Note that no tx buffer being used (eg: for hdr or control) means - that we have to be able to serialize access to the connection and retry with - the same stream. It also means that a stream that times out while waiting for - the connector to read the second half of its request has to stay there, or at - least needs to be handled gracefully. However if the connector cannot read - the data to be sent, it means that the buffer is congested and the connection - is dead, so that probably means it can be killed. - -- Rx buffers have to be pre-allocated just before calling recv(). A connection - will first try to pick a buffer and disable reception if it fails, then - subscribe to the list of tasks waiting for an Rx buffer. - -- full Rx buffers might sometimes be moved around to the next buffer instead of - experiencing a copy. That means that channels and connectors must use the - same format of buffer, and that only the channel will have to see its - pointers adjusted. - -- Tx of data should be made as much as possible without copying. That possibly - means by directly looking into the connection buffer on the other side if - the local Tx buffer does not exist and the stream buffer is not allocated, or - even performing a splice() call between the two sides. One of the problem in - doing this is that it requires proper ordering of the operations (eg: when - multiple readers are attached to a same buffer). If the splitting occurs upon - receipt, there's no problem. If we expect to retrieve data directly from the - original buffer, it's harder since it contains various things in an order - which does not even indicate what belongs to whom. Thus possibly the only - mechanism to implement is the buffer permutation which guarantees zero-copy - and only in the 100% safe case. Also it's atomic and does not cause HOL - blocking. - -It makes sense to chose the frontend_accept() function right after the -handshake ended. It is then possible to check the ALPN, the SNI, the ciphers -and to accept to switch to the h2_conn_accept handler only if everything is OK. -The h2_conn_accept handler will have to deal with the connection setup, -initialization of the header table, exchange of the settings frames and -preparing whatever is needed to fire new streams upon receipt of unknown -stream IDs. Note: most of the time it will not be possible to splice() because -we need to know in advance the amount of bytes to write the header, and here it -will not be possible. - -H2 health checks must be seen as regular transactions/streams. The check runs a -normal client which seeks an available stream from a server. The server then -finds one on an existing connection or initiates a new H2 connection. The H2 -checks will have to be configurable for sharing streams or not. Another option -could be to specify how many requests can be made over existing connections -before insisting on getting a separate connection. Note that such separate -connections might end up stacking up once released. So probably that they need -to be recycled very quickly (eg: fix how many unused ones can exist max). - diff --git a/doc/design-thoughts/rate-shaping.txt b/doc/design-thoughts/rate-shaping.txt deleted file mode 100644 index ca0940832..000000000 --- a/doc/design-thoughts/rate-shaping.txt +++ /dev/null @@ -1,90 +0,0 @@ -2010/01/24 - Design of multi-criteria request rate shaping. - -We want to be able to rate-shape traffic on multiple cirteria. For instance, we -may want to support shaping of per-host header requests, as well as per source. - -In order to achieve this, we will use checkpoints, one per criterion. Each of -these checkpoints will consist in a test, a rate counter and a queue. - -A request reaches the checkpoint and checks the counter. If the counter is -below the limit, it is updated and the request continues. If the limit is -reached, the request attaches itself into the queue and sleeps. The sleep time -is computed from the queue status, and updates the queue status. - -A task is dedicated to each queue. Its sole purpose is to be woken up when the -next task may wake up, to check the frequency counter, wake as many requests as -possible and update the counter. All the woken up requests are detached from -the queue. Maybe the task dedicated to the queue can be avoided and replaced -with all queued tasks's sleep counters, though this looks tricky. Or maybe it's -just the first request in the queue that should be responsible for waking up -other tasks, and not to forget to pass on this responsibility to next tasks if -it leaves the queue. - -The woken up request then goes on evaluating other criteria and possibly sleeps -again on another one. In the end, the task will have waited the amount of time -required to pass all checkpoints, and all checkpoints will be able to maintain -a permanent load of exactly their limit if enough streams flow through them. - -Since a request can only sleep in one queue at a time, it makes sense to use a -linked list element in each session to attach it to any queue. It could very -well be shared with the pendconn hooks which could then be part of the session. - -This mechanism could be used to rate-shape sessions and requests per backend -and per server. - -When rate-shaping on dynamic criteria, such as the source IP address, we have -to first extract the data pattern, then look it up in a table very similar to -the stickiness tables, but with a frequency counter. At the checkpoint, the -pattern is looked up, the entry created or refreshed, and the frequency counter -updated and checked. Then the request either goes on or sleeps as described -above, but if it sleeps, it's still in the checkpoint's queue, but with a date -computed from the criterion's status. - -This means that we need 3 distinct features : - - - optional pattern extraction - - per-pattern or per-queue frequency counter - - time-ordered queue with a task - -Based on past experiences with frequency counters, it does not appear very easy -to exactly compute sleep delays in advance for multiple requests. So most -likely we'll have to run per-criterion queues too, with only the head of the -queue holding a wake-up timeout. - -This finally leads us to the following : - - - optional pattern extraction - - per-pattern or per-queue frequency counter - - per-frequency counter queue - - head of the queue serves as a global queue timer. - -This brings us to a very flexible architecture : - - 1 list of rule-based checkpoints per frontend - - 1 list of rule-based checkpoints per backend - - 1 list of rule-based checkpoints per server - -Each of these lists have a lot of rules conditioned by ACLs, just like the -use-backend rules, except that all rules are evaluated in turn. - -Since we might sometimes just want to enable that without setting any limit and -just for enabling control in ACLs (or logging ?), we should probably try to -find a flexible way of declaring just a counter without a queue. - -These checkpoints could be of two types : - - rate-limit (described here) - - concurrency-limit (very similar with the counter and no timer). This - feature would require to keep track of all accounted criteria in a - request so that they can be released upon request completion. - -It should be possible to define a max of requests in the queue, above which a -503 is returned. The same applies for the max delay in the queue. We could have -it per-task (currently it's the connection timeout) and abort tasks with a 503 -when the delay is exceeded. - -Per-server connection concurrency could be converted to use this mechanism -which is very similar. - -The construct should be flexible enough so that the counters may be checked -from ACLs. That would allow to reject connections or switch to an alternate -backend when some limits are reached. - diff --git a/doc/design-thoughts/sess_par_sec.txt b/doc/design-thoughts/sess_par_sec.txt deleted file mode 100644 index e9363747e..000000000 --- a/doc/design-thoughts/sess_par_sec.txt +++ /dev/null @@ -1,13 +0,0 @@ -Graphe des nombres de traitements par seconde unité de temps avec - - un algo linéaire et très peu coûteux unitairement (0.01 ut) - - un algo en log(2) et 5 fois plus coûteux (0.05 ut) - -set yrange [0:1] -plot [0:1000] 1/(1+0.01*x), 1/(1+0.05*log(x+1)/log(2)) - -Graphe de la latence induite par ces traitements en unités de temps : - -set yrange [0:1000] -plot [0:1000] x/(1+0.01*x), x/(1+0.05*log(x+1)/log(2)) - - diff --git a/doc/internals/entities-v2.txt b/doc/internals/entities-v2.txt deleted file mode 100644 index 86782c34d..000000000 --- a/doc/internals/entities-v2.txt +++ /dev/null @@ -1,193 +0,0 @@ -An FD has a state : - - CLOSED - - READY - - ERROR (?) - - LISTEN (?) - -A connection has a state : - - CLOSED - - ACCEPTED - - CONNECTING - - ESTABLISHED - - ERROR - -A stream interface has a state : - - INI, REQ, QUE, TAR, ASS, CON, CER, EST, DIS, CLO - -Note that CON and CER might be replaced by EST if the connection state is used -instead. CON might even be more suited than EST to indicate that a connection -is known. - - -si_shutw() must do : - - data_shutw() - if (shutr) { - data_close() - ctrl_shutw() - ctrl_close() - } - -si_shutr() must do : - data_shutr() - if (shutw) { - data_close() - ctrl_shutr() - ctrl_close() - } - -Each of these steps may fail, in which case the step must be retained and the -operations postponed in an asynchronous task. - -The first asynchronous data_shut() might already fail so it is mandatory to -save the other side's status with the connection in order to let the async task -know whether the 3 next steps must be performed. - -The connection (or perhaps the FD) needs to know : - - the desired close operations : DSHR, DSHW, CSHR, CSHW - - the completed close operations : DSHR, DSHW, CSHR, CSHW - - -On the accept() side, we probably need to know : - - if a header is expected (eg: accept-proxy) - - if this header is still being waited for - => maybe both info might be combined into one bit - - - if a data-layer accept() is expected - - if a data-layer accept() has been started - - if a data-layer accept() has been performed - => possibly 2 bits, to indicate the need to free() - -On the connect() side, we need to know : - - the desire to send a header (eg: send-proxy) - - if this header has been sent - => maybe both info might be combined - - - if a data-layer connect() is expected - - if a data-layer connect() has been started - - if a data-layer connect() has been completed - => possibly 2 bits, to indicate the need to free() - -On the response side, we also need to know : - - the desire to send a header (eg: health check response for monitor-net) - - if this header was sent - => might be the same as sending a header over a new connection - -Note: monitor-net has precedence over proxy proto and data layers. Same for - health mode. - -For multi-step operations, use 2 bits : - 00 = operation not desired, not performed - 10 = operation desired, not started - 11 = operation desired, started but not completed - 01 = operation desired, started and completed - - => X != 00 ==> operation desired - X & 01 ==> operation at least started - X & 10 ==> operation not completed - -Note: no way to store status information for error reporting. - -Note2: it would be nice if "tcp-request connection" rules could work at the -connection level, just after headers ! This means support for tracking stick -tables, possibly not too much complicated. - - -Proposal for incoming connection sequence : - -- accept() -- if monitor-net matches or if mode health => try to send response -- if accept-proxy, wait for proxy request -- if tcp-request connection, process tcp rules and possibly keep the - pointer to stick-table -- if SSL is enabled, switch to SSL handshake -- then switch to DATA state and instantiate a session - -We just need a map of handshake handlers on the connection. They all manage the -FD status themselves and set the callbacks themselves. If their work succeeds, -they remove themselves from the list. If it fails, they remain subscribed and -enable the required polling until they are woken up again or the timeout strikes. - -Identified handshake handlers for incoming connections : - - HH_HEALTH (tries to send OK and dies) - - HH_MONITOR_IN (matches src IP and adds/removes HH_SEND_OK/HH_SEND_HTTP_OK) - - HH_SEND_OK (tries to send "OK" and dies) - - HH_SEND_HTTP_OK (tries to send "HTTP/1.0 200 OK" and dies) - - HH_ACCEPT_PROXY (waits for PROXY line and parses it) - - HH_TCP_RULES (processes TCP rules) - - HH_SSL_HS (starts SSL handshake) - - HH_ACCEPT_SESSION (instantiates a session) - -Identified handshake handlers for outgoing connections : - - HH_SEND_PROXY (tries to build and send the PROXY line) - - HH_SSL_HS (starts SSL handshake) - -For the pollers, we could check that handshake handlers are not 0 and decide to -call a generic connection handshake handler instead of usual callbacks. Problem -is that pollers don't know connections, they know fds. So entities which manage -handlers should update change the FD callbacks accordingly. - -With a bit of care, we could have : - - HH_SEND_LAST_CHUNK (sends the chunk pointed to by a pointer and dies) - => merges HEALTH, SEND_OK and SEND_HTTP_OK - -It sounds like the ctrl vs data state for the connection are per-direction -(eg: support an async ctrl shutw while still reading data). - -Also support shutr/shutw status at L4/L7. - -In practice, what we really need is : - -shutdown(conn) = - conn.data.shut() - conn.ctrl.shut() - conn.fd.shut() - -close(conn) = - conn.data.close() - conn.ctrl.close() - conn.fd.close() - -With SSL over Remote TCP (RTCP + RSSL) to reach the server, we would have : - - HTTP -> RTCP+RSSL connection <-> RTCP+RRAW connection -> TCP+SSL connection - -The connection has to be closed at 3 places after a successful response : - - DATA (RSSL over RTCP) - - CTRL (RTCP to close connection to server) - - SOCK (FD to close connection to second process) - -Externally, the connection is seen with very few flags : - - SHR - - SHW - - ERR - -We don't need a CLOSED flag as a connection must always be detached when it's closed. - -The internal status doesn't need to be exposed : - - FD allocated (Y/N) - - CTRL initialized (Y/N) - - CTRL connected (Y/N) - - CTRL handlers done (Y/N) - - CTRL failed (Y/N) - - CTRL shutr (Y/N) - - CTRL shutw (Y/N) - - DATA initialized (Y/N) - - DATA connected (Y/N) - - DATA handlers done (Y/N) - - DATA failed (Y/N) - - DATA shutr (Y/N) - - DATA shutw (Y/N) - -(note that having flags for operations needing to be completed might be easier) --------------- - -Maybe we need to be able to call conn->fdset() and conn->fdclr() but it sounds -very unlikely since the only functions manipulating this are in the code of -the data/ctrl handlers. - -FDSET/FDCLR cannot be directly controlled by the stream interface since it also -depends on the DATA layer (WANT_READ/wANT_WRITE). - -But FDSET/FDCLR is probably controlled by who owns the connection (eg: DATA). - diff --git a/doc/internals/entities.txt b/doc/internals/entities.txt deleted file mode 100644 index cdde82e34..000000000 --- a/doc/internals/entities.txt +++ /dev/null @@ -1,96 +0,0 @@ -2011/02/25 - Description of the different entities in haproxy - w@1wt.eu - - -1) Definitions --------------- - -Listener --------- - -A listener is the entity which is part of a frontend and which accepts -connections. There are as many listeners as there are ip:port couples. -There is at least one listener instantiated for each "bind" entry, and -port ranges will lead to as many listeners as there are ports in the -range. A listener just has a listening file descriptor ready to accept -incoming connections and to dispatch them to upper layers. - - -Initiator ---------- - -An initiator is instantiated for each incoming connection on a listener. It may -also be instantiated by a task pretending to be a client. An initiator calls -the next stage's accept() callback to present it with the parameters of the -incoming connection. - - -Session -------- - -A session is the only entity located between an initiator and a connector. -This is the last stage which offers an accept() callback, and all of its -processing will continue with the next stage's connect() callback. It holds -the buffers needed to forward the protocol data between each side. This entity -sees the native protocol, and is able to call analysers on these buffers. As it -is used in both directions, it always has two buffers. - -When transformations are required, some of them may be done on the initiator -side and other ones on the connector side. If additional buffers are needed for -such transforms, those buffers cannot replace the session's buffers, but they -may complete them. - -A session only needs to be instantiated when forwarding of data is required -between two sides. Accepting and filtering on layer 4 information only does not -require a session. - -For instance, let's consider the case of a proxy which receives and decodes -HTTPS traffic, processes it as HTTP and recodes it as HTTPS before forwarding -it. We'd have 3 layers of buffers, where the middle ones are used for -forwarding of the protocol data (HTTP here) : - - <-- ssl dec --> <-forwarding-> <-- ssl enc --> - - ,->[||||]--. ,->[||||]--. ,->[||||]--. - client (|) (|) (|) (|) server - ^--[||||]<-' ^--[||||]<-' ^--[||||]<-' - - HTTPS HTTP HTTPS - -The session handling code is only responsible for monitoring the forwarding -buffers here. It may declare the end of the session once those buffers are -closed and no analyser wants to re-open them. The session is also the entity -which applies the load balancing algorithm and decides the server to use. - -The other sides are responsible for propagating the state up to the session -which takes decisions. - - -Connector ---------- - -A connector is the entity which permits to instantiate a connection to a known -destination. It presents a connect() callback, and as such appears on the right -side of diagrams. - - -Connection ----------- - -A connection is the entity instantiated by a connector. It may be composed of -multiple stages linked together. Generally it is the part of the stream -interface holding a file descriptor, but it can also be a processing block or a -transformation block terminated by a connection. A connection presents a -server-side interface. - - -2) Sequencing -------------- - -Upon startup, listeners are instantiated by the configuration. When an incoming -connection reaches a listening file descriptor, its read() callback calls the -corresponding listener's accept() function which instantiates an initiator and -in turn recursively calls upper layers' accept() callbacks until -accept_session() is called. accept_session() instantiates a new session which -starts protocol analysis via process_session(). When all protocol analysis is -done, process_session() calls the connect() callback of the connector in order -to get a connection. diff --git a/doc/internals/header-parser-speed.txt b/doc/internals/header-parser-speed.txt deleted file mode 100644 index 285e2fab7..000000000 --- a/doc/internals/header-parser-speed.txt +++ /dev/null @@ -1,92 +0,0 @@ -TEST 3: - - printf "GET /\r\nbla: truc\r\n\r\n" - - -NO SPEEDUP : - -WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071080, r=0x8071094 -WHL: hdr_st=0x01, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071080, r=0x8071094 -WHL: hdr_st=0x32, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071086, r=0x8071094 -WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071087, r=0x8071094 -WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071091, r=0x8071094 -WHL: hdr_st=0x03, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8071092, lr=0x8071092, r=0x8071094 -WHL: hdr_st=0x34, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8071092, lr=0x8071093, r=0x8071094 -WHL: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8071092, lr=0x8071093, r=0x8071094 -END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8071092, lr=0x8071094, r=0x8071094 -=> 9 trans - - -FULL SPEEDUP : - -WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x806a770, lr=0x806a770, r=0x806a784 -WHL: hdr_st=0x32, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x806a770, lr=0x806a776, r=0x806a784 -WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806a777, lr=0x806a777, r=0x806a784 -WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806a777, lr=0x806a781, r=0x806a784 -WHL: hdr_st=0x26, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x806a782, lr=0x806a783, r=0x806a784 -END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x806a782, lr=0x806a784, r=0x806a784 -=> 6 trans - - - -TEST 4: - - - printf "GET /\nbla: truc\n\n" - - -NO SPEEDUP : - -WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x80750d0, lr=0x80750d0, r=0x80750e1 -WHL: hdr_st=0x01, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x80750d0, lr=0x80750d0, r=0x80750e1 -WHL: hdr_st=0x02, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x80750d0, lr=0x80750d5, r=0x80750e1 -WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x80750d6, lr=0x80750d6, r=0x80750e1 -WHL: hdr_st=0x04, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x80750d6, lr=0x80750df, r=0x80750e1 -WHL: hdr_st=0x03, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x80750e0, lr=0x80750e0, r=0x80750e1 -WHL: hdr_st=0x04, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x80750e0, lr=0x80750e0, r=0x80750e1 -WHL: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x80750e0, lr=0x80750e0, r=0x80750e1 -END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x80750e0, lr=0x80750e1, r=0x80750e1 -=> 9 trans - - -FULL SPEEDUP : - -WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8072010, lr=0x8072010, r=0x8072021 -WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8072016, lr=0x8072016, r=0x8072021 -END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8072020, lr=0x8072021, r=0x8072021 -=> 3 trans - - -TEST 5: - - - printf "GET /\r\nbla: truc\r\n truc2\r\n\r\n" - - -NO SPEEDUP : - -WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071080, r=0x807109d -WHL: hdr_st=0x01, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071080, r=0x807109d -WHL: hdr_st=0x32, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071086, r=0x807109d -WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071087, r=0x807109d -WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071091, r=0x807109d -WHL: hdr_st=0x05, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071092, r=0x807109d -WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071094, r=0x807109d -WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x807109a, r=0x807109d -WHL: hdr_st=0x03, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x807109b, lr=0x807109b, r=0x807109d -WHL: hdr_st=0x34, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x807109b, lr=0x807109c, r=0x807109d -WHL: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x807109b, lr=0x807109c, r=0x807109d -END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x807109b, lr=0x807109d, r=0x807109d -=> 12 trans - - -FULL SPEEDUP : - -WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x806dfc0, lr=0x806dfc0, r=0x806dfdd -WHL: hdr_st=0x32, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x806dfc0, lr=0x806dfc6, r=0x806dfdd -WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806dfc7, lr=0x806dfc7, r=0x806dfdd -WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806dfc7, lr=0x806dfd1, r=0x806dfdd -WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806dfc7, lr=0x806dfda, r=0x806dfdd -WHL: hdr_st=0x26, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x806dfdb, lr=0x806dfdc, r=0x806dfdd -END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x806dfdb, lr=0x806dfdd, r=0x806dfdd -=> 7 trans diff --git a/doc/internals/header-tree.txt b/doc/internals/header-tree.txt deleted file mode 100644 index 9a9736129..000000000 --- a/doc/internals/header-tree.txt +++ /dev/null @@ -1,124 +0,0 @@ -2007/03/30 - Header storage in trees - -This documentation describes how to store headers in radix trees, providing -fast access to any known position, while retaining the ability to grow/reduce -any arbitrary header without having to recompute all positions. - -Principle : - We have a radix tree represented in an integer array, which represents the - total number of bytes used by all headers whose position is below it. This - ensures that we can compute any header's position in O(log(N)) where N is - the number of headers. - -Example with N=16 : - - +-----------------------+ - | | - +-----------+ +-----------+ - | | | | - +-----+ +-----+ +-----+ +-----+ - | | | | | | | | - +--+ +--+ +--+ +--+ +--+ +--+ +--+ +--+ - | | | | | | | | | | | | | | | | - - 0 1 2 3 4 5 6 7 8 9 A B C D E F - - To reach header 6, we have to compute hdr[0]+hdr[4]+hdr[6] - - With this method, it becomes easy to grow any header and update the array. - To achieve this, we have to replace one after the other all bits on the - right with one 1 followed by zeroes, and update the position if it's higher - than current position, and stop when it's above number of stored headers. - - For instance, if we want to grow hdr[6], we proceed like this : - - 6 = 0110 (BIN) - - Let's consider the values to update : - - (bit 0) : (0110 & ~0001) | 0001 = 0111 = 7 > 6 => update - (bit 1) : (0110 & ~0011) | 0010 = 0110 = 6 <= 6 => leave it - (bit 2) : (0110 & ~0111) | 0100 = 0100 = 4 <= 6 => leave it - (bit 4) : (0110 & ~1111) | 1000 = 1000 = 8 > 6 => update - (bit 5) : larger than array size, stop. - - -It's easy to walk through the tree too. We only have one iteration per bit -changing from X to the ancestor, and one per bit from the ancestor to Y. -The ancestor is found while walking. To go from X to Y : - - pos = pos(X) - - while (Y != X) { - if (Y > X) { - // walk from Y to ancestor - pos += hdr[Y] - Y &= (Y - 1) - } else { - // walk from X to ancestor - pos -= hdr[X] - X &= (X - 1) - } - } - -However, it is not trivial anymore to linearly walk the tree. We have to move -from a known place to another known place, but a jump to next entry costs the -same as a jump to a random place. - -Other caveats : - - it is not possible to remove a header, it is only possible to empty it. - - it is not possible to insert a header, as that would imply a renumbering. - => this means that a "defrag" function is required. Headers should preferably - be added, then should be stuffed on top of destroyed ones, then only - inserted if absolutely required. - - -When we have this, we can then focus on a 32-bit header descriptor which would -look like this : - -{ - unsigned line_len :13; /* total line length, including CRLF */ - unsigned name_len :6; /* header name length, max 63 chars */ - unsigned sp1 :5; /* max spaces before value : 31 */ - unsigned sp2 :8; /* max spaces after value : 255 */ -} - -Example : - - Connection: close \r\n - <---------+-----+-----+-------------> line_len - <-------->| | | name_len - <-----> | sp1 - <-------------> sp2 -Rem: - - if there are more than 31 spaces before the value, the buffer will have to - be moved before being registered - - - if there are more than 255 spaces after the value, the buffer will have to - be moved before being registered - - - we can use the empty header name as an indicator for a deleted header - - - it would be wise to format a new request before sending lots of random - spaces to the servers. - - - normal clients do not send such crap, so those operations *may* reasonably - be more expensive than the rest provided that other ones are very fast. - -It would be handy to have the following macros : - - hdr_eon(hdr) => end of name - hdr_sov(hdr) => start of value - hdr_eof(hdr) => end of value - hdr_vlen(hdr) => length of value - hdr_hlen(hdr) => total header length - - -A 48-bit encoding would look like this : - - Connection: close \r\n - <---------+------+---+--------------> eoh = 16 bits - <-------->| | | eon = 8 bits - <--------------->| | sov = 8 bits - <---> vlen = 16 bits - diff --git a/doc/internals/http-cookies.txt b/doc/internals/http-cookies.txt deleted file mode 100644 index 6d65c5483..000000000 --- a/doc/internals/http-cookies.txt +++ /dev/null @@ -1,45 +0,0 @@ -2010/08/31 - HTTP Cookies - Theory and reality - -HTTP cookies are not uniformly supported across browsers, which makes it very -hard to build a widely compatible implementation. At least four conflicting -documents exist to describe how cookies should be handled, and browsers -generally don't respect any but a sensibly selected mix of them : - - - Netscape's original spec (also mirrored at Curl's site among others) : - http://web.archive.org/web/20070805052634/http://wp.netscape.com/newsref/std/cookie_spec.html - http://curl.haxx.se/rfc/cookie_spec.html - - Issues: uses an unquoted "Expires" field that includes a comma. - - - RFC 2109 : - http://www.ietf.org/rfc/rfc2109.txt - - Issues: specifies use of "Max-Age" (not universally implemented) and does - not talk about "Expires" (generally supported). References quoted - strings, not generally supported (eg: MSIE). Stricter than browsers - about domains. Ambiguous about allowed spaces in values and attrs. - - - RFC 2965 : - http://www.ietf.org/rfc/rfc2965.txt - - Issues: same as RFC2109 + describes Set-Cookie2 which only Opera supports. - - - Current internet draft : - https://datatracker.ietf.org/wg/httpstate/charter/ - - Issues: as of -p10, does not explain how the Set-Cookie2 header must be - emitted/handled, while suggesting a stricter approach for Cookie. - Documents reality and as such reintroduces the widely used unquoted - "Expires" attribute with its error-prone syntax. States that a - server should not emit more than one cookie per Set-Cookie header, - which is incompatible with HTTP which says that multiple headers - are allowed only if they can be folded. - -See also the following URL for a browser * feature matrix : - http://code.google.com/p/browsersec/wiki/Part2#Same-origin_policy_for_cookies - -In short, MSIE and Safari neither support quoted strings nor max-age, which -make it mandatory to continue to send an unquoted Expires value (maybe the -day of week could be omitted though). Only Safari supports comma-separated -lists of Set-Cookie headers. Support for cross-domains is not uniform either. - diff --git a/doc/internals/http-docs.txt b/doc/internals/http-docs.txt deleted file mode 100644 index 4ed24806d..000000000 --- a/doc/internals/http-docs.txt +++ /dev/null @@ -1,5 +0,0 @@ -Many interesting RFC and drafts linked to from this site : - - http://www.web-cache.com/Writings/protocols-standards.html - - diff --git a/doc/internals/http-parsing.txt b/doc/internals/http-parsing.txt deleted file mode 100644 index 8b3f23960..000000000 --- a/doc/internals/http-parsing.txt +++ /dev/null @@ -1,335 +0,0 @@ ---- Relevant portions of RFC2616 --- - -OCTET = -CHAR = -UPALPHA = -LOALPHA = -ALPHA = UPALPHA | LOALPHA -DIGIT = -CTL = -CR = -LF = -SP = -HT = -<"> = -CRLF = CR LF -LWS = [CRLF] 1*( SP | HT ) -TEXT = -HEX = "A" | "B" | "C" | "D" | "E" | "F" - | "a" | "b" | "c" | "d" | "e" | "f" | DIGIT -separators = "(" | ")" | "<" | ">" | "@" - | "," | ";" | ":" | "\" | <"> - | "/" | "[" | "]" | "?" | "=" - | "{" | "}" | SP | HT -token = 1* - -quoted-pair = "\" CHAR -ctext = -qdtext = > -quoted-string = ( <"> *(qdtext | quoted-pair ) <"> ) -comment = "(" *( ctext | quoted-pair | comment ) ")" - - - - - -4 HTTP Message -4.1 Message Types - -HTTP messages consist of requests from client to server and responses from -server to client. Request (section 5) and Response (section 6) messages use the -generic message format of RFC 822 [9] for transferring entities (the payload of -the message). Both types of message consist of : - - - a start-line - - zero or more header fields (also known as "headers") - - an empty line (i.e., a line with nothing preceding the CRLF) indicating the - end of the header fields - - and possibly a message-body. - - -HTTP-message = Request | Response - -start-line = Request-Line | Status-Line -generic-message = start-line - *(message-header CRLF) - CRLF - [ message-body ] - -In the interest of robustness, servers SHOULD ignore any empty line(s) received -where a Request-Line is expected. In other words, if the server is reading the -protocol stream at the beginning of a message and receives a CRLF first, it -should ignore the CRLF. - - -4.2 Message headers - -- Each header field consists of a name followed by a colon (":") and the field - value. -- Field names are case-insensitive. -- The field value MAY be preceded by any amount of LWS, though a single SP is - preferred. -- Header fields can be extended over multiple lines by preceding each extra - line with at least one SP or HT. - - -message-header = field-name ":" [ field-value ] -field-name = token -field-value = *( field-content | LWS ) -field-content = - - -The field-content does not include any leading or trailing LWS occurring before -the first non-whitespace character of the field-value or after the last -non-whitespace character of the field-value. Such leading or trailing LWS MAY -be removed without changing the semantics of the field value. Any LWS that -occurs between field-content MAY be replaced with a single SP before -interpreting the field value or forwarding the message downstream. - - -=> format des headers = 1*(CHAR & !ctl & !sep) ":" *(OCTET & (!ctl | LWS)) -=> les regex de matching de headers s'appliquent sur field-content, et peuvent - utiliser field-value comme espace de travail (mais de préférence après le - premier SP). - -(19.3) The line terminator for message-header fields is the sequence CRLF. -However, we recommend that applications, when parsing such headers, recognize -a single LF as a line terminator and ignore the leading CR. - - - - - -message-body = entity-body - | - - - -5 Request - -Request = Request-Line - *(( general-header - | request-header - | entity-header ) CRLF) - CRLF - [ message-body ] - - - -5.1 Request line - -The elements are separated by SP characters. No CR or LF is allowed except in -the final CRLF sequence. - -Request-Line = Method SP Request-URI SP HTTP-Version CRLF - -(19.3) Clients SHOULD be tolerant in parsing the Status-Line and servers -tolerant when parsing the Request-Line. In particular, they SHOULD accept any -amount of SP or HT characters between fields, even though only a single SP is -required. - -4.5 General headers -Apply to MESSAGE. - -general-header = Cache-Control - | Connection - | Date - | Pragma - | Trailer - | Transfer-Encoding - | Upgrade - | Via - | Warning - -General-header field names can be extended reliably only in combination with a -change in the protocol version. However, new or experimental header fields may -be given the semantics of general header fields if all parties in the -communication recognize them to be general-header fields. Unrecognized header -fields are treated as entity-header fields. - - - - -5.3 Request Header Fields - -The request-header fields allow the client to pass additional information about -the request, and about the client itself, to the server. These fields act as -request modifiers, with semantics equivalent to the parameters on a programming -language method invocation. - -request-header = Accept - | Accept-Charset - | Accept-Encoding - | Accept-Language - | Authorization - | Expect - | From - | Host - | If-Match - | If-Modified-Since - | If-None-Match - | If-Range - | If-Unmodified-Since - | Max-Forwards - | Proxy-Authorization - | Range - | Referer - | TE - | User-Agent - -Request-header field names can be extended reliably only in combination with a -change in the protocol version. However, new or experimental header fields MAY -be given the semantics of request-header fields if all parties in the -communication recognize them to be request-header fields. Unrecognized header -fields are treated as entity-header fields. - - - -7.1 Entity header fields - -Entity-header fields define metainformation about the entity-body or, if no -body is present, about the resource identified by the request. Some of this -metainformation is OPTIONAL; some might be REQUIRED by portions of this -specification. - -entity-header = Allow - | Content-Encoding - | Content-Language - | Content-Length - | Content-Location - | Content-MD5 - | Content-Range - | Content-Type - | Expires - | Last-Modified - | extension-header -extension-header = message-header - -The extension-header mechanism allows additional entity-header fields to be -defined without changing the protocol, but these fields cannot be assumed to be -recognizable by the recipient. Unrecognized header fields SHOULD be ignored by -the recipient and MUST be forwarded by transparent proxies. - ----------------------------------- - -The format of Request-URI is defined by RFC3986 : - - URI = scheme ":" hier-part [ "?" query ] [ "#" fragment ] - - hier-part = "//" authority path-abempty - / path-absolute - / path-rootless - / path-empty - - URI-reference = URI / relative-ref - - absolute-URI = scheme ":" hier-part [ "?" query ] - - relative-ref = relative-part [ "?" query ] [ "#" fragment ] - - relative-part = "//" authority path-abempty - / path-absolute - / path-noscheme - / path-empty - - scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." ) - - authority = [ userinfo "@" ] host [ ":" port ] - userinfo = *( unreserved / pct-encoded / sub-delims / ":" ) - host = IP-literal / IPv4address / reg-name - port = *DIGIT - - IP-literal = "[" ( IPv6address / IPvFuture ) "]" - - IPvFuture = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" ) - - IPv6address = 6( h16 ":" ) ls32 - / "::" 5( h16 ":" ) ls32 - / [ h16 ] "::" 4( h16 ":" ) ls32 - / [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32 - / [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32 - / [ *3( h16 ":" ) h16 ] "::" h16 ":" ls32 - / [ *4( h16 ":" ) h16 ] "::" ls32 - / [ *5( h16 ":" ) h16 ] "::" h16 - / [ *6( h16 ":" ) h16 ] "::" - - h16 = 1*4HEXDIG - ls32 = ( h16 ":" h16 ) / IPv4address - IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet - dec-octet = DIGIT ; 0-9 - / %x31-39 DIGIT ; 10-99 - / "1" 2DIGIT ; 100-199 - / "2" %x30-34 DIGIT ; 200-249 - / "25" %x30-35 ; 250-255 - - reg-name = *( unreserved / pct-encoded / sub-delims ) - - path = path-abempty ; begins with "/" or is empty - / path-absolute ; begins with "/" but not "//" - / path-noscheme ; begins with a non-colon segment - / path-rootless ; begins with a segment - / path-empty ; zero characters - - path-abempty = *( "/" segment ) - path-absolute = "/" [ segment-nz *( "/" segment ) ] - path-noscheme = segment-nz-nc *( "/" segment ) - path-rootless = segment-nz *( "/" segment ) - path-empty = 0 - - segment = *pchar - segment-nz = 1*pchar - segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims / "@" ) - ; non-zero-length segment without any colon ":" - - pchar = unreserved / pct-encoded / sub-delims / ":" / "@" - - query = *( pchar / "/" / "?" ) - - fragment = *( pchar / "/" / "?" ) - - pct-encoded = "%" HEXDIG HEXDIG - - unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~" - reserved = gen-delims / sub-delims - gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@" - sub-delims = "!" / "$" / "&" / "'" / "(" / ")" - / "*" / "+" / "," / ";" / "=" - -=> so the list of allowed characters in a URI is : - - uri-char = unreserved / gen-delims / sub-delims / "%" - = ALPHA / DIGIT / "-" / "." / "_" / "~" - / ":" / "/" / "?" / "#" / "[" / "]" / "@" - / "!" / "$" / "&" / "'" / "(" / ")" / - / "*" / "+" / "," / ";" / "=" / "%" - -Note that non-ascii characters are forbidden ! Spaces and CTL are forbidden. -Unfortunately, some products such as Apache allow such characters :-/ - ----- The correct way to do it ---- - -- one http_session - It is basically any transport session on which we talk HTTP. It may be TCP, - SSL over TCP, etc... It knows a way to talk to the client, either the socket - file descriptor or a direct access to the client-side buffer. It should hold - information about the last accessed server so that we can guarantee that the - same server can be used during a whole session if needed. A first version - without optimal support for HTTP pipelining will have the client buffers tied - to the http_session. It may be possible that it is not sufficient for full - pipelining, but this will need further study. The link from the buffers to - the backend should be managed by the http transaction (http_txn), provided - that they are serialized. Each http_session, has 0 to N http_txn. Each - http_txn belongs to one and only one http_session. - -- each http_txn has 1 request message (http_req), and 0 or 1 response message - (http_rtr). Each of them has 1 and only one http_txn. An http_txn holds - information such as the HTTP method, the URI, the HTTP version, the - transfer-encoding, the HTTP status, the authorization, the req and rtr - content-length, the timers, logs, etc... The backend and server which process - the request are also known from the http_txn. - -- both request and response messages hold header and parsing information, such - as the parsing state, start of headers, start of message, captures, etc... - diff --git a/doc/internals/naming.txt b/doc/internals/naming.txt deleted file mode 100644 index 0c3d5b537..000000000 --- a/doc/internals/naming.txt +++ /dev/null @@ -1,54 +0,0 @@ -Naming rules for manipulated objects and structures. - -Previously, there were ambiguities between sessions, transactions and requests, -as well as in the way responses are noted ("resp", "rep", "rsp"). - -Here is a proposal for a better naming scheme. - -The "session" is above the transport level, which means at ISO layer 5. -We can talk about "http sessions" when we consider the entity which lives -between the accept() and the close(), or the connect() and the close(). - -=> This demonstrates that it is not possible to have the same http session from - the client to the server. - -A session can carry one or multiple "transactions", which are each composed of -one "request" and zero or one "response". Both "request" and "response" are -described in RFC2616 as "HTTP messages". RFC2616 also seldom references the -word "transaction" without explicitly defining it. - -An "HTTP message" is composed of a "start line" which can be either a -"request line" or a "status line", followed by a number of "message headers" -which can be either "request headers" or "response headers", and an "entity", -itself composed of "entity headers" and an "entity body".Most probably, -"message headers" and "entity headers" will always be processed together as -"headers", while the "entity body" will design the payload. - -We must try to always use the same abbreviations when naming objects. Here are -a few common ones : - - - txn : transaction - - req : request - - rtr : response to request - - msg : message - - hdr : header - - ent : entity - - bdy : body - - sts : status - - stt : state - - idx : index - - cli : client - - srv : server - - svc : service - - ses : session - - tsk : task - -Short names for unions or cascaded structs : - - sl : start line - - sl.rq : request line - - sl.st : status line - - cl : client - - px : proxy - - sv : server - - st : state / status - diff --git a/doc/internals/repartition-be-fe-fi.txt b/doc/internals/repartition-be-fe-fi.txt deleted file mode 100644 index 261d073fe..000000000 --- a/doc/internals/repartition-be-fe-fi.txt +++ /dev/null @@ -1,20 +0,0 @@ -- session : ajouter ->fiprm et ->beprm comme raccourcis -- px->maxconn: ne s'applique qu'au FE. Pour le BE, on utilise fullconn, - initialisé par défaut à la même chose que maxconn. - - - \ from: proxy session server actuellement -field \ -rules px->fiprm sess->fiprm - -srv,cookies px->beprm sess->beprm srv->px -options(log) px-> sess->fe - -options(fe) px-> sess->fe - -options(be) px->beprm sess->beprm srv->px -captures px-> sess->fe - ->fiprm - - -logs px-> sess->fe srv->px -errorloc px-> sess->beprm|fe - -maxconn px-> sess->fe - ->be -fullconn px-> sess->beprm srv->px - -