1berry/haproxy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

CLEANUP: doc: remove 21 totally obsolete docs

Browse Source 
These were docs for very old design thoughts or internal subsystems
which are now totally irrelevant and even misleading. Those with some
outdated ideas mixed with useful stuff were kept though.
This commit is contained in:
Willy Tarreau 2023-05-31 15:13:29 +02:00
parent 7663301ea7
commit 431c62cbbf
21 changed files with 0 additions and 2279 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

39
doc/close-options.txt
View File

@ -1,39 +0,0 @@
2011/04/20 - List of keep-alive / close options with associated behaviours.
PK="http-pretend-keepalive", HC="httpclose", SC="http-server-close",
0 = option not set
1 = option is set
* = option doesn't matter
Options can be split between frontend and backend, so some of them might have
a meaning only when combined by associating a frontend to a backend. Some forms
are not the normal ones and provide a behaviour compatible with another normal
form. Those are considered alternate forms and are marked "(alt)".
SC HC PK Behaviour
0 0 X tunnel mode
0 1 0 passive close, only set headers then tunnel
0 1 1 forced close with keep-alive announce (alt)
1 0 0 server close
1 0 1 server close with keep-alive announce
1 1 0 forced close (alt)
1 1 1 forced close with keep-alive announce (alt)
At this point this results in 4 distinct effective modes for a request being
processed :
- tunnel mode : Connection header is left untouched and body is ignored
- passive close : Connection header is changed and body is ignored
- server close : Connection header set, body scanned, client-side keep-alive
is made possible regardless of server-side capabilities
- forced close : Connection header set, body scanned, connection closed.
The "close" modes may be combined with a fake keep-alive announce to the server
in order to workaround buggy servers that disable chunked encoding and content
length announces when the client does not ask for keep-alive.
Note: "http-pretend-keepalive" alone has no effect. However, if it is set in a
backend while a frontend is in "http-close" mode, then the combination of
both will result in a forced close with keep-alive announces for requests
passing through both.

27
doc/design-thoughts/backends-v0.txt
View File

@ -1,27 +0,0 @@
1 type générique "entité", avec les attributs suivants :
- frontend *f
- l7switch *s
- backend *b
des types spécifiques sont simplement des entités avec certains
de ces champs remplis et pas forcément tous :
listen = f [s] b
frontend = f [s]
l7switch = s
backend = [s] b
Ensuite, les traitements sont évalués dans l'ordre :
- listen -> s'il a des règles de l7, on les évalue, et potentiellement on branche vers d'autres listen, l7 ou back, ou on travaille avec le back local.
- frontend -> s'il a des règles de l7, on les évalue, et potentiellement on branche vers d'autres listen, l7 ou back
- l7switch -> on évalue ses règles, potentiellement on branche vers d'autres listen, l7 ou backends
- backend -> s'il a des règles l7, on les évalue (quitte à changer encore de backend) puis on traite.
Les requêtes sont traitées dans l'ordre des chaînages f->s*->b, et les réponses doivent être
traitées dans l'ordre inverse b->s*->f. Penser aux réécritures de champs Host à l'aller et
Location en retour.
D'autre part, prévoir des "profils" plutôt que des blocs de nouveaux paramètres par défaut.
Ca permettra d'avoir plein de jeux de paramètres par défaut à utiliser dans chacun de ces
types.

125
doc/design-thoughts/backends.txt
View File

@ -1,125 +0,0 @@
There has been a lot of confusion during the development because of the
backends and frontends.
What we want :
- being able to still use a listener as it has always been working
- being able to write a rule stating that we will *change* the backend when we
match some pattern. Only one jump is allowed.
- being able to write a "use_filters_from XXX" line stating that we will ignore
any filter in the current listener, and that those from XXX will be borrowed
instead. A warning would be welcome for options which will silently get
masked. This is used to factor configuration.
- being able to write a "use_backend_from XXX" line stating that we will ignore
any server and timeout config in the current listener, and that those from
XXX will be borrowed instead. A warning would be welcome for options which
will silently get masked. This is used to factor configuration.
Example :
---------
| # frontend HTTP/80
| listen fe_http 1.1.1.1:80
| use_filters_from default_http
| use_backend_from appli1
|
| # frontend HTTPS/443
| listen fe_https 1.1.1.1:443
| use_filters_from default_https
| use_backend_from appli1
|
| # frontend HTTP/8080
| listen fe_http-dev 1.1.1.1:8080
| reqadd "X-proto: http"
| reqisetbe "^Host: www1" appli1
| reqisetbe "^Host: www2" appli2
| reqisetbe "^Host: www3" appli-dev
| use_backend_from appli1
|
|
| # filters default_http
| listen default_http
| reqadd "X-proto: http"
| reqisetbe "^Host: www1" appli1
| reqisetbe "^Host: www2" appli2
|
| # filters default_https
| listen default_https
| reqadd "X-proto: https"
| reqisetbe "^Host: www1" appli1
| reqisetbe "^Host: www2" appli2
|
|
| # backend appli1
| listen appli1
| reqidel "^X-appli1:.*"
| reqadd "Via: appli1"
| balance roundrobin
| cookie app1
| server srv1
| server srv2
|
| # backend appli2
| listen appli2
| reqidel "^X-appli2:.*"
| reqadd "Via: appli2"
| balance roundrobin
| cookie app2
| server srv1
| server srv2
|
| # backend appli-dev
| listen appli-dev
| reqadd "Via: appli-dev"
| use_backend_from appli2
|
|
Now we clearly see multiple things :
------------------------------------
- a frontend can EITHER have filters OR reference a use_filter
- a backend can EITHER have servers OR reference a use_backend
- we want the evaluation to cross TWO levels per request. When a request is
being processed, it keeps track of its "frontend" side (where it came
from), and of its "backend" side (where the server-side parameters have
been found).
- the use_{filters|backend} have nothing to do with how the request is
decomposed.
Conclusion :
------------
- a proxy is always its own frontend. It also has 2 parameters :
- "fi_prm" : pointer to the proxy holding the filters (itself by default)
- "be_prm" : pointer to the proxy holding the servers (itself by default)
- a request has a frontend (fe) and a backend (be). By default, the backend
is initialized to the frontend. Everything related to the client side is
accessed through ->fe. Everything related to the server side is accessed
through ->be.
- request filters are first called from ->fe then ->be. Since only the
filters can change ->be, it is possible to iterate the filters on ->be
only and stop when ->be does not change anymore.
- response filters are first called from ->be then ->fe IF (fe != be).
When we parse the configuration, we immediately configure ->fi and ->be for
all proxies.
Upon session creation, s->fe and s->be are initialized to the proxy. Filters
are executed via s->fe->fi_prm and s->be->fi_prm. Servers are found in
s->be->be_prm.

74
doc/design-thoughts/be-fe-changes.txt
View File

@ -1,74 +0,0 @@
- PR_O_TRANSP => FE !!! devra peut-être changer vu que c'est un complément du mode dispatch.
- PR_O_NULLNOLOG => FE
- PR_O_HTTP_CLOSE => FE. !!! mettre BE aussi !!!
- PR_O_TCP_CLI_KA => FE
- PR_O_FWDFOR => BE. FE aussi ?
- PR_O_FORCE_CLO => BE
- PR_O_PERSIST => BE
- PR_O_COOK_RW, PR_O_COOK_INS, PR_O_COOK_PFX, PR_O_COOK_POST => BE
- PR_O_COOK_NOC, PR_O_COOK_IND => BE
- PR_O_ABRT_CLOSE => BE
- PR_O_REDISP => BE
- PR_O_BALANCE, PR_O_BALANCE_RR, PR_O_BALANCE_SH => BE
- PR_O_CHK_CACHE => BE
- PR_O_TCP_SRV_KA => BE
- PR_O_BIND_SRC => BE
- PR_O_TPXY_MASK => BE
- PR_MODE_TCP : BE côté serveur, FE côté client
- nbconn -> fe->nbconn, be->nbconn.
Pb: rendre impossible le fait que (fe == be) avant de faire ça,
sinon on va compter les connexions en double. Ce ne sera possible
que lorsque les FE et BE seront des entités distinctes. On va donc
commencer par laisser uniquement fe->nbconn (vu que le fe ne change
pas), et modifier ceci plus tard, ne serait-ce que pour prendre en
compte correctement les minconn/maxconn.
=> solution : avoir beconn et feconn dans chaque proxy.
- failed_conns, failed_secu (réponses bloquées), failed_resp... : be
Attention: voir les cas de ERR_SRVCL, il semble que parfois on
indique ça alors qu'il y a un write error côté client (ex: ligne
2044 dans proto_http).
=> be et pas be->beprm
- logs du backup : ->be (idem)
- queue : be
- logs/debug : srv toujours associé à be (ex: proxy->id:srv->id). Rien
pour le client pour le moment. D'une manière générale, les erreurs
provoquées côté serveur vont sur BE et celles côté client vont sur
FE.
- logswait & LW_BYTES : FE (puisqu'on veut savoir si on logue tout de suite)
- messages d'erreurs personnalisés (errmsg, ...) -> fe
- monitor_uri -> fe
- uri_auth -> (fe->firpm puis be->fiprm). Utilisation de ->be
- req_add, req_exp => fe->fiprm, puis be->fiprm
- req_cap, rsp_cap -> fe->fiprm
- rsp_add, rsp_exp => be->fiprm, devrait être fait ensuite aussi sur fe->fiprm
- capture_name, capture_namelen : fe->fiprm
Ce n'est pas la solution idéale, mais au moins la capture et configurable
par les filtres du FE et ne bouge pas lorsque le BE est réassigné. Cela
résoud aussi un pb d'allocation mémoire.
- persistance (appsessions, cookiename, ...) -> be
- stats:scope "." = fe (celui par lequel on arrive)
!!!ERREUR!!! => utiliser be pour avoir celui qui a été validé par
l'uri_auth.
--------- corrections à effectuer ---------
- remplacement de headers : parser le header et éventuellement le supprimer puis le(les) rajouter.
- session->proto.{l4state,l7state,l7substate} pour CLI et SRV
- errorloc : si définie dans backend, la prendre, sinon dans front.
- logs : faire be sinon fe.

262
doc/design-thoughts/config-language.txt
View File

@ -1,262 +0,0 @@
Prévoir des commandes en plusieurs mots clés.
Par exemple :
timeout connection XXX
connection scale XXX
On doit aussi accepter les préfixes :
tim co XXX
co sca XXX
Prévoir de ranger les combinaisons dans un tableau. On doit même
pouvoir effectuer un mapping simplifiant le parseur.
Pour les filtres :
<direction> <where> <what> <operator> <pattern> <action> [ <args>* ]
<direction> = [ req | rsp ]
<where> = [ in | out ]
<what> = [ line | LINE | METH | URI | h(hdr) | H(hdr) | c(cookie) | C(cookie) ]
<operator> = [ == | =~ | =* | =^ | =/ | != | !~ | !* | !^ | !/ ]
<pattern> = "<string>"
<action> = [ allow | permit | deny | delete | replace | switch | add | set | redir ]
<args> = optional action args
examples:
req in URI =^ "/images" switch images
req in h(host) =* ".mydomain.com" switch mydomain
req in h(host) =~ "localhost(.*)" replace "www\1"
alternative :
<direction> <where> <action> [not] <what> [<operator> <pattern> [ <args>* ]]
req in switch URI =^ "/images" images
req in switch h(host) =* ".mydomain.com" mydomain
req in replace h(host) =~ "localhost(.*)" "www\1"
req in delete h(Connection)
req in deny not line =~ "((GET|HEAD|POST|OPTIONS) /)|(OPTIONS *)"
req out set h(Connection) "close"
req out add line "Server: truc"
<direction> <action> <where> [not] <what> [<operator> <pattern> [ <args>* ]] ';' <action2> <what2>
req in switch URI =^ "/images/" images ; replace "/"
req in switch h(host) =* ".mydomain.com" mydomain
req in replace h(host) =~ "localhost(.*)" "www\1"
req in delete h(Connection)
req in deny not line =~ "((GET|HEAD|POST|OPTIONS) /)|(OPTIONS *)"
req out set h(Connection) "close"
req out add line == "Server: truc"
Extension avec des ACL :
req in acl(meth_valid) METH =~ "(GET|POST|HEAD|OPTIONS)"
req in acl(meth_options) METH == "OPTIONS"
req in acl(uri_slash) URI =^ "/"
req in acl(uri_star) URI == "*"
req in deny acl !(meth_options && uri_star || meth_valid && uri_slash)
Peut-être plus simplement :
acl meth_valid METH =~ "(GET|POST|HEAD|OPTIONS)"
acl meth_options METH == "OPTIONS"
acl uri_slash URI =^ "/"
acl uri_star URI == "*"
req in deny not acl(meth_options uri_star, meth_valid uri_slash)
req in switch URI =^ "/images/" images ; replace "/"
req in switch h(host) =* ".mydomain.com" mydomain
req in replace h(host) =~ "localhost(.*)" "www\1"
req in delete h(Connection)
req in deny not line =~ "((GET|HEAD|POST|OPTIONS) /)|(OPTIONS *)"
req out set h(Connection) "close"
req out add line == "Server: truc"
Prévoir le cas du "if" pour exécuter plusieurs actions :
req in if URI =^ "/images/" then replace "/" ; switch images
Utiliser les noms en majuscules/minuscules pour indiquer si on veut prendre
en compte la casse ou non :
if uri =^ "/watch/" setbe watch rebase "/watch/" "/"
if uri =* ".jpg" setbe images
if uri =~ ".*dll.*" deny
if HOST =* ".mydomain.com" setbe mydomain
etc...
Another solution would be to have a dedicated keyword to URI remapping. It
would both rewrite the URI and optionally switch to another backend.
uriremap "/watch/" "/" watch
uriremap "/chat/" "/" chat
uriremap "/event/" "/event/" event
Or better :
uriremap "/watch/" watch "/"
uriremap "/chat/" chat "/"
uriremap "/event/" event
For the URI, using a regex is sometimes useful (eg: providing a set of possible prefixes.
Sinon, peut-être que le "switch" peut prendre un paramètre de mapping pour la partie matchée :
req in switch URI =^ "/images/" images:"/"
2007/03/31 - Besoins plus précis.
1) aucune extension de branchement ou autre dans les "listen", c'est trop complexe.
Distinguer les données entrantes (in) et sortantes (out).
Le frontend ne voit que les requetes entrantes et les réponses sortantes.
Le backend voir les requêtes in/out et les réponses in/out.
Le frontend permet les branchements d'ensembles de filtres de requêtes vers
d'autres. Le frontend et les ensembles de filtres de requêtes peuvent brancher
vers un backend.
-----------+--------+----------+----------+---------+----------+
\ Where | | | | | |
\______ | Listen | Frontend | ReqRules | Backend | RspRules |
\| | | | | |
Capability | | | | | |
-----------+--------+----------+----------+---------+----------+
Frontend | X | X | | | |
-----------+--------+----------+----------+---------+----------+
FiltReqIn | X | X | X | X | |
-----------+--------+----------+----------+---------+----------+
JumpFiltReq| X | X | X | | | \
-----------+--------+----------+----------+---------+----------+ > = ReqJump
SetBackend | X | X | X | | | /
-----------+--------+----------+----------+---------+----------+
FiltReqOut | | | | X | |
-----------+--------+----------+----------+---------+----------+
FiltRspIn | X | | | X | X |
-----------+--------+----------+----------+---------+----------+
JumpFiltRsp| | | | X | X |
-----------+--------+----------+----------+---------+----------+
FiltRspOut | | X | | X | X |
-----------+--------+----------+----------+---------+----------+
Backend | X | | | X | |
-----------+--------+----------+----------+---------+----------+
En conclusion
-------------
Il y a au moins besoin de distinguer 8 fonctionnalités de base :
- capacité à recevoir des connexions (frontend)
- capacité à filtrer les requêtes entrantes
- capacité à brancher vers un backend ou un ensemble de règles de requêtes
- capacité à filtrer les requêtes sortantes
- capacité à filtrer les réponses entrantes
- capacité à brancher vers un autre ensemble de règles de réponses
- capacité à filtrer la réponse sortante
- capacité à gérer des serveurs (backend)
Remarque
--------
- on a souvent besoin de pouvoir appliquer un petit traitement sur un ensemble
host/uri/autre. Le petit traitement peut consister en quelques filtres ainsi
qu'une réécriture du couple (host,uri).
Proposition : ACL
Syntaxe :
---------
acl <name> <what> <operator> <value> ...
Ceci créera une acl référencée sous le nom <name> qui sera validée si
l'application d'au moins une des valeurs <value> avec l'opérateur <operator>
sur le sujet <what> est validée.
Opérateurs :
------------
Toujours 2 caractères :
[=!][~=*^%/.]
Premier caractère :
'=' : OK si test valide
'!' : OK si test échoué.
Second caractère :
'~' : compare avec une regex
'=' : compare chaîne à chaîne
'*' : compare la fin de la chaîne (ex: =* ".mydomain.com")
'^' : compare le début de la chaîne (ex: =^ "/images/")
'%' : recherche une sous-chaîne
'/' : compare avec un mot entier en acceptant le '/' comme délimiteur
'.' : compare avec un mot entier en acceptant el '.' comme délimiteur
Ensuite on exécute une action de manière conditionnelle si l'ensemble des ACLs
mentionnées sont validées (ou invalidées pour celles précédées d'un "!") :
<what> <where> <action> on [!]<aclname> ...
Exemple :
---------
acl www_pub host =. www www01 dev preprod
acl imghost host =. images
acl imgdir uri =/ img
acl imagedir uri =/ images
acl msie h(user-agent) =% "MSIE"
set_host "images" on www_pub imgdir
remap_uri "/img" "/" on www_pub imgdir
remap_uri "/images" "/" on www_pub imagedir
setbe images on imghost
reqdel "Cookie" on all
Actions possibles :
req {in|out} {append|delete|rem|add|set|rep|mapuri|rewrite|reqline|deny|allow|setbe|tarpit}
resp {in|out} {append|delete|rem|add|set|rep|maploc|rewrite|stsline|deny|allow}
req in append <line>
req in delete <line_regex>
req in rem <header>
req in add <header> <new_value>
req in set <header> <new_value>
req in rep <header> <old_value> <new_value>
req in mapuri <old_uri_prefix> <new_uri_prefix>
req in rewrite <old_uri_regex> <new_uri>
req in reqline <old_req_regex> <new_req>
req in deny
req in allow
req in tarpit
req in setbe <backend>
resp out maploc <old_location_prefix> <new_loc_prefix>
resp out stsline <old_sts_regex> <new_sts_regex>
Les chaînes doivent être délimitées par un même caractère au début et à la fin,
qui doit être échappé s'il est présent dans la chaîne. Tout ce qui se trouve
entre le caractère de fin et les premiers espace est considéré comme des
options passées au traitement. Par exemple :
req in rep host /www/i /www/
req in rep connection /keep-alive/i "close"
Il serait pratique de pouvoir effectuer un remap en même temps qu'un setbe.
Captures: les séparer en in/out. Les rendre conditionnelles ?

31
doc/design-thoughts/connection-sharing.txt
View File

@ -1,31 +0,0 @@
2014/10/28 - Server connection sharing
For HTTP/2 we'll have to use multiplexed connections to the servers and to
share them between multiple streams. We'll also have to do this for H/1, but
with some variations since H1 doesn't offer connection status verification.
In order to validate that an idle connection is still usable, it is desirable
to periodically send health checks over it. Normally, idle connections are
meant to be heavily used, so there is no reason for having them idle for a long
time. Thus we have two possibilities :
- either we time them out after some inactivity, this saves server resources ;
- or we check them after some inactivity. For this we can send the server-
side HTTP health check (only when the server uses HTTP checks), and avoid
using that to mark the server down, and instead consider the connection as
dead.
For HTTP/2 we'll have to send pings periodically over these connections, so
it's worth considering a per-connection task to validate that the channel still
works.
In the current model, a connection necessarily belongs to a session, so it's
not really possible to share them, at best they can be exchanged, but that
doesn't make much sense as it means that it could disturb parallel traffic.
Thus we need to have a per-server list of idle connections and a max-idle-conn
setting to kill them when there are too many. In the case of H/1 it is also
advisable to consider that if a connection was created to pass a first non-
idempotent request while other idle connections were still existing, then a
connection will have to be killed in order not to exceed the limit.

41
doc/design-thoughts/dynamic-buffers.txt
View File

@ -1,41 +0,0 @@
2014/10/30 - dynamic buffer management
Since HTTP/2 processing will significantly increase the need for buffering, it
becomes mandatory to be able to support dynamic buffer allocation. This also
means that at any moment some buffer allocation will fail and that a task or an
I/O operation will have to be paused for the time needed to allocate a buffer.
There are 3 places where buffers are needed :
- receive side of a stream interface. A connection notifies about a pending
recv() and the SI calls the receive function to put the data into a buffer.
Here the buffer will have to be picked from a pool first, and if the
allocation fails, the I/O will have to temporarily be disabled, the
connection will have to subscribe for buffer release notification to be
woken up once a buffer is available again. It's important to keep in mind
that buffer availability doesn't necessarily mean a desire to enable recv
again, just that recv is not paused anymore for resource reasons.
- receive side of a stream interface when the other end point is an applet.
The applet wants to write into the buffer and for this the buffer needs to
be allocated as well. It is the same as above except that it is the applet
which is put to a pause. Since the applet might be at the core of the task
itself, it could become tricky to handle the situation correctly. Stats and
peers are in this situation.
- Tx of a task : some tasks perform spontaneous writes to a buffer. Checks
are an example of this. The checks will have to be able to sleep while a
buffer is being awaited.
One important point is that such pauses must not prevent the task from timing
out. There it becomes difficult because in the case of a time out, we could
want to emit a timeout error message and for this, require a buffer. So it is
important to keep the ability not to send messages upon error processing, and
to be able to give up and stop waiting for buffers.
The refill mechanism needs to be designed in a thread-safe way because this
will become one of the rare cases of inter-task activity. Thus it is important
to ensure that checking the state of the task and passing of the freshly
released buffer are performed atomically, and that in case the task doesn't
want it anymore, it is responsible for passing it to the next one.

276
doc/design-thoughts/entities-v2.txt
View File

@ -1,276 +0,0 @@
2012/07/05 - Connection layering and sequencing
An FD has a state :
- CLOSED
- READY
- ERROR (?)
- LISTEN (?)
A connection has a state :
- CLOSED
- ACCEPTED
- CONNECTING
- ESTABLISHED
- ERROR
A stream interface has a state :
- INI, REQ, QUE, TAR, ASS, CON, CER, EST, DIS, CLO
Note that CON and CER might be replaced by EST if the connection state is used
instead. CON might even be more suited than EST to indicate that a connection
is known.
si_shutw() must do :
data_shutw()
if (shutr) {
data_close()
ctrl_shutw()
ctrl_close()
}
si_shutr() must do :
data_shutr()
if (shutw) {
data_close()
ctrl_shutr()
ctrl_close()
}
Each of these steps may fail, in which case the step must be retained and the
operations postponed in an asynchronous task.
The first asynchronous data_shut() might already fail so it is mandatory to
save the other side's status with the connection in order to let the async task
know whether the 3 next steps must be performed.
The connection (or perhaps the FD) needs to know :
- the desired close operations : DSHR, DSHW, CSHR, CSHW
- the completed close operations : DSHR, DSHW, CSHR, CSHW
On the accept() side, we probably need to know :
- if a header is expected (eg: accept-proxy)
- if this header is still being waited for
=> maybe both info might be combined into one bit
- if a data-layer accept() is expected
- if a data-layer accept() has been started
- if a data-layer accept() has been performed
=> possibly 2 bits, to indicate the need to free()
On the connect() side, we need to know :
- the desire to send a header (eg: send-proxy)
- if this header has been sent
=> maybe both info might be combined
- if a data-layer connect() is expected
- if a data-layer connect() has been started
- if a data-layer connect() has been completed
=> possibly 2 bits, to indicate the need to free()
On the response side, we also need to know :
- the desire to send a header (eg: health check response for monitor-net)
- if this header was sent
=> might be the same as sending a header over a new connection
Note: monitor-net has precedence over proxy proto and data layers. Same for
health mode.
For multi-step operations, use 2 bits :
00 = operation not desired, not performed
10 = operation desired, not started
11 = operation desired, started but not completed
01 = operation desired, started and completed
=> X != 00 ==> operation desired
X & 01 ==> operation at least started
X & 10 ==> operation not completed
Note: no way to store status information for error reporting.
Note2: it would be nice if "tcp-request connection" rules could work at the
connection level, just after headers ! This means support for tracking stick
tables, possibly not too much complicated.
Proposal for incoming connection sequence :
- accept()
- if monitor-net matches or if mode health => try to send response
- if accept-proxy, wait for proxy request
- if tcp-request connection, process tcp rules and possibly keep the
pointer to stick-table
- if SSL is enabled, switch to SSL handshake
- then switch to DATA state and instantiate a session
We just need a map of handshake handlers on the connection. They all manage the
FD status themselves and set the callbacks themselves. If their work succeeds,
they remove themselves from the list. If it fails, they remain subscribed and
enable the required polling until they are woken up again or the timeout strikes.
Identified handshake handlers for incoming connections :
- HH_HEALTH (tries to send OK and dies)
- HH_MONITOR_IN (matches src IP and adds/removes HH_SEND_OK/HH_SEND_HTTP_OK)
- HH_SEND_OK (tries to send "OK" and dies)
- HH_SEND_HTTP_OK (tries to send "HTTP/1.0 200 OK" and dies)
- HH_ACCEPT_PROXY (waits for PROXY line and parses it)
- HH_TCP_RULES (processes TCP rules)
- HH_SSL_HS (starts SSL handshake)
- HH_ACCEPT_SESSION (instantiates a session)
Identified handshake handlers for outgoing connections :
- HH_SEND_PROXY (tries to build and send the PROXY line)
- HH_SSL_HS (starts SSL handshake)
For the pollers, we could check that handshake handlers are not 0 and decide to
call a generic connection handshake handler instead of usual callbacks. Problem
is that pollers don't know connections, they know fds. So entities which manage
handlers should update change the FD callbacks accordingly.
With a bit of care, we could have :
- HH_SEND_LAST_CHUNK (sends the chunk pointed to by a pointer and dies)
=> merges HEALTH, SEND_OK and SEND_HTTP_OK
It sounds like the ctrl vs data state for the connection are per-direction
(eg: support an async ctrl shutw while still reading data).
Also support shutr/shutw status at L4/L7.
In practice, what we really need is :
shutdown(conn) =
conn.data.shut()
conn.ctrl.shut()
conn.fd.shut()
close(conn) =
conn.data.close()
conn.ctrl.close()
conn.fd.close()
With SSL over Remote TCP (RTCP + RSSL) to reach the server, we would have :
HTTP -> RTCP+RSSL connection <-> RTCP+RRAW connection -> TCP+SSL connection
The connection has to be closed at 3 places after a successful response :
- DATA (RSSL over RTCP)
- CTRL (RTCP to close connection to server)
- SOCK (FD to close connection to second process)
Externally, the connection is seen with very few flags :
- SHR
- SHW
- ERR
We don't need a CLOSED flag as a connection must always be detached when it's closed.
The internal status doesn't need to be exposed :
- FD allocated (Y/N)
- CTRL initialized (Y/N)
- CTRL connected (Y/N)
- CTRL handlers done (Y/N)
- CTRL failed (Y/N)
- CTRL shutr (Y/N)
- CTRL shutw (Y/N)
- DATA initialized (Y/N)
- DATA connected (Y/N)
- DATA handlers done (Y/N)
- DATA failed (Y/N)
- DATA shutr (Y/N)
- DATA shutw (Y/N)
(note that having flags for operations needing to be completed might be easier)
--------------
Maybe we need to be able to call conn->fdset() and conn->fdclr() but it sounds
very unlikely since the only functions manipulating this are in the code of
the data/ctrl handlers.
FDSET/FDCLR cannot be directly controlled by the stream interface since it also
depends on the DATA layer (WANT_READ/WANT_WRITE).
But FDSET/FDCLR is probably controlled by who owns the connection (eg: DATA).
Example: an SSL conn relies on an FD. The buffer is full, and wants the conn to
stop reading. It must not stop the FD itself. It is the read function which
should notice that it has nothing to do with a read wake-up, which needs to
disable reading.
Conversely, when calling conn->chk_rcv(), the reader might get a WANT_READ or
even WANT_WRITE and adjust the FDs accordingly.
------------------------
OK, the problem is simple : we don't manipulate the FD at the right level.
We should have :
->connect(), ->chk_snd(), ->chk_rcv(), ->shutw(), ->shutr() which are
called from the upper layer (buffer)
->recv(), ->send(), called from the lower layer
Note that the SHR is *reported* by lower layer but can be forced by upper
layer. In this case it's like a delayed abort. The difficulty consists in
knowing the output data were correctly read. Probably we'd need to drain
incoming data past the active shutr().
The only four purposes of the top-down shutr() call are :
- acknowledge a shut read report : could probably be done better
- read timeout => disable reading : it's a delayed abort. We want to
report that the buffer is SHR, maybe even the connection, but the
FD clearly isn't.
- read abort due to error on the other side or desire to close (eg:
http-server-close) : delayed abort
- complete abort
The active shutr() is problematic as we can't disable reading if we expect some
exchanges for data acknowledgement. We probably need to drain data only until
the shutw() has been performed and ACKed.
A connection shut down for read would behave like this :
1) bidir exchanges
2) shutr() => read_abort_pending=1
3) drain input, still send output
4) shutw()
5) drain input, wait for read0 or ack(shutw)
6) close()
--------------------- 2012/07/05 -------------------
Communications must be performed this way :
connection <-> channel <-> connection
A channel is composed of flags and stats, and may store data in either a buffer
or a pipe. We need low-layer operations between sockets and buffers or pipes.
Right now we only support sockets, but later we might support remote sockets
and maybe pipes or shared memory segments.
So we need :
- raw_sock_to_buf() => receive raw data from socket into buffer
- raw_sock_to_pipe => receive raw data from socket into pipe (splice in)
- raw_sock_from_buf() => send raw data from buffer to socket
- raw_sock_from_pipe => send raw data from pipe to socket (splice out)
- ssl_sock_to_buf() => receive ssl data from socket into buffer
- ssl_sock_to_pipe => receive ssl data from socket into a pipe (NULL)
- ssl_sock_from_buf() => send ssl data from buffer to socket
- ssl_sock_from_pipe => send ssl data from pipe to socket (NULL)
These functions should set such status flags :
#define ERR_IN 0x01
#define ERR_OUT 0x02
#define SHUT_IN 0x04
#define SHUT_OUT 0x08
#define EMPTY_IN 0x10
#define FULL_OUT 0x20

60
doc/design-thoughts/how-it-works.txt
View File

@ -1,60 +0,0 @@
How it works ? (unfinished and inexact)
For TCP and HTTP :
- listeners create listening sockets with a READ callback pointing to the
protocol-specific accept() function.
- the protocol-specific accept() function then accept()'s the connection and
instantiates a "server TCP socket" (which is dedicated to the client side),
and configures it (non_block, get_original_dst, ...).
For TCP :
- in case of pure TCP, a request buffer is created, as well as a "client TCP
socket", which tries to connect to the server.
- once the connection is established, the response buffer is allocated and
connected to both ends.
- both sockets are set to "autonomous mode" so that they only wake up their
supervising session when they encounter a special condition (error or close).
For HTTP :
- in case of HTTP, a request buffer is created with the "HOLD" flag set and
a read limit to support header rewriting (may be this one will be removed
eventually because it's better to limit only to the buffer size and report
an error when rewritten data overflows)
- a "flow analyzer" is attached to the buffer (or possibly multiple flow
analyzers). For the request, the flow analyzer is "http_lb_req". The flow
analyzer is a function which gets called when new data is present and
blocked. It has a timeout (request timeout). It can also be bypassed on
demand.
- when the "http_lb_req" has received the whole request, it creates a client
socket with all the parameters needed to try to connect to the server. When
the connection establishes, the response buffer is allocated on the fly,
put to HOLD mode, and a an "http_lb_resp" flow analyzer is attached to the
buffer.
For client-side HTTPS :
- the accept() function must completely instantiate a TCP socket + an SSL
reader. It is when the SSL session is complete that we call the
protocol-specific accept(), and create its buffer.
Conclusions
-----------
- we need a generic TCP accept() function with a lot of flags set by the
listener, to tell it what info we need to get at the accept() time, and
what flags will have to be set on the socket.
- once the TCP accept() function ends, it wakes up the protocol supervisor
which is in charge of creating the buffers, etc, switch states, etc...

277
doc/design-thoughts/http2.txt
View File

@ -1,277 +0,0 @@
2014/10/23 - design thoughts for HTTP/2
- connections : HTTP/2 depends a lot more on a connection than HTTP/1 because a
connection holds a compression context (headers table, etc...). We probably
need to have an h2_conn struct.
- multiple transactions will be handled in parallel for a given h2_conn. They
are called streams in HTTP/2 terminology.
- multiplexing : for a given client-side h2 connection, we can have multiple
server-side h2 connections. And for a server-side h2 connection, we can have
multiple client-side h2 connections. Streams circulate in N-to-N fashion.
- flow control : flow control will be applied between multiple streams. Special
care must be taken so that an H2 client cannot block some H2 servers by
sending requests spread over multiple servers to the point where one server
response is blocked and prevents other responses from the same server from
reaching their clients. H2 connection buffers must always be empty or nearly
empty. The per-stream flow control needs to be respected as well as the
connection's buffers. It is important to implement some fairness between all
the streams so that it's not always the same which gets the bandwidth when
the connection is congested.
- some clients can be H1 with an H2 server (is this really needed ?). Most of
the initial use case will be H2 clients to H1 servers. It is important to keep
in mind that H1 servers do not do flow control and that we don't want them to
block transfers (eg: post upload).
- internal tasks : some H2 clients will be internal tasks (eg: health checks).
Some H2 servers will be internal tasks (eg: stats, cache). The model must be
compatible with this use case.
- header indexing : headers are transported compressed, with a reference to a
static or a dynamic header, or a literal, possibly huffman-encoded. Indexing
is specific to the H2 connection. This means there is no way any binary data
can flow between both sides, headers will have to be decoded according to the
incoming connection's context and re-encoded according to the outgoing
connection's context, which can significantly differ. In order to avoid the
parsing trouble we currently face, headers will have to be clearly split
between name and value. It is worth noting that neither the incoming nor the
outgoing connections' contexts will be of any use while processing the
headers. At best we can have some shortcuts for well-known names that map
well to the static ones (eg: use the first static entry with same name), and
maybe have a few special cases for static name+value as well. Probably we can
classify headers in such categories :
- static name + value
- static name + other value
- dynamic name + other value
This will allow for better processing in some specific cases. Headers
supporting a single value (:method, :status, :path, ...) should probably
be stored in a single location with a direct access. That would allow us
to retrieve a method using hdr[METHOD]. All such indexing must be performed
while parsing. That also means that HTTP/1 will have to be converted to this
representation very early in the parser and possibly converted back to H/1
after processing.
Header names/values will have to be placed in a small memory area that will
inevitably get fragmented as headers are rewritten. An automatic packing
mechanism must be implemented so that when there's no more room, headers are
simply defragmented/packet to a new table and the old one is released. Just
like for the static chunks, we need to have a few such tables pre-allocated
and ready to be swapped at any moment. Repacking must not change any index
nor affect the way headers are compressed so that it can happen late after a
retry (send-name-header for example).
- header processing : can still happen on a (header, value) basis. Reqrep/
rsprep completely disappear and will have to be replaced with something else
to support renaming headers and rewriting url/path/...
- push_promise : servers can push dummy requests+responses. They advertise
the stream ID in the push_promise frame indicating the associated stream ID.
This means that it is possible to initiate a client-server stream from the
information coming from the server and make the data flow as if the client
had made it. It's likely that we'll have to support two types of server
connections: those which support push and those which do not. That way client
streams will be distributed to existing server connections based on their
capabilities. It's important to keep in mind that PUSH will not be rewritten
in responses.
- stream ID mapping : since the stream ID is per H2 connection, stream IDs will
have to be mapped. Thus a given stream is an entity with two IDs (one per
side). Or more precisely a stream has two end points, each one carrying an ID
when it ends on an HTTP2 connection. Also, for each stream ID we need to
quickly find the associated transaction in progress. Using a small quick
unique tree seems indicated considering the wide range of valid values.
- frame sizes : frame have to be remapped between both sides as multiplexed
connections won't always have the same characteristics. Thus some frames
might be spliced and others will be sliced.
- error processing : care must be taken to never break a connection unless it
is dead or corrupt at the protocol level. Stats counter must exist to observe
the causes. Timeouts are a great problem because silent connections might
die out of inactivity. Ping frames should probably be scheduled a few seconds
before the connection timeout so that an unused connection is verified before
being killed. Abnormal requests must be dealt with using RST_STREAM.
- ALPN : ALPN must be observed on the client side, and transmitted to the server
side.
- proxy protocol : proxy protocol makes little to no sense in a multiplexed
protocol. A per-stream equivalent will surely be needed if implementations
do not quickly generalize the use of Forward.
- simplified protocol for local devices (eg: haproxy->varnish in clear and
without handshake, and possibly even with splicing if the connection's
settings are shared)
- logging : logging must report a number of extra information such as the
stream ID, and whether the transaction was initiated by the client or by the
server (which can be deduced from the stream ID's parity). In case of push,
the number of the associated stream must also be reported.
- memory usage : H2 increases memory usage by mandating use of 16384 bytes
frame size minimum. That means slightly more than 16kB of buffer in each
direction to process any frame. It will definitely have an impact on the
deployed maxconn setting in places using less than this (4..8kB are common).
Also, the header list is persistent per connection, so if we reach the same
size as the request, that's another 16kB in each direction, resulting in
about 48kB of memory where 8 were previously used. A more careful encoder
can work with a much smaller set even if that implies evicting entries
between multiple headers of the same message.
- HTTP/1.0 should very carefully be transported over H2. Since there's no way
to pass version information in the protocol, the server could use some
features of HTTP/1.1 that are unsafe in HTTP/1.0 (compression, trailers,
...).
- host / :authority : ":authority" is the norm, and "host" will be absent when
H2 clients generate :authority. This probably means that a dummy Host header
will have to be produced internally from :authority and removed when passing
to H2 behind. This can cause some trouble when passing H2 requests to H1
proxies, because there's no way to know if the request should contain scheme
and authority in H1 or not based on the H2 request. Thus a "proxy" option
will have to be explicitly mentioned on HTTP/1 server lines. One of the
problem that it creates is that it's not longer possible to pass H/1 requests
to H/1 proxies without an explicit configuration. Maybe a table of the
various combinations is needed.
:scheme :authority host
HTTP/2 request present present absent
HTTP/1 server req absent absent present
HTTP/1 proxy req present present present
So in the end the issue is only with H/2 requests passed to H/1 proxies.
- ping frames : they don't indicate any stream ID so by definition they cannot
be forwarded to any server. The H2 connection should deal with them only.
There's a layering problem with H2. The framing layer has to be aware of the
upper layer semantics. We can't simply re-encode HTTP/1 to HTTP/2 then pass
it over a framing layer to mux the streams, the frame type must be passed below
so that frames are properly arranged. Header encoding is connection-based and
all streams using the same connection will interact in the way their headers
are encoded. Thus the encoder *has* to be placed in the h2_conn entity, and
this entity has to know for each stream what its headers are.
Probably that we should remove *all* headers from transported data and move
them on the fly to a parallel structure that can be shared between H1 and H2
and consumed at the appropriate level. That means buffers only transport data.
Trailers have to be dealt with differently.
So if we consider an H1 request being forwarded between a client and a server,
it would look approximately like this :
- request header + body land into a stream's receive buffer
- headers are indexed and stripped out so that only the body and whatever
follows remain in the buffer
- both the header index and the buffer with the body stay attached to the
stream
- the sender can rebuild the whole headers. Since they're found in a table
supposed to be stable, it can rebuild them as many times as desired and
will always get the same result, so it's safe to build them into the trash
buffer for immediate sending, just as we do for the PROXY protocol.
- the upper protocol should probably provide a build_hdr() callback which
when called by the socket layer, builds this header block based on the
current stream's header list, ready to be sent.
- the socket layer has to know how many bytes from the headers are left to be
forwarded prior to processing the body.
- the socket layer needs to consume only the acceptable part of the body and
must not release the buffer if any data remains in it (eg: pipelining over
H1). This is already handled by channel->o and channel->to_forward.
- we could possibly have another optional callback to send a preamble before
data, that could be used to send chunk sizes in H1. The danger is that it
absolutely needs to be stable if it has to be retried. But it could
considerably simplify de-chunking.
When the request is sent to an H2 server, an H2 stream request must be made
to the server, we find an existing connection whose settings are compatible
with our needs (eg: tls/clear, push/no-push), and with a spare stream ID. If
none is found, a new connection must be established, unless maxconn is reached.
Servers must have a maxstream setting just like they have a maxconn. The same
queue may be used for that.
The "tcp-request content" ruleset must apply to the TCP layer. But with HTTP/2
that becomes impossible (and useless). We still need something like the
"tcp-request session" hook to apply just after the SSL handshake is done.
It is impossible to defragment the body on the fly in HTTP/2. Since multiple
messages are interleaved, we cannot wait for all of them and block the head of
line. Thus if body analysis is required, it will have to use the stream's
buffer, which necessarily implies a copy. That means that with each H2 end we
necessarily have at least one copy. Sometimes we might be able to "splice" some
bytes from one side to the other without copying into the stream buffer (same
rules as for TCP splicing).
In theory, only data should flow through the channel buffer, so each side's
connector is responsible for encoding data (H1: linear/chunks, H2: frames).
Maybe the same mechanism could be extrapolated to tunnels / TCP.
Since we'd use buffers only for data (and for receipt of headers), we need to
have dynamic buffer allocation.
Thus :
- Tx buffers do not exist. We allocate a buffer on the fly when we're ready to
send something that we need to build and that needs to be persistent in case
of partial send. H1 headers are built on the fly from the header table to a
temporary buffer that is immediately sent and whose amount of sent bytes is
the only information kept (like for PROXY protocol). H2 headers are more
complex since the encoding depends on what was successfully sent. Thus we
need to build them and put them into a temporary buffer that remains
persistent in case send() fails. It is possible to have a limited pool of
Tx buffers and refrain from sending if there is no more buffer available in
the pool. In that case we need a wake-up mechanism once a buffer is
available. Once the data are sent, the Tx buffer is then immediately recycled
in its pool. Note that no tx buffer being used (eg: for hdr or control) means
that we have to be able to serialize access to the connection and retry with
the same stream. It also means that a stream that times out while waiting for
the connector to read the second half of its request has to stay there, or at
least needs to be handled gracefully. However if the connector cannot read
the data to be sent, it means that the buffer is congested and the connection
is dead, so that probably means it can be killed.
- Rx buffers have to be pre-allocated just before calling recv(). A connection
will first try to pick a buffer and disable reception if it fails, then
subscribe to the list of tasks waiting for an Rx buffer.
- full Rx buffers might sometimes be moved around to the next buffer instead of
experiencing a copy. That means that channels and connectors must use the
same format of buffer, and that only the channel will have to see its
pointers adjusted.
- Tx of data should be made as much as possible without copying. That possibly
means by directly looking into the connection buffer on the other side if
the local Tx buffer does not exist and the stream buffer is not allocated, or
even performing a splice() call between the two sides. One of the problem in
doing this is that it requires proper ordering of the operations (eg: when
multiple readers are attached to a same buffer). If the splitting occurs upon
receipt, there's no problem. If we expect to retrieve data directly from the
original buffer, it's harder since it contains various things in an order
which does not even indicate what belongs to whom. Thus possibly the only
mechanism to implement is the buffer permutation which guarantees zero-copy
and only in the 100% safe case. Also it's atomic and does not cause HOL
blocking.
It makes sense to chose the frontend_accept() function right after the
handshake ended. It is then possible to check the ALPN, the SNI, the ciphers
and to accept to switch to the h2_conn_accept handler only if everything is OK.
The h2_conn_accept handler will have to deal with the connection setup,
initialization of the header table, exchange of the settings frames and
preparing whatever is needed to fire new streams upon receipt of unknown
stream IDs. Note: most of the time it will not be possible to splice() because
we need to know in advance the amount of bytes to write the header, and here it
will not be possible.
H2 health checks must be seen as regular transactions/streams. The check runs a
normal client which seeks an available stream from a server. The server then
finds one on an existing connection or initiates a new H2 connection. The H2
checks will have to be configurable for sharing streams or not. Another option
could be to specify how many requests can be made over existing connections
before insisting on getting a separate connection. Note that such separate
connections might end up stacking up once released. So probably that they need
to be recycled very quickly (eg: fix how many unused ones can exist max).

90
doc/design-thoughts/rate-shaping.txt
View File

@ -1,90 +0,0 @@
2010/01/24 - Design of multi-criteria request rate shaping.
We want to be able to rate-shape traffic on multiple cirteria. For instance, we
may want to support shaping of per-host header requests, as well as per source.
In order to achieve this, we will use checkpoints, one per criterion. Each of
these checkpoints will consist in a test, a rate counter and a queue.
A request reaches the checkpoint and checks the counter. If the counter is
below the limit, it is updated and the request continues. If the limit is
reached, the request attaches itself into the queue and sleeps. The sleep time
is computed from the queue status, and updates the queue status.
A task is dedicated to each queue. Its sole purpose is to be woken up when the
next task may wake up, to check the frequency counter, wake as many requests as
possible and update the counter. All the woken up requests are detached from
the queue. Maybe the task dedicated to the queue can be avoided and replaced
with all queued tasks's sleep counters, though this looks tricky. Or maybe it's
just the first request in the queue that should be responsible for waking up
other tasks, and not to forget to pass on this responsibility to next tasks if
it leaves the queue.
The woken up request then goes on evaluating other criteria and possibly sleeps
again on another one. In the end, the task will have waited the amount of time
required to pass all checkpoints, and all checkpoints will be able to maintain
a permanent load of exactly their limit if enough streams flow through them.
Since a request can only sleep in one queue at a time, it makes sense to use a
linked list element in each session to attach it to any queue. It could very
well be shared with the pendconn hooks which could then be part of the session.
This mechanism could be used to rate-shape sessions and requests per backend
and per server.
When rate-shaping on dynamic criteria, such as the source IP address, we have
to first extract the data pattern, then look it up in a table very similar to
the stickiness tables, but with a frequency counter. At the checkpoint, the
pattern is looked up, the entry created or refreshed, and the frequency counter
updated and checked. Then the request either goes on or sleeps as described
above, but if it sleeps, it's still in the checkpoint's queue, but with a date
computed from the criterion's status.
This means that we need 3 distinct features :
- optional pattern extraction
- per-pattern or per-queue frequency counter
- time-ordered queue with a task
Based on past experiences with frequency counters, it does not appear very easy
to exactly compute sleep delays in advance for multiple requests. So most
likely we'll have to run per-criterion queues too, with only the head of the
queue holding a wake-up timeout.
This finally leads us to the following :
- optional pattern extraction
- per-pattern or per-queue frequency counter
- per-frequency counter queue
- head of the queue serves as a global queue timer.
This brings us to a very flexible architecture :
- 1 list of rule-based checkpoints per frontend
- 1 list of rule-based checkpoints per backend
- 1 list of rule-based checkpoints per server
Each of these lists have a lot of rules conditioned by ACLs, just like the
use-backend rules, except that all rules are evaluated in turn.
Since we might sometimes just want to enable that without setting any limit and
just for enabling control in ACLs (or logging ?), we should probably try to
find a flexible way of declaring just a counter without a queue.
These checkpoints could be of two types :
- rate-limit (described here)
- concurrency-limit (very similar with the counter and no timer). This
feature would require to keep track of all accounted criteria in a
request so that they can be released upon request completion.
It should be possible to define a max of requests in the queue, above which a
503 is returned. The same applies for the max delay in the queue. We could have
it per-task (currently it's the connection timeout) and abort tasks with a 503
when the delay is exceeded.
Per-server connection concurrency could be converted to use this mechanism
which is very similar.
The construct should be flexible enough so that the counters may be checked
from ACLs. That would allow to reject connections or switch to an alternate
backend when some limits are reached.

13
doc/design-thoughts/sess_par_sec.txt
View File

@ -1,13 +0,0 @@
Graphe des nombres de traitements par seconde unité de temps avec
- un algo linéaire et très peu coûteux unitairement (0.01 ut)
- un algo en log(2) et 5 fois plus coûteux (0.05 ut)
set yrange [0:1]
plot [0:1000] 1/(1+0.01*x), 1/(1+0.05*log(x+1)/log(2))
Graphe de la latence induite par ces traitements en unités de temps :
set yrange [0:1000]
plot [0:1000] x/(1+0.01*x), x/(1+0.05*log(x+1)/log(2))

193
doc/internals/entities-v2.txt
View File

@ -1,193 +0,0 @@
An FD has a state :
- CLOSED
- READY
- ERROR (?)
- LISTEN (?)
A connection has a state :
- CLOSED
- ACCEPTED
- CONNECTING
- ESTABLISHED
- ERROR
A stream interface has a state :
- INI, REQ, QUE, TAR, ASS, CON, CER, EST, DIS, CLO
Note that CON and CER might be replaced by EST if the connection state is used
instead. CON might even be more suited than EST to indicate that a connection
is known.
si_shutw() must do :
data_shutw()
if (shutr) {
data_close()
ctrl_shutw()
ctrl_close()
}
si_shutr() must do :
data_shutr()
if (shutw) {
data_close()
ctrl_shutr()
ctrl_close()
}
Each of these steps may fail, in which case the step must be retained and the
operations postponed in an asynchronous task.
The first asynchronous data_shut() might already fail so it is mandatory to
save the other side's status with the connection in order to let the async task
know whether the 3 next steps must be performed.
The connection (or perhaps the FD) needs to know :
- the desired close operations : DSHR, DSHW, CSHR, CSHW
- the completed close operations : DSHR, DSHW, CSHR, CSHW
On the accept() side, we probably need to know :
- if a header is expected (eg: accept-proxy)
- if this header is still being waited for
=> maybe both info might be combined into one bit
- if a data-layer accept() is expected
- if a data-layer accept() has been started
- if a data-layer accept() has been performed
=> possibly 2 bits, to indicate the need to free()
On the connect() side, we need to know :
- the desire to send a header (eg: send-proxy)
- if this header has been sent
=> maybe both info might be combined
- if a data-layer connect() is expected
- if a data-layer connect() has been started
- if a data-layer connect() has been completed
=> possibly 2 bits, to indicate the need to free()
On the response side, we also need to know :
- the desire to send a header (eg: health check response for monitor-net)
- if this header was sent
=> might be the same as sending a header over a new connection
Note: monitor-net has precedence over proxy proto and data layers. Same for
health mode.
For multi-step operations, use 2 bits :
00 = operation not desired, not performed
10 = operation desired, not started
11 = operation desired, started but not completed
01 = operation desired, started and completed
=> X != 00 ==> operation desired
X & 01 ==> operation at least started
X & 10 ==> operation not completed
Note: no way to store status information for error reporting.
Note2: it would be nice if "tcp-request connection" rules could work at the
connection level, just after headers ! This means support for tracking stick
tables, possibly not too much complicated.
Proposal for incoming connection sequence :
- accept()
- if monitor-net matches or if mode health => try to send response
- if accept-proxy, wait for proxy request
- if tcp-request connection, process tcp rules and possibly keep the
pointer to stick-table
- if SSL is enabled, switch to SSL handshake
- then switch to DATA state and instantiate a session
We just need a map of handshake handlers on the connection. They all manage the
FD status themselves and set the callbacks themselves. If their work succeeds,
they remove themselves from the list. If it fails, they remain subscribed and
enable the required polling until they are woken up again or the timeout strikes.
Identified handshake handlers for incoming connections :
- HH_HEALTH (tries to send OK and dies)
- HH_MONITOR_IN (matches src IP and adds/removes HH_SEND_OK/HH_SEND_HTTP_OK)
- HH_SEND_OK (tries to send "OK" and dies)
- HH_SEND_HTTP_OK (tries to send "HTTP/1.0 200 OK" and dies)
- HH_ACCEPT_PROXY (waits for PROXY line and parses it)
- HH_TCP_RULES (processes TCP rules)
- HH_SSL_HS (starts SSL handshake)
- HH_ACCEPT_SESSION (instantiates a session)
Identified handshake handlers for outgoing connections :
- HH_SEND_PROXY (tries to build and send the PROXY line)
- HH_SSL_HS (starts SSL handshake)
For the pollers, we could check that handshake handlers are not 0 and decide to
call a generic connection handshake handler instead of usual callbacks. Problem
is that pollers don't know connections, they know fds. So entities which manage
handlers should update change the FD callbacks accordingly.
With a bit of care, we could have :
- HH_SEND_LAST_CHUNK (sends the chunk pointed to by a pointer and dies)
=> merges HEALTH, SEND_OK and SEND_HTTP_OK
It sounds like the ctrl vs data state for the connection are per-direction
(eg: support an async ctrl shutw while still reading data).
Also support shutr/shutw status at L4/L7.
In practice, what we really need is :
shutdown(conn) =
conn.data.shut()
conn.ctrl.shut()
conn.fd.shut()
close(conn) =
conn.data.close()
conn.ctrl.close()
conn.fd.close()
With SSL over Remote TCP (RTCP + RSSL) to reach the server, we would have :
HTTP -> RTCP+RSSL connection <-> RTCP+RRAW connection -> TCP+SSL connection
The connection has to be closed at 3 places after a successful response :
- DATA (RSSL over RTCP)
- CTRL (RTCP to close connection to server)
- SOCK (FD to close connection to second process)
Externally, the connection is seen with very few flags :
- SHR
- SHW
- ERR
We don't need a CLOSED flag as a connection must always be detached when it's closed.
The internal status doesn't need to be exposed :
- FD allocated (Y/N)
- CTRL initialized (Y/N)
- CTRL connected (Y/N)
- CTRL handlers done (Y/N)
- CTRL failed (Y/N)
- CTRL shutr (Y/N)
- CTRL shutw (Y/N)
- DATA initialized (Y/N)
- DATA connected (Y/N)
- DATA handlers done (Y/N)
- DATA failed (Y/N)
- DATA shutr (Y/N)
- DATA shutw (Y/N)
(note that having flags for operations needing to be completed might be easier)
--------------
Maybe we need to be able to call conn->fdset() and conn->fdclr() but it sounds
very unlikely since the only functions manipulating this are in the code of
the data/ctrl handlers.
FDSET/FDCLR cannot be directly controlled by the stream interface since it also
depends on the DATA layer (WANT_READ/wANT_WRITE).
But FDSET/FDCLR is probably controlled by who owns the connection (eg: DATA).

96
doc/internals/entities.txt
View File

@ -1,96 +0,0 @@
2011/02/25 - Description of the different entities in haproxy - w@1wt.eu
1) Definitions
--------------
Listener
--------
A listener is the entity which is part of a frontend and which accepts
connections. There are as many listeners as there are ip:port couples.
There is at least one listener instantiated for each "bind" entry, and
port ranges will lead to as many listeners as there are ports in the
range. A listener just has a listening file descriptor ready to accept
incoming connections and to dispatch them to upper layers.
Initiator
---------
An initiator is instantiated for each incoming connection on a listener. It may
also be instantiated by a task pretending to be a client. An initiator calls
the next stage's accept() callback to present it with the parameters of the
incoming connection.
Session
-------
A session is the only entity located between an initiator and a connector.
This is the last stage which offers an accept() callback, and all of its
processing will continue with the next stage's connect() callback. It holds
the buffers needed to forward the protocol data between each side. This entity
sees the native protocol, and is able to call analysers on these buffers. As it
is used in both directions, it always has two buffers.
When transformations are required, some of them may be done on the initiator
side and other ones on the connector side. If additional buffers are needed for
such transforms, those buffers cannot replace the session's buffers, but they
may complete them.
A session only needs to be instantiated when forwarding of data is required
between two sides. Accepting and filtering on layer 4 information only does not
require a session.
For instance, let's consider the case of a proxy which receives and decodes
HTTPS traffic, processes it as HTTP and recodes it as HTTPS before forwarding
it. We'd have 3 layers of buffers, where the middle ones are used for
forwarding of the protocol data (HTTP here) :
<-- ssl dec --> <-forwarding-> <-- ssl enc -->
,->[||||]--. ,->[||||]--. ,->[||||]--.
client (|) (|) (|) (|) server
^--[||||]<-' ^--[||||]<-' ^--[||||]<-'
HTTPS HTTP HTTPS
The session handling code is only responsible for monitoring the forwarding
buffers here. It may declare the end of the session once those buffers are
closed and no analyser wants to re-open them. The session is also the entity
which applies the load balancing algorithm and decides the server to use.
The other sides are responsible for propagating the state up to the session
which takes decisions.
Connector
---------
A connector is the entity which permits to instantiate a connection to a known
destination. It presents a connect() callback, and as such appears on the right
side of diagrams.
Connection
----------
A connection is the entity instantiated by a connector. It may be composed of
multiple stages linked together. Generally it is the part of the stream
interface holding a file descriptor, but it can also be a processing block or a
transformation block terminated by a connection. A connection presents a
server-side interface.
2) Sequencing
-------------
Upon startup, listeners are instantiated by the configuration. When an incoming
connection reaches a listening file descriptor, its read() callback calls the
corresponding listener's accept() function which instantiates an initiator and
in turn recursively calls upper layers' accept() callbacks until
accept_session() is called. accept_session() instantiates a new session which
starts protocol analysis via process_session(). When all protocol analysis is
done, process_session() calls the connect() callback of the connector in order
to get a connection.

92
doc/internals/header-parser-speed.txt
View File

@ -1,92 +0,0 @@
TEST 3:
printf "GET /\r\nbla: truc\r\n\r\n"
NO SPEEDUP :
WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071080, r=0x8071094
WHL: hdr_st=0x01, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071080, r=0x8071094
WHL: hdr_st=0x32, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071086, r=0x8071094
WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071087, r=0x8071094
WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071091, r=0x8071094
WHL: hdr_st=0x03, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8071092, lr=0x8071092, r=0x8071094
WHL: hdr_st=0x34, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8071092, lr=0x8071093, r=0x8071094
WHL: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8071092, lr=0x8071093, r=0x8071094
END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8071092, lr=0x8071094, r=0x8071094
=> 9 trans
FULL SPEEDUP :
WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x806a770, lr=0x806a770, r=0x806a784
WHL: hdr_st=0x32, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x806a770, lr=0x806a776, r=0x806a784
WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806a777, lr=0x806a777, r=0x806a784
WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806a777, lr=0x806a781, r=0x806a784
WHL: hdr_st=0x26, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x806a782, lr=0x806a783, r=0x806a784
END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x806a782, lr=0x806a784, r=0x806a784
=> 6 trans
TEST 4:
printf "GET /\nbla: truc\n\n"
NO SPEEDUP :
WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x80750d0, lr=0x80750d0, r=0x80750e1
WHL: hdr_st=0x01, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x80750d0, lr=0x80750d0, r=0x80750e1
WHL: hdr_st=0x02, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x80750d0, lr=0x80750d5, r=0x80750e1
WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x80750d6, lr=0x80750d6, r=0x80750e1
WHL: hdr_st=0x04, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x80750d6, lr=0x80750df, r=0x80750e1
WHL: hdr_st=0x03, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x80750e0, lr=0x80750e0, r=0x80750e1
WHL: hdr_st=0x04, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x80750e0, lr=0x80750e0, r=0x80750e1
WHL: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x80750e0, lr=0x80750e0, r=0x80750e1
END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x80750e0, lr=0x80750e1, r=0x80750e1
=> 9 trans
FULL SPEEDUP :
WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8072010, lr=0x8072010, r=0x8072021
WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8072016, lr=0x8072016, r=0x8072021
END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x8072020, lr=0x8072021, r=0x8072021
=> 3 trans
TEST 5:
printf "GET /\r\nbla: truc\r\n truc2\r\n\r\n"
NO SPEEDUP :
WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071080, r=0x807109d
WHL: hdr_st=0x01, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071080, r=0x807109d
WHL: hdr_st=0x32, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x8071080, lr=0x8071086, r=0x807109d
WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071087, r=0x807109d
WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071091, r=0x807109d
WHL: hdr_st=0x05, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071092, r=0x807109d
WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x8071094, r=0x807109d
WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x8071087, lr=0x807109a, r=0x807109d
WHL: hdr_st=0x03, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x807109b, lr=0x807109b, r=0x807109d
WHL: hdr_st=0x34, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x807109b, lr=0x807109c, r=0x807109d
WHL: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x807109b, lr=0x807109c, r=0x807109d
END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x807109b, lr=0x807109d, r=0x807109d
=> 12 trans
FULL SPEEDUP :
WHL: hdr_st=0x00, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x806dfc0, lr=0x806dfc0, r=0x806dfdd
WHL: hdr_st=0x32, hdr_used=1 hdr_tail=0 hdr_last=1, h=0x806dfc0, lr=0x806dfc6, r=0x806dfdd
WHL: hdr_st=0x03, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806dfc7, lr=0x806dfc7, r=0x806dfdd
WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806dfc7, lr=0x806dfd1, r=0x806dfdd
WHL: hdr_st=0x34, hdr_used=2 hdr_tail=1 hdr_last=2, h=0x806dfc7, lr=0x806dfda, r=0x806dfdd
WHL: hdr_st=0x26, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x806dfdb, lr=0x806dfdc, r=0x806dfdd
END: hdr_st=0x06, hdr_used=3 hdr_tail=2 hdr_last=3, h=0x806dfdb, lr=0x806dfdd, r=0x806dfdd
=> 7 trans

124
doc/internals/header-tree.txt
View File

@ -1,124 +0,0 @@
2007/03/30 - Header storage in trees
This documentation describes how to store headers in radix trees, providing
fast access to any known position, while retaining the ability to grow/reduce
any arbitrary header without having to recompute all positions.
Principle :
We have a radix tree represented in an integer array, which represents the
total number of bytes used by all headers whose position is below it. This
ensures that we can compute any header's position in O(log(N)) where N is
the number of headers.
Example with N=16 :
+-----------------------+
| |
+-----------+ +-----------+
| | | |
+-----+ +-----+ +-----+ +-----+
| | | | | | | |
+--+ +--+ +--+ +--+ +--+ +--+ +--+ +--+
| | | | | | | | | | | | | | | |
0 1 2 3 4 5 6 7 8 9 A B C D E F
To reach header 6, we have to compute hdr[0]+hdr[4]+hdr[6]
With this method, it becomes easy to grow any header and update the array.
To achieve this, we have to replace one after the other all bits on the
right with one 1 followed by zeroes, and update the position if it's higher
than current position, and stop when it's above number of stored headers.
For instance, if we want to grow hdr[6], we proceed like this :
6 = 0110 (BIN)
Let's consider the values to update :
(bit 0) : (0110 & ~0001) | 0001 = 0111 = 7 > 6 => update
(bit 1) : (0110 & ~0011) | 0010 = 0110 = 6 <= 6 => leave it
(bit 2) : (0110 & ~0111) | 0100 = 0100 = 4 <= 6 => leave it
(bit 4) : (0110 & ~1111) | 1000 = 1000 = 8 > 6 => update
(bit 5) : larger than array size, stop.
It's easy to walk through the tree too. We only have one iteration per bit
changing from X to the ancestor, and one per bit from the ancestor to Y.
The ancestor is found while walking. To go from X to Y :
pos = pos(X)
while (Y != X) {
if (Y > X) {
// walk from Y to ancestor
pos += hdr[Y]
Y &= (Y - 1)
} else {
// walk from X to ancestor
pos -= hdr[X]
X &= (X - 1)
}
}
However, it is not trivial anymore to linearly walk the tree. We have to move
from a known place to another known place, but a jump to next entry costs the
same as a jump to a random place.
Other caveats :
- it is not possible to remove a header, it is only possible to empty it.
- it is not possible to insert a header, as that would imply a renumbering.
=> this means that a "defrag" function is required. Headers should preferably
be added, then should be stuffed on top of destroyed ones, then only
inserted if absolutely required.
When we have this, we can then focus on a 32-bit header descriptor which would
look like this :
{
unsigned line_len :13; /* total line length, including CRLF */
unsigned name_len :6; /* header name length, max 63 chars */
unsigned sp1 :5; /* max spaces before value : 31 */
unsigned sp2 :8; /* max spaces after value : 255 */
}
Example :
Connection: close \r\n
<---------+-----+-----+-------------> line_len
<-------->| | | name_len
<-----> | sp1
<-------------> sp2
Rem:
- if there are more than 31 spaces before the value, the buffer will have to
be moved before being registered
- if there are more than 255 spaces after the value, the buffer will have to
be moved before being registered
- we can use the empty header name as an indicator for a deleted header
- it would be wise to format a new request before sending lots of random
spaces to the servers.
- normal clients do not send such crap, so those operations *may* reasonably
be more expensive than the rest provided that other ones are very fast.
It would be handy to have the following macros :
hdr_eon(hdr) => end of name
hdr_sov(hdr) => start of value
hdr_eof(hdr) => end of value
hdr_vlen(hdr) => length of value
hdr_hlen(hdr) => total header length
A 48-bit encoding would look like this :
Connection: close \r\n
<---------+------+---+--------------> eoh = 16 bits
<-------->| | | eon = 8 bits
<--------------->| | sov = 8 bits
<---> vlen = 16 bits

45
doc/internals/http-cookies.txt
View File

@ -1,45 +0,0 @@
2010/08/31 - HTTP Cookies - Theory and reality
HTTP cookies are not uniformly supported across browsers, which makes it very
hard to build a widely compatible implementation. At least four conflicting
documents exist to describe how cookies should be handled, and browsers
generally don't respect any but a sensibly selected mix of them :
- Netscape's original spec (also mirrored at Curl's site among others) :
http://web.archive.org/web/20070805052634/http://wp.netscape.com/newsref/std/cookie_spec.html
http://curl.haxx.se/rfc/cookie_spec.html
Issues: uses an unquoted "Expires" field that includes a comma.
- RFC 2109 :
http://www.ietf.org/rfc/rfc2109.txt
Issues: specifies use of "Max-Age" (not universally implemented) and does
not talk about "Expires" (generally supported). References quoted
strings, not generally supported (eg: MSIE). Stricter than browsers
about domains. Ambiguous about allowed spaces in values and attrs.
- RFC 2965 :
http://www.ietf.org/rfc/rfc2965.txt
Issues: same as RFC2109 + describes Set-Cookie2 which only Opera supports.
- Current internet draft :
https://datatracker.ietf.org/wg/httpstate/charter/
Issues: as of -p10, does not explain how the Set-Cookie2 header must be
emitted/handled, while suggesting a stricter approach for Cookie.
Documents reality and as such reintroduces the widely used unquoted
"Expires" attribute with its error-prone syntax. States that a
server should not emit more than one cookie per Set-Cookie header,
which is incompatible with HTTP which says that multiple headers
are allowed only if they can be folded.
See also the following URL for a browser * feature matrix :
http://code.google.com/p/browsersec/wiki/Part2#Same-origin_policy_for_cookies
In short, MSIE and Safari neither support quoted strings nor max-age, which
make it mandatory to continue to send an unquoted Expires value (maybe the
day of week could be omitted though). Only Safari supports comma-separated
lists of Set-Cookie headers. Support for cross-domains is not uniform either.

5
doc/internals/http-docs.txt
View File

@ -1,5 +0,0 @@
Many interesting RFC and drafts linked to from this site :
http://www.web-cache.com/Writings/protocols-standards.html

335
doc/internals/http-parsing.txt
View File

@ -1,335 +0,0 @@
--- Relevant portions of RFC2616 ---
OCTET = <any 8-bit sequence of data>
CHAR = <any US-ASCII character (octets 0 - 127)>
UPALPHA = <any US-ASCII uppercase letter "A".."Z">
LOALPHA = <any US-ASCII lowercase letter "a".."z">
ALPHA = UPALPHA | LOALPHA
DIGIT = <any US-ASCII digit "0".."9">
CTL = <any US-ASCII control character (octets 0 - 31) and DEL (127)>
CR = <US-ASCII CR, carriage return (13)>
LF = <US-ASCII LF, linefeed (10)>
SP = <US-ASCII SP, space (32)>
HT = <US-ASCII HT, horizontal-tab (9)>
<"> = <US-ASCII double-quote mark (34)>
CRLF = CR LF
LWS = [CRLF] 1*( SP | HT )
TEXT = <any OCTET except CTLs, but including LWS>
HEX = "A" | "B" | "C" | "D" | "E" | "F"
| "a" | "b" | "c" | "d" | "e" | "f" | DIGIT
separators = "(" | ")" | "<" | ">" | "@"
| "," | ";" | ":" | "\" | <">
| "/" | "[" | "]" | "?" | "="
| "{" | "}" | SP | HT
token = 1*<any CHAR except CTLs or separators>
quoted-pair = "\" CHAR
ctext = <any TEXT excluding "(" and ")">
qdtext = <any TEXT except <">>
quoted-string = ( <"> *(qdtext | quoted-pair ) <"> )
comment = "(" *( ctext | quoted-pair | comment ) ")"
4 HTTP Message
4.1 Message Types
HTTP messages consist of requests from client to server and responses from
server to client. Request (section 5) and Response (section 6) messages use the
generic message format of RFC 822 [9] for transferring entities (the payload of
the message). Both types of message consist of :
- a start-line
- zero or more header fields (also known as "headers")
- an empty line (i.e., a line with nothing preceding the CRLF) indicating the
end of the header fields
- and possibly a message-body.
HTTP-message = Request | Response
start-line = Request-Line | Status-Line
generic-message = start-line
*(message-header CRLF)
CRLF
[ message-body ]
In the interest of robustness, servers SHOULD ignore any empty line(s) received
where a Request-Line is expected. In other words, if the server is reading the
protocol stream at the beginning of a message and receives a CRLF first, it
should ignore the CRLF.
4.2 Message headers
- Each header field consists of a name followed by a colon (":") and the field
value.
- Field names are case-insensitive.
- The field value MAY be preceded by any amount of LWS, though a single SP is
preferred.
- Header fields can be extended over multiple lines by preceding each extra
line with at least one SP or HT.
message-header = field-name ":" [ field-value ]
field-name = token
field-value = *( field-content | LWS )
field-content = <the OCTETs making up the field-value and consisting of
either *TEXT or combinations of token, separators, and
quoted-string>
The field-content does not include any leading or trailing LWS occurring before
the first non-whitespace character of the field-value or after the last
non-whitespace character of the field-value. Such leading or trailing LWS MAY
be removed without changing the semantics of the field value. Any LWS that
occurs between field-content MAY be replaced with a single SP before
interpreting the field value or forwarding the message downstream.
=> format des headers = 1*(CHAR & !ctl & !sep) ":" *(OCTET & (!ctl | LWS))
=> les regex de matching de headers s'appliquent sur field-content, et peuvent
utiliser field-value comme espace de travail (mais de préférence après le
premier SP).
(19.3) The line terminator for message-header fields is the sequence CRLF.
However, we recommend that applications, when parsing such headers, recognize
a single LF as a line terminator and ignore the leading CR.
message-body = entity-body
| <entity-body encoded as per Transfer-Encoding>
5 Request
Request = Request-Line
*(( general-header
| request-header
| entity-header ) CRLF)
CRLF
[ message-body ]
5.1 Request line
The elements are separated by SP characters. No CR or LF is allowed except in
the final CRLF sequence.
Request-Line = Method SP Request-URI SP HTTP-Version CRLF
(19.3) Clients SHOULD be tolerant in parsing the Status-Line and servers
tolerant when parsing the Request-Line. In particular, they SHOULD accept any
amount of SP or HT characters between fields, even though only a single SP is
required.
4.5 General headers
Apply to MESSAGE.
general-header = Cache-Control
| Connection
| Date
| Pragma
| Trailer
| Transfer-Encoding
| Upgrade
| Via
| Warning
General-header field names can be extended reliably only in combination with a
change in the protocol version. However, new or experimental header fields may
be given the semantics of general header fields if all parties in the
communication recognize them to be general-header fields. Unrecognized header
fields are treated as entity-header fields.
5.3 Request Header Fields
The request-header fields allow the client to pass additional information about
the request, and about the client itself, to the server. These fields act as
request modifiers, with semantics equivalent to the parameters on a programming
language method invocation.
request-header = Accept
| Accept-Charset
| Accept-Encoding
| Accept-Language
| Authorization
| Expect
| From
| Host
| If-Match
| If-Modified-Since
| If-None-Match
| If-Range
| If-Unmodified-Since
| Max-Forwards
| Proxy-Authorization
| Range
| Referer
| TE
| User-Agent
Request-header field names can be extended reliably only in combination with a
change in the protocol version. However, new or experimental header fields MAY
be given the semantics of request-header fields if all parties in the
communication recognize them to be request-header fields. Unrecognized header
fields are treated as entity-header fields.
7.1 Entity header fields
Entity-header fields define metainformation about the entity-body or, if no
body is present, about the resource identified by the request. Some of this
metainformation is OPTIONAL; some might be REQUIRED by portions of this
specification.
entity-header = Allow
| Content-Encoding
| Content-Language
| Content-Length
| Content-Location
| Content-MD5
| Content-Range
| Content-Type
| Expires
| Last-Modified
| extension-header
extension-header = message-header
The extension-header mechanism allows additional entity-header fields to be
defined without changing the protocol, but these fields cannot be assumed to be
recognizable by the recipient. Unrecognized header fields SHOULD be ignored by
the recipient and MUST be forwarded by transparent proxies.
----------------------------------
The format of Request-URI is defined by RFC3986 :
URI = scheme ":" hier-part [ "?" query ] [ "#" fragment ]
hier-part = "//" authority path-abempty
/ path-absolute
/ path-rootless
/ path-empty
URI-reference = URI / relative-ref
absolute-URI = scheme ":" hier-part [ "?" query ]
relative-ref = relative-part [ "?" query ] [ "#" fragment ]
relative-part = "//" authority path-abempty
/ path-absolute
/ path-noscheme
/ path-empty
scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )
authority = [ userinfo "@" ] host [ ":" port ]
userinfo = *( unreserved / pct-encoded / sub-delims / ":" )
host = IP-literal / IPv4address / reg-name
port = *DIGIT
IP-literal = "[" ( IPv6address / IPvFuture ) "]"
IPvFuture = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" )
IPv6address = 6( h16 ":" ) ls32
/ "::" 5( h16 ":" ) ls32
/ [ h16 ] "::" 4( h16 ":" ) ls32
/ [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32
/ [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
/ [ *3( h16 ":" ) h16 ] "::" h16 ":" ls32
/ [ *4( h16 ":" ) h16 ] "::" ls32
/ [ *5( h16 ":" ) h16 ] "::" h16
/ [ *6( h16 ":" ) h16 ] "::"
h16 = 1*4HEXDIG
ls32 = ( h16 ":" h16 ) / IPv4address
IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet
dec-octet = DIGIT ; 0-9
/ %x31-39 DIGIT ; 10-99
/ "1" 2DIGIT ; 100-199
/ "2" %x30-34 DIGIT ; 200-249
/ "25" %x30-35 ; 250-255
reg-name = *( unreserved / pct-encoded / sub-delims )
path = path-abempty ; begins with "/" or is empty
/ path-absolute ; begins with "/" but not "//"
/ path-noscheme ; begins with a non-colon segment
/ path-rootless ; begins with a segment
/ path-empty ; zero characters
path-abempty = *( "/" segment )
path-absolute = "/" [ segment-nz *( "/" segment ) ]
path-noscheme = segment-nz-nc *( "/" segment )
path-rootless = segment-nz *( "/" segment )
path-empty = 0<pchar>
segment = *pchar
segment-nz = 1*pchar
segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims / "@" )
; non-zero-length segment without any colon ":"
pchar = unreserved / pct-encoded / sub-delims / ":" / "@"
query = *( pchar / "/" / "?" )
fragment = *( pchar / "/" / "?" )
pct-encoded = "%" HEXDIG HEXDIG
unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"
reserved = gen-delims / sub-delims
gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
/ "*" / "+" / "," / ";" / "="
=> so the list of allowed characters in a URI is :
uri-char = unreserved / gen-delims / sub-delims / "%"
= ALPHA / DIGIT / "-" / "." / "_" / "~"
/ ":" / "/" / "?" / "#" / "[" / "]" / "@"
/ "!" / "$" / "&" / "'" / "(" / ")" /
/ "*" / "+" / "," / ";" / "=" / "%"
Note that non-ascii characters are forbidden ! Spaces and CTL are forbidden.
Unfortunately, some products such as Apache allow such characters :-/
---- The correct way to do it ----
- one http_session
It is basically any transport session on which we talk HTTP. It may be TCP,
SSL over TCP, etc... It knows a way to talk to the client, either the socket
file descriptor or a direct access to the client-side buffer. It should hold
information about the last accessed server so that we can guarantee that the
same server can be used during a whole session if needed. A first version
without optimal support for HTTP pipelining will have the client buffers tied
to the http_session. It may be possible that it is not sufficient for full
pipelining, but this will need further study. The link from the buffers to
the backend should be managed by the http transaction (http_txn), provided
that they are serialized. Each http_session, has 0 to N http_txn. Each
http_txn belongs to one and only one http_session.
- each http_txn has 1 request message (http_req), and 0 or 1 response message
(http_rtr). Each of them has 1 and only one http_txn. An http_txn holds
information such as the HTTP method, the URI, the HTTP version, the
transfer-encoding, the HTTP status, the authorization, the req and rtr
content-length, the timers, logs, etc... The backend and server which process
the request are also known from the http_txn.
- both request and response messages hold header and parsing information, such
as the parsing state, start of headers, start of message, captures, etc...

54
doc/internals/naming.txt
View File

@ -1,54 +0,0 @@
Naming rules for manipulated objects and structures.
Previously, there were ambiguities between sessions, transactions and requests,
as well as in the way responses are noted ("resp", "rep", "rsp").
Here is a proposal for a better naming scheme.
The "session" is above the transport level, which means at ISO layer 5.
We can talk about "http sessions" when we consider the entity which lives
between the accept() and the close(), or the connect() and the close().
=> This demonstrates that it is not possible to have the same http session from
the client to the server.
A session can carry one or multiple "transactions", which are each composed of
one "request" and zero or one "response". Both "request" and "response" are
described in RFC2616 as "HTTP messages". RFC2616 also seldom references the
word "transaction" without explicitly defining it.
An "HTTP message" is composed of a "start line" which can be either a
"request line" or a "status line", followed by a number of "message headers"
which can be either "request headers" or "response headers", and an "entity",
itself composed of "entity headers" and an "entity body".Most probably,
"message headers" and "entity headers" will always be processed together as
"headers", while the "entity body" will design the payload.
We must try to always use the same abbreviations when naming objects. Here are
a few common ones :
- txn : transaction
- req : request
- rtr : response to request
- msg : message
- hdr : header
- ent : entity
- bdy : body
- sts : status
- stt : state
- idx : index
- cli : client
- srv : server
- svc : service
- ses : session
- tsk : task
Short names for unions or cascaded structs :
- sl : start line
- sl.rq : request line
- sl.st : status line
- cl : client
- px : proxy
- sv : server
- st : state / status

20
doc/internals/repartition-be-fe-fi.txt
View File

@ -1,20 +0,0 @@
- session : ajouter ->fiprm et ->beprm comme raccourcis
- px->maxconn: ne s'applique qu'au FE. Pour le BE, on utilise fullconn,
initialisé par défaut à la même chose que maxconn.
\ from: proxy session server actuellement
field \
rules px->fiprm sess->fiprm -
srv,cookies px->beprm sess->beprm srv->px
options(log) px-> sess->fe -
options(fe) px-> sess->fe -
options(be) px->beprm sess->beprm srv->px
captures px-> sess->fe - ->fiprm
logs px-> sess->fe srv->px
errorloc px-> sess->beprm|fe -
maxconn px-> sess->fe - ->be
fullconn px-> sess->beprm srv->px -