Changes between Initial Version and Version 1 of ApplicationDeployment


Ignore:
Timestamp:
08/11/2012 02:12:56 AM (5 years ago)
Author:
apoirier
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ApplicationDeployment

    v1 v1  
     1{{{ 
     2#!rst 
     3Deploying Nagare applications in a production environment 
     4========================================================= 
     5 
     6Serving a Nagare application in a development setup is easy. You just have to launch 
     7the following command: 
     8 
     9.. code-block:: sh 
     10 
     11    nagare-admin serve [options] <app_name> 
     12 
     13It starts a standalone threaded server which stores the sessions data in memory. 
     14 
     15This standalone server is very convenient when developing Nagare applications, especially when 
     16using the ``--reload`` and ``--debug`` options that respectively restart the server when source 
     17files are changed and show the exceptions stacktraces as Web pages instead of in the console 
     18shell. 
     19 
     20However, this standalone mode is not suitable in a production environment, because it is not 
     21efficient nor fail-safe: 
     22 
     23  - Since it's a threaded program, it does not fully utilize the CPU cores due to the GIL (see `this 
     24    discussion <http://stackoverflow.com/questions/4496680/python-threads-all-executing-on-a-single-core>`_). 
     25    To solve this problem, we need to use multiprocessing. 
     26  - Static files are served by the standalone server and are not cached. They would be better served 
     27    by a faster dedicated Web server such as Apache, Nginx, Lighttpd, ... 
     28  - The server processes are not monitored, and they don't cope with failures very well. For example, in a 
     29    production setup, it's important to detect zombie processes and restart them to prevent attacks or to limit 
     30    the impact of unnoticed programmation errors in the application. Of course, you should also use logging 
     31    to trace the origin of the problems in order to solve them. 
     32 
     33So, in a production environment, the recommended way to serve Nagare applications 
     34is through *FastCGI* associated to a frontend Web server such as Apache, Nginx or 
     35Lighttpd.  We are now going to describe how to do that. 
     36 
     37 
     38Serving an application through FastCGI 
     39-------------------------------------- 
     40 
     41In a FastCGI setup, a Web server such as `Apache`_, `Nginx`_ or `Lighttpd`_ is needed. It will serve 
     42the static contents of the application, such as images, CSS and javascript files, and pass the other requests 
     43(i.e. those with a "dynamic" nature) to the Nagare application through the `FastCGI`_ protocol. 
     44 
     45FastCGI is a variation of the CGI protocol, which defines an interface layer between a Web server and external 
     46applications (such as shell or python scripts). The main difference between FastCGI and CGI is that FastCGI 
     47doesn't create a new worker process at the start of each request but use a pool of processes that are created 
     48when the FastCGI service starts, and uses sockets for exchanging informations (i.e. request and response data) 
     49between the Web server and the external applications. 
     50 
     51This setup is particularly efficient when serving Nagare applications because 
     52the pool of processes fully utilize the CPU cores. Furthermore, Web servers are 
     53far more efficient than the standalone threaded server when serving the static 
     54contents of the application since they cache the data and they are mostly written 
     55in C. 
     56 
     57 
     58The publisher configuration 
     59~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
     60 
     61In order to serve a Nagare application through FastCGI, we must first tell the ``nagare-admin serve`` utility 
     62that we don't want to use the standalone server but FastCGI instead. This is done by creating a publisher 
     63configuration file (see :wiki:`PublisherConfiguration`) and passing it to the ``nagare-admin serve`` command 
     64with the ``-c`` option. 
     65 
     66Here is a typical publisher configuration file to serve an application through FastCGI: 
     67 
     68.. code-block:: ini 
     69 
     70    [publisher] 
     71    type = fastcgi 
     72    host = 127.0.0.1 
     73    port = 9000 
     74 
     75    [sessions] 
     76    type = memcache 
     77    host = 127.0.0.1 
     78    port = 11211 
     79 
     80We use the ``fastcgi`` backend for the publisher (with the default settings) and the ``memcache`` backend for 
     81the sessions management. 
     82 
     83Then, the FastCGI server can be launched by running this command: 
     84 
     85.. code-block:: sh 
     86 
     87    nagare-admin serve -c publisher.conf <app_name> 
     88 
     89By default, the FastCGI server forks 5 worker processes at startup to handle the incoming requests received on 
     90port 9000. However, the number of worker processes can grow or shrink depending on the workload. 
     91 
     92.. warning:: 
     93 
     94  When an application is served through FastCGI, you must use a ``memcache`` 
     95  backend for storing the sessions data. The default ``standalone`` session 
     96  backend will not work with FastCGI because the session data is stored 
     97  in memory, and since the processes spawned by the FastCGI server don't 
     98  share memory (by definition), the sessions data stored in one process would 
     99  not be available in another process. 
     100 
     101  That's why a kind of distributed "shared memory" session backend should be 
     102  used with the FastCGI publisher. So don't forget to launch your ``memcached`` 
     103  server. 
     104 
     105As the sessions are shared through the memcached server, the same FastCGI 
     106application can be launched on several machines thus easily creating a scalable 
     107applicative cluster. 
     108 
     109 
     110Configuring the Web server 
     111~~~~~~~~~~~~~~~~~~~~~~~~~~ 
     112 
     113The next step is to configure the Web server so that it serves the static files and pass the other requests to the 
     114FastCGI processes we just launched. Of course, the procedure depend on the Web server used, so we are going to 
     115show you how to achieve that with `Apache`_, `Nginx`_ and `Lighttpd`_. However, it should be easy to use another Web 
     116server supporting FastCGI if you want to. 
     117 
     118 
     119Apache 
     120++++++ 
     121 
     122On Apache, the FastCGI support is provided by the `mod_fastcgi`_ module which is not installed by default. 
     123You should download and install it by yourself, as described `here <http://www.fastcgi.com/mod_fastcgi/INSTALL>`_. 
     124Then, the ``mod_fastcgi`` module should be activated by including it in the main apache configuration file as shown 
     125below: 
     126 
     127.. code-block:: apache 
     128 
     129    LoadModule fastcgi_module /path/to/mod_fastcgi.so 
     130 
     131 
     132The rewrite rules serving the static files of your application can be generated with this command: 
     133 
     134.. code-block:: sh 
     135 
     136    nagare-admin create-rules --apache <application> > rewrite_rules.apache 
     137 
     138 
     139You should obtain something like this: 
     140 
     141.. code-block:: apache 
     142 
     143    RewriteEngine On 
     144    RewriteRule ^/static/nagare/(.*)$ /path/to/python/site-packages/nagare-0.4.1-py2.7.egg/static/$1 [L] 
     145    RewriteRule ^/static/myapp/(.*)$ /path/to/python/site-packages/myapp-0.0.1-py2.7.egg/static/$1 [L] 
     146 
     147Then, you must create a ``VirtualHost`` for your application and include the rewrite rules file 
     148into the configuration, as shown below: 
     149 
     150.. code-block:: apache 
     151 
     152    # virtualhost configuration for http://www.myapp.com 
     153    <VirtualHost *:80> 
     154        ServerName www.myapp.com 
     155 
     156        FastCGIExternalServer /path/to/python/site-packages/fcgi -host 127.0.0.1:9000 
     157 
     158        Include /path/to/rewrite_rules.apache 
     159        RewriteRule ^/(.*)$ /fcgi/$1 [QSA,L] 
     160    </VirtualHost> 
     161 
     162The ``FastCGIExternalServer`` directive instructs Apache to forward the requests directed to 
     163``/path/to/python/site-packages/fcgi`` to the FastCGI server listening at ``127.0.0.1`` on port 
     164``9000`` which has been launched externally, and the last ``RewriteRule`` directive redirects 
     165the requests not handled by the other rewrite rules to ``/path/to/python/site-packages/fcgi``, 
     166thus forwarding those requests to the FastCGI processes. 
     167 
     168Finally, you should ``include`` the ``VirtualHost`` configuration file into the main 
     169apache configuration file, or put it in the ``conf.d`` directory of your apache installation, 
     170and restart Apache. 
     171 
     172 
     173Nginx 
     174+++++ 
     175 
     176Nginx has builtin support for FastCGI thanks to the `Nginx's HttpFastcgiModule`_, so no 
     177manual installation is necessary. 
     178 
     179The rewrite rules serving the static files can be generated with this command: 
     180 
     181.. code-block:: sh 
     182 
     183    nagare-admin create-rules --nginx <application> > rewrite_rules.nginx 
     184 
     185 
     186You should obtain something like this: 
     187 
     188.. code-block:: nginx 
     189 
     190    location /static/nagare/ { 
     191      alias /path/to/python/site-packages/nagare-0.4.1-py2.7.egg/static/; 
     192    } 
     193 
     194    location /static/ipages/ { 
     195      alias /path/to/python/site-packages/myapp-0.0.1-py2.7.egg/static/; 
     196    } 
     197 
     198Then, you must create a server configuration, as shown below: 
     199 
     200.. code-block:: nginx 
     201 
     202    server { 
     203        listen 80; 
     204        server_name www.myapp.com; 
     205        access_log /var/log/nginx/myapp.access.log; 
     206        error_log /var/log/nginx/myapp.error.log; 
     207 
     208        # serve the static files 
     209        include /path/to/rewrite_rules.nginx; 
     210 
     211        # serve the application 
     212        location / { 
     213            include /etc/nginx/fastcgi_params; 
     214            # Nagare applications need a properly set PATH_INFO variable 
     215            # either pass the request URI or use fastcgi_split_path_info to split the URI 
     216            fastcgi_param PATH_INFO $fastcgi_script_name; 
     217            fastcgi_pass 127.0.0.1:9000; 
     218        } 
     219    } 
     220 
     221Note that, in addition to the default FastCGI parameters specified in ``/etc/nginx/fastcgi_params``, 
     222we must also set the ``PATH_INFO`` parameter which is required by Nagare. 
     223 
     224Finally, you must copy the configuration file in ``/etc/nginx/conf.d/``, or create a symbolic link 
     225there pointing to your configuration file, and restart Nginx. 
     226 
     227 
     228Lighttpd 
     229++++++++ 
     230 
     231Lighttpd has also builtin support for FastCGI thanks to the `Lighttpd's mod_fastcgi`_ module. 
     232 
     233The rewrite rules serving the static files can be generated with this command: 
     234 
     235.. code-block:: sh 
     236 
     237    nagare-admin create-rules --lighttpd <application> > rewrite_rules.lighttpd 
     238 
     239However, it's easier the generate the rewrite rules in an ``include_shell`` directive. 
     240An example of Lighttpd server configuration is show below: 
     241 
     242.. code-block:: lighttpd 
     243 
     244    server.modules = ( "mod_rewrite", "mod_fastcgi" ) 
     245    server.errorlog = "/tmp/lighttpd.log" 
     246    server.indexfiles = ( "index.html" ) 
     247 
     248    #debug.log-request-handling = "enable" 
     249 
     250    server.port = 8080 
     251    server.bind = "0.0.0.0" 
     252    #server.event-handler = "linux-sysepoll" 
     253 
     254    fastcgi.server = ( 
     255                     "/fcgi" => 
     256                     ( 
     257                         ( 
     258                             "host" => "127.0.0.1", 
     259                             "port" => 9000, 
     260                             "check-local" => "disable", 
     261                         ) 
     262                     ) 
     263                 ) 
     264 
     265    # Here, change the path to the ``nagare-admin`` command 
     266    include_shell "path/to/nagare-admin create-rules --lighttpd" 
     267    url.rewrite += ( 
     268        "^(.*)" => "/fcgi/$1" 
     269    ) 
     270 
     271    # --------------------------------------------------------------------------------- 
     272 
     273    mimetype.assign = ( 
     274      ".pdf"          =>      "application/pdf", 
     275      ".sig"          =>      "application/pgp-signature", 
     276      ".spl"          =>      "application/futuresplash", 
     277      ".class"        =>      "application/octet-stream", 
     278      ".ps"           =>      "application/postscript", 
     279      ".torrent"      =>      "application/x-bittorrent", 
     280      ".dvi"          =>      "application/x-dvi", 
     281      ".gz"           =>      "application/x-gzip", 
     282      ".pac"          =>      "application/x-ns-proxy-autoconfig", 
     283      ".swf"          =>      "application/x-shockwave-flash", 
     284      ".tar.gz"       =>      "application/x-tgz", 
     285      ".tgz"          =>      "application/x-tgz", 
     286      ".tar"          =>      "application/x-tar", 
     287      ".zip"          =>      "application/zip", 
     288      ".mp3"          =>      "audio/mpeg", 
     289      ".m3u"          =>      "audio/x-mpegurl", 
     290      ".wma"          =>      "audio/x-ms-wma", 
     291      ".wax"          =>      "audio/x-ms-wax", 
     292      ".ogg"          =>      "application/ogg", 
     293      ".wav"          =>      "audio/x-wav", 
     294      ".gif"          =>      "image/gif", 
     295      ".jpg"          =>      "image/jpeg", 
     296      ".jpeg"         =>      "image/jpeg", 
     297      ".png"          =>      "image/png", 
     298      ".xbm"          =>      "image/x-xbitmap", 
     299      ".xpm"          =>      "image/x-xpixmap", 
     300      ".xwd"          =>      "image/x-xwindowdump", 
     301      ".css"          =>      "text/css", 
     302      ".html"         =>      "text/html", 
     303      ".htm"          =>      "text/html", 
     304      ".js"           =>      "text/javascript", 
     305      ".asc"          =>      "text/plain", 
     306      ".c"            =>      "text/plain", 
     307      ".conf"         =>      "text/plain", 
     308      ".text"         =>      "text/plain", 
     309      ".txt"          =>      "text/plain", 
     310      ".dtd"          =>      "text/xml", 
     311      ".xml"          =>      "text/xml", 
     312      ".mpeg"         =>      "video/mpeg", 
     313      ".mpg"          =>      "video/mpeg", 
     314      ".mov"          =>      "video/quicktime", 
     315      ".qt"           =>      "video/quicktime", 
     316      ".avi"          =>      "video/x-msvideo", 
     317      ".asf"          =>      "video/x-ms-asf", 
     318      ".asx"          =>      "video/x-ms-asf", 
     319      ".wmv"          =>      "video/x-ms-wmv", 
     320      ".bz2"          =>      "application/x-bzip", 
     321      ".tbz"          =>      "application/x-bzip-compressed-tar", 
     322      ".tar.bz2"      =>      "application/x-bzip-compressed-tar" 
     323     ) 
     324 
     325Then, place this config file in ``/etc/lighttpd/lighttpd.conf`` and restart Lighttpd. 
     326 
     327 
     328Handling the FastCGI processes 
     329------------------------------ 
     330 
     331Since the FastCGI processes are launched externally from the Web server, you can use any monitoring 
     332tool to handle the processes, such as `Supervisor`_. Here is a typical configuration file that starts 
     333a Nagare application as a daemon with Supervisor: 
     334 
     335.. code-block:: ini 
     336 
     337    [unix_http_server] 
     338    file=/path/to/supervisord/supervisor.sock 
     339 
     340    [supervisord] 
     341    logfile=/path/to/supervisord/supervisord.log 
     342    pidfile=/path/to/supervisord/supervisord.pid 
     343    directory=/path/to/supervisord 
     344 
     345    [rpcinterface:supervisor] 
     346    supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface 
     347 
     348    [supervisorctl] 
     349    serverurl=unix:///path/to/supervisord/supervisor.sock 
     350 
     351    [program:myapp] 
     352    command=/path/to/nagare-admin serve /path/to/application/conf/myapp.conf -c /path/to/application/conf/fastcgi.conf 
     353    process_name=myapp 
     354    autostart=true 
     355    autorestart=true 
     356    stdout_logfile=/path/to/application/logs/myapp.log 
     357    redirect_stderr=true 
     358 
     359Then, you can start the application by running this command: 
     360 
     361.. code-block:: sh 
     362 
     363    /path/to/supervisord -c /path/to/supervisord.conf 
     364 
     365The ``supervisord`` process daemonizes itself and starts the FastCGI processes through 
     366the ``nagare-admin serve`` command. Furthermore, they are restarted automatically when something goes 
     367wrong thanks to the ``autorestart`` option. 
     368 
     369 
     370.. _`FastCGI`: http://www.fastcgi.com/ 
     371.. _`Apache`: http://httpd.apache.org/ 
     372.. _`Nginx`: http://nginx.org/ 
     373.. _`Supervisor`: http://supervisord.org/ 
     374.. _`Lighttpd`: http://www.lighttpd.net/ 
     375.. _`mod_fastcgi`: http://www.fastcgi.com/mod_fastcgi/docs/mod_fastcgi.html 
     376.. _`Nginx's HttpFastcgiModule`: http://wiki.nginx.org/HttpFastcgiModule 
     377.. _`Lighttpd's mod_fastcgi`: http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs:ModFastCGI 
     378 
     379.. wikiname: ApplicationDeployment 
     380}}}