Simple python web server to demonstrate GET/POST handling

This blog describes a simple open source (MIT licensed) web server that demonstrates how browser/server interactions work for GET and POST requests using the python BaseHTTPServer package. It was updated to version 1.2 on 20-May-2015. This software is licensed under the term of the MIT license. Feel free to use it however you wish.

Use it as a starting point to understand the package when creating your own custom web server for handling specific types of requests but recognize that it is not suitable for a production environment. To use it in such an environment you would, at the very least, want to add improve the error handling and add threading support. You would probably also want to add SSL/TLS support.

Download

To use the simple web server download one of the archives and extract the files.

The files in the archives are shown below.

The example below shows how to download and install using the zip archive.

$ mkdir -p ~/tmp/webserver
$ cd ~/tmp/webserver
$ wget http://projects.joelinoff.com/webserver/webserver-1.2.zip  # download
$ unzip webserver.zip                                             # extract
$ chmod a+x webserver.py                                          # make it executable
$ ls -1
webserver-start.sh
webserver-stop.sh
webserver.css
webserver.html
webserver.js
webserver.png
webserver.py*

Running the Server

After you download the server, you simply run it by typing “./webserver.py” or “python2.7 ./webserver.py“. That will start the server on your local host using port 8080. If that port is already in use, then use the -p option to specify another port like this “./webserver.py -p 9000“. Do not try to use port 80. It will interfere with the normal operation of your system.

$ ./myserver.py

As the simple web server runs, it will print out informational messages as it is running to show you what is happening.

This is the list of options available from the server.

Accessing the Server

Now that the server is running you can access the webserver.html page by entering the following URL: http://127.0.0.1/webserver.html (you can also use 0.0.0.0 if that is easier to type). If server is running properly you will see something like this:

If you enter data in the “Form Using GET” fieldset, the argument values will appear in the webserver output log. If you enter data in the “Form Using POST” fieldset, the argument values will in the webserver output log and on a new page. In both case, you inspect the source code to see how the form data was captured and processed from the browser request.

Now try entering “http://127.0.0.1:8080/info” and you will see internal server information. This shows, in a crude way, how you can enter custom URLs.

At this point you have verified that the simple web server is working. If you want to go further, you try creating your own HTML or javascript files.

Implementation Discussion

This implementation uses the BaseHTTPServer and cgi packages for creating the server and processing browser requests. It uses argparse to manage the command lines arguments. There are a number of interesting features of this example.

The first is the use of the class factory make_request_handler_class to allow the request object to have access to the command line options, specifically the root directory for accessing files to display for GET requests. You could add any arbitrary class arguments this way. For example you add the content headers for handling different file extensions which in the current implementation are in the do_GET() handler method.

Another interesting feature is setting the content type based on the file extension.

Yet another interesting feature is that the POST request handler redirects to a custom page with a back button. This is simply to demonstrate that it can be done.

The source code for webserver.py is presented in full below for interactive reference but it probably makes more sense to download it so that you can search through it in your favorite editor.

#!/usr/bin/env python2.7
# -*- coding: utf-8 -*-
'''
Simple web server that demonstrates how browser/server interactions
work for GET and POST requests. Use it as a starting point to create a
custom web server for handling specific requests but don't try to use
it for any production work.

You start by creating a simple index.html file in web directory
somewhere like you home directory: ~/www.

You then add an HTML file: ~/www/index.html. It can be very
simple. Something like this will do nicely:

   
   
     
       
       
     
     
       
Hello, world!
     
   

At this point you have a basic web infrastructure with a single file
so you start the server and point to the ~/www root directory:

   $ webserver.py -r ~/www

This will start the web server listening on your localhost on port
8080. You can change both the host name and the port using the --host
and --port options. See the on-line help for more information (-h,
--help).

If you do not specify a root directory, it will use the directory that
you started the server from.

Now go to your browser and enter http://0.0.0.0:8080 on the command
line and you will see your page.

Try entering http://0.0.0.0:8080/info to see some server information.

You can also use http://127.0.0.1.

By default the server allows you to see directory listings if there is
no index.html or index.htm file. You can disable this by specifying
the --no-dirlist option.

If you want to see a directory listing of a directory that contains a
index.html or index.htm directory, type three trailing backslashes in
the URL like this: http://foo/bar/spam///. This will not work if the
--no-dirlist option is specified.

The default logging level is "info". You can change it using the
"--level" option.

The example below shows how to use a number of the switches to run a
server for host foobar on port 8080 with no directory listing
capability and very little output serving files from ~/www:

  $ hostname
  foobar
  $ webserver --host foobar --port 8080 --level warning --no-dirlist --rootdir ~/www

To daemonize a process, specify the -d or --daemonize option with a
process directory. That directory will contain the log (stdout), err
(stderr) and pid (process id) files for the daemon process. Here is an
example:

  $ hostname
  foobar
  $ webserver --host foobar --port 8080 --level warning --no-dirlist --rootdir ~/www --daemonize ~/www/logs
  $ ls ~/www/logs
  webserver-foobar-8080.err webserver-foobar-8080.log webserver-foobar-8080.pid
'''

# LICENSE
#   Copyright (c) 2015 Joe Linoff
#   
#   Permission is hereby granted, free of charge, to any person obtaining a copy
#   of this software and associated documentation files (the "Software"), to deal
#   in the Software without restriction, including without limitation the rights
#   to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
#   copies of the Software, and to permit persons to whom the Software is
#   furnished to do so, subject to the following conditions:
#   
#   The above copyright notice and this permission notice shall be included in
#   all copies or substantial portions of the Software.
#   
#   THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
#   IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
#   FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
#   AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
#   LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
#   OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
#   THE SOFTWARE.

# VERSIONS
#   1.0  initial release
#   1.1  replace req with self in request handler, add favicon
#   1.2  added directory listings, added --no-dirlist, fixed plain text displays, logging level control, daemonize
VERSION = '1.2'

import argparse
import BaseHTTPServer
import cgi
import logging
import os
import sys


def make_request_handler_class(opts):
    '''
    Factory to make the request handler and add arguments to it.

    It exists to allow the handler to access the opts.path variable
    locally.
    '''
    class MyRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
        '''
        Factory generated request handler class that contain
        additional class variables.
        '''
        m_opts = opts

        def do_HEAD(self):
            '''
            Handle a HEAD request.
            '''
            logging.debug('HEADER %s' % (self.path))
            self.send_response(200)
            self.send_header('Content-type', 'text/html')
            self.end_headers()

        def info(self):
            '''
            Display some useful server information.

            http://127.0.0.1:8080/info
            '''
            self.wfile.write('')
            self.wfile.write('  ')
            self.wfile.write('    ')
            self.wfile.write('  ')
            self.wfile.write('  ')
            self.wfile.write('    ')
            self.wfile.write('      ')
            self.wfile.write('        ')
            self.wfile.write('          ')
            self.wfile.write('          ' % (repr(self.client_address)))
            self.wfile.write('        ')
            self.wfile.write('        ')
            self.wfile.write('          ')
            self.wfile.write('          ' % (repr(self.command)))
            self.wfile.write('        ')
            self.wfile.write('        ')
            self.wfile.write('          ')
            self.wfile.write('          ' % (repr(self.headers)))
            self.wfile.write('        ')
            self.wfile.write('        ')
            self.wfile.write('          ')
            self.wfile.write('          ' % (repr(self.path)))
            self.wfile.write('        ')
            self.wfile.write('        ')
            self.wfile.write('          ')
            self.wfile.write('          ' % (repr(self.server_version)))
            self.wfile.write('        ')
            self.wfile.write('        ')
            self.wfile.write('          ')
            self.wfile.write('          ' % (repr(self.sys_version)))
            self.wfile.write('        ')
            self.wfile.write('      ')
            self.wfile.write('    
client_address
%r
command
%r
headers
%r
path
%r
server_version
%r
sys_version
%r
')
            self.wfile.write('  ')
            self.wfile.write('')

        def do_GET(self):
            '''
            Handle a GET request.
            '''
            logging.debug('GET %s' % (self.path))

            # Parse out the arguments.
            # The arguments follow a '?' in the URL. Here is an example:
            #   http://example.com?arg1=val1
            args = {}
            idx = self.path.find('?')
            if idx >= 0:
                rpath = self.path[:idx]
                args = cgi.parse_qs(self.path[idx+1:])
            else:
                rpath = self.path

            # Print out logging information about the path and args.
            if 'content-type' in self.headers:
                ctype, _ = cgi.parse_header(self.headers['content-type'])
                logging.debug('TYPE %s' % (ctype))

            logging.debug('PATH %s' % (rpath))
            logging.debug('ARGS %d' % (len(args)))
            if len(args):
                i = 0
                for key in sorted(args):
                    logging.debug('ARG[%d] %s=%s' % (i, key, args[key]))
                    i += 1

            # Check to see whether the file is stored locally,
            # if it is, display it.
            # There is special handling for http://127.0.0.1/info. That URL
            # displays some internal information.
            if self.path == '/info' or self.path == '/info/':
                self.send_response(200)  # OK
                self.send_header('Content-type', 'text/html')
                self.end_headers()
                self.info()
            else:
                # Get the file path.
                path = MyRequestHandler.m_opts.rootdir + rpath
                dirpath = None
                logging.debug('FILE %s' % (path))

                # If it is a directory look for index.html
                # or process it directly if there are 3
                # trailing slashed.
                if rpath[-3:] == '///':
                    dirpath = path
                elif os.path.exists(path) and os.path.isdir(path):
                    dirpath = path  # the directory portion
                    index_files = ['/index.html', '/index.htm', ]
                    for index_file in index_files:
                        tmppath = path + index_file
                        if os.path.exists(tmppath):
                            path = tmppath
                            break

                # Allow the user to type "///" at the end to see the
                # directory listing.
                if os.path.exists(path) and os.path.isfile(path):
                    # This is valid file, send it as the response
                    # after determining whether it is a type that
                    # the server recognizes.
                    _, ext = os.path.splitext(path)
                    ext = ext.lower()
                    content_type = {
                        '.css': 'text/css',
                        '.gif': 'image/gif',
                        '.htm': 'text/html',
                        '.html': 'text/html',
                        '.jpeg': 'image/jpeg',
                        '.jpg': 'image/jpg',
                        '.js': 'text/javascript',
                        '.png': 'image/png',
                        '.text': 'text/plain',
                        '.txt': 'text/plain',
                    }

                    # If it is a known extension, set the correct
                    # content type in the response.
                    if ext in content_type:
                        self.send_response(200)  # OK
                        self.send_header('Content-type', content_type[ext])
                        self.end_headers()

                        with open(path) as ifp:
                            self.wfile.write(ifp.read())
                    else:
                        # Unknown file type or a directory.
                        # Treat it as plain text.
                        self.send_response(200)  # OK
                        self.send_header('Content-type', 'text/plain')
                        self.end_headers()

                        with open(path) as ifp:
                            self.wfile.write(ifp.read())
                else:
                    if dirpath is None or self.m_opts.no_dirlist == True:
                        # Invalid file path, respond with a server access error
                        self.send_response(500)  # generic server error for now
                        self.send_header('Content-type', 'text/html')
                        self.end_headers()

                        self.wfile.write('')
                        self.wfile.write('  ')
                        self.wfile.write('    ')
                        self.wfile.write('  ')
                        self.wfile.write('  ')
                        self.wfile.write('    
Server access error.
')
                        self.wfile.write('    
%r
' % (repr(self.path)))
                        self.wfile.write('    
Back
' % (rpath))
                        self.wfile.write('  ')
                        self.wfile.write('')
                    else:
                        # List the directory contents. Allow simple navigation.
                        logging.debug('DIR %s' % (dirpath))

                        self.send_response(200)  # OK
                        self.send_header('Content-type', 'text/html')
                        self.end_headers()
                        
                        self.wfile.write('')
                        self.wfile.write('  ')
                        self.wfile.write('    ' % (dirpath))
                        self.wfile.write('  ')
                        self.wfile.write('  ')
                        self.wfile.write('    Home
' % ('/'));

                        # Make the directory path navigable.
                        dirstr = ''
                        href = None
                        for seg in rpath.split('/'):
                            if href is None:
                                href = seg
                            else:
                                href = href + '/' + seg
                                dirstr += '/'
                            dirstr += '%s' % (href, seg)
                        self.wfile.write('    
Directory: %s
' % (dirstr))

                        # Write out the simple directory list (name and size).
                        self.wfile.write('    ')
                        self.wfile.write('      ')
                        fnames = ['..']
                        fnames.extend(sorted(os.listdir(dirpath), key=str.lower))
                        for fname in fnames:
                            self.wfile.write('        ')
                            self.wfile.write('          ')
                            self.wfile.write('          ')
                            self.wfile.write('          ' % (os.path.getsize(fpath)))
                            self.wfile.write('        ')
                        self.wfile.write('      ')
                        self.wfile.write('    
')
                            path = rpath + '/' + fname
                            fpath = os.path.join(dirpath, fname)
                            if os.path.isdir(path):
                                self.wfile.write('            %s/' % (path, fname))
                            else:
                                self.wfile.write('            %s' % (path, fname))
                            self.wfile.write('          
  
%d
')
                        self.wfile.write('  ')
                        self.wfile.write('')

        def do_POST(self):
            '''
            Handle POST requests.
            '''
            logging.debug('POST %s' % (self.path))

            # CITATION: http://stackoverflow.com/questions/4233218/python-basehttprequesthandler-post-variables
            ctype, pdict = cgi.parse_header(self.headers['content-type'])
            if ctype == 'multipart/form-data':
                postvars = cgi.parse_multipart(self.rfile, pdict)
            elif ctype == 'application/x-www-form-urlencoded':
                length = int(self.headers['content-length'])
                postvars = cgi.parse_qs(self.rfile.read(length), keep_blank_values=1)
            else:
                postvars = {}

            # Get the "Back" link.
            back = self.path if self.path.find('?') < 0 else self.path[:self.path.find('?')]

            # Print out logging information about the path and args.
            logging.debug('TYPE %s' % (ctype))
            logging.debug('PATH %s' % (self.path))
            logging.debug('ARGS %d' % (len(postvars)))
            if len(postvars):
                i = 0
                for key in sorted(postvars):
                    logging.debug('ARG[%d] %s=%s' % (i, key, postvars[key]))
                    i += 1

            # Tell the browser everything is okay and that there is
            # HTML to display.
            self.send_response(200)  # OK
            self.send_header('Content-type', 'text/html')
            self.end_headers()

            # Display the POST variables.
            self.wfile.write('')
            self.wfile.write('  ')
            self.wfile.write('    ')
            self.wfile.write('  ')
            self.wfile.write('  ')
            self.wfile.write('    
POST variables (%d).
' % (len(postvars)))

            if len(postvars):
                # Write out the POST variables in 3 columns.
                self.wfile.write('    ')
                self.wfile.write('      ')
                i = 0
                for key in sorted(postvars):
                    i += 1
                    val = postvars[key]
                    self.wfile.write('        ')
                    self.wfile.write('          ' % (i))
                    self.wfile.write('          ' % key)
                    self.wfile.write('          ' % val)
                    self.wfile.write('        ')
                self.wfile.write('      ')
                self.wfile.write('    
%d
%s
%s
')

            self.wfile.write('    
Back
' % (back))
            self.wfile.write('  ')
            self.wfile.write('')

    return MyRequestHandler


def err(msg):
    '''
    Report an error message and exit.
    '''
    print('ERROR: %s' % (msg))
    sys.exit(1)


def getopts():
    '''
    Get the command line options.
    '''

    # Get the help from the module documentation.
    this = os.path.basename(sys.argv[0])
    description = ('description:%s' % '\n  '.join(__doc__.split('\n')))
    epilog = ' '
    rawd = argparse.RawDescriptionHelpFormatter
    parser = argparse.ArgumentParser(formatter_class=rawd,
                                     description=description,
                                     epilog=epilog)

    parser.add_argument('-d', '--daemonize',
                        action='store',
                        type=str,
                        default='.',
                        metavar='DIR',
                        help='daemonize this process, store the 3 run files (.log, .err, .pid) in DIR (default "%(default)s")')

    parser.add_argument('-H', '--host',
                        action='store',
                        type=str,
                        default='localhost',
                        help='hostname, default=%(default)s')

    parser.add_argument('-l', '--level',
                        action='store',
                        type=str,
                        default='info',
                        choices=['notset', 'debug', 'info', 'warning', 'error', 'critical',],
                        help='define the logging level, the default is %(default)s')

    parser.add_argument('--no-dirlist',
                        action='store_true',
                        help='disable directory listings')

    parser.add_argument('-p', '--port',
                        action='store',
                        type=int,
                        default=8080,
                        help='port, default=%(default)s')

    parser.add_argument('-r', '--rootdir',
                        action='store',
                        type=str,
                        default=os.path.abspath('.'),
                        help='web directory root that contains the HTML/CSS/JS files %(default)s')

    parser.add_argument('-v', '--verbose',
                        action='count',
                        help='level of verbosity')

    parser.add_argument('-V', '--version',
                        action='version',
                        version='%(prog)s - v' + VERSION)

    opts = parser.parse_args()
    opts.rootdir = os.path.abspath(opts.rootdir)
    if not os.path.isdir(opts.rootdir):
        err('Root directory does not exist: ' + opts.rootdir)
    if opts.port < 1 or opts.port > 65535:
        err('Port is out of range [1..65535]: %d' % (opts.port))
    return opts


def httpd(opts):
    '''
    HTTP server
    '''
    RequestHandlerClass = make_request_handler_class(opts)
    server = BaseHTTPServer.HTTPServer((opts.host, opts.port), RequestHandlerClass)
    logging.info('Server starting %s:%s (level=%s)' % (opts.host, opts.port, opts.level))
    try:
        server.serve_forever()
    except KeyboardInterrupt:
        pass
    server.server_close()
    logging.info('Server stopping %s:%s' % (opts.host, opts.port))


def get_logging_level(opts):
    '''
    Get the logging levels specified on the command line.
    The level can only be set once.
    '''
    if opts.level == 'notset':
        return logging.NOTSET
    elif opts.level == 'debug':
        return logging.DEBUG
    elif opts.level == 'info':
        return logging.INFO
    elif opts.level == 'warning':
        return logging.WARNING
    elif opts.level == 'error':
        return logging.ERROR
    elif opts.level == 'critical':
        return logging.CRITICAL


def daemonize(opts):
    '''
    Daemonize this process.

    CITATION: http://stackoverflow.com/questions/115974/what-would-be-the-simplest-way-to-daemonize-a-python-script-in-linux
    '''
    if os.path.exists(opts.daemonize) is False:
        err('directory does not exist: ' + opts.daemonize)

    if os.path.isdir(opts.daemonize) is False:
        err('not a directory: ' + opts.daemonize)

    bname = 'webserver-%s-%d' % (opts.host, opts.port)
    outfile = os.path.abspath(os.path.join(opts.daemonize, bname + '.log'))
    errfile = os.path.abspath(os.path.join(opts.daemonize, bname + '.err'))
    pidfile = os.path.abspath(os.path.join(opts.daemonize, bname + '.pid'))

    if os.path.exists(pidfile):
        err('pid file exists, cannot continue: ' + pidfile)
    if os.path.exists(outfile):
        os.unlink(outfile)
    if os.path.exists(errfile):
        os.unlink(errfile)

    if os.fork():
        sys.exit(0)  # exit the parent

    os.umask(0)
    os.setsid()
    if os.fork():
        sys.exit(0)  # exit the parent

    print('daemon pid %d' % (os.getpid()))

    sys.stdout.flush()
    sys.stderr.flush()

    stdin = file('/dev/null', 'r')
    stdout = file(outfile, 'a+')
    stderr = file(errfile, 'a+', 0)

    os.dup2(stdin.fileno(), sys.stdin.fileno())
    os.dup2(stdout.fileno(), sys.stdout.fileno())
    os.dup2(stderr.fileno(), sys.stderr.fileno())

    with open(pidfile, 'w') as ofp:
        ofp.write('%i' % (os.getpid()))


def main():
    ''' main entry '''
    opts = getopts()
    if opts.daemonize:
        daemonize(opts)
    logging.basicConfig(format='%(asctime)s [%(levelname)s] %(message)s', level=get_logging_level(opts))
    httpd(opts)


if __name__ == '__main__':
    main()  # this allows library functionality

Enjoy!