HTTP Request/Response Basics

The following introductory topics will be discussed in this article:

§  The life-cycle of an HTTP request & response.

§  Anatomy of an HTTP request & response.

§  HTTP Methods & best practices.

The life-cycle of an HTTP request commonly looks like this:

1.      A user visits the URL of a website.

2.      This creates a request which is routed to a web server via the internet (a network of DNS’s, routers and switches) over HTTP (Hypertext Transfer Protocol).

3.      The web server receives the HTTP request and responds to the user with the web page (or content) which was requested.

Every time you click on a link and visit a web page, behind the scenes you are making a request, and in turn receiving a response from a web server. Note that HTTP requests can be made via many channels, not just web browsers. For example, an HTTP request could be made using TELNET, or a client written in JAVA or C# etc.

To see an example of what an HTTP request and response looks like do the following:

1.      Go to the website http://web-sniffer.net/

2.      Type in www.google.com (or any website you wish) in the “HTTP(S)-URL:” input field. When you click on “Submit” you will see the HTTP request and response data for www.google.com.

The anatomy of an HTTP request:
As a web developer, an important area to understand is the method portion of an HTTP request. The method tells the web server what kind of request is being performed on a URI.
So if you type in the URL www.google.com/finance (for example). You are requesting the /finance URI. Within the /finance URI the HTTP request has to define an HTTP method.

The method portion of an HTTP request contains the following definition options:

       Method         = "OPTIONS"

                      | "GET"

                      | "HEAD"

                      | "POST"

                      | "PUT"

                      | "DELETE"

                      | "TRACE"

                      | "CONNECT"

                      | extension-method

       extension-method = token

OPTIONS
Options is useful for finding out which HTTP methods are accessible by a client. Depending on how the web server you are trying to connect to is configured, the administrator may only have the POST and GET HTTP methods accessible. While other HTTP methods such as DELETE, TRACE, etc are disabled.

GET
A GET request retrieves data from a web server by specifying parameters in the URL portion of the request. If you examine the example HTTP request below, we are asking for index.html, and passing the parameter report_id.

        GET /index.html?report_id=34543222 HTTP/1.1

        Host: www.awebsite.com

        User-Agent: Safari/4.0

Examples of when to use GET:

1.      You are accessing a URL purely for the sake of viewing data. You could think of it as using an SQL SELECT statement. You are asking for data from the web server without the intent of updating any data.

2.      You need a URL to be ‘bookmarkable’. Basically HTTP GET is considered to be repeatable, which allows requests to be retried safely and responses to be cached.

3.      You don’t mind the request being repeated. For example a user visiting the same URL more than once.

Examples of when not to use GET:

1.      You are passing sensitive data such as usernames, passwords, social security numbers, etc.

2.      You are sending large amounts of data. Although there isn’t a character limit defined in the HTTP specification for the length of a URL, IE 4 for example only supports a max URL length of ~2000 characters using a GET request.

3.      You need to update something on a server, for example submitting a form which will update a users address or shopping cart.

POST
A POST HTTP request utilizes a message body to send data to a web server. If you examine the example HTTP POST request below, you will see that we are passing a POST HTTP request with the message body of ‘userid=mo&password=mypassw’ to login.jsp (login.jsp would be an application that the web server forwards requests to).
Examples of when to use POST:

1.      You have a large amount of data to send to a web server (the size of data would exceed URL limits of the GET method).

2.      You are sending sensitive data such as uesrnames, passwords, social security numbers etc.

3.      You are altering the state of data in a web application. For example, a shopping cart keeping track of items which you are purchasing.

Examples of when not to use POST:

1.      The URL that you are passing has a requirement of being ‘bookmarkable’. If the state of the URL changes, then the user will not be able to retrieve, or view the data it it’s former state.

2.      Your request needs to be idempotent. Note that POST requests can be idempotent, however it’s better practice to use PUT (if this HTTP request method is supported by your web server and client)

        POST /login.jsp HTTP/1.1

        Host: www.awebsite.com

        User-Agent: Safari/4.0

        Content-Length: 27

        Content-Type: application/x-www-form-urlencoded

        userid=mo&password=mypassw

PUT
PUT similar to POST utilizes a message body to transfer data. However, there are some fundamental differences between the two. Firstly PUT is considered to be idempotent, secondly a PUT’s action is always defined for a specific URI, finally a PUT is for loading the data for that resource. In other words you should know the exact location of where the data you are sending will be retrieved later.
Example of when to use PUT:

1.      Put is idempotent, so basically if you need to accommodate for a scenario where a request is submitted multiple times but the result needs to be identical for each submission; use PUT. This could be useful for creating a new user for instance. If you send a PUT request to create a user Joe Smith multiple times, the last request should have the same results as if it were sent first.

2.      You have a specific URI which you are sending data to. For example:

POST URI:

http://hostname.com/users/new

PUT URI:

http://hostname.com/users/joesmith

Example of when not to use PUT:

1.      PUT should not be used for non idempotent requests (if the state of the resource is likely to change each time a request is sent).

2.      It’s good to keep in mind that in the case of html forms, most browsers do not support the PUT/DELETE methods. It is expected that POST/GET are used. Some Restful frameworks such as Ruby on Rails for example requires the use of PUT/DELETE, however these HTTP Methods are simply tunneled through the HTTP POST Method.

        PUT /somedatabase/some_doc_id HTTP/1.1

        Content-Length: 240

        Content-Type: application/json

        {

          "Subject":"Resume",

          "Author":"Mo",

          "Body":"Find my resume attached"

        }

HEAD
The HTTP HEAD Method is used to retrieve information about a URL from a web server. So for example if you sent a HEAD request, you would receive a response from the web server containing the same information as you would with an HTTP POST excluding the body data. Here is an example:

        HEAD /de HTTP/1.1[CRLF]

        Host: www.google.com[CRLF]

        Connection: close[CRLF]

        User-Agent: Web-sniffer/1.0.37 [CRLF]

        Accept-Encoding: gzip[CRLF]

        Accept-Charset: ISO-8859-1,UTF-8;q=0.7,*;q=0.7[CRLF]

        Cache-Control: no-cache[CRLF]

        Accept-Language: de,en;q=0.7,en-us;q=0.3[CRLF]

        Referer: http://web-sniffer.net/[CRLF]

DELETE
The HTTP DELETE method can be used to delete a resource from a server. Generally it is used in two scenarios. Fist scenario is if you are following RESTful standards in developing your web application. Secondly it can be used when DELETE is enabled on a web-server and you would like to follow the HTTP standard for deleting a resource. It’s important to note however that you can use HTTP POST to handle an HTTP DELETE action also, the decision is narrowed down to the options described above.

TRACE
If you attempt to execute an HTTP TRACE method on most web-servers you will likely see this message:

        Status: HTTP/1.1 501 Not Implemented

HTTP TRACE is used to eacho the contents of an HTTP Request back to the requester (which can be useful for debugging). This however may pose a security threat because malicious code can abuse HTTP TRACE functionality to gain access to information in HTTP headers such as cookies and authentication data, if an HTTP TRACE request is sent the original request data will be returned in addition to any user specific data. An example HTTP TRACE response can look like this:

        TRACE / HTTP/1.1

        Host: www.google.com

        HTTP/1.1 200 OK

        Server: Microsoft-IIS/5.0

        Date: Tue, 31 Oct 2012 03:01:44 GMT

        Connection: close

        Content-Type: message/http

        Content-Length: 39

        TRACE / HTTP/1.1

        Host: www.google.com

CONNECT
HTTP CONNECT can be used to establish a network connection to a web server over HTTP. It’s primarily used in cases where a secure/encrypted HTTP connection (tunnel) needs to be established between a client and a web server such as an SSL connection.
Simple HTTP tunnels are an unencrypted connection through an HTTP proxy to an arbitrary destination. The tunnel takes advantage of the HTTP CONNECT method normally used for HTTPS (secure web traffic) to connect to the destination server. A typical HTTPS connection through a proxy should look like:

        CONNECT remote-server:443 HTTP/1.0

        User-Agent: Mozilla/4.0 (compatible; MSIE 6.0;..

        Host: remote-server

        Content-Length: 0

        Proxy-Connection: Keep-Alive

        Pragma: no-cach

HTTP Response Codes
Whenever a request is made to an HTTP server, a response code is sent back to the client accompanying the requested data. It’s important to understand what these response codes are as they will be useful for managing errors in your web applications. A list of HTTP response codes and their meanings can be found here:

Additional Reading:
HTTP 1.1 specification
PUT vs. POST
HTTP TRACE Vulnerability Blog Post
Restful HTTP PUT/POST Article
HTTP CONNECT Security White Paper