Our final paper summarizing our work will be posted here upon its
completion. This page contains a modified version of the Google Document
that we used to collaborate during the course of this project.

Automated verification has long since proven itself a valuable tool in
verifying correctness of traditional applications and embedded systems.
With the advent of ubiquitous web applications replacing
traditional applications and playing an increased role in critical
domains, verification of this new type of program has become of
significant importance.

Verification of web applications is a relatively recent area of study,
in which classic verification techniques are applied to web application
code in order to prove correctness of various aspects of the
system.  However, the challenges of verifying web applications are
quite different from traditional applications.  Perhaps the most
significant differences are the nature of the state and state
transitions.  A typical application can be modeled as a state
machine in a relatively straight-forward way, as the program code has
control over execution flow.  The state of the machine is the
position within the execution flow, and a union of the values of program
variables.  The state of a web application, however, is less
straight-forward.  A use case of a typical web application is
composed of dozens of transactions, where each transaction consists of
an HTTP request followed by a response issued by the server after some
computation.

The inherent developmental and security implications of this segmented
nature of execution has led to the adoption of Model-View-Controller
(MVC) frameworks that greatly ease development and maintenance of web
applications in many popular scripting languages.  One such
framework, Zend Framework, lends an MVC architecture to web applications
written in PHP.  The Zend_Controller and Zend_View objects are used
in tandem to segment the application into controllers, models, and
views, similar to the classic MVC design pattern that has dominated
desktop application GUI development for years.

Web applications developed using the MVC design pattern feature a
logical separation of message dispatch logic, business logic, and user
interface logic that has proven to easy software maintenance
significantly.  In this work, we examine how this organizational
design pattern may also make automated verification for web applications
more applicable and powerful.

A review of current literature in the field of web
application verification found little on the topic of exploiting
applications that employ the Model-View-Controller design pattern. 
Current literature focuses on web applications constructed in an
arbitrary manner, where there is no contract between the server and the
client that enforces a particular navigation “flow” between pages. 
With these types of programs, verification at the application scale was
extraordinarily difficult due to the lack of well-defined paths of
execution.  Whereas entire stand-alone applications may be modeled
as a
straightforward state machine, modeling a web application in a similar
way is very difficult due to the nature of the “state” in an
intrinsically stateless protocol such as HTTP.  While almost any
functional web application overcomes this limitation by maintaining
client state information in the form of cookies or URL encodings, these
methods are primarily designed to maintain datums such as
identification, and are not typically used to enforce a flow of
execution.

The primary limitation of the work done by others is that it assumes the
navigation flow intended by the programmer is the only possible path of
execution.  As security researchers know all too well, this is
hardly the case.  Web applications are exceptionally unique in
their model of execution flow, in that application logic is segmented
into discrete fragments that get invoked with each HTTP request.
 The sequence of requests that the user should send is merely >suggested by the UI, and is in no way enforced.
 Thus, models used for verification by others and others simply
serve
to model check UI components and general application architecture;
however, based on the assumptions made about execution flow, these
models are all but worthless within the context of security.

Our work thus serves largely as an extension of the work done by others,
where we exploit architectural traits specific to the MVC design pattern
in an
attempt to further what can be accomplished with automated
verification.  In order to build an automated verification
framework for modern MVC-based web applications, we investigated how to
enhance the nature of the “state” in a web application.  Using
cookies to enforce a position and path of execution along a web
application state machine would seem to prove quite useful within the
context of automated verification.  The Spring Framework for Java
provides many out-of-the-box solutions for enterprise web
application development, including a novel feature called “WebFlow”,
which allows the developer to segment the web application into distinct
“flows” that each have an enforced flow of execution.

There are a few (simple) hurdles to overcome. Primarily, HttpServletRequest’s parameter map is immutable, so you can’t simply add new parameters to an existing HttpServletRequest object and pass it on to the controller. Instead, you use a HttpServletRequestWrapper to wrap an existing HttpServletRequest to extract the existing params into a mutable map, that you can then programmatically add new parameters. (see the examples below)

Once you have your HttpServletRequestWrapper wrapped around your HttpServletRequest, it’s simply a matter of passing it to your controller and letting it bind and validate the values in the request. Eventually, Spring will return a ModelAndView. Within that ModelAndView will be any binding/validation errors that occurred. Now you have to extract those errors and pass them back to DWR to dynamically present on the page. This is all as simple as extracting the BindingResult from the ModelAndView and checking for any errors. Once you have the errors, you can place them back into a HashMap and return it to DWR.

This option is head and shoulders above anything else we’ve seen. It leaves DWR to do what DWR does, and leaves Spring to do what Spring does. We’re just providing some glue between the two to make them work. Very little code is actually involved in making this work, and, ultimately, is reusable across all your applications. It can be used to save individual form fields, or an entire form. It’s simply up to you what form fields you want to pass to the DWR Endpoint to wrap into the HttpServletRequest to expose to Spring.

The following is some code examples for the DWR Endpoint and DWR client-side javascript.

HttpRequestWithModifiableParams – HttpServletRequestWrapper
Here’s the Servlet Request Wrapper you can use to make the Request object’s parameters mutable. You can simply use this as-is:

package edu.csun.sa.elections.util;

import java.util.Collections;
import java.util.Enumeration;
import java.util.HashMap;
import java.util.Map;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletRequestWrapper;

public class HttpRequestWithModifiableParams extends HttpServletRequestWrapper{
    Map params;

    /**
     * @param request
     */
    public HttpRequestWithModifiableParams(HttpServletRequest request) {
        super(request);
        this.params = new HashMap(request.getParameterMap());
    }

    public String getParameter(String name) {
        String returnValue = null;
        String[] paramArray = getParameterValues(name);
        if (paramArray != null && paramArray.length > 0){
          returnValue = paramArray[0];
        }
        return returnValue;
    }

    public Map getParameterMap() {
        return Collections.unmodifiableMap(params);
    }

    public Enumeration getParameterNames() {
        return Collections.enumeration(params.keySet());
    }

    public String[] getParameterValues(String name) {
    	String[] result = null;
    	String[] temp = (String[])params.get(name);
    	if (temp != null){
    		result = new String[temp.length];
    		System.arraycopy(temp, 0, result, 0, temp.length);
    	}
        return result;
    }

    /**
     * Sets the a single value for the parameter.  Overwrites any current values.
     * @param name Name of the parameter to set
     * @param value Value of the parameter.
     */
    public void setParameter(String name, String value){
      String[] oneParam = {value};
      setParameter(name, oneParam);
    }

    /**
     * Sets multiple values for a parameter.
     * Overwrites any current values.
     * @param name Name of the parameter to set
     * @param values String[] of values.
     */
    public void setParameter(String name, String[] values){
      params.put(name, values);
    }
}

DWR Endpoint
We created a simple Java bean as an endpoint for DWR to proxy requests through – although, this would probably be better served as a Spring AOP Proxy. DWR 2.0 allows you to declare that a method will receive an HttpServletRequest and an HttpServletResponse, which does not show up in the DWR signature. It automatically knows to include these implicitly:

public class DwrEndpoint {

  // This is the controller we want to pass the modified request through
  private Controller formController = null;

  // This is the DWR endpoint we'll be using to proxy the request through
  // Notice that it takes an HttpServletRequest and HttpServletResponse, which DWR will handle automatically
  // Ultimately, all we need to pass to this method in DWR is a javascript suedo-hashmap (see below)
  public Map saveFormField(HttpServletRequest req, HttpServletResponse res, Map formParams) {

    // This map will hold errors mapping to the field (key) and the error (value)
    // The intrinsic issue with this is that only one error per field can be returned
    Map errors = new HashMap();

      try {
        // See above for the custom HttpServletRequestWrapper
        HttpRequestWithModifiableParams newReq = new HttpRequestWithModifiableParams(req);

        if( formParams != null ) {
          // We will iterate through the form params to place them in the servlet params
          Iterator i = formParams.keySet().iterator();
          while( i.hasNext() ) {
            String key = (String) i.next();
            key = URLDecoder.decode(key, "UTF-8");

            String value = (String) formParams.get(key);
            String[] values = value.split(",");
            for(int x=0; x < values.length; x++)
              values[x] = URLDecoder.decode(values[x], "UTF-8");

            // Parameters separated by "," need to be stored as an array
            if(values.length > 1)
              newReq.setParameter(key, values);
            else
              newReq.setParameter(key, value);
          }
        }

        // Send our modified request to the form - at this point Spring takes over
        // and we'll receive a ModelAndView in response.
        ModelAndView mav = formController.handleRequest(newReq, res);
        ModelMap map = mav.getModelMap();

        // Extract any errors and place them in a map back to the JS client.
        // As a note: this needs work, since only one error per field can be returned as-is.
        BindingResult br = (BindingResult) map.get("org.springframework.validation.BindingResult." + formController.getCommandName());
        if( br != null ) {
          if( br.getErrorCount() > 0 ) {
            for( Object obj : br.getFieldErrors() ) {
              FieldError error = (FieldError) obj;
              errors.put(error.getField(), error.getDefaultMessage());
            }
          }
        }

      } catch( Exception e ) {
        log.error("Error in DWR call", e);
      }
    }

    return errors;
  }

Lastly, the JS code to call DWR with. Remember, our JS needs to pass the NAME and VALUE of the form to DWR, just as a POST would pass them to the Form Controller. These pairs will eventually be converted into Request Parameters, and Spring will bind them to the command object appropriately.

    <script type='text/javascript'>
      function saveFormField(id) {
        var form = document.myForm;

        // obj will simulate a HashMap to be passed via DWR to our endpoint
        var obj = new Object();
        // get the form object we want to save
        elem = form.elements[id];

        // add the name and value to the "hashmap"
        obj[elemName] = elem.value;

        // Create a hashmap to store an the ID to be passed to the callback
        // (this format is for DWR 2.x, there's an easier format for 3.x)
        var params = new Object();
        params['id'] = id;

        // Define a callback function from the saveFormField call, this will allow us to
        // dynamically update the page with Spring Validation errors.
        var callbackProxy = function(response){
          saveFormFieldCallback(response, params);
        };

        // Define the callback variable
        var callMetaData = { callback:callbackProxy };

        // Call DWR, passing the object containing our "hashmap" of form name & value,
        // and the callback variable to execute after DWR returns.
        DwrEndpoint.saveFormField( obj, callMetaData );
      }

      // The callback will output the validation error from Spring, or clear the field if no error occurred.
      function saveVoteCallback(obj, params){
        if(typeof obj != 'undefined'){
          var key = "";
          for(key in obj){}
          if(key == '')
            dwr.util.setValue("errors_" + params["id"], "");
          else
            dwr.util.setValue("errors_" + key, obj[key]);
        }
      }
    </script>

In the end, you have a Javascript call to DWR passing name/value pairs representing form fields, to which DWR passes to your DWR-exposed Java bean that translates the name/value pairs into HttpServletRequest parameters and passes it on to Spring to bind and validate. On the return, Spring passes back a ModelAndView containing any binding/validation errors that are returned to your Javascript to present to the user. DWR doesn’t know about Spring, and Spring doesn’t know about DWR, just as it should be.

I recently had the dubious pleasure of working with Spring-Modules’ Declarative Caching with EHCache. The package I’m developing involves database integration through iBATIS, Spring Web Services, and caching to increase performance. I chose to use Spring Modules’ Declarative Caching because it obviously integrates well with the rest of the Spring architecture, and allows for the use of more than one caching engine, of which, I chose EHCache.

Let me just say, as simple as Spring has made things to work with in the past, the Spring-Modules package has been less than simple to work with. The Spring core, and many of it’s sub-projects, are clearly documented and inclusive of examples, however, the Declarative Caching module seems to lack both. It took many hours of trolling through the Internet and posting in forums to finally get something to work. Ultimately, I’m pleased with the outcome, but not content with the effort it took to get here. There are a few “gotchas” to take note of, which I’ll explain below.

After getting my own project to work, I decided to build a functional example of Declarative Caching to hopefully save someone else time in the future. What is attached to this post is the source and configurations for a fully functional servlet utilizing Spring-Modules’ Declarative Caching. All that is provided is the java source and configuration files. I originally built this in Netbeans 6 on Tomcat 6 with Java 1.5, however, with tweaks, it should work everywhere.

The “gotchas” I alluded to earlier is three-part:

1. Spring-Modules v0.8 is the latest release, but v0.9 was slated for early 2007, which did not occur.
2. Spring-Modules’ Caching provides functions for caching a single object, and flushing the entire cache, but it does not provide a means of removing a single object from the cache even though EHCache does. There are some ways around this (get access to the CacheProvider in your code and call it yourself, or create a custom interceptor). It is rumored that the fix would be out in v0.9, which as noted in #1, did not occur.
3. Namespaces – There’s presently a namespace issue with the XSDs for the caching bean definitions. This is due to the fact that www.springmodules.org is not the URL for the Spring-Modules project, and therefore, the XSD’s cannot be found there. To work around it, I placed the XSD’s in the “classes” folder of the web-app and referenced them as such in the AppContext file. You’ll see it when you look at the namespaces in “cache-AppContext.xml”

Also, the attached package does not include all of the necessary libraries. You can find out what libraries and versions I used by looking at the documentation in the source of “ExampleController.java”. The source and config files are clearly documented. So, if you’re planning on using Declarative Caching, make sure you have a firm understanding of how Spring works before you delving into this sub-project. If you have a firm understanding already, first look at “cache-AppContext.xml” to understand how the beans are defined, then “CachedClass.java” to understand how the bean definitions reference the cached methods, and finally, the “ExampleController.java” class to understand how you call the cached methods.

And if you’re like me, and you read the documentation at the Spring-Modules site regarding Declarative Caching, you’ll notice that the documentation does not mesh with the example configuration. The example configuration is MUCH less verbose than what the documentation shows, so don’t panic, it still works.

Enjoy!

Source Code for Funcational Example

[cgb@cell pfd]$ ./withRapidMind
terminate called after throwing an instance of ‘rapidmind::Exception’
what(): The type ‘double’ is not supported by the Cell backend.
Aborted

What the hell indeed. Much thanks to the RapidMind guys for an awesome function called “what” and not supporting the double type (although I’m sure there’s a long explanation about that I haven’t found yet).

So reinstalling Yellow Dog and RapidMind by their instructions works fine. But I do have a couple gripes which may just be standard Linux issues that I never realized until now for some reason:

If I sudo up to root, then I should be able to do anything that root could do if root logged in! But I can’t! I can’t boot into the ps3 game OS, I can’t even add groups! What the hell! Completely ridiculous.

(Not a Linux problem) X/Gnome looks like shit on my monitor. Holy crap I never realized how bad the analog connections on the PS3 look until I fired up X. If you have the choice and can get a digital connection and a monitor that suports HDCP, buy it. I’ll put up a screenshot of how miserable this looks sooner or later.

I’m sure there are other things, and I’ll put them up as I remember them. Asides from those little gripes, the environment is actually very nice. Here’s a nice how-to that helped me out a bit: CellPerformance.Com. I’d link directly to the RapidMind page too, but you have to login for that (although it is free to sign up).

Anywho, we’re gonna benchmark the Cell versus a slightly-faster-than-average PC and do some other cool stuff with that. Check back later for how that turns out.

UPDATE (10/30/07): Turns out installing SDK 2.1 went terrible. More on this later.

So I tried it in Ruby. I really wanted to do it in Ruby. The new Netbeans beta supports Ruby development (a whole ‘nother blog post on that may be coming soon…), and it got me really excited about Ruby. Unfortunately, that whole predicament went even worse, and the error messages that either of these give you will make sure that you’re not fixing these problems on your own.

I’m not asking for a quantum computer simulator that works on all systems. Ironically enough, that is available here (jQuantum) for all systems and it looks very interesting on first glance. What I am saying is that MySQL and SMTP are not new players in their fields, and in fact are very well-established in their fields. These things need to work on all platforms in their languages, especially on the common Linux distributions.

For anyone who cared long enough to get here, I ended up with a Perl script that parses the file and does the MySQL stuff, and produces a file of e-mail addresses that a bash script takes in and e-mails out using sendmail (that script was readily on hand). Better luck next time. Sigh.

The Perl script below takes in each word from the dictionary file, gets the MD5 hex value associated with it, and keeps the first 6 characters of it. The “6″ can be changed to whatever you like, but lowering it will increase the number of false positives and increasing it will reduce the savings in file size you’re getting from using the Bloom Filter in the first place.

We then (as the Kata post above recommends) generate random 5 character words and see if they match up in our Bloom Filtered Array. If so, we make sure that we didn’t accidentally generate a real word. My Perl script is set to do this 10000 times, taking on average 10 minutes.

I ran this script 3 times, getting 159 false positives the first time, 145 the second, and 141 the third time. This averages to 147 false positives when checking 10000 random words, yielding a 1.47% error rate. That’s pretty good, if I don’t say so myself. The savings in file size is not too shabby as well. The dictionary file is 2.49MB, while the Bloom Filtered Array is 1.64 MB (a 34% reduction in size). This may not look too great, but on bigger data sets, a 1.47% error rate may be justifiable in exchange for cutting your data set’s size down by a third. Below is my code if you wish to try it out (requires a *NIX of some sort):

#!/usr/bin/perl
use strict;
use warnings;
use Digest::MD5 qw(md5 md5_hex md5_base64);

# Chris Bunch
# Bloom Filter implemented along these guidelines:
# http://codekata.pragprog.com/2007/01/kata_five_bloom.html
# Creates an array and for each word in /usr/dict/words
# MD5’s the word and places the first n bits of it in the array

# Do not create array if -nocreate is seen

# Create array, size = wc -l /usr/share/dict/words

my $arraySize = `wc -l /usr/share/dict/words`;
my @wordList = `cat /usr/share/dict/words`;
my @bloomFilter;

# Length of words in the Bloom Filter

my $md5Length = 6;

# For each line of /usr/share/dict/words

foreach my $thisWord (@wordList) {

# MD5 this word, save the first n bits

chomp ($thisWord);
my $hash = md5_hex($thisWord);
my $substring = substr($hash, 0, $md5Length);

# Place it into the array at this place

push(@bloomFilter, $substring);
}

my $i = 0;
my @letters = qw(a b c d e f g h i j k l m n o p q r s t u v w x y z);

while ($i < 10000) {
my $searchWord = $letters[int(rand(@letters))] . $letters[int(rand(@letters))] . $letters[int(rand(@letters))] . $letters[int(rand(@letters))] . $letters[int(rand(@letters))];

# map the word to the first n bits of its MD5

my $hashedWord = md5_hex($searchWord);
my $substring = substr($hashedWord, 0, $md5Length);

# look it up in the array

my $found = “false”;
foreach my $thisWord (@bloomFilter) {

if ($substring eq $thisWord) {
$found = “true”;
}
}

# if present, print present, otherwise, print not present

if ($found eq “true”) {
my @grepResults = `grep $searchWord /usr/share/dict/words`;
foreach my $result (@grepResults) {
chomp ($result);
if ($result eq $searchWord) {
$found = “false”;
}
}

if ($found eq “true”) {
print “\n$searchWord was found”;
}

}
else {

}

$i++;
}

As always, if you see something miserably wrong with the code or have any comments, feel free to drop me a line.

The documentation is incomplete and inconsistent. We have placed most of it on a central server, but the organization of it does not easily lend itself to be found. I had to dive through all the folders to find documentation that until now, I didn’t even know existed or had seen before long ago. Furthermore, it lacks the documentation needed for several key infrastructure services such as our identity management daemon (RMWD).

The knowledge transfer process was definitely lacking. This is the main part that I believe is everyone’s fault and no one’s. It’s like that feeling you get when you’re supposed to hang out with your friends and they don’t call you that day, but you don’t call them either. Neither the leaving developers nor the management made the knowledge transfer a priority, but we didn’t know the gravity of the situation. But then again, how could we have? It’s no excuse for sure, it’s just my rebuttal.

I suppose I could go on, but I think that pretty much covers it. Of course, please give me your input on this, especially if you feel differently.