Version: 0.9
Author: Michael D. Thomas, mdthomas@mindspring.com
This document outlines the high-level concepts, design, user experience and roadmap for the InternetworkersPollingApp project. Implementation and technical details -- including database schemas, object models, languages, etc. -- are not covered in this document. These details will be covered in the version-specific Technical Specification.
The InternetworkersPollingApp provides collaborative online surveying with an emphasis on gauging the opinions and energies of an online community in a trustworthy and credible manner. The approach of the app is also aimed to cover two subordinate cases of surveying: polling and voting.
The App includes a module for question creation so that the community can build the survey. Rather than a 1xN arrangement typical of surveying -- where one person or one group decide on the questions to ask a larger group -- this app allows an NxN arrangement where anyone in the community can submit questions to ask the entire community.
The App addresses a key problem of online surveying -- result validity. It's easy to hack online surveys and the prevention of hacks restricts the ease of survey taking by legitimate users. Rather than attempt to prevent hack attempts, the App focuses on establishing the validity of each particular survey, separating the individual results based on their judged validity, and aggregating the results based on their validity. The survey is open to anyone, including hackers, but the hackers' illegitimate effect on the results are (hopefully) relegated to the less valid result sets thus limiting "hacker pollution" of the results.
The benefits of online surveying are obvious and well documented:
The challenges of online surveying are more complex.
The key challenge of online surveying (and online polling, voting, etc.) was best illustrated by The New Yorker in the early 90's: "On the Internet, no one knows your a dog." How do you know if all of your survey takers are dogs? Do you put in a question about cats and try to interpolate?
It would be easy to focus on the different aspects of HTTP and talk about the problems of stateless network protocols for this kind of thing. However, it must be observed that this problem isn't new. Look at the history of voting. In America, there are a lot of documented cases of deceased voters somehow making in to the polls. In current times, democracies early in their evolution are besieged by foreign officials monitoring the voting process and ostensibly preventing fraud.
The challenge is to map a person to a vote (or survey response) while keeping the vote anonymous. There is no failsafe system for doing this that scales. The only systems I can think of require all voters to be physically present at the time and place of the voting and completely observant of the entire process.
For larger communities, total voting integrity is approachable if not attainable. It isn't that important that the system have absolute integrity -- only that the system is trusted by its participants to produce representative results. I.e., the perception that the system is good (if not perfect) is the key to a successful voting or surveying system.
Highly trusted online surveying systems are certainly possible with current Internet technologies. User/password schemes are widely used for this kind of thing and on the high end SSL and bioauthentication could be used. (Personally, I've never been a fan of bioauthentication. Anything important enough for this kind of security would probably motivate someone to just cut off my finger.) However, all of these high integrity strategies impose difficulty on the survey taker. At the least, the survey employing these methods bias the results towards those users willing to go to the trouble of registering and overcoming the technological hurdles imposed.
The dichotomy is credibility vs. representation. If you try too hard to make your survey credible, it won't represent your community. If your survey is too accessible, it will have little or no credibility.
There are schemes that attempt to strike a balance using approaches involving HTTP cookies and IP blocking. But to quote a popular site that uses one of these schemes:
"This whole thing is wildly inaccurate. Rounding errors, ballot stuffers, dynamic IPs, firewalls. If you're using these numbers to do anything important, you're insane."
This project attempts to attack the problem of survey hacking by recognizing it instead of trying to prevent it. The underlying protocols (and the dynamics of the Internet experience) make it difficult to build an online system where each survey going in to the system can be trusted. Instead, this app attempts to produce survey results that can be trusted in their context.
Rather than attempting to prevent hack attempts on the front end (when the survey is taken), my strategy is to expose hack attempts on the back end (when the results are presented).
The strategy of this app is to classify the results in to sets of differing validity. The sets are based on how well it can be established that its individual survey results are provided by a particular individual and that this particular individual will not be able to submit another survey in to this particular set.
In this implementation of this strategy, extra effort is required of those survey takers whose results are fed in to the set of highest validity. However, any voting system is biased towards those willing to put more effort towards the process. In America (and other democracies), those that bother to register and make it to the polls are those whose voices are heard. People that don't register to vote and don't go to the polls aren't represented in the aggregate results. It is judged to be acceptable that the opinions of non-voters aren't factored in when our political leaders are chosen. I'm not sure I've ever even heard this debated.
The challenge in voting is to keep the effort required to vote low enough so that it is acceptable to dismiss those that don't vote. For this survey app, the effort required to achieve the highest level of validity should also be kept low. If the effort required is low, then that particular result set will be perceived and trusted as being representative. If truly polluting hack attempts occur, they will be easily detected by examining against the representative result set. In a worst case scenario, result sets of lower validity can simply be discarded. Alternatively, the results can be judged with a caveat such as the quote above -- meaning that the sets with the lowest validity of this survey app will be perceived as having the same integrity as popular online surveys available today.
There are three key strategies in the development of this application:
This is achieved by allowing and facilitating anyone in the community to submit questions for the survey. There are certainly questions that shouldn't be asked. E.g., if someone asks the question: "What is your credit card number and expiration date?" someone might actually answer it and this very private information would automatically be displayed on the web. Our approach is to appoint a benevolent Survey Moderator that will filter out poor questions prior to the beginning of the survey in a manner that the community accepts and to make it easy for the Survey Moderator to perform these duties.
As discussed in the previous section, higher integrity can be achieved while still having an accessible survey by presenting results as sets of varying validity. This is achieved through a combination of registration and confirmation while still providing survey taker anonymity. Before taking the survey, the user is asked to submit their email address. Upon submission, it is compared against a set of pre-registered email addresses (in this case, the list of email address of the various Internetworker's email lists). After completion, an email is sent to the email address so that the survey can be confirmed. There is also enough survey specific data in the email so that the survey result can be removed if the survey taker didn't actually take the survey -- more on the importance of this in the next section. This yields the following four sets of results:
The app will also create result validity by storing all available data that the HTTP transaction provides, including originating IP address, browser type, etc. By storing this information the Survey Administrator can more easily detect hack attempts and how they relate to particular surveys.
As covered in the previous section, if the underlying voting system isn't trusted the results of that system are irrelevant. Of course, complete unanimous trust is an impossible goal. The best that can be hoped for is that the threshold can be crossed where the community believes that the system can create collective opinions and that the system has no discouraging ulterior purpose. For this app, this is attempted as follows:
This section outlines the different types of users of the survey system and how they interact with the system.
The Survey Question Submitter creates questions for the survey by using an online tool. They can choose between multiple types of questions. All questions are reviewed by a Survey Moderator before the survey begins.
The survey taker is the key user of the system and their flow through the system is the most complex:
The Survey Results Consumer visits the results page to see the results. Anyone can be a Survey Results Consumer. The results are broken down according to the validity sets described previously.
The Survey Moderator uses their judgment to edit and/or delete questions so that the survey is able to achieve its goals.
The Survey Hacker attempts to mainpulate the results to their liking by either ballot stuffing, denying service (DOS attacks), and/or attacking the data stores on the back end directly. We don't like Survey Hackers.
The Survey Administrator maintains the system while the survey is running. This includes auditing the system to ensure that results are being collected correctly. Besides the Survey Hacker, only the survey administrator has the ability to compromise the integrity of the system. To ensure higher integrity, it may be possible to spread administration duties across several individuals and limit the power of each.
The system should be able to be downloaded and installed elsewhere. Care is taken to develop the survey as an application that can be packaged and deployed easily. The interface can be easily modified for a variety of purposes.
The code is developed so that other developers can make use of its part and its aggregate.
This section looks at the architecture at a very high level. Those interested in more detail should look at the technical specification.
Version 0.9 has two data stores:
The Add Question Module is used by Survey Question Submitters to create questions. Editing and deleting won't be available for version 0.9. However, since the questions are store in a simple text XML file, editing and deleting of questions can be easily accomplished by a Survey Administrator on request.
This module accepts the responses generated by the Survey Taker, puts them in to the database and sends the confirmation email.
This retrieves the results from the database and presents them.
This module accepts the responses of confirmation emails and processes confirmations and refutations. For 0.9 this step will require some level of manual administration.
This module allows the Survey Moderator to moderate and edit questions online. This module isn't available for 0.9.
This sections serves as a framework to address present and future design ideas.
This version will have the following features:
This version would have incremental improvements to the current system and would continue to house the entire survey in a single xml file. This architecture is very limiting to the future of the project and should be considered a dead-end version. The only reason that I would pursue this version (and not skip directly to version 2.0) is if there is no demand for what I outline for version 2.0 but there is demand for an improved version.
The additional known features for version 1.0:
The key change for version 2.0 is that the survey questions are moved in to their own database schema. This immediately requires the creation of survey moderation tool/process and a new Survey Question Editor. Despite these costs, there are profound benefits to storing questions in the database. One example: dynamic question types can be created. For instance, the Survey Taker can enter a new answer or chose an answer previously entered by another Survey Taker.
The additional known features for version 2.0:
Version 2.0 represents a base framework. From this framework, this project will hopefully proceed on the evolutionary path of other open source projects. I see the paths of evolution as follows: