Methodology

AN OVERVIEW

The 2007 State Liability Systems Ranking Study was conducted for the U.S. Chamber Institute for Legal Reform by Harris Interactive, Inc. The final results are based on interviews with a nationally representative sample of 1,599 in-house general counsel or other senior litigators at companies with annual revenues of at least $100 million. Interviews averaging 22 minutes in length were conducted by telephone and took place between December 27, 2006 and March 2, 2007.

SAMPLE DESIGN

A representative sample of companies with annual revenues of at least $100 million annually was drawn using sample from idExec, Dun & Bradstreet, and AMI. Alert letters were sent to the general counsel at each company. These letters provided general information about the study, notified them that an interviewer from Harris Interactive would be contacting them and requested their participation. A copy of both letters appears in Appendix C.

The sample was segmented into two main groups. Of the 1,599 respondents, 77 were from insurance companies, with the remaining 1,522 interviews being conducted among public corporations from other industries. The proportion of interviews with insurance companies represents 5% of the total sample. Typically, in the universe of companies with $100 million or more in revenues, insurance companies represent 6% of this population. Since property casualty insurance companies have extensive experience with state liability systems, for the purposes of this study we worked to ensure that our proportion of insurance companies matched the overall population.

Respondents had an average of 22 years of relevant legal experience (including their current position), had been with their company an average of 13.8 years, and had been in their current position an average of 11.4 years.

TELEPHONE INTERVIEWING PROCEDURES

The 2007 State Liability Systems Ranking Study utilized Harris' computer-assisted telephone interviewing (CATI) system, whereby trained interviewers call and immediately input responses into the computer. This system greatly enhances reporting reliability. It reduces clerical error by eliminating the need for keypunching, since interviewers enter respondent answers directly into a computer terminal during the interview itself. This data entry program does not permit interviewers to inadvertently skip questions, since each question must be answered before the computer moves on to the next question. The data entry program also ensures that all skip patterns are correctly followed. The on-line data editing system refuses to accept punches that are out-of-range, it demands confirmation of responses that exceed expected ranges, and asks for explanations for inconsistencies between certain key responses.

In order to achieve high respondent participation, in addition to the alert letters, numerous telephone callbacks were made in order to reach the respondent and conduct the interview at a convenient time for the respondent.

Once a qualified respondent was identified, the respondent was first asked about his/her familiarity with several states. First, 24 states out of the list of 50 possible states were presented to the respondent. Within these 24 states, the 17 states presented were the following: Alaska, Hawaii, Idaho, Iowa, Kansas, Maine, Montana, Nebraska, New Hampshire, North Dakota, Rhode Island, South Dakota, Utah, Vermont, West Virginia, Washington and Wyoming. These states were prioritized in order to get a sufficient number of evaluations, since in the past years of this study, data for these states were based on fewer evaluations. The remaining 7 states were randomly selected from the remaining states not mentioned above.

Respondents were then given the opportunity to name any other state, aside from the states already presented, and specify if they are very or somewhat familiar with that state.

If the respondent was very or somewhat familiar with a given state, the respondent was then given the opportunity to evaluate that state's liability system. The maximum number of states a respondent had the opportunity to evaluate was 10. On average, each respondent evaluated 3 states. This represents a change from 2006 when respondents were given an opportunity to evaluate a maximum of 15 states, evaluating an average of 6 states. This change was made in order to reduce the burden on respondents and increase the likelihood that they were really familiar with the states.

SIGNIFICANCE TESTING

Reliability of Survey Percentages

It is important to bear in mind that the results from any sample survey are subject to sampling variation. The magnitude of this variation (or error) is affected both by the number of interviews-the base size-and by the level of the percentages expressed in the results.

Table A-1 shows the possible sample variation that applies to percentage results for this survey. The chances are 95 in 100 that a survey result does not vary, plus or minus, by more than the indicated number of percentage points from the result that would have been obtained if interviews were conducted with all persons in the universe represented by the sample. For example, if the response for a sample size of 300 is 30%, then in 95 cases out of 100, the response in the total population would have been between 25% and 35% (+/-5%). Note that survey results based on subgroups of small size can be subject to large sampling error.

Table A-1: Recommended Allowance for Sampling Error of Proportions (Plus or Minus)

  Survey Percentage Result
Sample Size 10% or 90% 20% or 80% 30% or 70% 40% or 60% 50%
1600 2 2 2 2 3
1500 2 2 2 3 3
1400 2 2 2 3 3
900 2 3 3 3 3
800 2 3 3 3 3
700 2 3 3 4 4
600 2 3 4 4 4
500 3 4 4 4 4
400 3 4 4 5 5

Significance of Differences Between Proportions

Sampling tolerances are also involved in the comparison of results from different surveys or from different parts of a sample from the same survey (subgroup analysis). Table A-2 shows the percentage difference that must be obtained before a difference can be considered statistically significant. These figures, too, represent the 95% confidence level.

To illustrate, suppose the two percentages in question are 34% and 25%. More specifically, suppose that one group of 300 has a response of 34% "yes" to a question, and an independent group has a response of 25% to the same question, for an observed difference of 9 percentage points. According to the table, this difference is subject to a potential sampling error of 6-7 percentage points. Since the observed difference is greater than the sampling error, the observed difference is significant.

Table A-2: Sampling Error of Difference between Proportions

Approximate Sampling Tolerances (at 95% Confidence Level) To Use in Evaluating Differences between Two Percentage Results

  Survey Percentage Result
Sample Sizes 10% or 90% 20% or 80% 30% or 70% 40% or 60% 50%
900 v.     900 3 4 4 5 5
                500 3 4 5 5 6
                300 4 5 6 7 7
                200 5 6 7 8 8
                100 6 8 10 10 10
                50 9 11 13 14 14
500 v.     500 4 4 6 6 6
                300 4 6 7 7 7
                200 6 7 8 8 8
                100 7 9 10 11 11
                50 9 12 13 14 15

Sampling error of the type so far discussed is only one type of error. Survey research is also susceptible to other types of error, such as refusals to be interviewed (non-response error), question wording and question order, interviewer error, and weighting by demographic control data. Although it is difficult or impossible to quantify these types of error, the procedures followed by Harris Interactive, Inc. keep errors of these types to a minimum.

NOTES ON READING TABLES

The base ("N") on each question is the total number of respondents answering that question. An asterisk (*) on a table signifies a value of less than one-half percent (0.5%). A dash represents a value of zero. Percentages may not always add up to 100% because of computer rounding or the acceptance of multiple answers from respondents answering that question. Note that in some cases results may be based on small sample sizes. Caution should be used in drawing any conclusion from results based on these small samples.

States were given a grade ("A", "B", "C", "D", "F") by respondents for each of the key elements of their liability systems. Tables show the ratings of the states by these grades, the percentage of respondents giving each grade, and the mean grade for each element. The mean grade was calculated by converting the letter grade using a 5.0 scale where "A" = 5.0, "B" = 4.0, "C" = 3.0, "D" = 2.0, "F" = 1.0. Therefore, the mean score displayed can also be interpreted as a letter grade. For example, a mean score of 2.8 could be seen as roughly a "C-" grade. Ties between states with matching mean scores were ranked by looking at the percentage of "A" grades, the base sizes and any rounding that may have taken place.

For the "Ranking on Key Elements" tables, states were ranked by their mean grades on that element.

The "Overall Ranking of State Liability Systems" table was calculated by creating an index using the scores given on each of the key elements. All of the key element items were highly correlated with one another, and with overall performance. The differences in the relationship between each item and overall performance were trivial, so it was determined that each item should contribute equally to the index score. The index was created from the mean across the 12 items, which was rescaled from 0 to 100 prior to averaging them together.

Punitive Damages:

This year the scores for the six states that have no punitive damages (Louisiana, Massachusetts, Nebraska, New Hampshire, New Jersey, and Washington) were calculated in two different ways. One way (used in our main tables and in all previous years) is to base the score on all the other criteria excluding punitive damages. The second way involved assigning to each of these six states the highest score achieved from other states on punitive damages (i.e., Delaware's). While this improves their overall raw score, it does not make a difference in their overall ranking.

PROJECT RESPONSIBILITY AND ACKNOWLEDGEMENTS

The Harris team responsible for the design and analysis of The State Liability Systems Ranking Study included Humphrey Taylor, Chairman, The Harris Poll, Vice President David Krane and Research Associate Chasson Gracie. We would like to acknowledge Linda Kelly from the U.S. Chamber Institute for Legal Reform and Judyth Pendell of Pendell Consulting, LLC, for their invaluable contributions to the design, content, focus and analysis of the project. Harris Interactive is responsible for the final determination of topics, question wording, collection of the data, statistical analysis and interpretation in the report.

PUBLIC RELEASE OF SURVEY FINDINGS

All Harris surveys are designed to comply with the code and standards of the Council of American Survey Research Organizations (CASRO) and the code of the National Council of Public Polls (NCPP). Should data from the survey be released to the public, any release must stipulate that the complete report is also available.