History

History of the Western Region Item Bank

Summary

In March 1981, the County of San Bernardino Human Resources Department established the Western Region Item Bank (WRIB). WRIB is a cooperative, computer-based test question item bank aimed both at increasing the productivity of professional staff and at producing higher quality written tests. While the equipment, computer programming and staff time required for item bank development represented considerable costs, the benefits of such a resource have been recognized for some years.

WRIB is based on a conceptual item bank model developed by T. Darany in Michigan in 1970 that was successfully implemented in 1975 in a single jurisdiction by the State of Missouri. At the time that development of WRIB began, no cooperative item bank implementation of the model had been achieved. Its item taxonomy was developed by J. French in 1981.

By December 1981, WRIB was operational and the County of San Bernardino began servicing requests for items from the 27 jurisdictions in California, Oregon, Nevada and Arizona that had joined together and financed the cooperative testing resource. The bank now contains over 54,000 items in 375 content categories. WRIB has grown to over 180 participating jurisdictions in sixteen states.

I. Objectives

While every employer endeavors to hire from among the best applicants available, there is a special impetus for the public sector where open competition, equality of opportunity and employment of the best qualified are specifically mandated. Written testing and test development are primary functions in carrying out this mandate for employment on the basis of merit. Therefore, written test questions, or items represent a major personnel selection resource.

Computer technology provides a means of maximizing the utility and quality of that resource. Items (questions) may be categorized according to subject matter content. They may then be stored in a computer system, along with data regarding history of use and item quality. As items are used in tests, new history data are added. Consequently, an item bank reduces test development time, improves the efficiency of professional staff, and produces higher quality written tests. A test which is quite customized to the specific case can be produced without the necessity of constructing all new items for every testing need. Previously used items can be selected from the appropriate subject matter categories based on their past performance. Based on past performance, items can be continually improved through revision.

Work on a cooperative item bank grew out of recognition that few small and medium sized jurisdictions, including the County of San Bernardino, could readily afford the equipment and computer programming costs involved in developing and operating an item bank. In addition, it would offer the advantages of item banking to small jurisdictions that have relatively few items. Even if cost were not a factor, single jurisdiction item banking would not be practical for such jurisdictions, since an item bank must contain a very large number of quality items to be a useful resource. Cooperation can overcome this barrier by pooling items from many jurisdictions.

Objectives: Therefore, the objectives that were intended to be met by WRIB were:

1. Development and implementation of a cooperative, interjurisdictional item bank resource with the flexibility to serve the needs of jurisdictions of all sizes. For small jurisdictions, services were intended to include production of booklet masters and test scoring.

2. Demonstration of the multi-jurisdiction application of an automated item bank model designed in 1970 by T. Darany (currently President of Darany & Associates). It was intended that this serve as an example for similar cooperative item banks in other parts of the country.

II. Results

Three requirements for membership in WRIB were established. First, is a $1,700.00 annual membership fee. The second is a commitment to submit edited, categorized items for inclusion in the bank. The third requirement is a commitment to dedicate 80 professional staff hours per year to qualitative review of designated item groups. As the bank grows, this effort will provide input for major item revision and deletion decisions and assure that items continue to be properly categorized.

Item bank programming and data entry began in March, 1981, with WRIB becoming operational in December 1981. Over 54,000 items in 375 content categories have been entered into the bank to respond to requests from WRIB members.

Specific Results: In terms of its primary goal, WRIB has been a complete success. Utilizing computers in the County of San Bernardino, computer programs have been written that provide for the following functions:

1. Item entry and correction.

2. Item selection based on requestor's parameters, i.e., content (category code), item type (multiple choice, true false, or alternate-choice), prior use statistics (difficulty or discrimination indices) and other parameters, such as agency source, agency most recently using, and job classification administered. It should be noted that certain pictorial or graphic based items may be referenced within the computer, but maintained off line.

3. Printout of items in review draft format, i.e., with correct answer and past use statistics included.

4. Printout of items in booklet master format, including instructions, time limits, scoring formula, and subpart headings.

5. Scoring of answer sheets for jurisdictions which utilize the booklet master service, including printout of examinees' raw scores by subpart, mean, standard deviation, reliability, frequency distribution and detailed item analysis, including upper middle lower group alternative response percentages, point bi serial and U L discrimination indices and a total percentage correct.

6. Automatic updating of individual item histories as part of item bank scoring.

The objective of flexibility to serve the needs of all sized users has been met. This results from the variety of options available. The smaller jurisdictions that do not have sophisticated booklet master production facilities and/or automated scoring processes are provided these services by WRIB. Yet, use of these services is not required. The larger jurisdictions can continue to use their own facilities for these functions and use the item bank solely as an item resource, drawing on greater breadth in both item content and item use statistics. Financial benefits, in addition to qualitative benefits, can be substantial. Members have developed estimates of savings in costs ranging from $800 to $3,000 per test when compared with using test development approaches not involving WRIB.

III. Operations

Startup: Concrete action toward making the benefits of an item bank available through a cooperative effort grew out of discussion of common problems at the meetings of the Western Region Intergovernmental Personnel Assessment Council (WRIPAC, in 1979. WRIPAC is a consortium of public jurisdictions within the states of Arizona, California and Nevada. Following unsuccessful efforts by WRIPAC to obtain grant funding for the development of WRIB, the County of San Bernardino pursued the idea of self funded cooperative effort. On December 5, 1980, written invitations to participate were sent to all WRIPAC jurisdictions and others, requesting letters of interest and support. At that time, the County of San Bernardino offered to house, oversee program development, and operate the cooperative item bank. Based upon response to that offer, the County of San Bernardino established a trust account to receive and disburse WRIB funds. On March 31, 1981, the first year's fees were requested of affirmatively responding jurisdictions. The first formal call for items to be entered into the bank went out in May 1981.

Development and Costs: In essence, the WRIB system consists of a set of computer programs and a large number of categorized, computer stored questions. This system resides in the San Bernardino County Human Resources Department. The County of San Bernardino already had all of the computer equipment initially needed for WRIB when the item bank project began. Consequently, the primary developmental cost was for computer programming. Programming was contracted and initial cost was approximately $16,000. Enhancements are financed directly by WRIB funds. While programming is a one time cost, item conversion and the servicing of requests is the major on going WRIB budget cost. (Item conversion refers to the keyboard entry of items into the computer). WRIB funded staff began with one full time clerical position and currently consists of one full-time analyst, two full-time clerical positions and two supporting analysts. Thus far, that staffing has been sufficient to provide timely response to requests and still continue item conversion activities. Based on an "expected utilization" survey, access to the bank was initially limited to two requests per jurisdiction, per week. This was the highest indicated need reported in the survey, and then by only one WRIB member. Many jurisdictions indicated needs well below that figure. However, from time to time, new item entry has slowed down to meet peaks in request volume.

Considerable professional staff time is required for item review, editing, and categorizing. This contribution is made by all users, as they prepare their own items for submission. The professional staff of the County of San Bernardino devotes additional staff time in "spot checking" submitted items for accuracy of categorization and duplicates.

A final, extremely significant component of WRIB is its categorization system. This Dewey Decimal like taxonomy was initially drafted in 1980 by San Bernardino County staff. Work with it to categorize items began prior to WRIB's formation and has provided several useful insights. It quickly became evident that a category title, by itself, would not generally provide sufficient specificity to assure that WRIB participants would categorize items into the same categories.

In addition, it became evident that refinement of the system would be a major part of item conversion. It was only by looking at the items which were available in a major content area (e.g., fiscal or law enforcement) that the amount of specificity and final category titles could be accurately determined.

Based upon these observations, the taxonomy was revised to include a category description, or definition, in addition to each title. Definitions are developed as part of the refinement or evolution of the categories. The evolving nature of the taxonomy has posed few problems. Through its early work with the system and by working ahead of the other participants, the County of San Bernardino has usually been enable to request items in groups of categories that have already been refined and described.

The conversion of items by subsets of the total system offers several advantages beyond its accommodation of the evolving nature of the taxonomy. It enabled WRIB to more quickly become operational. By putting all efforts into specified expected high use categories, usefully sized pools of items were converted much more quickly than if item conversion has proceeded in a less directed manner. Also, remaining problems with the requested categories or their descriptions surface quickly and can be corrected as all participants attend to them at the same point in time. Finally, it's more efficient with respect to procedures for detection of duplicates.

In addition to the item taxonomy, French has developed an efficient job categorization system for labeling item use history records. This facilitates item selection based on prior use with appropriate examinee groups.

Services to Participants: Requests for items and services from WRIB participants are received by online requests, mail, or fax, utilizing forms which permit designation of specific needs. Review Draft item request parameters consist primarily of category code, item type and source (contributing jurisdiction).

In requesting Review Drafts, users are encouraged to request more items than actually needed to permit local, subject matter review and item selection from among the items provided. The important point here is that WRIB does not provide test development services nor intact tests. WRIB is a resource for improved test development at the local member level.

Therefore, a returned and annotated Review Draft is required for (camera ready) Booklet Master production. Production of the Booklet Master creates the basis for later scoring of answer sheets. The scoring programs utilize the record of the Booklet Master to retrieve the item keys (correct answers) and determine subpart location on the answer sheets.