|
Christopher Johnstone, Jason Altman, Joe Timmons, Martha Thurlow, and Cara Cahalan-Laitusis
August 2009
All rights reserved. Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:
Johnstone, C., Altman, J., Timmons, J., Thurlow, M., & Laitusis, C. (2009). Field-based perspectives on technology assisted reading assessments: Results of an interview study with teachers of students with visual impairments (TVIs). Minneapolis, MN: University of Minnesota, Technology Assisted Reading Assessment.
Technology applications for education have increased dramatically in the 21st century. Computers and other technological tools now are often used in instruction for all students. Likewise, students are increasingly expected to know skills such as how to find information on the Internet, how to use audio materials for educational purposes, and how to compose essays on word processors.
Similarly, technology options for students with visual impairments are plentiful. During a typical school day, a student with low vision may use computer-based technologies (such as screen readers or magnifiers), handheld magnification, closed-circuit televisions (CCTVs), or handheld devices that can be used for downloading and listening to text files. Likewise, students who are blind are likely to read refreshable braille, use braille note-taking devices, and use audio files from a variety of sources (Cox & Dykes, 2001).
The use of these tools in conjunction with one another has implications for the large-scale assessment of students with visual impairments. Federal legislation, including the Individuals with Disabilities Education Act, 2004 (IDEA 04) and the No Child Left Behind Act of 2001 (NCLB) requires the participation of all students in statewide achievement testing. Specifically, NCLB mandates participation and achievement reporting for subgroups in the school population, including students with disabilities. Both IDEA 04 and NCLB permit assessment accommodations as appropriate for those students who need them. Prior to new technologies, students with visual impairments had four options of test formats: take the test with standard size print, take the test with enlarged print, take the test in braille, or have the test read by a proctor.
Each of these testing options has its own set of problems. Reading standard print is not a possibility for many students with visual impairments. Large print tests are appropriate for students with low vision, but print quality may be problematic. Braille tests are appropriate for students who have learned braille, but may be problematic for those who have not learned braille or only know non-contracted braille. Finally, oral administration of reading tests causes scoring implications in most states because it introduces possible validity problems, both because of compromising the intent of the measures and also because of other threats to validity raised when human readers inadvertently lead students toward or away from the correct answer (Lazarus, Thurlow, Lail, Eisenbraun, & Kato, 2006; Thurlow, 2007).
Because students with visual impairments use so many different tools during the school day, providing students only limited assessment choices creates a disconnect between instructional and assessment accommodations (Thurlow, Johnstone, Timmons, & Altman, 2007). Therefore, there is a need to understand the role of technology in the instruction of students with visual impairments. From there, further questions arise about the applications of various technologies in the assessment process.
The purpose of this study was to gather information to inform the design and development of a technology-assisted reading assessment for students with visual impairments or blindness.
Top of Page | Table of Contents
Assessment is used for a variety of purposes in schools. Black and William (1998) described classroom assessment as a mechanism for providing explicit feedback to students in regard to their curriculum objectives. Assessment for system accountability has slightly different goals, but is still designed to provide the general public with information about achievement levels on specific types of tasks. A continuum of assessment options (ranging from highly individualized classroom assessments to highly standardized large-scale assessments) represents a variety of ways for stakeholders to measure students’ learning.
As assessments change from classroom-based to system-based, there is a perceived need for greater standardization to support reaching defensible conclusions about how a system is performing. Increased standardization may be problematic for students with disabilities. States often allow specific accommodations for such students as one way to ease the barriers sometimes created by standardization. In terms of testing, accommodations are changes in procedures or materials during the testing process that are specifically designed to minimize the effects of disability and to better represent students’ understanding of tested constructs. Thurlow, Elliott, and Ysseldyke (2003) noted that accommodations are often allowed in one of six categories: presentation, response, timing, scheduling, setting, and other.
Examples of accommodations for students with visual impairments include:
Each of these categories has implications for the use of assistive technology. For example, a presentation accommodation may allow a student to use computer-generated speech as a read-aloud accommodation (Dolan, Hall, Bannerjee, Chun, & Strangman, 2005). Likewise, a response accommodation may allow a student to word process (Higgins, Russell, & Hoffman, 2005) or use braille writing technology (Landau, Russell, Gourgey, Erin, & Cowan, 2003) to respond to test items. Timing, scheduling, and setting accommodations also have relevance to assistive technology. For example, a student who uses a non-portable CCTV may need additional time to adjust assessments on XY fields, may need to take a test in the room where the CCTV is located, and may need to share use of the CCTV with other students if there is not a machine for each student.
The use of accommodations in states is guided by statewide policy (Thurlow, Thompson, & Johnstone, 2007). Such policies often are determined by values and psychometric concerns about whether particular accommodations change or do not change constructs. Although use of technologies such as calculators for mathematics or word processors for writing tests are allowed for all students in some states, accommodations in the area of reading are still highly disputed in policy discussions (Johnstone, Thurlow, Thompson, & Clapper, 2008; Lazarus, Thurlow, Eisenbraun, Lail, Matchett, & Quenemoen, 2006).
There is a clear need to understand how students are accessing text and what ramifications such access has for large-scale assessment, especially in the area of reading. Discussions about allowable accommodations for reading impact all students with disabilities, but are especially germane to students with visual impairments. Currently, options for technology-based accommodations are somewhat limited for students with visual impairments, even though these students may rely extensively on assistive technology during instruction. For example, Lazarus et al. (2006) found that across states braille and large-print assessments are generally accepted while technology-based accommodations are not widely embraced.
One reason hesitation exists in allowing technology-based accommodations is that research on testing accommodations is inconsistent. In a review of 49 empirical research studies, Johnstone, Altman, Thurlow, and Thompson (2006) concluded that findings across studies are often contradictory in both the effects of accommodations on student performance and on the validity of test results.
Specific research relevant to students with visual impairments and assistive technology may help to clarify some issues. Unfortunately, recent reviews of accommodations literature have found a paucity of research about such topics. Between 2002 and 2004, Johnstone et al. (2006) found only three studies that examined the effects of accommodations on students with visual impairments. In one of these studies, Hansen, Lee, and Forer (2002) reported that speech-output was positively received by adult students with visual impairments. Although no investigation of student achievement was attempted in this study, the presentation of test materials in auditory formats was well-received by its end users. Landau et al. (2002) conducted a feasibility evaluation study for a device that voiced and allowed braille input for mathematics exams. Finally, Jackson’s (2003) doctoral dissertation investigated accommodations use among students with visual impairments and found that assessment results were not affected by student demographic characteristic (i.e., accommodations had similar effects across all students with visual impairments). In a more recent analysis of accommodations research published in 2005 and 2006, Zenisky and Sireci (2007) did not find any research that specifically examined effects for students with visual impairments.
Despite the lack of concrete evidence on the effectiveness of technology-based accommodations for students, the use of technology-based instructional accommodations is widespread. In a national survey of teachers of students with visual impairments (TVIs), Thurlow et al. (2007) found that they were using a wide variety of assistive technologies in daily reading instruction. Respondents reported student use of the following computer-based technologies as part of reading instruction: CCTV or video magnifier (25% of students with visual impairments), screen reader (21%), computer magnifier/reader combination (21%), digital talking book (18%), and text reader (8%). Although technologies vary, it is evident that technology is a part of the process of “reading” for many students with visual impairments.
At the statewide level, 17 states mentioned that the increased availability of assistive technology used during assessments is partially responsible for the improved performance of students with disabilities on statewide testing (Altman, Lazarus, Thurlow, Quenemoen, Cuthbert, & Cormier, 2008). Altman et al.’s survey of state special education directors also found that nine states noted limited availability of assessment technology as responsible for the lack of improvement shown by students with disabilities.
It is clear that there is still much that is unknown about how assistive technologies may be used in the context of large-scale assessment. It is also clear, however, that such technologies are in place and are being used for both instruction and informal classroom assessment. The leap from highly individualized classroom instruction and assessment to system-level assessments requires careful consideration of policy and practice.
To better understand the implications for technology-assisted reading assessment based on the knowledge and experiences of teachers of students with visual impairments, we conducted follow-up interviews with many of the TVIs who completed the survey. The interviews addressed the use of technology in reading and language arts instruction for students who are blind or visually impaired and the role of assistive technology in assessments of language arts skills. TVIs were chosen as the focus of this effort in order
to obtain the perspectives of those working on a daily basis with students with visual impairments or blindness.
Top of Page | Table of Contents
Teachers of students with visual impairments for this study were drawn from a group of TVIs who previously agreed to complete a survey on the same topic (see Thurlow et al., 2007). In total, we interviewed 27 TVIs. We carefully sampled from the population of survey respondents to ensure that at least two TVIs were interviewed in the following categories:
All interviews were conducted by telephone. In most cases, there were two (and up to four) researchers present during telephone interviews. All interviews were recorded for review at a later time. Research participants were asked to answer a series of structured interview questions. Structured interviews allowed for researchers to create a standardized data set for particular questions, but allowed research participants to elaborate on particular points that were most salient to their work (Bogdan & Biklen, 1992). We sought to gather specific information about whether there was a perceived need for assessments of technology-assisted reading, what types of constructs are most appropriate for a test of technology-assisted reading, what proficiency requirements for such a test would be, what technologies TVIs currently used, and any other information they wished to share.
Each participant answered the same questions, but additional questions were included (if needed) for clarification. Interviews generally lasted 30 minutes, although some interviews lasted as long as one hour. One interview lasted 1.5 hours and needed to be conducted over two sessions. The interview protocol for this study is provided in the Appendix.
Qualitative data were analyzed by two of the authors of this report. Authors listened to interview transcripts as a team and noted all relevant responses from participants using a coding form that highlighted the target data for this study. After listening to the first several interviews in a team, we ensured that inter-rater reliability was present by discussing any discrepancies in data interpretation. Throughout this process, data fit into emerging categories. We placed TVI responses and direct quotes into themes that captured their essence and saved data for a secondary analysis.
After the initial coding phase terminated, one researcher completed the analysis process using themes and rules established by the coding team.
Top of Page | Table of Contents
Interview results suggested that there is a pressing need for a technology-based reading assessment. All 27 TVIs indicated that they would use a technology-based assessment if one were readily available. Results reported here address recommendations on format, structure, and assessment. Illustrative quotes and summaries are provided as starting points from which to build a test of technology-assisted reading.
One of the reasons TVIs were so interested in a test of technology-assisted reading was because they use some form of assistive technology in their teaching. Specifically, all TVIs who taught students with low-vision used at least some form of magnification equipment. All TVIs with low vision students used CCTVs, and almost all used some form of computer screen magnification. TVIs who taught students who use braille readers also reported using a variety of technologies such as audio devices and braille note takers. The TVIs who used devices in addition to CCTVs reported the specific brand names of products. These included:
It was apparent, through conversations with TVIs, that technology and reading go hand-in-hand for students with visual impairments. There was, as in all fields, a continuum of perceived comfort with using assistive technologies. Some TVIs used technology fluidly in instruction. Those who had wide-ranging knowledge of the tools available (and had access to such tools) seemed able to effortlessly match instructional tasks with technology tools. On the other hand, there were five TVIs who spent most of their time teaching students how to use CCTVs and were hesitant to engage in teaching students using computer-based technology tools (especially those that added voicing to files). This finding echoes Thurlow et al.’s (2007) finding that noted the gate-keeping function to assistive technology that TVIs sometimes have. In all cases, however, TVIs used some form of technology to assist teaching. Despite widespread use of technology, TVIs reported that in some states, students were not able to use any technologies on reading assessments.
Accommodations policies vary widely in states. Lazarus et al. (2006) investigated the use of “audio video equipment” as a reading accommodation in statewide testing and found that 17 states permitted the use of audio or video equipment without restriction. In addition, 10 states allowed the use of the accommodation in certain circumstances; specifically, several of these states did not allow the use of audio/video on a reading test. In three additional states, there were implications for scoring when the audio/video accommodation was used. Our research produced similar findings. One TVI from a southern state noted that students can use any accommodation for state testing that they do for instruction. A TVI from an eastern state, however, lamented that students only have three choices for assessment: large print, braille, or standard print. One TVI, faced with this dilemma, explained how students often cannot demonstrate knowledge under such constraints.
“If a student is a strong braille reader and the test is accurate, it is not accurate for students who are good at comprehension but don’t read braille or large print well, or kids who haven’t picked up braille yet.”
Beyond the specific use of technology, additional problems were expressed by teachers about accommodations use. One TVI from a southeastern state revealed that her state allows students to use magnifying technology (such as CCTVs), but does not allow for students to take the time needed to navigate documents with a CCTV. When asked about her statewide assessment she said,
“I don’t think it is very accurate in the sense that the kids read the test themselves. It takes a long time and they fatigue easily. My student is very intelligent but he has to take breaks and gets tired of doing it [the test]....”
It was apparent through discussions with TVIs that there is, in some states, a disconnect between assistive technology usage in instruction and assistive technology allowances in assessment. Although Lazarus et al.’s (2006) review of statewide accommodation policies reflected vast disparities in allowances, TVIs’ reports provided valuable insights into how particular accommodations decisions affect students. In addition, timing considerations for students who use various technologies are necessary. Students who use devices like CCTVs, according to TVIs, fatigue more easily than other students and may need additional time and breaks beyond the time and one-half accommodation (e.g., a student is allowed 90 minutes on a 60-minute test) offered in at least one state’s assessment.
Because technology is so prevalent in schools and because statewide assessments may not accurately measure the true reading proficiency of students with visual impairments, we asked TVIs what assessments they used to examine student proficiency in either reading or use of technology. There was a wide range of answers indicating that there is not a nationwide standard for assessing reading with technology. All teachers used some form of informal assessment. One TVI’s comments reflected on the often hit-or-miss nature of teacher assessment of technology. “If one technology doesn’t work,” she said, “we try something else. One student was having trouble with CCTV so we used ZoomText on a laptop. Sometimes it is just trial and error.” Another TVI used an informal performance assessment with error analysis to determine her students’ current level of functioning. “I get them on the computer,” she said, “and I tell them to do an assignment. While they are trying to do different things, I watch. If they are having trouble with something, I show them—I give them direct instruction.”
Other teachers used a more standardized approach. Five teachers specifically mentioned using the “SETT Framework” originally developed by Joy Zabala (2002). The purpose of the SETT Framework is to aid practitioners in making effective assistive technology decisions. The SETT Framework is not an assessment tool, but a series of considerations designed to guide assessment decisions. Zabala’s questions are listed below:
Student Considerations
Environmental Considerations
Task Considerations
Tool Considerations
(Source, Zabala, 2002).
As noted, Zabala’s SETT framework is not a specific tool that educators can use, but a series of questions that aid in decision making. The SETT program may not fit the needs of a technology-assisted reading assessment, but some of the questions asked are pertinent in test development. For example, forcing educators and test developers alike to consider the students, their special needs, and environmental implications of technology use are important beginning steps in test development.
A second assessment tool described by TVIs was the Texas School for the Blind Assistive Technology Assessment Summary for Students with Visual Impairments. This freely accessible assessment provides practitioners with a series of Yes/No questions that can be used to determine appropriate uses for assistive technology. The Texas School for the Blind (TSB) assessment is applicable for students with low and no vision, and is organized by specific tasks, described briefly below.
Accessing Print
The TSB assessment begins by asking a series of brief background questions about a student’s vision, and then asks a series of questions about how the student accesses print. Specifically, the assessment asks how regular print is accessed in different formats (e.g., “dittos,” paper with felt tip pens, papers enlarged on photocopiers, etc.). The assessment next asks about how students read large print at a variety of print sizes and distances.
Each of the questions is designed to better understand each student’s print reading potential.
Optical Aids
The next section of the TSB assessment asks about the types of magnification devices a student uses, including glasses, contact lenses, telescopes (by power), and CCTVs. Detailed questions are also presented relevant to CCTV usage. Specifically, the assessment asks about specific skills a student must have in order to benefit from CCTVs, including adjusting size of image, focusing image, adjusting vertical brakes on X/Y tables, and adjusting left/right margin stops.
Braille and Tactile
A short section on braille and tactile access to print follows the lengthier sections just described. In this section, teachers are asked to comment on whether students can use simple tactile graphics, read print materials in braille, and read using electronic braille display.
Auditory
The TSB assessment then asks a series of questions about student comprehension of material that is presented in auditory formats. Specifically, the assessment asks whether students are able to comprehend passages when read to them, paraphrase information when read orally, write/type/or braille information presented orally, comprehend “fast” speech, use auditory technology, and utilize variable pitch and speed controls.
Reading Rates
Another section relevant to reading asks practitioners a series of questions about student reading rates under different conditions. Conditions include 12-point print, student optimum-size print, reading with CCTV, reading braille, and reading rate when presented with recorded materials. The Texas assessment makes no distinction between reading and how it is traditionally defined (print and braille) and reading as an act of listening to recorded materials.
Accessing the Computer
The final section relevant to reading in the Texas School for the Blind assessment asks practitioners to determine whether students can access a computer. Specific questions are asked about how a student might visually access a screen (at what size font, etc.), how students are able to understand synthesized speech (auditory access), how students access refreshable braille (tactile access), students’ proficiency on keyboards, and students’ ability to navigate using a mouse. These, along with other data from areas mentioned above, are used to produce a series of assistive technology recommendations for individual students.
The Texas School for the Blind assessment was deemed “very useful” by the teachers who regularly used it. Among the seven teachers who mentioned it, all were satisfied with the level of information derived from using the instrument. In addition to the general instrument, the Texas School for the Blind also has a Web site that contains specific assessments for the following technologies available for public consumption (http://www.tsbvi. edu/technology/tech-assess.htm):
An important lesson learned from teacher interviews was that only about one-fourth of teachers were using formal assessment protocols or tools to assess students’ capacity to use assistive technology. Among those most frequently used are the SETT Protocol and the Texas School for the Blind Assistive Technology Assessment Summary for Students with Visual Impairments. One teacher who used the Texas assessment also mentioned using the Georgia Project for Assistive Technology’s (n.d.) Assistive Technology Evaluation and the Wisconsin Assistive Technology Initiatives (2004) WATI Assistive Technology Checklist. Each of these has elements of the SETT and Texas assessments.
Those TVIs who used formal assessments understood that assistive technology use is dependent on student capacity to interact with the technology, the needs of a particular environment, and whether the student’s primary input mechanism is visual, auditory, tactile, or any combination of these; at the same time, the technology use must have an end goal of increasing efficiency to do educational tasks. Teachers who did not use any formal approaches to assessment had a trial-and-error type approach that may not have captured the breadth of skills and functions assessed by a more formal tool such as the Texas School for the Blind Assistive Technology Assessment Summary for Students with Visual Impairments.
TVIs participating in the interviews suggested that the Texas School for the Blind assessments and SETT Frameworks may provide a model for developing a test of technology-assisted reading. This developed test then might be used to help determine the level of assistive technology accommodations a student may use for accountability tests.
According to teachers, a test of technology-assisted reading should be flexible enough to encompass a wide variety of students (students who are totally blind may be tested on refreshable braille displays while students with low vision may be tested on how to use a closed circuit television). The common ground between students with low and no vision appears to be computer-based assistive technologies.
Teachers suggested that a variety of “access” skills should be tested as part of a technology-assisted reading assessment. These access skills include basic keyboarding skills, basic keyboarding keystrokes (especially for students who cannot use a computer mouse), and powering computers on and off. Once students demonstrate they are able to access computers, teachers recommended that skills such as activating screen enlargers, accessing audio files, accessing internet files, sorting files into folders (file management), and self-advocacy skills such as ordering materials via the Internet be included in a technology-assisted reading assessment. Many of the technologies discussed by teachers were mentioned as possible formats for a technology-assisted reading assessment, but JAWS, Windows Eyes, Intellikeys, Intellitools, Open Book, and ZoomText were most frequently mentioned. For students who are completely blind, supplemental materials such as Book Port or Freedom Box were also considered important.
The scope of a technology-assisted reading assessment would depend on several factors. First, the student’s vision must be considered. The repertoire of assistive technologies used by students with low vision is different from those used by students with no vision (although many tools overlap). In addition, the possibility of additional disabilities and student keyboarding skill impacts the usefulness of what technologies can be used (computer-based technologies depend heavily on keyboarding skills). Finally, state accommodation policy will impact assistive technology decisions (e.g., allowances for auditory accommodations) and will affect the breadth of skills that may be tested.
Interview questions ranged from very concrete questions about the daily practices of TVIs to open-ended questions about how to define proficiency on a test of technology assisted reading. Those teachers who commented on what made a student a proficient technology-assisted reader examined proficiency from two different angles: fluency with technology and reading-related results.
Fluency with Technology
Teachers recognized that advanced proficiency in technology-assisted reading meant the ability to move fluidly between technologies in order to optimize the potential of various technologies. One TVI’s expectation of an advanced proficient technology-assisted reader was a student who could use technologies to complete tasks, and know enough about different technologies to use the most effective tool.
This teacher said:
I would ask, “Can they work all the aspects and features of a device?” For example, if a print is not a good size or if the copy is not good, can the reader adjust? Screen readers may not work on a Web page or if a program keeps freezing, a student would have to work it out. Having a test [of technology-assisted reading] would encompass all the basic features of a technology. They could have six or seven devices to use in addition to the content they are working on. I am not talking about just high-tech either. This could be a handheld magnifier, a high contrast roller, a monocular for distance. These are things that typical kids don’t have to deal with that our kids do.
TVIs explained proficiency as less a matter of efficiency and more a matter of understanding the basic functions of a technology (for example, how to work a CCTV). One TVI considered proficiency if “the student is able to care for the equipment, i.e., the cleaning; the storage, that’s the first thing I think of ” while another said that if a student could “scan and read something” he/she would be proficient.
There was general agreement on the differences between below proficient, proficient, and advanced proficient users of assistive technology. Teachers generally agreed that “independence” was a cut point for proficiency. No matter what piece of equipment a student used, if it could not be used independently (without the assistance of teachers), the student would not be considered proficient. In most cases, “advanced” was described as a student who could make choices and self-advocate in terms of technology choices. For example, another teacher said:
I think the advanced proficient person would be able to go back and forth between modalities with a clear conception of what is needed. For example, when I go into a picture, do I want it described or do I want to enlarge it to get the image into my eyeballs. What is most useful? They would know the technology enough so that they could flip back and forth between modalities.
An interesting finding from questions related to proficiency is that very few students were purely visual, auditory, or tactile readers. Most students had a primary reading modality supported by other modalities. Therefore, teachers’ logic about fluidity with technologies is apt to the discussion of proficiency. If a student typically uses different technologies for different purposes, an assessment that tested a student’s ability in decision making would have face validity in the realm of technology-assisted reading outcomes. Likewise, if a student was able to independently use a variety of tools (but perhaps was not fluid between tools) that student might be considered proficient. Finally, if a student needed assistance from a teacher or other adult in order to access programs, that student would be considered not proficient.
The second indicator of proficiency reported by teachers was students’ performance at grade level. One TVI said “my kids are academic. They should be able to access grade-level stuff.” This thought was echoed by several teachers in slightly different words.
The push for an academic curriculum guided many TVIs in their interactions with mainstream teachers, their professional development, and their desire to understand a wide variety of assistive technology options. For these teachers, a proficient user of technology-assisted reading would be a reader who read and understood grade-level material. For teachers, this meant that visual and tactile readers would decode and read words fluently enough to understand what was written. For audio readers, the end-goal was comprehension of recorded text.
Although this element of proficiency may seem simplistic to readers, only two TVIs expressed an outright desire for their students to take out-of-grade-level assessments. Specifically, these TVIs suggested out-of-level assessments for students with motoric issues who could not access most assistive technologies and for students with comorbid cognitive disabilities. Most TVIs were concerned that their students be assessed on grade-level, with a fair opportunity to demonstrate grade-level skills. As noted previously, TVIs were concerned about the validity of tests for students who use technology as a regular part of their reading routine but are not allowed to use technology on tests.
One teacher described the types of things she would like to see on a test of technology-assisted reading:
A measure of fluency, words per minute, comprehension, are they dropping details, are they inferring, are they missing things that are blindness related—for example, what color is the sky? I’d want the errors to be delineated finely. To find out if the words per minute are down because he doesn’t know technique. I would want some of the similar things that were found in regular tests. I think I would want to have more retention foils—if I am on page 3, can I still remember what I read in the first paragraph, etc.
It was evident that TVIs had interest in a technology-assisted reading assessment that is a general reading assessment their students could access in the way they do instruction. If this was possible, then students would be held to the same expectations of reading proficiency. Teachers’ perceptions indicated that students could achieve in school, but were more likely to achieve on assessments if they had the breadth of tools available to them in instruction.
Top of Page | Table of Contents
The purpose of this research was to solicit opinions and capture experiences from field-based practitioners that had implications for the development of a technology-assisted reading assessment. From our discussions with TVIs, four main themes emerged to guide the development of such an assessment. Although none of these themes answers specific technical questions about the proposed assessment, each frames a larger assessment perspective that is informative in the design of new assessment tools.
The use of assistive technology (AT) is often considered an instructional or assessment “accommodation” even though TVIs report that AT is so integrated into the activities of students with visual impairments that use of AT seems like a typical function of academic life. There is, however, a gulf between use of AT in instruction and allowances for AT on statewide assessment in some states. Some teachers reported that their students are only allowed to take state tests in standard, large print, or braille formats. In these states, a test of technology assisted reading will challenge current assessment policy because policies currently do not allow technology use. A follow-up research study to this report might examine state accommodation policy from the lens of a technology-assisted reading assessment. This report would clarify the states where such an assessment may be acceptable and those where it would not.
Although most TVIs in this study used informal observational methods to gauge their students’ assistive technology proficiency, some teachers used more formalized protocols. For these teachers, the SETT protocol and the Texas School for the Blind assessments were most popular. Zabala’s (2002) SETT framework is useful because it grounds AT considerations in an environmental perspective. Such a perspective aligns with international norms on terms such as “impairment” and “disability.” According to the World Health Organization (2001), an impairment is a dysfunction in part of the body, but a disability is caused by an interaction of impairment and environmental barriers. A test of technology-assisted reading would benefit from careful consideration of the barriers to accessing the general education curriculum and general education assessments, and which AT platforms and skills are necessary to minimize barriers.
The second assessment from which a technology-assisted reading assessment might benefit is the Texas School for the Blind and Visually Impaired Assistive Technology Assessment Summary for Students with Visual Impairments. This particular assessment provides a checklist approach for determining how a student accesses print or braille in a variety of formats. The assessment is not intended to “measure” students’ proficiency as a to-be-developed technology-assisted reading assessment might, but rather to determine the conditions under which a student would succeed in academic tasks. Despite the differences in purposes of the Texas instrument and a new technology-assisted reading assessment, a checklist approach may be a useful design strategy for the new assessment. Furthermore, the Texas School for the Blind has checklist assessments for specific technologies, such as: Beginner Braille n’ Speak, Braille and Speak/Type and Speak, e-mail and Internet, Functional Braille and Speak, JAWS, Keyboarding, Low Vision Technology, MS Word, Scanner, Speech Technology, Telebraille, Windows Environment, and Zoom-Text. Checklists for tasks may be informative in the development of a technology-assisted reading assessment.
Teachers reported that their students generally fell into three categories: academic students who used assistive technology, students who were unable to use assistive technology because of motor impairments, and students who had visual impairments along with cognitive disabilities. Teachers generally agreed that for the students who participated in mainstream academic curriculum, a technology-assisted reading assessment would be very informative. For these students, reading print was possible but challenging because of fatigue factors and limited assistive technology allowances on statewide test accommodations. TVIs welcomed a technology-assisted reading assessment for their academic students because it would provide students access to the technologies they use every day in reading activities and would produce a more valid inference of the students’ reading abilities.
There needs to be further research on the small population of students with comorbid visual and motoric disabilities. These students may well be very competent in academic tasks, but because they are not able to access information visually or through assistive technology (because of motor limitations) it is unknown how these students may be functioning. Further research to examine adapted mice, keyboards, switches, or other technologies may be necessary as this project progresses. Finally, for students with significant cognitive disabilities, there may be a future line of research in the use of assistive technology for alternate assessments based on alternate achievement standards.
In our interviews, we asked teachers how one would differentiate between advanced proficient, proficient, and below proficient assistive technology users who use visual, auditory, and multi-modal reading strategies. According to teachers, there are few students who use only one type of input for reading. Across teachers and types of students, there seemed to be agreement that students who could independently use various platforms would be considered proficient. Students who could not use platforms independently (i.e., they needed to ask help from teachers when they got stuck) would be considered below proficient. Finally, students who are able to move fluidly between platforms as needed (e.g., a student who examines an assignment and decides that a CCTV is appropriate for viewing diagrams and headings, but will listen to the text for content) would be considered advanced proficient.
Each of the four themes provides a starting point in the development of a technology-based reading assessment. First, it is clear from both interviews and surveys with TVIs that technology usage varies widely among teachers and students. Because of this, specific platforms will need to be carefully planned. Second, it is also evident from our findings that considerations for statewide accommodations are an important first step in examining the feasibility of a technology-assisted reading assessment. Third, respondents indicated that there are existing tools from which to build an assessment. Finally, assessments should focus on grade-level strategies and student independence. Students with visual impairments “read” in a variety of ways throughout their school day. The development of an assessment that measures their proficiency in both how they use technology and reading outcomes holds great promise for an assessment system that seeks to include all students.
Top of Page | Table of Contents
Altman, J. R., Lazarus, S. L., Thurlow, M. L., Quenemoen, R. F., Cuthbert, M., & Cormier, D. C. (2008). 2007 survey of states: Activities, changes, and challenges for special education. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
Black, P., & William, D. (1998). Inside the black box. London: King’s College Press.
Bogdan, R.C., & Biklen, S.K. (1992). Qualitative research for education. An introduction to theory and methods. Boston: Allyn and Bacon.
Cox, P. R., & Dyikes, M. K. (2001). Effective classroom adaptations for students with visual impairments. Teaching Exceptional Children, 33(6), 68-74.
Dolan, R. P., Hall, T. E., Bannerjee, M., Chun, E., & Strangman, N. (2005). Applying principles of universal design to test design: The effect of computer-based read-aloud on test performance of high school students with learning disabilities. The Journal of Technology, Learning, and Assessment, 3(7). Retrieved August 5, 2006, from http://escholarship. bc.edu/jtla/
Georgia Project for Assistive Technology (n.d.). Assistive technology evaluation. Retrieved from the World Wide Web http://www.gpat.org/GPAT%20Resource%20Topic%20Pages/ GPAT%20Resource%20Main%20Page.htm
Hansen, E. G., Lee, M. J., & Forer, D. C. (2002). A ‘self-voicing’ test for individuals with visual impairments. Journal of Visual Impairment and Blindnesss, 96(4), 273–275.
Higgins, J., Russell, M., & Hoffman, T. (2005). Examining the effect of computer-based passage presentation on reading test performance. The Journal of Learning, Technology, and Assessment, 3(4). Retrieved August 5, 2006, from http://escholarship.bc.edu/jtla/
Jackson, L. M. (2003). The effects of testing adaptations on students’ standardized test scores for students with visual impairments in Arizona (Doctoral dissertation, University of Arizona, 2003). Dissertation Abstracts International, 64/10, 3644
Johnstone, C. J., Altman, J., Thurlow, M. L., & Thompson, S. J. (2006). A summary of research on the effects of test accommodations: 2002 through 2004 (Technical Report 45). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
Johnstone, C.J., Thurlow, M.L., Thompson, S.J., & Clapper, A.T. (2008). The potential for multi-modal approaches to reading for students with disabilities as found in state reading standards. Journal of Disability Policy Studies, 18(4), 219-229.
Landau, S., Russell, M., Gourgey, K., Erin, J. N., & Cowan, J. (2003). Use of talking tactile tablet in mathematics testing. Journal of Visual Impairment and Blindness, 97(2), 85–96.
Lazarus, S.S., Thurlow, M.L., Lail, K.E., Eisenbraun, K.D., & Kato, K. (2006). 2005 state policies on participation and accommodation for students with disabilities (Synthesis Report 64). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
Lazarus, S. S., Thurlow, M. L., Eisenbraun, K. D., Lail, K. E., Matchett, D. L., & Quenemoen, M. (2006). State accommodations policies: Implications for the assessment of reading. Minneapolis, MN: University of Minnesota, Partnership for Accessible Reading Assessment.
Texas School for the Blind and Visually Impaired (n.d.). Assistive technology assessment summary for students with visual impairments. Retrieved February 18, 2008, from the World Wide Web: http://www.tsbvi.edu/technology/tech-assess.htm
Thurlow, M., Elliott, J., & Ysseldyke, J. (2003). Testing students with disabilities: Practical strategies for complying with state requirements (2nd Edition). Thousand Oaks, CA: Corwin Press.
Thurlow, M.L., Johnstone, C.J., Timmons, J., & Altman, J.R. (2007). Survey of teachers of students with visual impairments. Students served and their access to state assessments of reading. Princeton, NJ: Educational Testing Service. Retrieved February 12, 2008, from the World Wide Web. http://www.naraptara.info/reports/surveyofteachersofstudentswithVI.pdf
Thurlow, M.L., Thompson, S.J., & Johnstone, C.J. (2007). Assessment accommodations. In Florian, L. (Ed.), Sage handbook of special education (pp.331–348). London: Sage.
Wisconsin Assistive Technology Initiative (2004). WATI Assistive Technology Checklist.
World Health Organization (2001). International classification of functioning, disability and health. Geneva: Author.
Zabala, J. (2002). Introduction to the SETT framework. Retrieved February 18, 2008, from the World Wide Web: http://sweb.uky.edu/~jszaba0/SETTintro.html
Zenisky, A. L., & Sireci, S. G. (2007). A summary of the research on the effects of test accommodations: 2005–2006 (Technical Report 47). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
Top of Page | Table of Contents
We are in the process of creating a prototype assessment of technology-assisted reading for students who are blind or visually impaired. The primary purpose of this test is to assess whether or not students can independently access text once they enter high school. Another purpose of this assessment is to evaluate a student’s readiness to participate in the state assessment using assistive technology such as refreshable braille displays and screen readers as an accommodation.
The purpose of this telephone interview is to gather information to inform the design and development of the test. We’ll begin with some questions pertaining generally to the use of assistive technology in reading and language arts instruction for students who are blind or visually impaired. We will then ask some questions that pertain specifically to the role of assistive technology in assessments of language arts skills. We expect this interview to take approximately one hour.
1. Do you provide direct instruction in language arts or play a supportive role? Explain.
2. Describe in detail how assistive technology decisions are made for your students, both in instruction and assessment.
3. How accurately do you think state tests measure your students’ ability to read?
4. How does the use or non-use of assistive technology in state tests affect the usefulness of these tests in assessing your students’ abilities in language arts?
5. Do you ever assess your students’ ability to independently access grade level assistive technology?
a. If yes, what types of tasks does a student need to do to independently access text (mention the names of skills if teacher does not)?
b. If no, how would you go about assessing these skills?
6. Can you tell us how you would view an assessment of technology-assisted reading? That is, would it be useful for you to know your students’ level of proficiency in using assistive technology to access text independently?
a. Do you have opinions about what it should contain?
b. Would you use it?
7. What do you see as the relative importance of a student’s process (e.g. choice of type of assistive technology, particular tasks performed using assistive technology) compared with the results the student achieves using assistive technology (e.g. displaying knowledge, understanding text, locating information in text)?
8. What formats of text do your students access?
9. Describe the assistive technologies you currently employ with your students for language arts instruction.
10. How do you use assistive technology to address the needs of students who have trouble with reading related to their visual impairment?
11. How do you address the needs of students who are having difficulty using assistive technology for reading?
12. What grade levels of students are you familiar with teaching? [Continue only if applicable.] What types of text do 7th to 10th graders typically access?
a. Ask specifically about NIMAS, Daisy, Scanned documents, Word documents, Internet documents, others.
b. Also ask specifically about types of text-content (e.g. novels, poetry, textbooks, reference material).
13. What does it mean to be a proficient visual reader who uses AT? (e.g., a proficient user who uses magnification or other visual reading tools) How about an advanced proficient? A below proficient?
14. What does it mean to be a proficient audio reader who uses AT? (e.g., a proficient reader who uses talking books, books on CD, etc.). How about an advanced proficient reader? A below proficient reader?
15. What does it mean to be a proficient audio/visual reader who uses AT? (e.g., a proficient reader who uses screen readers with visual and audio output, talking text, etc.). How about an advanced proficient reader? A below proficient reader?
16. What advice would you give the assistive technology industry for improving services for your students?
17. What advice would you give your state department of education for enhancing support for the use of assistive technology to improve educational outcomes of your students with blindness or visual impairments? And specifically with respect to improving a student’s ability to read?
Top of Page | Table of Contents
Contact readingassessment@ets.org
with questions or assistance with this site.
© 20010 Regents of the University of Minnesota
The University of Minnesota is an equal opportunity educator and employer.
Online Privacy Policy