Sample of a Design Analysis

[excerpted] 

Standards Mastery: Summary of Assumptions and Unknowns

As I embark on this project, I've undertaken a careful comparison of existing functional requirements (noting that they represent obligations to existing customers and assurances widely circuculated in our marketing materials) with my analysis of Benchmark Standards and standardized assessment practices in various key states. My analysis reveals serious disconnects and troubling assumptions which underscore the importance of vetting design decisions with our instructional designers and subject-matter experts, just as marketing and engineering must inform decisions on content delivery and interactivity.

From existing marketing materials:
“[redacted] takes your state tests and does an item-by-item analysis of alignment with state standards […] It presents data at the student, class & school level and aligns test results to state standards.”

“The Standardized Testing Analysis module allows principals and teachers to view data from the last standardized test given to students, in most states last spring, at the school, class and student level.”

“[redacted] gathers each student's collection of standardized test results and compares them to your state's standards.”

Class Detailed Report “I want to know whether one or two students are pulling the class average down or if the entire class is having a difficult time
“ Drill Down

From previously signed contracts “Attachment E”
“Standards Mastery Reporting including standardized test scores and drill down for up to five states.”

“Standardized Test Score Analysis. End users will be able to view student results on the previous year’s state administered standardized test (for tests correlated to the state standards) [See assumption 4] End users will be able to benchmark such results on standardized tests to students’ current performance based on classroom assessments. [See assumptions 5 & 8]

Assumptions:

1. Standardized test results will appear on all Standards Mastery Reports.
2. Standardized test results will be imported to [redacted] by [redacted] staff. States will supply results as data files stored on CD-ROM, Zip Disk, or accessible via FTP or Web standard protocols.
3. Student ID’s that have been entered in [redacted] during school setup will match Student ID’s in test results data files. (see pg. 6 AAD 3.2 Import Std Test Results) Currently Student ID is an optional field in Setup Wizard, need to investigate how schools match ID’s with those used by testing services.
4. [Redacted] Content Analysts will create correlations between Standardized Test scores and the previous year’s standards. These correlations display on drill down into each test score from (any or just individual?) Standards Mastery reports.
5. Standardized test results more than one year old are not reported. For subjects tested every 2-3 years, no scores will be displayed and therefore no benchmark will be possible.
6. Each result reported for individual students must correlate to at least one standard.
7. Each result reported for individual students will correlate to fewer than five(?) standards. (This is a concern of content granularity rather than technical limits)
8. Drill-down from Individual reports will display a report of the test’s score-type (scaled, grade equivalent, local Stanine, etc.) and text of previous year’s standard.
9. Mid-year exams, even if they are designed to resemble statewide and national standardized tests will be treated as classroom assessments. Classroom assessments can manually be correlated to standards within the Achieve app.
10. One column for each standardized test will be necessary unlike classroom assessments where scores may be summarized and displayed in a single column (and therefore may need to be weighted?)

New questions and assumptions based on [redacted] datafile:

Some standards will have more than one score for a given test.

Scores may not apply to the lowest level of standard, but to a parent standard. Repeat score for each substandard? Create way to group substandards that apply? 

Do xsubjects respect the same grouping of substandards that the test report uses? What if a substandard is included in the test report but excluded by the xsubject?

 

Great Unknowns

• What data will states provide to us and when?  Can it be correlated to specific students and specific standards? Will [redacted] need to correlate the previous year’s standards to the current years standards (since these correlations provide our basis for displaying standardized test results alongside classroom assessments of current year’s standards)?
• If Norm-referenced tests are not correlated to state standards will importing them to [redacted] serve any purpose at all?   Do we plan to hire staff to correlate items on standardized tests?
• Can Standardized tests provide the same drill-down capability that we will provide with classroom assessments?   Some states may not offer reports on individuals that detail to the standard level.
• We’re saying specific test scores need to drill down to show score-types, do they also need to indicate assessment type? Who determines/assigns assessment type? How does this affect validity of benchmarking classroom assessments of one type against standardized assessments of another? Non-issue?
• Could some test results want to be weighted differently or be excluded from individual, class, or school summary reports?
• How many standardized assessments can each state ask us to import? Potentially there is no end to alternative/formative assessments.
• What if data is not delivered via standard electronic media?
• What if the Student ID’s entered during Setup Wizard DO NOT match Student ID’s in standardized test reports? 
• If a tested standard has no equivalent in the current year’s standards, is it ok to not display that result anywhere? Do we strive to map as many test results as possible or prefer only dead-ringer correlations?
• If different tests use fancy calculations for longitudinal scores, do we report their numbers or calculate our own as a flat average of individual scores?
• Will we want reports to be able to filter on cut scores to show only problems?
• Do school and class summaries need to report Standard Deviation or Standard Error of Measurement to be meaningful?
• Since standardized scores administered for the current year are reported in April, we assume that they cannot be included in Standards Mastery Reports for the grade-year in which they are administered.
• Are there scientific reasons why it would not be valid to convert some criterion-referenced scores to percentages?  How will we accommodate the following score types (taken from pg. 18 3.1 AAD Stds Mastery Report)
  • Scale Score
  • Grade Equivalent
  • Local Percentile Rank
  • Local Stanine
  • Local Normal Curve Equivalent
  • Achievement/Ability Comparison
  • Strand Performance Category Cut
  • Performance Standard
  • Alternative Assessment Allowances/Special Considerations

 

Future Considerations

What to do with standards or sub-standards that aren’t assessed? A specific report on just this could be useful– test makers will hate us.

Could a standards-centric report be created as a useful way to plan a roster for a tutoring session? The report could list names of students in any class that would benefit from a tutorial on a specific standard.

Could a principal’s Standards Coverage report list all standards in a subject and list the sections where each is taught including the hit rate. This way, two teachers that avoid teaching a particular standard can be referred to one who teaches it willingly. Also a willing teacher can be prompted to publish his or her lesson plans.

Do we need Unique ID’s for districts and schools, as well as students? Do we want to adopt the unique ID’s used by the National Center for Education Statistics?  http://nces.ed.gov NCES lists 17k districts in U.S. and territories.

Noticed dual terminology in UI and documentation, can we get consistent and use either Standards Coverage or Hit Rate but not both?

 

State Standardized-Test Types
N = norm referenced C = criterion referenced

AK

(http://www.eed.state.ak.us/tls/assessment/)

k-1                   Development Profile ?
3, 6, 8              Benchmark Tests C
4, 5, 7, 9          TerraNova Cat-6 (CTB/McGraw Hill) N
Grade 10         High School Grad Exam C

AK by subject:
English: 3, 4, 6, 7, 8, 10
Math: 3, 4, 6, 7, 8, 10
Science: No testing
Social Studies: No testing
AFT says tests align to standards? YES

 

IL

(http://www.isbe.state.il.us/assessment/)

2                      Illinois Assessment of Reading and Math C
3, 5, 8              ISAT N
4, 7                  ISAT (diff subjects) N
9, 10                ISAT –optional N
11                    PSAE (Prairie State Achievement Exam) C
??                     Illinois Measure of Annual Growth in English C & N
??                     Illinois Alternative Assessment  alternative

IL by subject:
English: 3, 5, 8, 11
Math: 3, 5, 8, 11
Science: 4, 7, 11
Social Studies: 4, 7, 11
AFT says tests align to standards? YES

 

MS

(http://www.mde.k12.ms.us/acad/osa/mccas.html)

2-8                   Mississippi Curriculum Test C
11                    Functional Literacy Exam C (will be phased out by subject tests)
6                      NRT (TerraNova)  N
8-12                 SATP Subj Tests (Algebra, Biology, English II, US History from 1877) C
4, 8                  NAEP (only some schools National Assessment of Education Progress)
4, 6, 7              Grade Level Testing Program (4, 7= Writing; 6= Normed?) N

hard to know MS assessments:
secondary        MS Career Planning and Assessment ?
?                      ACT N
9-12                 Harcourt Test for HS mentioned by Steve He?
           
MS by subject:
English: K-8, 10, 11
Math: K-5, 7-9, 11
Science: 10
Social Studies: 11
AFT says tests align to standards? YES

 

PA

N.B. - No Statewide  Student ID#

(http://www.pde.state.pa.us/)

3, 5, 8, 11        PSSA C Reading and Math (March-April)
6, 9, 11            PSSA C, open-ended rubric Writing (February)
4, 7, 10            PSSA C Science, Environment, Technology (beg. 2004-2005)

PA by subject:
English: 5, 6, 8, 9, 11
Math: 5, 8, 11
Science: No testing
Social Studies: No testing
AFT says tests align to standards? YES

 

TN

 (http://www.state.tn.us/education/tsintro.htm)

3-8                   TCAP Achievement N
3, 5, 8              in 2002-2003 TCAP Achievement criterion-ref portion C (see paragr. 4)
5, 8, 11            TCAP Writing performance rubric
9-12                 TCAP Competence prob N (to be scrapped in 2004)
9-12                 TCAP Gateway Tests C (since test must be passed to graduate has to be C)
9-12                 TCAP End of Course prob C (required to be 15% of course grade!)

TN by subject:
English: 3-9, 11
Math: 3-10
Science: 3-8, 10
Social Studies: 3-8, 10
AFT says tests align to standards? No

 

WV

 (http://wvde.state.wv.us/assessment/)

K                     Metropolitan Readiness ? (kindergarten only)
8                      ACT Explore N
4, 8, 12            NAEP (random studts) (No indiv. student, school, or county results are generated.)
1-11                 SAT9 N

No Criterion Ref Tests?!

CTB/McGraw will dev for 2003-4 http://wvde.state.wv.us/news/410/

WV by subject:
English: K-11
Math: K-11
Science: 3-11
Social Studies: 3-11
AFT says tests align to standards? NO

 

Test Makers

CTB/Mc-Graw Hill  http://www.ctb.com/
CTB's unique Customized Reporting from a Standardized Test, Second Edition (CRST|2) can provide reports that are directly related to your state content standards. CTB's TestMate(r) Clarity provides you with the ability to aggregate and disaggregate assessment data, and to reorganize the data for optimum utility.
See also Classroom Manager
http://www.ctb.com/products/product_detail.jsp?FOLDER%3C%3Efolder_id=34969

 

Pearson NCS http://www.ncspearson.com/

 

Riverside (Houghton-Mifflin) http://www.riverpub.com/
NY, NJ, and OH state tests

 

Harcourt  - 

SAT-9 Standard Achievement Test