Assessing Critical Thinking in Writing

Our Writing Initiative rubric for evaluating critical thinking in student writing is very simple. It has three competencies that we feel should be made evident in the assignment, and that should be evidenced in the resulting papers. There are many aspects of critical thinking that might have been used as measures, but we chose three that we felt were common to almost all good writing assignments.

The three competencies are:
1) Find appropriate evidence and accurately interpret it
2) Identify pertinent arguments pro and con and rebut those arguments that counter to your thesis
3) Draw a thoroughly justified conclusion(s)

We ask instructors to have students indicate when they submit assignments to their portfolios which assignments best address critical thinking competencies. Part of that request is because we want students (and faculty!) to be very conscious of critical thinking elements in the assignment.

At the start of the Initiative in 2008, we realized early on that instructors too quickly said that "all my assignments use critical thinking" but were then unable to actually point out where that critical thinking occurred.

The assessment of this area in writing samples has shown improvement the past five years. Students are most capable in achieving competency in evidence gathering and interpretation and in drawing conclusions. This can partially be explained because these skills are addressed through research in not only WI classes but in other courses as well. The competencies we stress in our information literacy component also reinforce these two areas. The development of argument and drawing conclusions are a focus in Composition I & II essays, and are stressed in the College Writing Exam essays.

The weakness that is evident in our analysis is one that keeps many papers in the “competent” part of our rubric rather than receiving a rating of “accomplished.” We observed in earlier years and have addressed in subsequent Faculty Institutes, that many instructors create assignments that require a single argument rather than the more “accomplished” criteria of addressing multiple and counter arguments.

That also affects the third competency, since a writer addressing only one point of view will most naturally arrive at only one conclusion.

In some writing situations, such as in the sciences and on the College Writing Exam, multiple arguments are actually seen as “going off topic.” We have made very conscious efforts this past year in our training and when working with faculty on assignment creation to have them create assignments which allow and encourage multiple perspectives on an argument.

In our analysis of portfolio samples from the fifth year, we again found improvement in all three competencies but competency 2 (argument) both showed the greatest improvement and still remains the weakest area. 

Our conclusion is that this is an area that needs to be addressed by more courses using writing across the curriculum. The problem may be more in the assignments that are given, than in students' ability to accomplish the task.


All of our rubrics, including the critical thining rubric, are available online.

Moving Assessments Online

As with so many other aspects of education, assessment is moving more and more online.

Assessment movements usually start in K-12 by mandates and sometimes trickle up to higher education later. In the 2014-15 academic year, more than forty states will implement their online testing programs. Thirty states already do their summative assessments online, but the new assessments will require more of the schools including changes in instruction, and possibly different tech devices and high-speed bandwidth.

These new assessments are being created by two major consortia of states, the Partnership for Assessment of Readiness for College and Careers (PARCC) and the Smarter Balanced Assessment Consortium (SBAC). They are based on the Common Core State Standards (CCSS) and will try to assess address higher-order thinking skills and problem solving.

As online assessments they will include the traditional multiple-choice questions but also simulations, computer-based items, short answers, and a lot of writing.

The objective is to cover all the standards, some of which are harder to measure, especially online. Part of the appeal of online testing is being able to obtain results quickly with the hope that teachers can use the results to affect instruction for classes and even specific students.

Can An Exam Really Evaluate Student Writing?

I am pretty sure that teachers have been complaining about poor student writing since the earliest days of having places that we would call schools. And it's difficult to get teachers to agree on what constitutes good writing and especially on how you can evaluate student writing.

The portfolio is a popular approach that has gained some traction, but many schools from elementary to college level still use (or are required to use) a writing essay.

Doug Hesse, director of the writing program at the University of Denver, prefers grading student portfolios assembled over several semesters.“Within the writing community,” he said, “there’s a lot of wariness especially of single-measure, single-sitting exams.”

A recent article on InsideHigherEd.com looked at two colleges that differ on the role of exams in evaluating student writing. But before I comment on those examples, let me talk about Passaic County Community College's own writing exam.

The exam came about decades ago when today's employers and faculty members were themselves college students. In PCCC's case (so the story goes), an important local business man complained to the College President that he was getting job applicants who had graduated from PCCC and "couldn't write."

And so, the PCCC College Writing Exam (CWE) was implemented. It was seen as away to validate that a graduating student was an adequate writer. It is an important graduation requirement for all PCCC students because failing it means you don't graduate, no matter what you have as a grade average. As far as we know, we are the only college in NJ (2 or 4 year) that has this requirement. That is seen by some as admirable, and by others as foolish.

For the CWE, students are asked to write that most basic of forms - the  five paragraph essay. That is about 450 words about a general topic or a topic from their major that they have not seen before. Students will be given the general theme (such as "survival" or "personal health" or "elections") but not the actual question so that they may prepare some ideas for their essay.

Students use a computer to type their essay, but it does not allow spell check, grammar check or Internet access. The testing allows for two hours.

The essay will be read and holistically evaluated by at least two readers. Readers are faculty and staff members from PCCC who have been trained in scoring the exam using a standard rubric.

The article I read talks about Old Dominion University and Hampden-Sydney College who also decided that the best way to make sure their graduates could write was to require them to pass a writing test before graduation.

Students at Hampden-Sydney’s have a 33% failure rate for students taking the test for the first time. That's not very different from our failure rate. The article states that faculty work with those students to "make sure they improve their writing and graduate on time."
Old Dominion University (a much larger public university) had problems with the test results. More than 600 of 42,000 seniors in the past decade finished every other degree requirement but couldn’t pass the test and graduate. Big universities and smaller community colleges don't like students not graduating because of one item.


So this year Old Dominion announced it was phasing the test out and instead asking students to instead earn at least a C in three classes -- two English courses and a writing-intensive class in their major. Current students who prefer to take the assessment still can.

It might sound like an easy way out, but the university Provost, Carol Simpson, said the assessment did a poor job of gauging writing ability. She believes writing should be evaluated in the classroom, not on a test that students take on their way out the door.

Simpson's quote in the article has been echoed many times on the PCCC campus by faculty: “Many students waited until they had completed all other requirements and then go, ‘Oh gosh, I have to take this exam.’ They didn’t prepare for it. It was a timed, stressful exam. It just wasn’t really a good example of their ability to write.”

Students don’t seem to be mourning the exam’s death. “Does one test, like a five-paragraph essay, really determine your ability to write analytically?” asked Student Body President Luis Ferreira, who passed the exam last year. “It didn’t really serve a serious purpose for me. It was just kind of a formality.”

Those colleges give a reading on a topic which students thyen need to take a position on and respond with a well-reasoned paper of around 500 words. The essays are graded by multiple faculty members.

Smaller and all-male Hampden-Sydney has a different approach. Before the exam, students must pass two rhetoric classes (3 if they test poorly as incoming freshmen) which are capped at 14 students and focus on grammar and essay composition.

If a student fails the test, generally taken late in his sophomore year, he has two opportunities to pass it again as a junior and to seek help from writing instructors. The rare student who hasn’t passed by senior year is placed into a writing-intensive course in which he is tutored. The student is then given an alternative assessment. The student then writes three essays but isn’t held to a time limit. 

Can one essay taken under conditions that do not resemble the way students do the majority of their academic writing really be a fair evaluation of their ability to write?

Course-Level Assessment Using Student Portfolios

The Writing Initiative team uses student portfolios to assess overall writing, the use of critical thinking and the use of information literacy in assignments. Writing Intensive students have been using an electronic portfolio product called eFolio.

Using a randomized selection of at least 10% of the student artifacts submitted to portfolios across all sections, the team evaluates assignments using the WI rubrics for writing, critical thinking and information literacy that are used in the WI course sections.

The rubrics for writing and critical thinking were aligned with the scoring methodology used for our College Writing Exam so that we could make some longitudinal comparisons with data on that exam that goes back well before the Initiative.

The information literacy rubric and scoring adheres to the methodology that has been used for a number of years prior to the grant to analyze IL in the College Experience classes.

Some instructors who find limited success having student use the electronic portfolios have been allowed to submit paper or electronic copies of student writing for analysis.


The percentage of WI students who upload at least one assignment to their portfolios each semester had consistently been about one-third, which was less than anticipated. However, during the Spring 2011 semester, portfolio use rose to 48%, nearly a 50% increase. There is certainly improved buy-in by faculty and students that has occurred as the Initiative progresses. Portfolios – and the Initiative as a whole – are a growing part of the institutional culture.

Students have access to eFolio tutorials and support documentation via LibGuides. Use of the online video tutorials, recorded with Echo360 lecture capture technology, has increased over 200% from 2010 to 2011. In-class eFolio sessions are conducted as needed, and students with eFolio questions can drop in or schedule an appointment with the Technology Resource Specialist.

The Writing Initiative has played a very important role in making the case for portfolios as a means of campus-wide assessment. Last year, the Vice-President of Academic Affairs outlined a five-year, three-phase plan to adopt portfolios for assessment in several academic programs. The Education/Early Childhood Education department has systematically adopted department-wide use of portfolios. Portfolios are also being piloted in a small sample of Composition I and ESL classes.

Institutionalizing the Writing Initiative at PCCC

During Year 4 (2010-2011), the Writing Initiative had a focus on the institutionalization of the Initiative efforts beyond the end of the grant in Fall 2012.  Determining how the Writing Initiative efforts will continue once grant funds are no longer available and what resources, personnel and services the College is willing to support are our primary concerns during this fifth and final year of the grant.

We have met or exceeded the goals set forth in the grant for Year 4 and implemented some activities that were not part of the original grant but have been warranted by circumstances and opportunities. For example, the expansion of faculty professional development by joining with the College Writing Committee in offering Roundtables, the continuation and expansion of the Year 3 “Connections” and additional support for the College Writing Exam (CWE) are all extensions of the original goals.

The Writing Center has been booked fully for the current staffing and hours the past year.  The original intent of the grant was that the Center would transition from a potential client base of about 1000 students who have taken a WI course to serving the entire college-level community of approximately 4000 students at the three campuses.

The College’s Academic Council Constitution went through a revision this spring and changes to committee duties currently performed by Initiative staff members indicates one direction the College is likely to take after the grant.

The Initiative team has created a best practices sustainability plan that examines the mission and objectives, personnel, and budget concerns for the Writing Initiative, and we will be posting information about that plan in upcoming post here.

Formative Student Writing Assessment

Everyone who has had an educational pedagogy course has studied formative and summative assessment.

Summative assessment is generally carried out at the end of a course, unit or project and in an educational setting, summative assessments are typically used to assign students a course grade.

Formative assessment is a self-reflective process intended to enhance, recognize and respond to the learning. It is a bi-directional process between teacher and student and should be designed to evaluate students on a frequent basis so that adjustments (by the teacher and by the student) can be made to help them reach target achievement goals.

Much research in the teaching of writing is about the use of formative assessment. Anyone who has taught writing knows how difficult (if not impossible) it is to have a final "test" to measure a student's ability to write.

Formative assessment improves student writing by monitoring it through the prewriting, writing, and revision processes.

I was looking over some notes I made and slipped into a book on assessment years ago. The notes still seem true today as I go into the fifth year of the Writing Initiative at PCCC. Here were my top 10 thoughts while reading that book.

1. monitor, diagnose, and provide continual feedback to your students
2. break large writing assignments into several smaller tasks
3. vary your feedback methods
4. discuss the myths and realities of writing in an academic setting
5. make sure students self-assess and reflect orally & in writing on their progress
6. teachers and students should create and use rubrics and checklists
7. need to support students in all stages of the writing process
8. each piece of writing a student creates, no matter how brief, formal or informal, is an opportunity to learn
9. use individual, small-group, and large-group discussions
10. feedback is key to a "writing workshop" approach

These are three of the relevant books on my shelf. They are probably intended for K-12 teachers but I find much of it relevant to teaching in higher education, especially at a community college. That is not meant as being at all negative about what we see in colleges today, but it does reflect on the observation that the same issues continue with students from secondary education as they move into higher education. And the same issues exist with the teaching of writing, and to a greater degree, in most college classrooms.


   

• Because Writing Matters: Improving Student Writing in Our Schools
• What Student Writing Teaches Us: Formative Assessment in the Writing Workshop
• Successful Student Writing Through Formative Assessment

Grant Wiggins on Assessing Student Performance

Assessing student performance has always been challenging and controversial.

Assessment is more than testing. Intellectual performance is more than right answers. New systems of assessment more closely examine students' habits of mind and provide teachers and policy makers with more useful and credible feedback.

Leaving aside the true politics of assessment, I want to look at the work that has been done by Grant Wiggins.

Grant Wiggins (http://www.grantwiggins.org) who is probably best known for being the co-author, with Jay McTighe, of Understanding by Design.

How do you know when students understand something and how can you design learning experiences that make it much more likely that students understand content and apply it in meaningful ways?

Understanding by Design (UbD) is a method that seeks to answer these questions. One thing the approach provides is a template for creating curriculum units based on the idea of "backward design" method.

Backward design is a method of designing curriculum by setting goals before choosing activities or content to teach. That seems so logical, but for much curriculum the goals are not considered until some assessment is needed. The content is often decided first, possibly by the selection of a textbook. The idea of backward design is to select content and methodologies and then teach towards those goals that were set first. It goes against the idea of "textbook coverage" which is so prevalent in higher education.

Backward design is "backward" only because it challenges the traditional methods of curriculum planning. In backward design, one starts with goals, then assessments and finally lesson plans.

UbD hs been used as a framework to maximize student understanding in grades K through college and Wiggins was lead consultant on many state assessment reform initiatives, such as the portfolio project in Vermont and performance assessment consortia in New Jersey and North Carolina.

In Educative Assessment, Wiggins challenges schools to adopt a different type of student assessment. Rather than the one-shot audit tests that are generally used, it examines assessment grounded in authentic tasks enabling students to self-correct their ongoing performance.

Many of these types of assessments - rubrics, portfolios, reflective practice - are being used by us in the Writing Initiative.

In Assessing Student Performance: Exploring the Purpose and Limits of Testing, Wiggins discusses how testing differs from assessment.

Why are performance tests, by themselves, not an adequate system of student assessment? How might we better "test our tests" beyond current technical standards?

Like many educators, he does not feel that increased national testing will offer the accountability of schools that is obviously needed. The book analyzes problematic practices in test design and formats like multiple choice that prevent students from explaining their answers.


Further Reading
Understanding by Design
Understanding by Design: Professional Development Workbook
Integrating Differentiated Instruction & Understanding by Design
Making the Most of Understanding by Design
Assessing Student Performance: Exploring the Purpose and Limits of Testing

The Writing Initiative Task Force

The Writing Initiative Task Force is a group of administrators and faculty charged with monitoring the creation and assessing the implementation of the Writing Intensive (WI) courses at the college.

As the Initiative is now in its fourth year (of the 5 year grant), this group becomes more important in assuring the sustainability of the Initiative goals beyond the time of the grant.

The group is headed by Greg Fallon, Associate Dean of Learning Resources and the Title V Project Coordinator for the Writing Initiative.

Matters addressed by the Task Force during this past year included:

1. Performance of WI students on CWE (presentation of data by Dr. Gurvinder Khaneja of Institutional Research; positive signs; findings to be shared at fall meeting of all full-time faculty)
2. Number of WI courses developed and to be developed (19 WI courses have been developed thus far, just one short of the number originally planned; Human Growth and Development and Economics are currently being considered; Computer Concepts and Applications needs another trained instructor)
3. Composition I as a pre-requisite (previously, students were enrolling in WI courses without this pre-requisite; the pre-requisite is now in place after intervention by registrar)
4. Need for additional sections of WI courses as well as additional instructors (announced at meetings of departmental chairpersons)
5. Insufficient use of ePortfolios (to increase use, Dr. Kineavy plans to introduce ePortfolios in the required first-year experience class so that students are comfortable with them by the time they take WI courses)
6. Expansion of evaluation committee (at present, WI staff does all of the assessment of student submissions to ePortfolio—plans to include faculty from Gen Ed, Assessment, and College Writing Committees to accommodate increasing number of students)
7. Review of WI courses to maintain WI standards (Task Force responsibility)
8. Alignment of rubrics (currently, the writing rubrics for the College Writing Exam (CWE) and the WI courses are perfectly aligned; Ken Ronkowitz and Elizabeth Nesius have further worked out the alignment of the CWE, WI, and WritePlacer rubrics to show how they correspond for assessment purposes)
9. Getting the word out (various efforts: WI faculty presentations, a dedicated blog, an anthology of student writings, and the branding of the Initiative through the logo and its use on promotional materials)
10. Involvement of General Education Committee (a co-chairperson is a member of the Task Force; the Gen Ed Committee doing inventory of the writing already done in Gen Ed courses; establishing outcomes expected of PCCC students with respect to writing and critical thinking)
11. Involvement of Assessment Committee (the chairperson is a member of the Task Force; the review of writing outcomes assessment for WI courses will fall to this committee)

The current Task Force members are:

1. Dr. Jacqueline Kineavy, Senior Vice President for Academic and Student Affairs
2. Prof. Edward Mosley, Chairperson for the Department of Humanities and Social Sciences
3. Ms. Donna Stankiewicz, Assistant Dean of Nursing Education
4. Mr. Todd Sorber, Executive Director of Institutional Advancement and Workforce Development
5. Dr. Gurvinder Khaneja, Executive Director of Institutional Research
6. Mr. Greg Fallon, Associate Dean of Learning Resources & Title V Project Coordinator
7. Mr. Ken Ronkowitz, Director of the Writing Initiative
8. Ms. Elizabeth Nesius, Coordinator of the Writing Center
9. Mr. Ken Karol, Technology Resource Specialist
10. Dr. Ida Greidanus, Professor of Biology
11. Prof. Mark Hillringhouse, Co-Chairperson of the College Writing Committee
12. Prof. Margaret Holland, Co-Chairperson of the College Writing Committee
13. Prof. Alan Mitnick, Chairperson of the English Department
14. Dr. Lonna Murphy, Co-Chairperson of the General Education Committee
15. Dr. Christine Redman-Waldeyer, Chairperson of the Assessment Committee

Rubric Alignment

There has been concern at PCCC in the Testing Department about the adoption of the Writeplacer test at all nineteen NJ community colleges this year. There has been discussion about the possibility of using it as our College Writing Exam (CWE).

Elizabeth Nesius and Ken Ronkowitz from the Initiative had aligned the CWE rubric with the WI writing rubric two years ago so that the course-based assessments hat we use can be better correlated to CWE baseline scores for the college at large.

This past summer, they also aligned both rubrics with the College Board's Writeplacer rubric that PCCC uses for initial college placement. We presented that alignment to the College Writing Committee and Dr. Stassis.

Although we can show an alignment of scores and in criteria, the Initiative team feels that it would be inappropriate to use an entry/placement exam as the exit exam as the CWE is now designated.

The Initiative will also continue to use our own modified criteria language for the WI rubric since the user audience is students and not professional essay readers.

Grant Year 3 Report

The year for our Writing Initiative ends at the close of October and so we are now into year 4 of the grant.

In the program narrative, 28 objectives are identified to be completed by the end of the five-year project period. During year three, the narrative directed the team to focus on the six specific objectives, which are summarized below and will be covered in more detail in future posts.

This past year the Team has worked with faculty to design seven new writing intensive (WI) courses. Five were piloted in year three and two are being piloted in the fall 2010 semester. We also trained 14 new WI instructors and launched 8 new sections of previously developed courses.

In addition to the program assessments done by Institutional Research, we have added our own course-based assessments of all WI courses offered through the artifacts in student ePortfolios.

We also conducted our first student and faculty surveys and focus groups following the spring 2010 semester.

In faculty development, we have now trained 38 full-time and part-time instructors in teaching WI courses and in using the technologies, critical thinking and information literacy.

We have served more than 600 WI students so far in the Writing Center, through one-on-one appointments and student workshops.

After the October 2009 visit by our outside evaluators, their report identified areas for improvement that included branding, the Task Force, writing rubric alignment, working with faculty, clarifying WI minimum requirements, faculty compensation and course-based assessment via portfolios.

We will address those areas in future posts as we prepare for our outside evaluation visit on November 30.

Documenting Teaching Effectiveness

We often have a sense in education that something is working (or not working) but it's difficult for us to produce the evidence that some practice or piece of technology is the reason.

The Writing Initiative at PCCC is a five-year federal Title V grant and we have numerous requirements to assess our success at improving writing and the teaching of writing at the college. Producing evidence of effectiveness beyond the anecdotal information from teachers, students and our writing consultants one of our biggest challenges.

Right now, the Initiative team is working on our year 3 report for the outside evaluators who review the grant each October.

The Seeking Evidence of Impact Initiative that is part of the EDUCAUSE Learning Initiative (ELI) is an ambitious new initiative intended to bring the teaching and learning community into a collective discussion about ways of gathering evidence of the impact of innovations and current practices.

They are trying to bring the latest research and commentary in the area of learning technology effectiveness together. This includes white papers, articles, websites, reports, and other resources.



I have been looking back into books on assessment that I used in the past. One of those is Classroom Assessment Techniques (Angelo and Cross) which is a general handbook that offers teachers at all levels how-to advice on classroom assessment, including what classroom assessment entails and how it works, and also how to plan, implement, and analyze assessment projects.

If you are new to formal assessment, the case studies that detail the real-life classroom experiences of teachers carrying out successful classroom assessment projects are useful. Although the book focuses on K-12 settings, the information often applies to higher levels too.



For higher education, Ken Bain's What the Best College Teachers Do (Harvard University Press, 2004) is also an interesting book to look to for some possible answers to what makes a great teacher great. The book was the result of a 15-year study of nearly a hundred college teachers in a wide variety of fields and universities.

One finding was that it's not what teachers do, it's what they understand. In general, lesson plans and lecture notes matter less than the way those teachers comprehend their subject and how much they value the learning process itself.

Think back on your own best teachers - especially those in high school and college - and they probably impressed you with how well they knew their subject but also with the way they engaged you and challenged their students.

For me, that always included interaction beyond the classroom Q&A. I knew that the best teachers felt that teaching really matters and that students can learn.

Classroom Research: Implementing the Scholarship of Teaching is a text more likely to be used by college faculty members in groups and in workshops. It covers topics like problem-based discussion and integrating teachers' experience with recent research and theory on learning. It also provides assessment and research projects that can be used in classes.

If you want or need to get more into the quantitative part of assessment, practitioners, Assessing Performance: Designing, Scoring, and Validating Performance Tasks has step-by-step guidance for developing, administering, scoring, and validating a range of performance tasks. Though I find developing scoring tools, training raters, reducing rater bias, reviewing scores and report results less interesting, it is a necessary part of the job for some people in any educational setting.

Part of our Writing Initiative at PCCC is to introduce students and faculty to technology. ePortfolios, eTutoring, lecture-capture technology, and supporting courses with Web materials like LibGuides are all part of our day-to-day work. Using Technology Evaluation to Enhance Student Learning, (Teachers College Press, 2004) is a look at the technologies that have real, long-term payoffs for student learning so that you can make research-based decisions concerning the use of educational technology.