Final Report
VALIDITY REPORT
Validity
What is this?
Validity is one of the most
important components of test usefulness, and it refers to the extent to which a
test measures what it intends to measure.
What is this used for?
It is used for providing a
real measure of the particular content or skill(s) that we intend to measure.
Exam context
The type of exam applied was
Progress Test, it contains from unit 1 to unit 3 of the Empower Student’s book
C1, the group that was evaluated Target Language V (7:00 am to 9:00 am), the
exam contains 5 sections, grammar, vocabulary, reading, listening and writing.
The methods that were used are: correcting mistakes, multiple choice,
underline, match, T or F and fill in the blanks.
Data base
We used excel to make our
data base to have the values of the exam, these values are: candidate’s number,
sections, item’s number, good and bad answers, specifying by 1 and 0.
First, we made the format in excel, making a table with the values: candidate’s
number and the sections specifying by the item’s number.
Then, we fill each gap with 1 and 0, where 1 represented right answers
and 0 wrong ones.
Next, we add the good items of each candidate and we add the good ones
of each item. We add the totals of the good items and we had the total of it
and we did the same with the good ones of each item.
The formula that we used to discriminate if the items were reliable or
not was: IF= No. correct.
No. total of candidates
Once all the data was collected the
interpretation began. Using a range scale from 1 to 10 the items were rated
according to the number given from the results of the columns divided to the
rows. In such scale the results of the division were placed in the range scale in
order to decide whether the item was easy or hard. If the resulting number was
an amount between 3 to 7 then it indicated that the item was within the
requirements, otherwise being a lower or higher level would indicate the
failure of such item being way too hard or way too easy.
Grammar Section
In the grammar section were
25 items divided in 4 parts, in the first part the method was used writing right sentence, in this part I
noticed a little difficulties because
the instruction was not clear causing confusion in the answers; in the second
part was multiple choice here all was good
the answers were understandable and comprehensible for the students, the third
part was underline the correct answer in
this part was the same, the instruction and the options were clear and the
students could answer well, the last part was rewrite the sentences correctly
here also I noticed difficulty to answer
because the instruction and the
sentences were written a little confused and the options were complicated to students.
Good Items
|
3-4-5-6-7-8-9-10-12-15-16-17-20-23
|
Bad items
|
1- 2- 11- 13- 14-18- 19- 21- 22- 24-25
|
Part I Correct the five mistakes with adverbs.
Original items:
1. - Please slowly try to speak. / 2. - We will be probably late.
Depending our database, items
1 and 2 were difficult for students with a 0.91 and 0-23 which tell us that
those items were difficult to answer, one of the reasons maybe it was a bad
item was because the instruction was not very objective with what was wanted to
achieve, so one of the recommendations that I suggest is the following
New modification of
instruction and items:
Order the following sentences using the adverbs of manner correctly.
1. He ate greedily the chocolate
cake. Correct: He ate the chocolate
cake greedily.
2. She gave us generously
the money. Correct: She
gave us the money generously.
With this new instruction and the items, we will see If the students are
able to order the sentences correctly using the adverbs of manner.
Part II Choose the most appropriate complement in each sentence.
Original item:
11.- The engineers said that the bridge….
a) will be ready by the end of the year. b) have finish yet. c) hasn’t build
Checking the result of this
item in the database, also it was a difficult item to answer for the students,
so the suggestion for this item is to change the sentence with a verb tense
correct, so the following item to change it is:
11.- At this time next year, John...
a) will be living in Boston. b) will be live in
Boston. c) has lived in Boston.
The correct option is A, because we are talking of a true fact in the
future about John.
Part III Underline the correct word in each sentence.
In this part, we find that
some elements are not understable or that, according to our database at the
classification level, they were so easy that even if the students did not know,
they could answer this item; the vocabulary used here could also have caused
confusion in the students, so my suggestions for each item would be the
following:
Original item:
13. - If he treated me like that I wouldn’t stand for him/it.
Modified item
13. - If I had been born in Mexico, I would speak Spanish more fluently/properly.
I decided to put this item because it uses adverbs of the book and is
attached to the subject of this, so based on this I think the students
shouldn’t have problems in answering because they have already seen. So, the
answer correct is fluently.
Original item:
14. - The government should set up a committee/a minister to sort the
matter out.
Modified item
14.-The Venezuelan government used to be relatively good/well off, but it has
invested more money in debts than in its city.
I chose this item in this part because it uses the vocabulary of the
book and brings it closer to the subject that was in the book.
Original item:
18. - I told her off/told off her for leaving the office unlocked.
Modified item
18. - I told the boys off/ told off the
boys for making so much noise.
I decided to leave the same
vocabulary of linking words in this item but change the sentence, so that the
students probably understand better the context in which the word is used.
Original item:
19. - After a week on the ice expedition ran into difficulties/potholes
Modified item:
19.-During our trip to Manhattan on the highway we came across with difficulties/potholes.
I left the same words of the
item just change the sentence because in the previous item without
understanding the meaning of the words was easy to answer based on the context
of this.
PART VI Rewrite the five sentences with the most appropriate word.
In this part of the exam I
would leave some same items, based on the database I think the error was in the
instruction so I would only change the instruction, the first item and the last
item for another.
The instruction that I would leave for this part is the following:
Order the following sentences and charge the
words that are misused, the words that are in italics are the ones that are
wrong.
Original item:
21. - It was an disaster outright!
The answer for this item is the following:
It was an outright disaster!
Original item:
22.-He makes by hand his homework.
The answer for this item is the following:
He does his homework by hand.
Original item:
24. - They will be probably in
time.
The answer for this item is the following:
They will probably be on time.
Original item
25.-My career is ruined outright!
The answer for this item is the following:
My career is outright ruined!
Vocabulary section
In the vocabulary section
were 20 items divided in 3 parts, in the first part the method was used matching, in this part we noticed a difficulties because there were bad spelling in the
options. The second part was right
however in the third part the method was used fill in the blanks, the
instructions confused students, some of them have difficulties because the
instructions mentioned that the options were of quantity and some of them
weren´t and they didn’t applied in the context. As result the items had bad
results.
Good Items
|
26-27-28-29-31-33-34-35-36-37-38-40-43-44
|
Bad items
|
30-39-41-42-45
|
PART I Match the word with the definition
The item #30, in the matching part, was founded to be under the regular
range indicating that such item was too easy for the student.
Original item.
5. - substantially “To a great or significant extent.”
This item could be more appropriate to the level of students.
Modified item.
5.- reluctant “On willing unhesitant”
This item could be more appropriate to the level of students.
PART III Complete each sentence with the correct quantity words from the
table.
In this part, first it is necessary to change the instruction.
Original instruction.
Complete each sentence with the correct quantity words from the table.
Modified instruction.
Complete the sentences below with the words in the box.
Original item.
14.- I’ve been checking things off the list one by one and now I
think I’ve done everything.
Modify item.
14.- Many species live only in the heart of the rainforest.
Original item.
16.- We’ve got an awful lot to do and precious little time to
do it in, so let’s started now.
Modify item.
Anything might be hiding in the dense vegetation.
Original item
17.- You can study online or go to a language academy. Either way,
I’m sure you’ll improve your English.
Modify item.
This is truly an untouched wilderness.
Original item.
20. - All too often I get students in my class who just don’t have the
level to pass the exam.
Modify item.
Tourist come for the preserve beaches and gentle sea.
Reading comprehension section
For Reading Comprehension an abstract from the book used in target language
V (Empower C1) was used. The abstract had four short paragraphs whose theme was
“Dirty Britain”. This section was divided in two parts. The first one consisted
of six questions following the ABC pattern having a one point value each
question. For each question four possible answers were given but only one was
correct. Students were asked to circle the correct answer according to what
they had comprehended from the text. Unfortunately, there was an error in the design
of question number five whose answers did not match the question and were the
same as in number one. In the second part of the section, four
true-or-false-type-of-questions were used to evaluate reading comprehension
from the text. Such question had a one point per correct item. There was no
comment or problem regarding this section.
|
Good Items
|
1 – 2 – 3 – 4 – 6 – 7 – 8 – 9 – 10
|
|
Bad items
|
5
|
Part I Circle the correct answer
Original item:
5. - The Irish Government introduced….
a) Britain looks perfect c) You can see how Britain is now
b) You can see Britain at its dirtiest d) The grass has thickened on the
verges
After applying the exam item number five had a validity report of 1.7 which
indicates that the question was so difficult that it was almost impossible to
answer because its answers were not correct. The possible answers given were
copied from question one in the same section so that it did not match at all to
question number five. It is recommendable to change the answers in this item
for the appropriate using substantives in the beginning of the answer.
Modified item:
5. - The Irish Government introduced….
a) A tax on non-recyclable carrier bags b) A law on leaving supermarkets
to give away plastic bags
c) An attempt to follow the example on protests d) A reduction on 90% of
their plastic bags
by checking the text it was found that the actual answer for this question
was “a tax on non-recyclable carrier bags” so this answer was provided along
with other three to the student in order to have a better chance to give the
correct answer.
LISTENING COMPREHENSION SECTION
This section used five questions with the format free answer according to
what was heard in the audio. Students were asked to listen to a five minutes
long audio about an interview of a filmmaking then they had to answer the
questions writing down the name of the character who said specific things in
the audio. They had five questions to answer.
DISCRIMINATION REPORT
The purpose of the discrimination index is to
determine if the items of an examination can be separated into different
values; good, regular or bad and based on that check whether the test meets the
objectives of the evaluation.
So, according to this in our database we realized that some of the
results of the items didn’t discriminate, only 7 items were in a range between
30 and more which means that they are at a good level but could improve, the
other items if they must be changed because they are below the range 19 which
means that they are poor reactive and therefore do not
discriminate in the exam, besides that they do not fulfill the objective of
knowing if with these reagents the knowledge can be measured previous of the
student when answering it, We could also find out that the results varied a lot
with the first and second versions of the formulas but through this we were
able to identify what the problems with the items were and we were able to
improve them.
With the help of the formulas, and the reagents that are the main ones
to evaluate, we could make our discrimination table and understand that
depending on the exam along with the items we can know who achieved a higher
score, as well as the opposite of who could have had a low score on each part
of the test.
We also realized that some parts of the exam instructions weren’t clear,
because some students told us that they didn’t understand it, and they just
answered by answering. We think that we as teachers, evaluators must to be
careful and aware of design a test. Another aspect that we need to be aware is
about the content of the exam, some of this exercises or tasks were not in the
units that we wanted to be value, and this can cause confusion between students
at the time of answering. We had to change the bad items for ones that contain
topics from the units; as a result we have a combination of quantity words and
vocabulary from the unit 3.
We realized that in the listening section, one item had to be changed
but we notice that the designers use an audio from one webpage, and the
instructions were clear, students only had to write the name of the person that
said the statement, and we think that maybe this kind of items don’t need to be
change, just we need to be aware of the length of the audio, because it was to
long and students were bored, the complained about it.
Comentarios
Publicar un comentario