Measurement in educational research and assessment

I. Preestablished Instruments and Archival Data
- data: numerical (attendance rate), verbal (interview transcript) and graphic (picture

- types of data: preestablished and self-developed
Preestablished Instruments
Preestablished instruments---standardized: already- made or piloted measuring tools including tests, observational rating scales, questionnaires or scoring protocols for interviews.
- Standardized instruments characteristics:
● a fixed set of questions or stimuli; fixed time frame, fixed set of instruction and identified responses; measures specific outcomes and is subjected to extensive research development and review;
● performance can be compared to a referent such as a norm group, a standard or criterion, or an individual’s own performance
> Norm-referenced tests: comparing student’s performance with that of a norm group (eg. TOEFL)
> Criterion-referenced tests: comparing against a predetermined standard of performance
> Self-referenced tests: measuring an individual student’s performance over time to see if it improves or declines when compared with past performance.
- Five Types of Preestablished measures: achievement, aptitude, personality, attitude or interest, behaviors.
1. achievement tests—measuring what has already been learnt
2. aptitude tests—predicting what one can do or how one will perform in the future
3. personality tests—measuring self-perception or personal characteristics, traits, or behaviors
4. attitude or interest scale—assessing attitude toward a topic or interests in areas
5. behavior rating scales—making diagnoses of problems, frequency or intensity of behavior
Archival Data
Archival data: have already been collected by individual teacher, school, or district rather than by a researcher. Examples of archival data are student absenteeism, graduation rates, suspensions, standardized state test scores, and teacher grade-book data.

II. Scales of Measurement
Specifically, there are 4 types of variables that can be measured quantitatively, and these are called levels or scales of measurement.
● Nominal scales—measuring variables that are categorical or classes.
● Ordinal scales—numberings of different levels or ranks. The distance is not equal; this limits the types of statistical tests that can be applied to the data and also limits the conclusions about differences between persons at different ranks.
● Interval scales—numberings of different level in which the distances, or intervals, between the levels are equal. It is absent of true zero score, in this case to say that a score of zero means the absence of something is not totally accurate.
● Ratio scales—(considered producing the most precise data) including the properties of nominal, ordinal, and interval and also include a true zero point.

III. Summarizing Data Using Descriptive Statistics
Descriptive statistics: to show the patterns in the data.
Frequency Distribution
- distribution: describe the range of scores and their overall frequencies. Scores may also be displayed in a graph. For ordinal scale of measurement of data, they are usually connected by a line, and the graph is referred to as a frequency polygon. If categorical, the graph is called histogram.
> a Normal distribution—looks like a bell-shaped and symmetrical, with the highest point on the curve and most frequent scores clustered in the middle of the distribution: when large groups are randomly sampled and measured.
> Skewed distributions: asymmetrical, meaning the scores are distributed differently at the two ends. – a negatively skewed distribution—most of the scores are high, but there are a small number of scores that are low.
+ a positively skewed distribution—most of the scores are low, but there are a small number of scores that high (Fig. 4.3, p. 78).
- The “outlier” in a skewed distribution pull the “tail” of the distribution out in that direction.
> Bimodal distributions have two clusters of frequent scores as the two humps in the distribution (Fig. 4.4, p. 79).
What is a Typical or Average Score? Measures of Central Tendency
Three common measures are the mode, mean, and median.
Mode: the score occurring most frequently. If the distribution is asymmetrical, the mode may not be a precise estimate of central tendency.
Mean: arithmetical average of a set of scores. In a skewed distribution, the mean can be misleading.
Median: the score that divides a distribution exactly in half when scores are arranged from the highest to the lowest. It is a stable measure of the central tendency of a set of scores although there are a few outlier scores in a distribution that are much different from the rest.

Measures of Variability
1. Range: the difference between the highest and lowest scores in a distribution. It is not a stable or precise measure of variability because it can be affected by a change in just one score.
2. Standard Deviation: is the average distance between each of the scores in a distribution and the mean. One way to describe variability is to consider how far each score is from that center score. It is called standard deviation because it represents the average amount by which the scores deviate from a mean. The larger the number, the more variable are the scores in the distribution (Fig. 4.5, p. 82). The standard deviation is widely used in part because of its relationship to the normal distribution, visually represented as the normal curve displayed in Fig. 4.6, which forms the basis for comparing individual scores with those of a larger group.
3. Normal Curve: One useful feature of normal curve is that a certain percentage of scores always falls between the mean and certain distances above and below the mean. These distances are described as how many standard deviations above or below the mean a score falls.

Types of Scores Used to Compare Performance
Raw scores: one cannot know what a raw score represents without knowing more about the mean and standard deviation of the distribution of scores.
Percentile ranks: are scores that indicate the percentage of persons scoring at or below a given score. So a percentile rank of 82% means that 82% of persons scored below that score.
Standard scores: telling how far a score is from the mean of distribution in terms of standard deviation units. Examples of standard scores are Z scores and college entrance exams (Fig. 4.6).
Z SCORE = (SCORE – MEAN)/STANDARD DEVIATION
Stanines: stanine scores are a type of standard score that divides a distribution into nine parts, each of which includes about one half of a standard deviation.
Percentage: Some types of data may be distorted when percentages are used. One should always examine both the total number and the percentage of persons when comparing measures.
Grade-Equivalent Scores: A grade-equivalent score is reported in years and months. So 3.4 means third grade, fourth month. A grade-equivalent score reports the grade placement for which that score would be considered average. This means that the average student at the reported grade level could be expected to get a similar score on this test.
Use of Correlation Coefficients in Evaluating Measure
Correlations: measures of the relationship between two variables. A correlational relationship is summarized using a descriptive statistic called a correlation coefficient. Regardless of sign (- / +), the size of the number shows how strong the relationship is between the variables.
- A positive correlation coefficient: one variable increases, the other also increases.
- A negative correlation coefficient: one variable increases, the other decreases.

VI. Evaluating the Quality of Education Measures: Reliability and Validity
Reliability: the consistency of scores—obtaining approximately the same score
Validity: what the instrument “claims” to measure is truly what it is measuring—accuracy
Reliability
- Even the most reliable test would not produce the exact same score. This difference is referred to measurement error. Factors that affect reliability include personal characteristics, variations in test setting, in administration and scoring of the test and in participant responses due to guessing.
- Standard error of measurement: SEM = SD √1 - r For example: SEM= 2.7, Observed score= 70, means that the student who received a score of 70 might score 2.7 points higher or lower if s/he retook the test.
- Stability or Test-Retest Reliability (Consistency across time): an average wait time of four or six weeks.
- Equivalent-Form Reliability (Consistency across form)
- Internal Consistency Reliability (Consistency within the instrument): the common method is through split-half reliability. Then Spearman-Brown prophecy formula is applied—the instrument must be long enough. Another approach is to examine the correlations between each item and the overall score on the instrument—a given score consistently measures the same amount of knowledge.
Validity
Does the instrument measure what it is designed to measure? So when constructing a test or using a standardized instrument, validity is the single most important characteristic.
- Content Validity:
1. sampling validity
2. content validity: involving examining each individual item to determine if it measures the content area.
- Criterion-Related Validity: examining a relationship of each measure. It reflects the degree to which two scores on two different measures are correlated.
1. concurrent validity: examines the degree to which one test correlates with another taken in the same time frame (the new and the old test produce similar result).
2. predictive validity: used to predict the future. After 1st time testing, waiting for a while, then if the correlation is high between the 1st test result and the next, then the 1st test is considered to have criterion-related validity in predicting future success.
- Construct Validity: involves a search for evidence that an instrument is accurately measuring an abstract trait or ability. Constructs are traits that are derived from variables and are nonobservable. Construct validity might include aspects of content, concurrent and predictive validity. Some questions might be involved in establishing construct validity:
> Does the measure clearly define the meaning of the construct?
> Can the measure be analyzed into component parts or processes that are appropriate to the construct?
> Is the instrument related to other measures of similar constructs, and is it not related to instruments measuring things that are different?
> Can the measure discriminate between (or identify separately) groups that are known to differ?

Finding Preestablished Measures
One source that many researchers use is the Mental Measurement Year-book (MMY) (Box 4.3. Web Sites with information on Preestablished, Standardized Tests, p. 97).

Criteria for Selecting Preestablished Instruments
A particular instrument may have a high degree of reliability and validity, but for a variety of reasons, it may not be suitable for the population you intend to study. Also, a researcher should examine past study to see what specific instruments other researchers in similar areas have employed. Finally, do not settle for the first measure that you find.

Reliability and Validity in Archival Data
It is important to consider possible inaccuracies that the data may contain. A researcher should validate or double-check the raw data. The researcher might conduct interviews with the individuals who originally collected the data. The researcher would also want to examine the instruments used to collect the data and any information about their piloting and administration. Another method for collecting the data should be employed too.

Complement Choice after ‘Opinion’

It is generally well known that there are other words and expressions coming after many nouns, especially abstract nouns, in order to complete the meaning of those nouns. Those words or expressions are called ‘complements’. These complements can be prepositional phrases, infinitive expressions or clauses (with/without prepositions). Since there are no simple rules for what should follow what specifically, it is necessary to learn, find or study what kind of structure can follow what noun.
In this study, trying to examine the choice of complement after the noun ‘opinion’, besides the study of the occurrence this noun, is the main aim of the researcher. To be more precise, the study tries to answer the question what kind of complement choice after ‘opinion’ made be the writers across the three-level essays composed of 82 (Low (25), Mid (29), and High (28)).
To answer this question, AntConc software is used; the incidence of ‘opinion’ is recorded across the three-level essays; then, whether and what complement after ‘opinion’ is used will be examined.
The occurrence of the noun ‘opinion’ was calculated. In Table 1 below, the result of the noun ‘opinion’ in the Low, Mid and High is presented. The frequency of the noun ‘opinion’ in the Low is 0.51% (n = 23); in the Mid, 0.46% (n = 27); and in the High, 0.53% (n = 30). It can be seen that the distribution of the noun ‘opinion’ among these three-level writer shows no significance fluctuation, when they get to higher level.
Table 1
The Frequency of ‘opinion’ in the Mid, Low and High
Low Mid High Total
F per 4477 F per 5828 F per 5643 F per 15948
Opinion 23 .51% 27 .46% 30 .53% 80 .50%
The figures in this table are rounded off to the first two decimals.
* F means Frequency

To further comment, it is worth noting that, based on Cambridge Advanced Learner’s Dictionary, two main kinds of complement after the noun ‘opinion’ are prepositional phrases (about/on) and that-clause. From the analysis, it is so remarkable that the three-level writers do not use any prepositional phrases as a complement after ‘opinion’ at all, except several that-clauses. Table 2 below shows how frequent that­-clauses are across the three-level essays.
As can be seen from Table 2, the highest choice of that-clause as a complement after ‘opinion’ is in the Low—17.39% (n = 4), followed by the Mid—14.81% (n = 4) and the High—10% (n = 3). From this, it can be explained that the Low to use more complements, specifically that-clause, after the noun ‘opinion’ than the others. Also, when they get higher they tend to reduce this complementizer.
In brief, no matter how frequently the noun ‘opinion’ is used, there is no significance difference among the three-level students. Furthermore, it is so surprising for they do not use prepositional phrases as a complement after the noun ‘opinion’, but a number of that-clauses instead, which among these three-level writers, the lower use the most and the higher use the least.
Table 2
Complement after ‘Opinion’
Low Mid High
F per 23 F per 27 F per 30
That-clause 4 17.39% 4 14.81% 3 10%
Prepositional phrase 0 0 0
The figures in this table are rounded off to the first two decimals.
* F means Frequency

Obstacles to the Educational Success of Cambodians in Japan: Hiroshima University

1- Introduction
100.000 foreign students in 21st century is a number expected by Japan in Higher Education setting, through government and private scholarships. Though there were only 75.800 foreign students as by 2002 (Badamsambuu, 2002), the number of foreign students has been increasing remarkably. At Graduate school of International Development and Cooperation (IDEC), Hiroshima University, over 60 percent of the students are foreigners. These foreign students are seen as the setting the educational pace for the rest of Japanese. However, taking a close look, it will be seen that some nationals (students of a nation) are not doing well as others. This may be true for Cambodian students at Hiroshima University and other universities as well.
The aim of this paper is to discover why it is that Cambodian students are not as academically successful as others. In order to answer to this question, several factors should be examined: historical and educational background, culture, family, language, and mental and physical health.
2- Historical and Educational Background
Tracing back and starting from the genocidal regime, the Cambodians had to live through the ‘killing field’, from 1975-1979, when Cambodia was under the Democratic Kampuchea, ruling by Khmer Rouge. Economy, education system, building, infrastructure were completely destroyed. Educated people were killed (about 2 millions); people, including family members were killed or tortured in front of one another. Those who survived had been living through starvation, pain and suffering. This caused Cambodians, young eligible for schooling in particular, to miss out their educational opportunities during and after that tragedy period. Moreover, education system of Cambodia nowadays has been worse, though in many respects, it has improved. The clear point in this is teacher’s demand for ‘informal fees’ from the students (Launey, 2007); this forces some students who cannot afford to drop out of class. This has been one of the obstacles for Cambodian children getting proper education, let alone studying English language.
Before civil war, Cambodia adopted French education system in which students had to pass exams in order to be promoted to the next grade. However, Cambodian students’ education was disrupted in many points. The educational system was completely devastated by Khmer Rouge when they took power. The Khmer Rouge killed the one who was educated. The only education provided at that time was only learning the time tables by memorizing and learning the Khmer Rouge’s ideals of citizenship. For the Khmer Rouge, education was the damage to its revolution. (Pak, 2001)
From 1975-1979, the children practically learned nothing about reading or writing (Khmer). In some areas, schools were not available for over 10 years. Cambodian children/students, thus, had no enough time to adjust themselves to a formal school structure in such a short period.
The Japanese school system is a new experience for them, not just because of the new language, but also there are multiple subjects to learn. Moreover, students have to be responsible for their own learning not like at lower level which students waited in the class for the teacher to come to teach and there is only one teacher for all subjects. The habit that teacher is the knowledge-giver and student is the receiver does not establish the critical thinking skills of the students. This system does not allow students to ask or create questions; this does not only make their creative thwart but also affect their interpersonal skills and relationship negatively (Pak, 201). This kind of negative experiences made them disadvantageous when entering the Japanese education system. This event is paralleled with event faced by foreign students of the University of the Ryukyu. 31.2% of 134 foreign students found the unfamiliarity with Japanese study methods one of the problems causing stress in their lives, and academic in particular (Randall, M., Naka, K., Yamamoto, K. Nakamoto, H., Arakaki, H., and Ogura, C., 1998).
Cambodian people/students do not like competition nor cooperation (Pak, 2001). However, at university level, in order to achieve good result, and some time to be accepted in idea, students have to fight against each other in ideas and/or to compete. By this, Cambodian students find it hard to adapt and adjust.
2- Culture
The same study by Randall et al. (1998) shows that culture is also one of significant causing problems for foreign students. In Cambodian culture, value and respect are given to high educated people. Particularly teachers receive great respect. Educating children responsibility is given to the teacher, even though the ultimate responsibility is on the children. Also, Cambodian culture believes that people are born with innate talent and disposition. Their success at school and in life is believed to be predetermined by these inborn abilities, therefore this belief makes parents hesitate in pushing their children in school. As a result, students are not so aggressive in studying that when facing with little obstacles they are likely not to pursue their study more. However, the Japanese school system is a competitive one, which students will be graded according to the performance. This system presents the obstacle to Cambodian students. During the sessions, though they know or have ideas to share, Cambodian students feel so shy that they will not talk unless they are asked or ordered to do so.
3- Family
Generally, student education does not depend on only the students themselves, teachers, or parents individually, but is a collective effort of individuals and family. Cambodian family-student interaction can be said superficial. Cambodian parents do/dare not push their children against their own interest and desire. Rather than put pressure on children to excel in education, parents tend to give advice and guidance only even though they may have high aspiration for the children. Whether single- or couple- parent family, this may also influence on the performance of the children. In 1990, percentage of Cambodian families that were survival together was 50% (Zhou, 1997). During Pol Pot regime, many Cambodian family spouses were killed, as a consequence, there has a high percentage of single-parent family. Raising children has become heavier burden for the family, let alone setting a role model for their children. Therefore, Cambodian children are lack of modality which they should look up to. A study by a sociologist, Zhou (1997), shows that couple-family provide better psychological environment, higher level of academic performance, and stronger educational aspirations than those in single-parent families. In short, lack of family modality, encouragement, and couple-parent may be the hinder in their academic pursuit, which may thus lead to lower performance level in their study.
4- Language
English is used as a medium of instruction at universities in Japan, at Hiroshima University in particular. Students must be English proficiency if they are to achieve academic success, needless to say about comprehension in the courses. Linguistically, it is believed that ‘sink or swim’, where the students are totally submerged in English in the classroom and have to learn on their own (Zhou, 1997). Students may find it hard to keep up and may be frustrated by not being able to understand or express themselves in the classroom. This language handicap is likely to be a considerable factor in preventing the success. It is even worse if the course inevitably attended by the students is conducted in Japanese language. Relating with Japanese language, those who possess high proficiency level of the language find studying and living in Japan more motivated, enjoyable and convenient than those who do not possess the language; as a consequence, they also study Japanese at the same time. Linguistically, Japanese is completely different from English, and Khmer specifically for several main points. To be more precise, the characters of the letters, grammar systems and even the syntaxes of both Japanese and Khmer are so different that Cambodian students find Japanese language hard to study. The hardship has become a stressor, which hinders the success in learning. As a result, those who fail in learning Japanese language find it less motivated in learning in other courses too. This point in case is also supported by the study of assessment of psychosocial stressors and maladjustment among foreign students of the University of the Ryukyu, which most students (55.2% of 134 informants) reported that language inadequacy also caused stress hindering their study (Randall et al., 1998).
5- Mental and Physical Health
Even though Japan is a part of Asia, it is completely different from Cambodia. It can be seen that Japan is a developed one which requires people to run along with a wheel of current socio-economic development and technology. Having no or little knowledge, in technology particularly, causes student life trouble. One more example of the difference can be seen in terms of climate. Cambodia has only two seasons: wet and dry, whereas Japan has four seasons: winter, summer, spring and fall. This climate difference may cause Cambodian students fairly weak in physical condition.
Additionally, living in this new environment requires them some time to adjust. They may need time to socialize with other foreign students, neighborhoods, and Japanese people in particular. Yet, being far away from motherland where students have been living for over twenty years will cause loneliness. In Okinawa, after the arrival, foreign students experienced serious mental health conditions; among 134 students, 39.6% had been very depressed, 11.9% were having a ‘nervous breakdown’, and 9.7% felt not want to live on (Randall et al., 1998). In order to overcome this problem, students may take some time to adapt, thus they may spend less time on their study, especially self-study. It is even more stressful when students cannot communicate successfully. A clear example for this point is that when students do shopping, they fail to get what they want to or to understand what is written on the label of the products. Otherwise, it is hard to find Japanese people, outside the university/school, to speak English; even though they can speak, their English will not help them to communicate effectively.
Once again, after Pol Pot regime, it was the time which children of that generation needed nutritious food the most in order to grow properly in terms of physical and mental health. Due to poverty, children of that time could not have enough food to eat. This missing period has resulted in making Cambodian students who are now at the university not so fully fledged physically and mentally.
6- Conclusion
Having observed the different factors that hinder the success in Education, we can see that a certain group of students are performing like this or like that in academic achievement. After looking at historical and educational background, culture, family, language, and mental and physical health, the answer why Cambodian students are not as good as others becomes clearer. However, there are many more factors that have not been examined in this paper that can help Cambodian students and other foreign students as a whole to be successful in their education, such as supervision of the supervisors; lecturers; other differences of the individuals including motivation, learning strategies, language aptitude; the management; other stakeholders and the community concerned. Overall, Cambodian students can improve their academic success once only they learn to adapt and adjust to living in Japan well.

How Should We Look at History?

The history, of Southeast Asia in particular, may be controversial, if different perspectives are taken into account. Thus, it requires us to rethink critically. One of the ways to view is doing the comparison study, which will help us to understand our history more accurately and from which we can learn from the experience of other countries dealing with problems and then we can select which better policies best applied to our country’s future problems.
One example of how we should look or compare attentively is the history of Thailand specifically. Thailand, in Thai National Museum, tried to reinvent it history by giving argument that Thai precedents were there in the early Pleistocen period in SEA; it rejected that Thai people originated from the southern China, Yunan by showing the difference in Thai and Chinese languages and the Chinese chronicle which showing that no Chinese immigration from China southward when Kublai Khaan invaded China. Another argument is that Thailand does not start from Sukhothai but traced back to Lopbury (it was doubtful based on the study of Phungtian ()); Thai was invaded by Mon, or Khmer but eventually it gained the land back. There are more explanations which can be questioned.
In order to get more accurate history or information, many different methods should be employed including oral history, cartoons, painting or movies. To be more precise, a case of a barber going along with Japanese army troops (Prof.’s Nakao’s lecture) is true; or, interviewing of anthropologists after the war provides more accurate account. This case was also true in Cambodian situation after Pol Pot regime (under Vietnamese influence) which Vietnamese barbers were everywhere in Cambodia at that time.
An idea of nationalism encourages the ruler to invent the history that is not relevant to the daily lives of people, but legitimize their role and land. This makes a case of what Manchuria is to be suspected. The same applies to what Thailand is.
Yet, a case of foreign researcher working on our history can be questioned in terms of reality. This point can be seen in Cambodian histories written by some foreign researchers including David Chandler, Milton Osborne and Francois Ponchaud. This can provide more or less true information, though, a further examination should be conducted.
In conclusion, comparison study helps us understand more clearly who history is written and rewritten (Cambodian history needs to be revised too), learn from the other countries and find out the best points to adapt or adopt.
It is not finished yet, another question can be raised: so, we should forget about the past or not! A case of Japanese army’s brutality to Chinese and Korean is somehow similar to the brutality of Thailand, Burma, Vietnam or Cambodia amongst each other.

Frequency and Function of 'that'

The Frequency and Function of relative pronoun/conjunction that: A Corpus Analysis of Three-Level Opinion Essays

To write concisely, one should pay attention to word choice and sentence structure. This is to eliminate unnecessary words, to use simple sentence as well as not to repeat the same things. However, Japanese writers tend to use the relative pronouns or the conjunction that heavily, which can be superfluous or distracting to English-speaking audiences (Fujii, n.d.).
This study aims to examine the use of the relative pronoun or the conjunction that and to see whether the incidence of that varies across the three-level opinion essays of the High, Mid and Low (based on the learners’ TOEIC scores).
This study is based on a set of English data composed of 82 learner essays in text format (High (28), Mid (29) and Low (25)). The total number of words in all the essays is 43240 (Word Types: 1663).
In order to meet this goal, AntConc software is employed; the main basic usages of that based on A Communicative Grammar of English (Leech and Svartvik, 2002) are used as a reference. After identifying the frequency in the three sets of data, the interpretation/close examine of that to identify its function/use is also provided. The incidence of that was recorded separately for the three levels.
The frequency of the relative pronoun/conjunction that was calculated. Table 1 (please see at the back page) presents the results of that across the three-level essays. As Table 1 shows, the frequency/occurrence of that in the Low is 1.56% (n = 70) per 4477 words followed by the High (1.39%, n = 5643) and the Mid (1.33%, n = 5828). The three different levels show tendencies for relative pronoun/ conjunction that. According to Table 1, that is higher in the Low than in the others. One can explain that the difference use of that appears that the Low students tend to use that by getting influence from spoken discourse. It can also be seen from the Table 1 that when they get go to higher level, they realize to use that moderately, though the discrepancy between the Mid (1.33%) and the High (1.39%) appears 0.06%.
The functions of that were determined and counted in this study, too. That not only introduces clause, but also is used a demonstrative adjective or pronoun. The prominent main function of that across the three level essays is in the relative clause as a Direct Object (e.g. I think that, I believe that, etc.). Table 2 (please refer to the back page) shows the distribution of the functions of that across the three-level essays.
Table 2 shows that the highest frequency of that relative clause as a direct object is in the Mid, 46.15% (n = 36), followed by the Low, 44.28% (n = 31) and the High only 37.97% ( n = 30). As can be seen in Table 2, the ratio of the Low/Mid and High shows a great tendency of natural spoken discourse. The lower level tends to write like a spoken discourse.
To conclude, the heavy use of that-relative clause in writing is quite predictable since it is the influence of spoken discourse. Moreover, the main function of the that-relative clauses is as a direct object of the verbs. However, as was mentioned earlier, the use of that-relative clause as a direct object with higher level students tends to decrease due to the awareness of the difference between the spoken and written English.