Norman Edmund’s Educational Blog

Personal Attributes. One of the first things to be considered is the personal attributes of the researcher. Is there evidence of

dishonesty
self-delusion
favoritism
extreme bias
emotional instability
recklessness
anti-social disorder
rigid thinking
sensation seeking

Technical Questions. Fairness requires that those involved in a misconduct investigation be technically qualified or have technical advisors so as to reach the correct decision.

Recommended Reading. Evidence of misconduct is so complex that, rather than continue, I recommend the following:

Responsible Science - Ensuring the Integrity of the Research Process Volumes I and II (1992). Published by the National Academies of Science.

Honor in Science (1991). Pamphlet published by Sigma Xi, the Scientific Research Society.

Federal Rules of Evidence. See www.judiciary.house.gov

Betrayers of the Truth by Broad and Wade (1982).

The Scientific Method Today - my pamphlet can be reviewed as to where there may be serious violations of the usual procedures of the research process.

In my research on the scientific method, the most frequent evidence of misconduct that I noticed was in the social sciences. The ignoring of contrary evidence seems to be widely prevalent.

This ends, for the time being, my series on integrity.

In addition to the two previous provisions defining findings of research misconduct, the one above applies. Charging anyone with misconduct is serious business. Thus, the government requires that there be a preponderance of evidence to support the charges.

Let’s define “research process”:

The American Heritage Dictionary defines the noun research as “(1) Scholarly or scientific investigation or inquiry. (2) Close, careful study.”

Let’s examine the definition.

“Scholarly” - The method of scholars is another name for the scientific method. Time and scholarly study have shown the scientific method to be the best method of obtaining reliable knowledge. Thus, “scholarly” means using the scientific method.

“Scientific investigation” - Back in 1863, T.H. Huxley told us “The method of scientific investigation is nothing but the expression of the necessary mode or working of the human mind. It is simply the mode at which all phenomena are reasoned about, rendered precise and exact.”

Please note that Huxley stated “the mode [method] at which all phenomena are reasoned about, rendered precise and exact.” Research is far from simply using common sense. It requires disciplined inquiry.

So again, research basically means using the method of investigation, which is the scientific method.

“Inquiry” - Inquiry is necessary in research, but just inquiry can result in aimless wandering unless the method of inquiry, which is the scientific method, is followed.

“Close, careful study” - To conduct close, careful study, you must follow, in complex matters, the method of study, the scientific method. Centuries of research have established this as reliable knowledge.

Conclusion. An accurate interpretation or definition of the word “research” is research based on and following the method of research, also called the scientific method.

The word “research” has often been carelessly or broadly used and interpreted. In addition, any time someone conducts research, he/she does not have to go through all the stages of the process. An accurate definition has generally not been understood.

The same dictionary defines “process” as “a series of actions, changes, or functions that bring about a result.”

Thus, “research process” properly defined based on the literature means the scientific method or process. Because of the misunderstandings about the scientific method described in my book End the Biggest Educational and Intellectual Blunder in History this is not well understood.

The Scientific Method Gains Legal Status

A historic event occurred in 1993. In the Daubert v. Merrell Dow Pharmaceuticals, Inc. [509 U.S. 579 (1993)] decision, the U.S. Supreme Court ruled:

“But, in order to qualify as ’scientific knowledge’ an inference or assertion must be derived by the scientific method. Proposed testimony must be supported by appropriate validation - i.e., ‘good grounds,’ based on what is known.” [emphasis mine]

The Court also stated:

“A new theory or explanation must generally survive a period of testing, review, and refinement before achieving scientific acceptance. This process does not merely reflect the scientific method, it is the scientific method.”

Subsequent to the Daubert decision of 1993, the question arose as to whether it applied only to scientific testimony. The U.S. Supreme Court held in the Kumbo Tire Co. Ltd. v. Carmichael [526 U.S. 137 (1999)] case that

“Daubert factors may apply to the testimony of engineers and other experts who are not scientists.”

I would interpret all this (although I am not an attorney) that research means following the scientific method and that the scientific method has legal standing and should be considered in cases of misconduct.

More on preponderance of evidence next week.

Misconduct must include three elements listed in my May 5, 2006 comments. This is the second necessary element. There are occasional frauds done intentionally or knowingly, but the ones that present the greater danger and uncertainty are those done recklessly.

In my comments of May 5, 2006 I point out that not doing a thorough search and ignoring contrary evidence could be defined as “reckless.” It could be also be defined as “significant departure from accepted practices of the relevant research community.”

The dangerous situation of research being done recklessly should be impressed on all researchers. I believe that it is not a big problem in the natural sciences. However, in the social sciences, including education, it is a huge problem. Evidence for this is the many comments about the unreliability of research in the social sciences.

What should be done? First, we need an educational campaign to teach researchers in the social sciences that they must follow the scientific method (SM-14 type) in their research.

Second, government agencies that make grants should enforce the requirement that the scientific method be used.

Third, government grantees should be careful to follow the scientific method (SM-14 type) for their own protection.

In my May 5, 2006 comments, the first finding of research misconduct required under Federal regulations was

“A finding of research misconduct requires that:
There be a significant departure from accepted practices of the relevant research community.”

In my May 12, 2006 comments, I established that educational research is covered by the regulations in footnote 2.

It is my opinion (and remember that I am not an attorney) that many educational researchers are making “a significant departure from accepted practices of the relevant research community.” This is based on my research showing that they are not following the scientific method and the general acknowledgment in the literature that educational research is not reliable.

In the Federal regulations, does “relevant research community” mean the individual fields covered under footnote 2, or does it mean the research communities covered by the regulations?

If it means the individual communities, then what is an act of misconduct in some communities is not in others. However, certainly no community of researchers would approve of widespread ignoring of contrary evidence that I believe exists in the educational field.

Whatever the case, it seems that clarification is needed on what applies and why. For, despite the Federal Regulations on Research Misconduct, there is much evidence and acknowledgment that educational research is not reliable.

From the Federal Register (vol. 65, no. 235, December 6, 2000):

“III. Responsibilities of Federal Agencies and Research Institutions
Agencies and research institutions are partners who share responsibility for the research process. Federal agencies have ultimate oversight authority for Federally funded research, but research institutions bear primary responsibility for prevention and detection of research misconduct and for the inquiry, investigation, and adjudication of research misconduct alleged to have occurred in association with their own institution.”

This places the primary responsibility for the prevention and detection of research misconduct on colleges, universities, and other grantees. Wow! This seems unfair if the Federal agencies have been sanctioning the theory that the scientific method doesn’t exist, isn’t necessary to be used, or ignoring it. The U.S. Department of Education and the National Science Foundation have never made a grant to research what the scientific method consists of.

In several recent acts, Congress has said that scientifically valid or scientifically based research must be done. I have shown that this means research following the scientific method. However, even though I have pointed this out (in a special report) to the U.S. Department of Education and some other agencies, they are not acknowledging it to be so.

Under the requirement that grantees have primary responsibility, they could have the possibility of billions of dollars of liability if a test case establishes that the scientific method should have been used in research and was not used. This doesn’t seem fair, as perpetuation of the Blunder has been primarily by leaders of our government agencies. Universities and other grantees should protect themselves by being sure that on current grants their employees follow the scientific method.

In the Federal regulations regarding research misconduct (Federal Register, vol. 65, no. 235, December 6, 2006), footnote 2 says:

“Research, as used herein, includes all basic, applied, and demonstration research in all fields of science, engineering, and mathematics. This includes, but is not limited to, research in economics, education, linguistics, medicine, psychology, social sciences, statistics, and research involving human subjects or animals.”

Question. The literature has numerous mentions by authoritative people that educational research is flawed and not dependable. Since much of this has been done under grants from U.S. government agencies, what is wrong? Are the regulations not being enforced? Or is there some other answer?

Under Federal regulations as published in the Federal Register, vol 65, no. 235, December 6, 2000, findings are defined as:

“A finding of research misconduct requires that:
There be a significant departure from accepted practices of the relevant research community; and
The misconduct be committed intentionally, or knowingly, or recklessly; and
The allegation be proven by a preponderance of evidence.”

These descriptions are so broad that great care must be exercised in claiming misconduct. Such a charge is a very serious matter.

Sigma Xi publishes an excellent booklet Honor in Science. It quotes C.P. Snow (The Search, 1959):

“The only ethical principle which has made science possible is that the truth shall be told all the time. If we do not penalise false statements made in error, we open up the way, don’t you see, for false statements by intention. And of course a false statement of fact, made deliberately, is the most serious crime a scientist can commit.”

It goes on to say:

“It is not sufficient for the scientist to admit that all human activity, including research, is liable to involve errors; he or she has a moral obligation to minimize the possibility of error by checking and rechecking the validity of the data and the conclusions that are drawn from the data.”

To this I would like to add a warning to do a thorough search and not ignore contrary evidence, since this could be defined as “reckless.”

Definition in the Federal Regulations - “Plagiarism is the appropriation of another person’s ideas, processes, results or words without giving appropriate credit.”

On the surface this may not seem as important as fabrication and falsification. However, the smooth functioning of the whole system of science depends on honesty and a devotion to cooperation, respect, proper credit, and communication. For a scientist to work hard and then find that some scoundrel has claimed credit for his ideas and research is a frustrating experience that generates outrage.

A charge of plagiarism is one of the major types of misconduct complaints to Federal agencies.

Here are some examples of plagiarism compiled from Stealing Into Print (1992) by Marcel C. La Follette:

Translating foreign works and claiming credit for the research
Unattributed stealing of someone else’s ideas by a peer reviewer, even rushing to get into print before the orginator of the ideas
Using paragraphs or pages from someone else’s papers without giving credit
Attributing quotes or results to another person which are incorrect or false
Misrepresenting authentic data
Plagiarism in computer programs
Using others’ drawings, pictures, art, etc.
Misrepresenting authors or co-authors
Lying about a manuscipt’s publication status
Failure to give credit to important contributing researchers

Another book with stories about plagiarism is Betrayers of the Truth (1982) by Broad and Wade.

Because plagiarism is such a touchy situation, it is important that guidance material be taught to students and its elementary nature understood by everyone. The internet has many sites devoted to plagiarism by students writing papers for school assignments.

For information on the web about Federal Policy on Research Misconduct see:

Office of Science and Technology Policy, Federal Register vol. 65, no. 235, Wednesday, December 6, 2000 at www.ori.dhhs.gov/policies/fed_research_misconduct.shtml

National Science Foundation:
www.nsf.gov/oig

U.S. Department of Health and Human Services:
www. oig.hhs.gov

U.S. Department of Education:
www.ed.gov/about/offices/list/oig/index.html?src=oc

Definition in the federal regulations:

“Falsification is making up data or results and recording or reporting them.”

There have been cases in which researchers deliberately and intentionally falsified their research. This type of activity is not tolerated in science. Researchers should realize that it will usually be detected sooner or later in the peer review process.

A Big Problem in Social Sciences

There are many references in the social sciences, which includes education, to the unreliability of research in these fields. For example, an article by Burkhardt and Schoenfeld in the December 2003 issue of Educational Researcher says:

“Roughly a decade has passed since Carl Kaestle (1993) wrote his well-known Educational Researcher article, ‘The Awful Reputation of Educational Research.’ Despite significant advances in theory and method, it is hard to claim that the situation has improved. Indeed research in education may be accorded even less respect now than a decade ago. Consider, for example, the following statement from the U.S. Department of Education’s Strategic Plan for 2002-2007 (2002):

Unlike medicine, agriculture and industrial production, the field of education operates largely on the basis of ideology and professional consensus. As such, it is subject to fads and is incapable of the cumulative progress that follows from the application of the scientific method and from the systematic collection and use of objective information in policy making. We will change education to make it an evidence-based field (p.48).”

The question is, why is this research so bad? Is it deliberate falsification and fabrication? The answer is no, but nevertheless, something is wrong. I believe that the basic cause of the unreliability of education research is the non-use of the scientific method, which is not only for scientists. The scientific method is the complete method of creative problem solving and decision making for all fields. Often education researchers

Do not go through all the mental activity stages of the scientific method
Ignore or do not search enough for contrary evidence
Fail to attempt to falsify their hypotheses before reaching a conclusion
Do not acknowledge or realize that they are subject to the same code of ethics as those in the natural sciences in terms of government grants

Brief History of the Scientific Method in the Field of Education

John Dewey’s book Sources of a Science of Education (1929) discusses education as a science. I quote from the 60th anniversary edition of Dewey’s book Experience and Education (1938):

“It is argued that science and its method must be subordinated; that we must return to the logic of ultimate first principles expressed in the logic of Aristotle and St. Thomas, in order that the young may have sure anchorage in their intellectual and moral life, and not be at the mercy of every passing breeze that blows.

I see at bottom but two alternatives between which education must choose if it is not to drift aimlessly. One of them is expressed by the attempt to induce educators to return to the intellectual methods and ideals that arose centuries before the scientific method was developed. The appeal may be temporarily successful in a period when general insecurity, emotional and intellectual as well as economic, is rife. For under these conditions the desire to lean on fixed authority is active. Nevertheless, it is so out of touch with all the conditions of modern life that I believe it is folly to seek salvation in this direction.

The other alternative is systematic utilization of the scientific method as the pattern and ideal of intelligent exploration and exploitation of the potentialities inherent in experience.”

For a number of years after 1938 educators were interested in the scientific method to the extent that it became one of the primary aims of modern education (Conant, 1947). Unfortunately, after the publication of On Understanding Science (Conant 1947), the situation changed dramatically. In a number of his books, Conant erroneously claimed that the scientific method does not exist.

Professor Jack Easley (1922-1994) wrote as his doctoral thesis a 30-page essay entitled Is the Teaching of Scientific Method a Significant Educational Objective? He challenged and, I believe, falsified Conant’s views and contrasted Dewey’s, Schwab’s, Conant’s, and his own views on scientific method. The essay was published in Philosophy and Education: Modern Readings (1958). Easley also wrote “Scientific Method as an Education Objective” (in Encyclopedia of Education, 1971). Unfortunately, his research was ignored.

An example of the change is the fact that national education reform programs from 1957 to the present do not include the teaching of the scientific method and a formula for its stages.

In contrast to the non-inclusion of the scientific method in national education reform programs, about 65% of science textbooks used in K-12 and universities have continued to cover the scientific method in discussions of 1 to 30 pages and often included a formula for it.

For a while, some books on education research continued to stress the use of the scientific method in education research; The Art and Science of Investigation (Mouly, 1978) and Educational Research - An Introduction (Borg and Gall, 1971) are two examples. Although I have not made a survey, I believe that current books on education research do not include the scientific method.

What happened in the education field and the other social sciences is described by Steve Fuller in Thomas Kuhn: A Philosophical History of Our Times (2000). He says:

“All of these revelations induced a collective sigh of relief from practitioners of the humanities and the social sciences, who had a hard enough time making sense of each other, let alone agreeing on a common method. They quickly latched on to Kuhn’s ideas and declared that they too were respectable knowledge producers laboring under paradigms.”

Operating under paradigms is not an efficient procedure. It is the scientific method that is the greatest quality control method ever recognized and developed. The complete story of this situation is told in my book End the Biggest Educational and Intellectual Blunder in History.

Here is what should be done to end this unintentional falsification:

Educational researchers and others in the social sciences must learn the scientific method.
Government officials should enforce the requirement that data not be omitted.

(Note: I will be posting only once per week while doing the series on integrity.)

Fabrication involves violation of honesty, the basic feature of the integrity of the research process.

In this process the researcher deceives by making up results or altering actual results and reporting them in an effort to deceive everyone. In many ways it is difficult to separate fabrication and falsification. For example, ignoring contrary evidence can cause either.

As reported in Honor in Science, published by Sigma Xi, centuries ago Charles Babbage (1792-1871) described trimming, cooking, and forging:

“Trimming: the smoothing of irregularities to make the data look extremely accurate and precise.
Cooking: retaining only those results that fit the theory and discarding others.
Forging: inventing some or all of the research data that are reported, and even reporting experiments to obtain those data that were never performed.”

Other acts of fabrication mentioned in the literature:
allowing self-deception, biases, political influence
false results in favor of expectations
gullibility
deliberate manipulation
propaganda for one’s theories
naming false co-authors
selective reporting of data
fabrication of bibliographic material
falsely reporting stage of research
failure to report what would make theory invalid
measurements never made
miscalibration of instruments
accepting work of unqualified assistant
elevating results or forecasts
false interpretations

Scientists have always been concerned about the integrity of the research process. This interest has increased now that the federal government has instituted more regulations about it. More information can be obtained by going to www.ostp.gov. From the site:

I. Research Misconduct Defined
Research misconduct is defined as fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results.
Fabrication is making up data or results and recording or reporting them.
Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record.
Plagiarism is the appropriation of another person’s ideas, processes, results, or words without giving appropriate credit.
Research misconduct does not include honest error or differences of opinion.