TOEFL^® Committee of Examiners 2024 Research Program

TOEFL COE research grants are intended to support research on topics related to the TOEFL iBT^® test, the TOEFL ITP^® Assessment Series and the TOEFL^® Essentials™ test, promoting high-quality language assessment research as judged by the COE and ETS. Applications that employ diverse and innovative research methodologies are encouraged, as are those focused on the TOEFL Essentials test.

Studies involving the collection of new empirical data can normally be funded up to US$100,000. This amount includes institutional overhead. ETS prefers to select bids which respect its nonprofit status. We therefore request that the overhead rate not exceed 15%.

1. Innovations in Language Learning, Teaching and Assessment

1.1 Investigate the application of the TOEFL Teaching Framework for curriculum, instruction and/or materials development in academic settings. (Please contact the grant coordinator at TOEFL_RC2@ets.org to obtain a copy of the TOEFL Teaching Framework.)

1.2 Explore how new technologies — including, for example, AI tools such as ChatGPT — can be harnessed to support innovation in test task design or in classroom-based language assessment.
 

2. Test Score Validity, Interpretation and Use

2.1 Provide support for core claims in the validity argument for the TOEFL Essentials test, as presented in the Design Framework for the TOEFL^® Essentials™ Test 2021 (PDF), in particular when it comes to the usefulness of test scores for making decisions related to English- language proficiency (extrapolation and utilization inferences).

2.2 Examine various uses of the TOEFL iBT, TOEFL ITP and TOEFL Essentials tests and their scores, in particular related to:

1. The perceptions and uses of certain test features by stakeholders involved in admissions and/or test preparation. Such features may include, but are not limited to, MyBest^® scores or the TOEFL Essentials Personal Video Statement
2. Admissions decisions makers’ use of language test scores (including TOEFL iBT, TOEFL ITP and TOEFL Essentials) to:
   • combine information from different sources (e.g., multiple test sessions) to make admission decisions
   • use language test scores for other purposes besides admissions (e.g., placement, monitoring, progress, exiting)
3. Common misconceptions that stakeholders may hold related to score interpretation, and the extent to which the provision of resources or training interventions may help to mitigate these misconceptions

3. English-medium Instruction

3.1 Investigate the English-language proficiency construct and related testing needs in diverse English-medium instructional (EMI) contexts worldwide. This includes the English needs of students, teachers and/or other educational stakeholders in EMI settings. (Proposals under this topic do not need to use TOEFL iBT, TOEFL ITP or TOEFL Essentials test data.)

4. Test-taker perspectives

4.1 Engage directly with test-taker populations who require accommodations for the TOEFL iBT, TOEFL ITP or TOEFL Essentials to explore their test-taking experiences, processes and perceptions

5. Other

Consideration will be given to research on other issues or innovations related to the TOEFL iBT, TOEFL ITP or TOEFL Essentials tests.

The datasets and research instrument described below will be made available as required for funded projects.

There are three separate TOEFL iBT Public Use datasets available to principal investigators funded by the TOEFL COE research program. The three separate TOEFL iBT datasets are from two test forms and include scores accompanied by demographic information about test takers (e.g., age, gender, native language, etc.) and copies of relevant test materials:

• Set 1. Item-level scores on all sections of the test for a random sample of 1,000 examinees from each form (2,000 examinees total).
• Set 2. Transcripts of spoken responses and final scores for each of the six speaking tasks for 240 examinees from each form, stratified by quartiles (480 examinees total).
• Set 3. Writing samples and final holistic scores for each of the two writing tasks for 240 examinees from each form, stratified by quartiles (480 examinees total).

There are two separate TOEFL ITP Public Use Datasets available to principal investigators funded by the TOEFL COE research program. The two TOEFL ITP datasets come from two test forms and include item-level scores and CEFR levels along with examinee demographic information (e.g., gender, native language, native country, etc.) and copies of relevant test materials. Each form includes data from 10,000 examinees, and the examinee populations for the two forms do not overlap.

Note: Requests for other data will be considered on a case-by-case basis and granted when feasible. Please be aware that legal concerns related to the protection of test-taker identity may limit what data can be shared with external researchers.

The TOEFL iBT and TOEFL ITP research forms will be available to principal investigators funded by the TOEFL COE research program.

For the TOEFL iBT test, two types of research forms are available:

1. two full, non-modifiable research forms delivered via internet
2. 3 full, modifiable research forms delivered via downloadable software (i.e., the TOEFL iBT Form Creator Software)

This software creates a stand-alone application which administers TOEFL iBT test items/sections/forms as specified by the researcher — up to three complete forms. The forms can be customized in a variety of ways (e.g., omitting items/sections, inserting pauses, changing timing) and can be administered on a local PC.

For TOEFL ITP tests, two types of research forms are available:

1. two paper-delivered research forms
2. three full, non-modifiable forms delivered via the internet

For data output, ETS will provide grantees with a scoring file for the Listening and the Reading sections (both TOEFL iBT and TOEFL ITP tests), and Structure and Written Expression section (TOEFL ITP tests only), as well as speaking samples (in .wav format) and/or writing samples (in .txt format) when applicable (TOEFL iBT test only). For each test section, either raw scores or scaled scores can be provided to grantees. For the Speaking and Writing test sections, ETS will arrange to have responses scored by TOEFL test raters. Scoring fees should be included in the project budget, and a scoring-fee estimate should be obtained by contacting the grant coordinator at TOEFL_RC2@ets.org.

For the TOEFL Essentials test, test vouchers for operational tests can be provided to grantees. Applicants must contact the Grant Coordinator at TOEFL_RC2@ets.org to obtain an estimate for test vouchers to include in their proposal budget.

Applications are invited from faculty or staff affiliated with nonprofit organizations and institutions (e.g., universities) with expertise in English language learning and assessment research. Proposals from current members of the TOEFL COE and for-profit organizations are not eligible. These studies are not intended to serve as the basis for student theses or dissertations.

Each awardee will be required to submit interim reports and at least one manuscript of publishable quality, in addition to appropriately acknowledging the support of the TOEFL program when disseminating the results of the funded work.

Note: Due to United States and European Union governmental sanctions, ETS may not be able to make payments to individuals or organizations in certain countries. The specific countries are determined by the United States and European Union governmental agencies and are subject to change. If your application is from one of these countries, you may be notified that your application cannot be considered for a TOEFL grant or award at this time.

Interested researchers must submit an intent form by April 15, 2023. Applicants should use the Intent Form Template (Word) and submit it electronically to TOEFL_RC2@ets.org, including "Intent to Apply — COE Research Grant (Last Name)" in the subject line. Selected applicants will be invited to submit full proposals.

Proposals must include a letter committing the applicant's institution to the proposal and stating its validity for a period of at least 90 days from the proposal's deadline date.The original letter (PDF) must be signed by a representative of the institution's contracting office with commitment authority and must be submitted with the proposal. Proposals will not be reviewed for consideration until such a letter has been received. The contracting officer may obtain a sample ETS grant agreement by submitting a request to TOEFL_RC2@ets.org.

Note: Some institutions are unable to comply with ETS's contractual language regarding intellectual property and data ownership. We recommend that the language be reviewed by the institution's contracting officer before signing the letter of institutional commitment.

For applicants who are invited to submit full proposals, proposal materials must be submitted via email to TOEFL_RC2@ets.org by 11:59 p.m. U.S. Eastern Time on July 15, 2023.

Submissions must include the following documents:

• two copies of the study proposal (one blind copy, one non-blind copy; see detailed requirements below)
• Principal Investigator's CV
• A completed Letter of Institutional Commitment (PDF)

The non-blind copy of your proposal must include all of the following key components, which are outlined in the below and are provided in the Grant Proposal Template (Word):

1. A cover page, which must include:
   • the title of the project
   • the name, affiliation and contact information of the principal investigator
   • the names and affiliations of co-investigators, if any
   • the name and contact information of institutional contract representative
   • the date of submission
   • identification of the research topic(s) addressed by the proposal
2. An abstract page, which must include:
   • the title of the project
   • a statement of the problem, including a brief description of the project proposed and a brief discussion of the implications or utility of the anticipated outcomes (250 words maximum)
   • identification of the research topic(s) addressed by the proposal
3. The body of the proposal should be approximately between 2,000 and 2,500 words and must include each of the following sections:
   • Rationale (a brief statement of the problem or the issue and relevance to one or more of the research topics above)
   • Literature review (a brief, focused summary of related research)
   • Research questions (the specific question(s) or research objectives to be addressed by the project)
   • Methodology (a detailed explanation of the proposed research methodology, including descriptions of the following):
     • study context and participants
     • variables investigated and instruments used
     • data collection procedures
     • data processing (data extraction and coding)
     • analyses to be conducted
   • Implications (a brief discussion of the implications or utility of anticipated or possible outcomes and relevance to practice or theory)
   • Limitations (a brief discussion of the limitations of the proposed study)
4. References
5. Budget and budget explanation (See the budget section of the proposal template for specifications.)
6. Schedule (list of the major activities of the project and the period in which each will be conducted; please assume April 15, 2024, as the start date for the project)
7. Proposed data management plan (one-paragraph description of the type of data to be collected, the type of permissions requested for collecting and storing the data (IRB/ethical approval), and processes for storing de-identified data)
8. Proposed dissemination plan (one-paragraph specification, in as much detail as possible, of where you plan to publish the manuscript(s) resulting from the funded research [e.g., as journal article, an ETS Research Report or other venue])

The blind copy of your proposal should be the same as the non-blind copy, except that it must not include:

• the cover page
• the author(s)' identifying information (including names and affiliations) anywhere in the abstract page, body, references, budget or schedule

Please use the proposal template, which contains the following formatting:

• Microsoft^® Word 20016 or later version (.docx)
• Times New Roman font size 12
• 8.5" x 11" (U.S. "Letter") page size
• 1-inch (1.0") margins on all sides
• Double (2.0) spaced
• For page numbering, paragraph indentation, references, graphs and tables, please follow APA style (American Psychological Association^®, 7th Edition)

Each proposal will be evaluated according to the following criteria:

• the proposal is relevant to the research topics identified above
• the rationale for conducting the research is sufficient
• the background or literature review is adequate and appropriate
• the research questions are clearly stated
• the data collection procedures are clearly explained
• the methods of data analysis are sufficiently detailed and appropriate
• the authors discuss the anticipated results and implications for the field of language learning and assessment
• the authors specify the limitations of the proposed study
• the principal investigator has the appropriate qualifications and background to conduct the proposed study

The review will take into account each of the criteria above. To view our review form, see TOEFL COE Proposal Review Form (PDF).

Applicants who are invited to revise and resubmit proposals will receive constructive comments. An invitation to the next step does not guarantee that applicants will receive a grant.

All grant winners will be required to submit two to three progress reports and at least one manuscript of publishable quality during their funded study. The exact due dates for these deliverables depend upon the individual project schedule. Please indicate the dates by which you intend to submit your progress reports and manuscript(s) in your project schedule.

Grant funds are generally disbursed in several payments, aligned to study milestones and submission of deliverables. The timing and amount of grant disbursements is flexible, depending on the project's needs. Please estimate in your budget and project schedule the specific amounts you will need, for specific purposes, at specific dates. For each payment that is disbursed, ETS must receive an invoice that itemizes the relevant project milestones and deliverables for which grant funds are requested.

Authors are expected to submit at least one manuscript of publishable quality as the final deliverable(s) of the grant. Typically, the research can be published either as an ETS Research Report or as an article in a scholarly journal. Please specify which publication venue you plan to pursue and be aware that the manuscript will have to pass ETS technical review before being submitted for publication, whether as an ETS Research Report or an article in a scholarly journal.

Grant recipients are required to appropriately acknowledge the support of the TOEFL program when disseminating the results of the funded work. The following standard disclaimer should be included in any publications, including conference presentations:
 

• April 15, 2023 — Deadline for submission of intent forms
• July 15, 2023 — Deadline for submission of full proposals
• October 15, 2023 — Responses to proposals mailed
• April 15, 2024 — Tentative start date for funded projects

1. I am a foreign national working in the United States. Can I apply for the grant?

Yes, but to make sure you will be able to perform the work associated with the project, you must first discuss potential involvement with your university or institution.

2. I am working outside of the United States. Can I apply for the grant?

Yes, but to make sure you will be able to perform the work associated with the project, you must first discuss potential involvement with your university or institution.

3. When can work on the project begin?

Please consider April 15, 2024, as the tentative start date for the project. The actual start date will vary depending on the duration of the contract process involving your university or organization and ETS. Contract negotiation and study planning may take several months depending on the regulations of your university or institution.

4. How long can the project last?

The expected duration for each project is 12 to 24 months. However, in special cases, projects may continue beyond 24 months. Decisions will be made on a case-by-case basis. In the grant proposal template, a section is provided in which you should detail the project timeline and milestones.

5. If the project continues beyond the initially agreed-upon end date, will additional funding be available to researchers?

No. No additional funding will be made available to grantees should their project extend beyond the original project timeline.

6. My project involves payments to participants. Should I include these in my budget?

Yes. In your proposal budget, please include details about the payments you plan to make to potential participants.

7. I need a data set for my study. Is there a dataset I can use?

Yes. Please refer to the TOEFL iBT and TOEFL ITP tests Public Use Dataset section above for details about available data.

8. I am interested in administering the TOEFL iBT test or the TOEFL ITP assessment series for my research. How are the assessments administered?

Research test forms of the TOEFL iBT and TOEFL ITP assessments will be available free of charge to principal investigators funded by the TOEFL COE research program. Researchers are responsible for administering the research forms to their study participants. ETS will provide researchers with test administration guidelines for their test administration sessions. For more information about the TOEFL iBT Research Forms, see the TOEFL iBT Research Forms section above. For more information about the TOEFL ITP Research Forms, which are retired paper-delivered TOEFL ITP test forms, see TOEFL ITP Test Content.

Note: ETS cannot provide live, secure assessments or constructed-response scoring services free of charge. If you plan to use the secure, live test in your research, or to use publicly available sample tasks to collect spoken or written responses to be scored at ETS, please contact TOEFL_RC2@ets.org to obtain more detailed information on fees you would need to budget in your proposal, and for any further questions regarding test administration for research purposes.

9. If I want to administer one or more of the assessments, how are the tests scored?

• For data output, the TOEFL iBT and TOEFL ITP online research forms produce a scoring file for the selected-response sections, as well as speaking samples (in .wav format) and/or writing samples (in .txt format) when applicable.
• For Speaking and Writing, the grant coordinator will arrange to have Speaking and Writing responses scored by TOEFL test raters.
• For the TOEFL ITP paper-delivered research forms, ETS will score the tests, as they currently include selected-response sections only.

For each test section, either raw scores or scaled scores can be provided to grantees. Scoring fees should be included in the project budget, and a scoring fee estimate should be obtained by contacting the grant coordinator at TOEFL_RC2@ets.org.

10. Will I be able to provide score reports to my participants who take a research form of the TOEFL iBT test or the TOEFL ITP tests?

While score reports will NOT be available for research forms of the TOEFL iBT test, ETS will provide score reports for participants who take a research form of the TOEFL ITP tests.

11. Who will own the data and draft manuscript resulting from the project?

ETS will own all data associated with the funded research. The principal investigator or the principal investigator's institution will own and retain copyright to any paper that is presented at a conference or accepted by a journal for publication. ETS will own and retain the copyright of any manuscript that is either published as an ETS Research Report or NOT accepted for publication. ETS encourages researchers to use the data and results for scientific publications; however, all conference papers and manuscripts will need to be subject to prior review and possible redaction by ETS.

A standard contract template is available to your grant officer upon request. If your grant officer would like to review the contract template before signing the Letter of Institutional Commitment (PDF), they should email a request to TOEFL_RC2@ets.org.

12. Will I be allowed to present the research at conferences?

Yes. We encourage our grantees to present their work at professional conferences. Grantees are required to submit an abstract and presentation slides to ETS for review at least 2 weeks prior to the date on which they are to be submitted or presented. In addition, grantees are required to include the standard disclaimer statement in their presentations (see Dissemination Plan above).

13. Can I request funds to travel to conference to present the research?

Yes. However, total travel expenses per calendar year should not exceed USD $1,000 for domestic travel or USD $1,500 for international travel.

14. How should I itemize the expenses that are listed in my budget?

For guidance on budgeting, please see the sample budget and budgeting notes given in the Grant Proposal Template (Word).

15. Can I request funds for computer software or hardware in the project budget?

No. We do not provide funds for software or hardware purchases, as most universities or institutions already provide all necessary software and equipment.