How to ask good questions for e-learning

Tests, quizzes and questions play a crucial role in digital learning

Tests, quizzes, and questions play a crucial role in digital learning as they serve two primary purposes.

• They allow participants to reflect on what they have learned . It is important to remember that it is not just the interaction itself that leads to learning, but the reflection on the interaction and the feedback that comes with it.
• Second, the tests are used to test the participant's understanding of the subject and ensure that they have learned what they were supposed to. These test results are also used to measure and document knowledge, which is valuable for both teachers and students.

If the purpose of the question is to promote reflection and learning, it is important that there is a real opportunity to make mistakes, as it is through corrective learning from our misunderstandings that we truly learn. Therefore, it is also crucial to provide feedback that explains why an answer is wrong, so that participants can understand their mistakes and learn from them.

Measure understanding rather than memory

Unfortunately, too many tests consist of simple multiple-choice questions with one clearly correct answer and three obviously incorrect answers.

An important key factor in designing questions is to focus on measuring understanding rather than just memory. Instead of testing the participant's ability to recall information from the past, tests should aim to measure the participant's understanding of the subject and their ability to apply the new knowledge in a meaningful way.

Ensuring compliance and validity

It is also important to consider whether the test is to be used to ensure compliance with certain guidelines or requirements. In this case, it is crucial to ensure that the test is valid and measures what it is supposed to. A test should not just give the impression of compliance, but actually ensure that the participant has the necessary knowledge and skills to meet the requirements.

Explore different types of tasks for interactive learning

In addition to multiple-choice questions, there are also other types of tasks, such as "drag and drop", where participants have to drag objects onto other objects or areas on the screen, and "hotspot" questions where they have to click on the right place on an image or screenshot. Drag and drop tasks require you to drag objects onto other objects or areas on the screen. These can be really good where the things to be understood are spatially related to each other – and it therefore makes sense for the user to answer by placing something spatial. All too often, however, drag and drop is used to sort things into boxes with "right" and "wrong" – I don't really think users learn much from that. It becomes more interaction for the sake of interaction than actual reflection. "Hotspot" questions can be especially useful if the user is learning to operate software or machines.

How to create good multiple-choice questions for e-learning

In the following, I will focus on multiple-choice questions, as they are the most common type of task in digital learning. I will provide steps and tips for creating good multiple-choice questions that accurately and effectively evaluate the participant's knowledge and understanding. I will also mention some typical pitfalls and mistakes to avoid to ensure that the tests are of high quality and promote deeper understanding among the participants.

Have clearly defined learning goals

Start by clearly defining the specific learning objectives you want to evaluate. Be specific about which key concepts or skills participants need to demonstrate.

Example - Subject: Mathematics

Learning objective: Understanding geometric shapes.

Question: What is the area of a circle?

1. The length of the circle's circumference
2. The area of the diameter of the circle
3. π times the radius squared (Correct answer)
4. Half the radius of the circle

Explanation: The clearly defined learning objective is understanding geometric shapes, specifically circles. The correct answer (C) represents the correct area of a circle, which evaluates the participant's knowledge of the specific concept.

Formulate clear and precise questions

Write the questions clearly and concisely so that they are easy for participants to understand. Avoid unnecessary complexity or ambiguous wording that may confuse participants.

Example - Subject: History

Question: When did the United States become independent?

1. 1776 (Correct answer)
2. 1492
3. 1865
4. 1914

Explanation: The question is formulated clearly and precisely without any ambiguity. The participant can easily identify that the answer is (A) 1776, which is the year of the independence of the United States. Incidentally, the answer 1492 is an example of using closely related concepts as a distractor (incorrect answer option), since 1492 is the year Christopher Columbus discovered America.

Consider different levels of difficulty

Mix questions with different levels of difficulty so you can evaluate the participant's knowledge at different levels. Include easy, moderate, and challenging questions to get a nuanced assessment.

Example - Subject: Biology

Learning objective: Understanding cell structure.

Question 1: Which part of the cell is responsible for the production of energy?

1. A) The cell membrane
2. B) The cell nucleus
3. C) Mitochondria (Correct answer)
4. D) Ribosomes

Question 2: Which part of the cell controls all cell activities and contains DNA?

1. A) The cell membrane
2. B) The cell nucleus (Correct answer)
3. C) Mitochondria
4. D) Ribosomes

Explanation: Question 1 is a moderately difficult question that tests the participant's knowledge of which part of the cell produces energy. Question 2 is easier as it tests basic knowledge of cell structure and the correct answer is more obvious.

Integrate cases and scenarios

Create multiple-choice questions that present participants with realistic cases or dilemmas where they must choose between different actions based on their knowledge and understanding.

Example: Case processing, decisions.

Question: Karen, who is retired, calls Anders, the case manager, and asks for home help. Anders promises to pass on her request to the visitation. At the same time, they discuss the possibility of a subsidy for dental treatment due to Karen's broken tooth. Anders assesses that it is only a minor problem and informs Karen that it is not relevant to help at this time. He assures her that she will hear more from the visitation about home help.

The next day, Karen calls Anders to ask if he has made a decision in the case.

What should Anders answer Karen?

1. A) That Karen, as mentioned, will hear from the visitation, and that a decision has not yet been made in the case.
2. B) That he and Karen have only spoken on the phone and that a decision will be communicated in writing.
3. C) That no decision has been made regarding home care, but that he has made an oral decision regarding subsidies for dental treatment. (Correct)

Explanation: All three statements are realistic and possible responses from the caseworker, Anders, when Karen calls and asks for home care. Choosing the correct response requires insight into the field and an understanding of how to best handle similar situations. The correct decision is to choose option C, where Anders explains that a decision regarding home care has not yet been made, but that he has made a verbal decision regarding dental care subsidies.

Avoid asking questions where the wrong course of action is unrealistic or obviously wrong.

Here is an example of a less good case:

Example: Environment

Question: You work as a project manager and are planning a major project that your company has won. You have just received two offers by email, where you can see that there is a big difference in price.

One product is environmentally certified and is the most expensive product, while the other product is not environmentally certified but is the cheapest.

What are you doing?

1. I choose the cheapest product.
2. I choose the environmentally certified product.
3. I enter into a dialogue with the advisor or colleagues about what expectations they have for the materials. (Correct)

Explanation: All three statements are realistic, but it is quite obvious that C is correct.

Consider whether you need a question bank

If you need to create many questions for a large test or training session, you might consider creating a question pool. A question pool is a collection of multiple questions that are relevant to the topic at hand. When participants take the test, they will be presented with a random selection of questions from this question pool.

The advantage of using a question bank is that it varies the questions for each participant, making it harder for them to cheat by sharing answers with others. It also reduces the possibility of participants knowing the exact questions in advance and thus avoiding learning the topic properly.

By following the aforementioned steps and tips, you can ensure that the questions you create are of high quality and accurately assess the participant’s knowledge and skills. By including realistic answer options and avoiding ambiguity, you can promote deeper understanding among participants. Always be open to feedback and evaluation to continually improve the questions and optimize their effectiveness in teaching and learning contexts.

Tips for finding good incorrect answer options

Address typical misunderstandings and errors related to the topic.

Introduce closely related concepts that require precise differentiation.

Use popular misconceptions or misleading information as distractors (incorrect answer options).

Subject: History

Question: Who was the first president of the United States?

1. A) Thomas Jefferson
2. B) George Washington (Correct Answer)
3. C) Abraham Lincoln
4. D) Franklin D. Roosevelt

Explanation: The distractors (A, C, and D) were all presidents of the United States, but they were not the first. The correct answer (B) is George Washington, who was the first president of the United States and played a crucial role in the country's early history. If I had used more recent presidents like Barack Obama, he would stand out and be blatantly wrong.

Include typical errors

Identify the most common misconceptions about the topic and use them as distractors. This provides an opportunity to correct participants' misconceptions through learning.

Example:

What is the capital of Australia?

1. A) Sydney
2. B) Melbourne
3. C) Canberra (Correct answer)
4. D) Brisbane

Explanation: This example focuses on identifying the capital of Australia. By including distractors such as Sydney and Melbourne, which are often mistakenly considered to be the capitals of Australia, participants are challenged to correct their misconceptions and gain a correct understanding that Canberra is the actual capital of the country.

If you can't think of good distractors, you should rethink the question and rewrite it.

Pitfalls and mistakes to avoid when designing multiple-choice questions

Vague or ambiguous wording

Question: What is the difference between an atom and a molecule?

Explanation: This question is ambiguous as the difference between atoms and molecules is not clearly defined. Participants will be confused about what to answer as the question does not clarify a specific difference.

Leading formulation

Question: Why are smartphones the best form of mobile technology?

Explanation: This question is leading as it already states that smartphones are the best form of mobile technology. It should be avoided to introduce an attitude or preference into the question.

For complex sentences

Question: Assuming that a person has bought two apples and a pear, and the price of each apple is 10 kr. and the pear is 15 kr., how much did the person pay in total?

Explanation: This question is too complex with long sentences and multiple elements, which may confuse participants. It is better to break the question into smaller parts to make it more manageable.

Too easy or obvious answer

Question: What is the capital of France?

1. A) Berlin
2. B) Madrid
3. C) Rome
4. D) Paris (Correct answer)

Explanation: This question is too easy as the correct answer (D) is obvious. It does not test the participant's knowledge, but only their memory of familiar information.

Too many negative statements

Question: What is not a property of a metal alloy?

1. A) Flexible
2. B) Leading (Correct answer)
3. C) Hard
4. D) Durable

Explanation: This question contains a negative wording, which may confuse the participant. It is better to phrase the question positively and ask about a property of a metal alloy.

Avoid questions with the answer options “all of the above” or “none of the above”

When designing multiple-choice questions, it is important to avoid including multiple correct answers or answer options that represent all or none of the options. These types of questions can create confusion and make it difficult to provide constructive feedback to participants. It is better to create clear and concise questions with well-defined answer options for a more effective learning experience.

Example: Which of the following are capitals of Scandinavia?

1. A) Stockholm
2. B) Oslo
3. C) Copenhagen
4. D) Helsinki
5. E) All of the above

Explanation: This example includes "All of the above" as an answer option, which creates a situation with multiple correct answers. Since Stockholm, Oslo, Copenhagen, and Helsinki are all capitals of different Scandinavian countries, it is incorrect to have an answer option that includes all of them. This creates confusion for participants and makes the feedback process difficult.

Avoid using questions where there are multiple correct answer options.

Example: Which of the following are capitals of Scandinavia?

1. A) Stockholm
2. B) Rome
3. C) Copenhagen
4. D) Paris

Explanation: When there are multiple correct answers, it can be difficult to give accurate feedback to participants. It can be unclear what their knowledge is if they choose one of the correct answers, but not all. Imagine that the user answers A and D, i.e. one right and one wrong? What should the feedback be? And how should the question be scored? Is it completely wrong or should you get half a point?

Avoid having the longest answer option as the correct one.

As a fun detail, I would like to mention that in my experience, the answer option that is the longest is often the correct one in most multiple choice questions. Probably because you have to take into account that reality is complicated.

Summary

Creating well-structured and meaningful multiple-choice questions is crucial to effectively evaluating participant knowledge and understanding.

Here are some important points to remember:

1. Clearly define your learning objectives to evaluate the participant's knowledge of specific concepts or skills.
2. Formulate questions clearly and precisely to avoid confusion and ambiguous wording.
3. Include questions with varying levels of difficulty to get a nuanced assessment of the participant's knowledge.
4. Avoid bias and clues in the questions and keep them objective.
5. Integrate realistic cases and scenarios to challenge participants to apply their knowledge in a practical context.
6. Consider creating a question bank to vary the questions and reduce the possibility of cheating.
7. Address common misconceptions and errors by including relevant distractors.
8. Avoid vague or ambiguous wording, overly complex sentences, and overly easy or obvious answers.
9. Be careful about having multiple correct answer options or the answer options "all of the above" or "none of the above".
10. Always remember to revise and improve your questions based on feedback and evaluation to optimize their effectiveness.

By following these guidelines, you can create high-quality multiple-choice questions that accurately assess participant knowledge and skills, and provide valuable data to inform teaching and learning. Always be open to feedback and evaluation to continually improve the questions and create a more effective learning experience.