How we are using AI
We at BMJ OnExamination are constantly evolving our content creation process. We are now using Artificial Intelligence (AI) to assist with content updating, streamlining our resources and allowing us to cover more topics more efficiently. Importantly, all of our content still starts and ends with human clinical experts.
Our AI only operates under careful direction of an expert human clinical team. Our skilled writers and medical professionals will always play an integral role in the creation, editing and quality assurance process. Each question assisted by AI is rigorously reviewed and verified by our experts to ensure that it maintains the highest levels of accuracy, relevancy, and quality you've come to expect from us. Embracing this technology doesn't mean we're replacing the human touch; rather, we're enhancing our capabilities to deliver the best possible content to you, our valued users, helping you pass your exams the first time.
We are also implementing a robust testing and assessment protocol to analyse how our AI-assisted questions perform in real-world scenarios. By carefully tracking user interactions, feedback, and performance metrics, we will continuously refine and optimise our AI program.
Embracing advances in medical education technology will allow us to provide you with high-quality questions that are up-to-date, accurate and cover all aspects of your exam revision.
Our Editorial process and roles
Stage 1 - Drafted by an educational expert
New ideas for questions are based on the exam curriculum and are drafted by a doctor who is an educational expert
Stage 2 - Editor with AI assistance
A BMJ editor then uses this question idea to generate a number of draft questions using an AI tool
Stage 3 - Editorial check
The draft questions are then checked by a BMJ editor and an independent qualified doctor. Unsuitable or incorrect questions are discarded at this stage.
Stage 4 - Edited by an educational expert
Selected draft questions are then edited by the educational expert, who checks the content against sources, clinical accuracy and relevance to the curriculum.
Stage 5 - Questions are published
Final questions are published to BMJ OnExamination
Stage 6 - Feedback, review & updates
User feedback and overall health of the question database is regularly reviewed and any changed made to ensure all exams remain current and up to date
Need more information? We have created an expanded version of our process as a PDF document here
FAQ
Why are you using AI to write questions?
We want to ensure that you have the widest and most up to date range of questions to answer to prepare you for your exams. By embedding AI into our question generation process we can generate questions quicker and with a similar degree of accuracy compared with human question writers.
How do you ensure questions made with assistance from artificial intelligence are accurate?
All our questions are checked by expert doctors who have experience in education and are familiar with how the curriculum of the individual exam is tested. They are responsible for checking the clinical accuracy of all questions in an exam and ensuring that they are kept up to date when, say, a new guideline comes out or the curriculum for an exam changes. In addition, all AI-drafted questions are checked by an independent expert and a BMJ editor before they are made live on the website. If you have concerns about any question, then please let us know and we will look at it. Our quality assurance process is shown below.
How do you ensure there is no bias or stereotypes used in questions?
As large language models are based on existing content with inherent biases, the outputs can include bias or stereotypes. That is why at OnExamination, our content production processes start and end with input from human clinical experts to minimise bias as much as possible. These are people with formal post-graduate qualifications and experience in that resource along with a strong educational background. They receive training on content production including how to minimise the risk of questions containing bias or stereotype.
What large language models do you use?
A number of different large language models exist and this is likely to grow further. It is likely that there will be different large language models that are more suited to specific tasks. OnExamination uses a number of different models for different purposes and we are continually assessing the best choices in light of ongoing quality evaluations.
Are your staff qualified or experienced in using artificial intelligence?
All of our staff that use artificial intelligence have received training in this field. The leads for this project have formal, further qualifications in artificial intelligence, one at doctorate level.
Why is your product not cheaper now that you are using artificial intelligence?
We will be using the additional resources to spend more time ensuring that all exams are up to date and provide a wider range of questions. This is important to ensure that you have the best possible preparation for the exam you are about to take and that we are providing value for money.
We understand that pricing is a significant consideration. However, our goal is to provide a high-quality, effective, and secure learning experience, and the cost reflects our commitment to achieving this goal.
Wouldn’t it be better for these roles to be given to humans?
The role of human experts is and will remain vital in our service. In fact, despite incorporating AI to assist, we still heavily rely on human expertise. In fact, with the extra content produced thanks to AI assistance, we will need to employ more humans to help in these key aspects not less.
With the explosion in medical literature, keeping up-to-date using traditional techniques is more difficult than ever. So using AI assistance for these tasks allows us to quickly incorporate the latest changes in medical knowledge into the revision questions, ensuring you are studying the most current information.
In summary, our approach is not about replacing humans with AI, but rather about blending the strengths of both to create an enhanced, efficient, and effective learning tool.