Most professional credentials were built on a sensible bargain. If a candidate can retrieve the right answer under exam conditions, without notes or help, they can probably apply it on the job. Recall was never the goal. It was a reasonable, scalable proxy for the knowledge that underpins practice. 

Generative AI weakens that bargain. When a correct answer is a prompt away, both on the job and, increasingly, around the edges of the exam, a test that mostly checks recall measures less of what employers, regulators and the public actually rely on a credential to signal. 

The credentialing bodies responding best are not trying to build AI-proof exams. They are asking a sharper question: what does competent practice look like when information is abundant, and how do we measure the part of it AI can’t do for the candidate? 

When the Machine Passes the Exam 

The evidence arrived quickly. In 2023, Microsoft researchers reported that GPT-4, with no specialized medical training, averaged about 87% on official USMLE sample and self-assessment materials, compared with 54–59% for its predecessor. A peer-reviewed study in Philosophical Transactions of the Royal Society A found that GPT-4 could pass the Uniform Bar Exam in every UBE jurisdiction, including the essay and performance components, not just multiple choice. 

These results need careful reading. They come from sample or released questions, under research conditions, and they say nothing about whether the model could practice medicine or law. But they establish something credentialing bodies can’t ignore. A large share of what many high-stakes exams test, accurate retrieval and application of codified knowledge to a well-formed question, is now something a general-purpose tool does at or above passing standard. 

Meanwhile, the skills employers say they value most are moving in the other direction. In the World Economic Forum’s Future of Jobs Report 2025, based on more than 1,000 employers, analytical thinking remains the top core skill, rated essential by seven in 10 companies. Employers also expect a substantial share of workers’ core skills to change by 2030. The gap between what recall-heavy exams measure and what the labor market prizes is widening from both ends. 

Why Recall Now Measures Less 

The problem isn’t that knowledge has stopped mattering. A clinician who has to look up everything is slow and unsafe, and no one can exercise judgment over facts they don’t understand. The problem is narrower, and more important: unaided recall is becoming a weaker signal of on-the-job competence, and it is under pressure from two directions at once. 

Validity. A credential is an inference: this person can be trusted to perform. When practitioners routinely work with references and AI tools, the inference from “recalled it alone in a test center” to “will perform well at work” gets longer. Some boards saw this early. In a 2017 study of more than 800 physicians, the American Board of Internal Medicine found that giving candidates access to a point-of-care reference improved the exam’s ability to separate high from low performers without changing the underlying skill measured. ABIM went on to add that resource to its maintenance-of-certification exams in 2018. 

Security. Recall items are also the easiest to compromise with AI. A short factual question can be pasted into a chatbot and answered in seconds. ABIM’s CEO wrote to diplomates in February 2026 that its policies prohibit importing question text into AI tools, that it monitors AI tools and forums for leaked content, and that a small number of participants have had access suspended or results invalidated. 

Put those together and the uncomfortable conclusion is that the items most exposed to AI-assisted misconduct are often the items least aligned with practice. That is where the redesign has to start. 

Start With the Competency Model, Not the Item Bank 

The instinct, when an exam looks vulnerable, is to write harder questions. The better move is upstream. Before a program can measure judgment, it has to define what good judgment looks like in that role, with enough precision that SMEs can write to it and psychometricians can score it. 

Nursing offers the clearest example. NCSBN’s 2013–2014 Strategic Practice Analysis showed that newly licensed nurses were increasingly expected to make complex decisions while caring for patients. Rather than adding more knowledge content, NCSBN built a framework for clinical judgment itself: the NCSBN Clinical Judgment Measurement Model, which breaks it into observable steps from recognizing cues and analyzing them, through prioritizing hypotheses and generating solutions, to taking action and evaluating outcomes. Every new item type that followed was written against that model. 

For most credentialing bodies, the equivalent work runs through the job analysis. In an AI-rich workplace, that analysis should now capture questions such as: 

Where do practitioners use AI or reference tools, and what do they still decide themselves? 

What does verifying an AI output look like in this role, and what errors must a competent practitioner catch? 

Which judgments carry accountability that cannot be delegated: prioritizing under uncertainty, recognizing when a situation falls outside protocol, explaining a decision to a client or patient? 

That last category is the durable core of most professions. A competency model that names it explicitly gives the exam something worth measuring. 

Scenario-Based Items: Testing Application, Not Memory 

Once judgment is defined, items can be built to observe it. On April 1, 2023, after more than a decade of research, NCSBN launched the Next Generation NCLEX. Its core is the unfolding case study. Candidates work through a client scenario that evolves across six items, each mapped to a step of the clinical judgment model, and every candidate sees three scored case studies. NCSBN also introduced new scoring methods that give credit for partially correct reasoning rather than treating every item as all-or-nothing. 

The principle travels well beyond nursing. Well-designed scenario items: 

Present ambiguity. Real cases contain irrelevant information, missing data and competing priorities. Recognizing what matters is the skill. 

Unfold over time. Later items depend on earlier decisions, so the candidate has to track a situation rather than answer isolated facts. 

Reward prioritization. Asking what to do first, or what to stop doing, separates judgment from knowledge. 

Score the reasoning. Partial credit and structured response formats reveal how a candidate thinks, not just whether they landed on the key. 

Two cautions apply. Scenario items are not automatically AI-resistant; a well-formed vignette can also be pasted into a chatbot, so security and delivery still matter. And they are expensive. Each case takes more SME time to write, review and validate than a stand-alone item, which is precisely where AI-assisted drafting, under strict SME review, can change the economics. 

Renewal That Checks Competence, Not Hours 

If initial certification is where recall is most visible, renewal is where the proxy problem is oldest. Many recertification programs still rest on accumulated continuing education hours. The Institute for Credentialing Excellence has been candid that many continuing competence programs rely mainly on untargeted, single-session CE, and research from Workcred notes that credits are sometimes awarded for attendance, or even registration, with no check on learning. Hours logged confirm participation. They don’t confirm current competence. 

Generative AI sharpens this. A field’s knowledge base, tools and standards of practice now shift faster than a five- or ten-year cycle. Renewal that checks whether a practitioner can still exercise judgment with today’s evidence and tools is more meaningful than renewal that checks whether they sat through sessions. 

ABIM’s Longitudinal Knowledge Assessment shows what the alternative can look like at scale. Launched in 2022, it replaces a single high-stakes exam with 30 questions a quarter over five years, and it is open-book: physicians may use the resources they would use in practice, but not other people. According to ABIM, around 80% of diplomates now choose it over the traditional exam, and nearly 90,000 physicians were participating as of early 2026. ABIM also reports a JAMA Internal Medicine study in which patients of physicians with higher LKA scores had lower post-hospitalization mortality and readmission rates. That is an association rather than proof of cause, but it is the kind of outcome evidence that CE hours have rarely been able to produce. 

The design lesson is general. Frequent, lower-stakes check-ins tied to the current competency model, with feedback that points to gaps, measure currency in a way that a transcript of hours cannot. 

Why “AI-Proof” Is the Wrong Goal 

It is tempting to frame all this as a security race: lock AI out of the testing room and the old exam keeps working. Security matters, and the ABIM experience shows it needs active enforcement. But as a strategy, “AI-proof” has two flaws. 

First, it is a race exam owners are unlikely to win for long. Tools improve faster than test-center policies. Second, and more important, it can quietly make an exam less valid. If competent practice now involves working alongside AI and reference tools, an exam that pretends those tools don’t exist is measuring a version of the job that no longer exists. 

The more durable goal is an exam that measures what AI can’t do for the candidate: noticing what matters in a messy situation, choosing what to do first, catching an error in a plausible answer, recognizing the limits of protocol, and owning the decision. In some programs that will mean secure, closed-book scenarios. In others it may mean deliberately allowing tools, as ABIM did with references, and measuring how well candidates use them. 

For credentialing leaders, that translates into five practical moves: 

Audit the blueprint for recall weight. Estimate what share of scored items test retrieval alone, and which of those reflect knowledge practitioners genuinely need unaided. 

Refresh the job analysis for AI-era practice. Capture where tools are used and which judgments stay with the practitioner. 

Pilot scenario items in one domain. Build a small set of unfolding cases, field-test them as unscored items, and compare their performance with existing items. 

Decide your tool policy on purpose. Closed-book or open-book should be a validity decision tied to practice, not a default inherited from 2010. 

Re-examine renewal. Ask what your recertification requirements actually confirm, and whether a longitudinal or scenario-based check would tell you more than hours. 

Connecting the Pieces: The OpenEyes Approach 

The three responses above, competency models, scenario-based items and competence-focused renewal, only work if they are connected. A scenario item is only as valid as the competency it was written to measure. A renewal check is only meaningful if it draws on the current model, not the one from the last job analysis. In many programs, those pieces live in separate systems maintained by separate teams. 

That connective problem is the one OpenEyes Technologies, a Washington, D.C.-based software company serving credentialing and assessment organizations, has organized its product suite around. Merit supports competency modeling. GenQue provides AI-assisted item generation. Vault handles item banking. Crown manages the credential lifecycle, from application through renewal. Census covers surveys and analytics. 

The logic is straightforward. When item generation starts from a defined competency model rather than a blank page, drafts are anchored to the judgments the program has decided to measure, and SMEs spend their time refining scenarios rather than writing them from scratch. That matters for the economics described above: scenario items are worth more and cost more, and AI-assisted drafting, with every item reviewed by qualified SMEs, is one way to make them affordable at volume. When the same competency model informs the item bank and the renewal process, a program can keep initial certification and recertification measuring the same, current definition of competence. 

For any credentialing body evaluating technology for this shift, OpenEyes included, the useful questions are about alignment rather than speed: 

Can every item be traced to a specific competency, and can we see coverage gaps across the model? 

Does AI-assisted drafting support scenario and multi-step formats, not just single-best-answer recall items? 

Do SMEs and psychometricians retain final authority over every item that reaches a live form? 

When the competency model changes, can we see which items and renewal requirements it affects? 

Tools that answer those questions well help a program measure what AI can’t do. Tools that simply produce more recall items faster make the underlying problem bigger. 

The Credential Worth Holding 

For most of the last century, a credential certified that its holder knew things. That was valuable because knowing things was scarce. Generative AI has made retrieval abundant, and abundance always shifts value to whatever remains scarce. 

What remains scarce is judgment: seeing what matters, deciding under uncertainty, catching what a confident machine got wrong, and standing behind the decision. Those are also the capabilities employers, regulators and the public most need a credential to vouch for. 

The credentialing bodies that redesign around that reality will not be issuing a weaker signal in an AI era. They will be issuing a stronger one, because their credential will say something a chatbot cannot. The ones that keep testing recall will find their exams still produce scores. The open question is how much longer anyone will treat those scores as proof of competence.