Open Certification of Software Testers

We are suspending activities related to the Open Certification project in support of the AST BBST courses which we feel will provide the industry with its first skills-based exam. We see it as a more impactful and difficult goal then the Open Certification project, and we hope that you’ll redirect your energies with us. We hope to reuse the work done to date in the Open Certification project and we hope to redirect some of the work we have yet to start. 

If you would be interested in any of the following, please let Cem or I know. We would love the help.

1) We need help in the Basic Concepts SIG. The Basic Concepts SIG is an effort to create a reference for testing terminology structured in the format of the Oxford English Dictionary or the Blacks Law Dictionary and ask, what are the many ways that this word is used and what are good examples of each usage? We want to mirror the field in its diversity, rather than impose a false uniformity. The Basic Concepts project is modeled on the content wiki project we spoke about at the last WOC meeting. 

2) We need help developing questions for the BBST course materials. As new videos and other course materials are created for BBST, we need volunteers to create questions in the question server for those materials. It’s a large need, and much of the research we did at the WOC meetings for question writing will come into play.

3) We would also be interested in knowing if anyone would be willing to take over care and feeding (maintenance) of the question server as we move forward. We are considering using the question server from WOC as an authoring tool for writing questions for BBST. This would create an open library of multiple choice questions, with the opportunity for peer review. A small number of instructors would gain question-author privileges, a larger pool of reviewers would gain critic's privileges.

The task of writing objective questions for BBST is much easier than for WOC because we can say, "According to the lecture..." rather than trying to figure out how to signal which of many possibly right answers is the right answer for a question asked (in the open certification exam) outside of any instructional context. (When there are 15 common definitions of "test case", how can we fairly signal which one is the "right" one in a given question? This became a crippling problem for open cert questions.)

If we continued this work, as a subproject of BBST, we would want to extend the question server in a few ways:

(a) possibly, an export to one or two standard formats importable into Moodle, Blackboard, Angel, etc.
(b) addition of some other objective-test formats
(c) automatic point assignments, according to the standards published on Cem’s blog (e.g. if all-of-the-above is correct and there are 3 choices, A, B, C (all correct), score is 100% for selection of all three, 25% for a choice that includes 2 of them, and 0 for a choice that includes 0 or 1 of them.) 

We would like to thank everyone for their help and energy over the last couple of years. We hope you will consider supporting the BBST courses and certification as we work to change the state of software testing certifications.  If you think you would like to help with one of the projects listed above, please contact Mike or Cem at mike@michaeldkelly.com and kaner@kaner.com.

Thank you,
Mike and Cem

1. The large pool of questions will be public, with references. They will form a study guide.
2. The pool is not derived from a single (antiquated) view of software testing. Different people with different viewpoints can add their own questions to the pool. If they have well-documented questions/answers, the questions will be accepted and can be included in a customizable exam.
3. The exam can be run any time, anywhere. Instead of relying on a certificate, an employer can ask a candidate to retake the test and then discuss the candidate’s answers with her. The discussion will be more informative than any number of multiple-false answers.
4. The exam is free.

• Questions, exams, and study guides posted with them are Creative-Commons (attribution) licensed.
• Open questions
• For now, all questions are “objective” (computer-scorable)
• All questions in the question database are visible to the public. Several people memorize the questions while they write an exam, then post the questions to mailing lists or web sites. Rather than calling it cheating to consult such information, we provide the same information—full information—to everyone.
• Anyone can create a question, so long as they follow our structural, annotating and referencing rules. All questions are signed by the author. Only the author can revise a question.
• Different authors can create identical questions with different designated-correct answers.
• Anyone can comment on a question and its grading scheme. System editors will occasionally reorganize a
• comment set to clarify and eliminate redundancy.
• Open exams
• An exam is a random stratified sample of the pool of questions.
• In creating an exam, an exam author can weight some topics heavily (more questions appear on those topics), designate preferred individual questions, and block other questions or topics.
• Different exams will reflect different visions of testing. Some employers will author custom exams.
• Exams are subject to review for redundancy with other exams, quality of description of the exam, and sensible selection of question (for example, it is unlikely that one would legitimately include two versions of the same question, differing in which answer is scored correct).
• Open body of knowledge
• Exam authors can publish study guides with their exams; others can critique or supplement them.
• Open / free references
• Authors must justify questions and answers, and critics justify comments, by linking to credible free-access documents on the Web. The insistence on free-access references is controversial and may not last. However, this is an internationally available project. Many of the stakeholders have limited access to the commonly cited books. And there is a remarkable selection of high-quality materials on the web.
• Free exam administration
• Examinees will log on at the open certification site and pick the type of exam desired. The software will select an appropriate group of questions, present a form (the exam) that the user fills out and submits, and score the exam.
• Exam feedback intended to promote reflection and discussion
• When someone writes the exam, the feedback they receive includes the list of questions, their answers, and the discussion associated with each of the questions.

• The emphasis on openness makes possible a more revealing and less prescriptive use of certification exams.
• An examinee who gets a sufficiently high score can claim to be “certified.” (We’ll print a certificate.)
• Rather than rely on this binary state, any employer can administer an exam during an interview and then discuss scored questions and answers with the examinee.
• Such discussions should be far more revealing than the raw score. Examinees who gave thoughtful “wrong” answers can demonstrate the worth of their answers.
• Discussions attached to a question help people study for the exam (and the possible interview) and provide references that candidate and interviewer can skimtogether during the post-exam discussion.
• Even though there are designated-correct answers, the entire process acknowledges the controversy of the simple answer and honors the desirable tendency of testers to challenge everything.

• Software testing is hardly the only subfield of CS/SE/CIS plagued with popular commercial certifications that assess examinees at low levels of the Bloom scale but market the image of the examinee who passes as an expert (or at least, as someone who has a practical clue).
• The structure we create for testing is generalizable to many other fields. Populating that structure with assessment and study materials will take substantial work, but for a group interested in doing that work, we have a model that is open source and available for free extension.