Skip to content

GitLab

Last edited by Dione Mentis
Page history

dp meet notes

Design phase meeting notes

Week 1: Question creation and submission

Focus: Question Creation and Submission

  1. Author​ ​creates new submission
  2. Author​ ​chooses a question type
  3. Author​ ​adds question content, answers, and metadata
  4. Author​ ​submits questions

Question types for sample content

  1. Multiple choice (single correct)
  2. Text (no question)- "question" will not be scored but can be useful for introducing a set of related questions.
  3. Complex Item Sets (This a is question with sub-questions)
  4. Multiple TF
  5. True/false
  6. Fill in the blank
  7. Fill in multiple blanks (one or more blank at any point in the question)
  8. Matching
  9. Essay
  10. Multiple answers
  11. Multiple dropdowns
  12. Numerical answer

Considerations in question creation design

  • Representative sample
  • Form accessibility
    • Image accessibility (uploads by authors vs production team created)
    • captions
    • alt-text
  • Question metadata
    • hierarchy (yes); defined list (yes); required vs optional?) list of levels can be shared soon, complete list to come later
    • what about question explainer? Guidance on why the correct answer is correct.
    • keywords: purpose to search by and leads to category to export by/make list by 4 levels (3200 words)
  • permissions acknowledgment (CC)
  • Question elements
    • ref/citation for image
    • could be more than one image in a question
    • images could be in the answers too (e.g. 4 graphs to choose from)
    • images possible (in question and answers)
    • tables in questions

Week 2: Managing Editors check of the submitted questions and the review process

User stories

  1. Managing editor assigns one or more questions to a handling editor
    • optionally does quick check of questions before assigning to handling editor
    • can skip handling editors (at least initially) and invite reviewers to review question(s).
  2. Handling editor invites Reviewer to review one or more questions
    • choose reviewer based on area of expertise (provided at time of sign up)
  3. Reviewer gets invitation (question lands in dashboard) and makes decision (4 or 5)
  4. Reviewer accepts the invitation. Status = In review
    • Reviewer takes the question
    • Reviewer completes survey (depends on correct/incorrect answer) which includes questions about metadata, i.e. does he agree/disagree with authors tags.
    • Reviewer submits review of questions (goes to handing editors dashboard who assigns production team)
  5. Reviewer rejects invitation. Status = Invite rejected.
    • reasons for rejection include: out of my area of expertise; don't have time to do it within the next two weeks. Free text area to provide reason/or list of defined options and ‘other’.
    • shows as ‘rejected invite’ row in reviewers dash
    • question goes back to handing editors dashboard with info of reviewer name and reason for rejection.

Teams

  1. Admin: Highest permissions.
  2. Managing editors: small group of people on the HHMI team, or contractors, who delegate submitted questions to Handing editors.
  3. Handing editors: small group of people on the HHMI team, or contractors, whose main responsibility is pushing content through the workflow and publishing questions.
  4. Reviewers: large group of people mostly external to HHMI. Serve as content specialists (universities lectures, authors, teachers, etc.). May also be an Author in the system.

Reviewer metadata

  • Reviewers have already agreed to be an HHMI before they are invited to review questions.
  • All users fill out a questionnaire on the courses they teach and their content expertise (this currently happens externally, but should happen in the system at signs up)
    • We might need an import script for current questionnaires (200-300). These users have an account created for them, they only need to accept terms to join.

Week 3: Queries from previous weeks and Production assignments

  1. Queries from week 1
  2. Queries from week 2
  3. Week 3 focus: Production assignments

Queries from week 1

  • Can any signed up user create and submit questions? Yes
  • Does this include Managing and handling Editors? Yes (and they can be reviewers too, by default)

Queries from week 2

When Handling editor receives submitted question, she does one of four things:

  1. Ask Author clarifying questions (Haven't covered yet)

Does handing editor make any necessary changes to the question or metadata?

  • Author can resubmit a question if a the handling editor asks for revisions (e.g. missing images) in chat.
  • Versioning isn't important, the resubmitted questions overwrites the original question.
  • Database could have backup for rare use case where author deletes question in error.
  1. Invite Reviewer to review a question
  • Is there a need to invite a reviewer to review questions in bulk? Yes, metadata on dashboard would need to show high level topic/expertise for this to make sense in bulk. (perhaps without expertise search?)
  • Is there a need to invite more than one reviewer to review a question at the same time? More than one person at a time (aim 2-3 reviewers)
  • Yes, combined above
  • Reviewers should be able to send notes to the Managing/handing editors
  • Reviewers should be able to contact the Author for clarification (No, authors are unknown to reviewers)
  • What happens if the reviewer neither accepts nor rejects invitation within x days? Suggest: reviewer gets reminder email "Your invitation will expire in X days". No response after 2 weeks means questions goes back the the handling editor.
    • Accept within a week (with reminder first)
    • Review within 2 weeks
    • Optionally have way to extent the 'deadline'/Due date (question level)
  1. Reject the question (Haven't covered yet)
  • reasons for rejection? doesn't meet req
  • what happens to a rejected question? Reject email. Could resubmit (same as point 1 above)
  1. Begins production process (Focus of this week, see below)

Week 3 focus: Production assignments

Start at Step 10d from proposal: Handling Editor begins production process

  • Is 'Review' a required step before this or can a question be with a reviewer at the same time as the production team works on question? Must review first. (outcome of review may be question is rejected totally/ no resubmission.
  1. Handling Editor assigns production tasks Production

People: * copy editors (always access) In order: * permissions researcher * illustrators - make finals from author's roughs * accessibility checker
- providing alt-text for images

Tracked changes instead of granular permissions on the elements of the question/answer.

  • Only see my assignments
  • Must be able to mark complete (missing from wireframe atm) ( Where is there space to have a conversation with handing editor about the task (rather direct chat with handling editor?).
  • handling editors will know the role of each person
  • Notifications from assignments?

  1. Production Roles complete production tasks (copy editing, accessibility check, illustrations etc)

  2. Handling Editor / Managing Editor makes a decision to publish question

  3. Question is published!

  4. Authors who submit 10 accepted higher-order questions will be offered certification of a peer-reviewed contribution. Email notification to verify. (Added for discussion)

  5. Authors can only be paid $500 per year for submission (a way to get this e.g. report would be helpful) (Added for discussion).

  • $10 per lo
  • $50 per ho

Week 4: Workflow steps review from week 1-3

sign-up-steps-20200204.pdf

submit-and-editor-review-question-20200204.pdf

resubmit-question-after-editor-review-20200204.pdf

reviewer-review-question-20200204.pdf

Week 5: Faculty and public use of published questions.

Week 5 Agenda

  1. Use of published question by faculty.
  2. Use of published question by the public.

Use of published question by faculty

Workflow steps:

  1. Community Member searches Question bank
  2. Community Member see the question with its metadata, can optionally take the question before deciding to save to a list for export.
  • SCORM compliant
  • PDF of questions for use in the classroom as a handout
  1. Community Member can Create, Rename, Update, Delete personal lists of questions
  2. Community Member can export a list to a chosen file format or external system

Added steps for discussion (Not included in the proposal):

  1. Community Member can edit an existing question and save the edited version. (can modify however they like.)

Faculty/community members are signed up users with roles: * Author (who can also be Reviewer, ME or HE) * Production team (no use case for these users to create questions or lists)

Relevant feedback from previous rounds:

The most common user will not be a reviewer or editor. We anticipate the general site users will save questions and lists and (possibly) author questions. We wonder if the “My Dashboard” page should be for this type of user. It would contain Authored Questions, Saved Questions, and Saved Lists all on one page. The second page could be called “Assignments” and it would include the Review and Edit tables. The Assignments page would only appear for users with the role “Reviewer” or “Editor."

Use of published question by the public (non-logged in user)

Workflow steps:

  1. User searches Question Bank
  2. User reads question(s)

Added steps for discussion (Not included in the proposal)

  1. We are ideally hoping that users will be able to "take" the question (select a response or enter text), then be shown the correct answer. If this is not within scope, our second choice would be to have the answer hidden and users can click a button to show the correct answer and why it is correct. For open-ended questions, since there will be no way to grade such an answer, show a rubric”

  2. User exports individual questions

    • Proposal states: “Questions exportable to learning management systems (SCORM compliant), Word documents, or other formats including CSV, Google docs, and QTIs.” (Error in Google doc, this relates to logged in users.)

Week 6: Sign up, User management, and other info pages

Week 6 Agenda

  1. Update: additional meet on Thursday 25 February to focus on production assignments and in-app chat
  2. Walkthrough of wireframes for sign up process (and integration with HHMI)
  3. Other pages

Walkthrough of wireframes for sign up process

Integrations with HHMI from proposal:

  1. The question database (Qdb) will be: Available to the community through HHMI BioInteractive, linked from website
  2. Log in using their BioInteractive website account, Google sign-in, or create a new account. New accounts created here should work on the BI website as well (better approach is to sign up on BI and then sign up with those credentials).

Other pages

  1. Managing users and their roles:

    • user management: a list of users in the system, showing basic info about the user; functionality to deactivate user
    • Admins “View how many/which questions teachers submit”
    • add users to 'teams': managing editors; handling editors; reviewers; production team

  2. Profile page

    • Change password or email?
    • Reviewer role is opt-in, but there needs to be a place to remove role, and to opt-in at a later date.

  3. Information pages:

    • Questions & Learning Objectives (same purpose as 'Discover' page) search by both questions and LOs.
    • Professional Learning (link to assessment courses and other pages on BioInteractive website)
    • About

  4. Home page is a landing page, that links to Discover page. Note: footer should be visible on all pages, not just landing page.

Not in proposal, added for discussion Admins can add a “basic page” and link it from the navigation. What are the options for styles on basic pages? (agreed this feature won't be developed. Coko could create addition 'information pages' in future). The Super/Sys admin will be able to edit all published questions (e.g. to fix a typo). Discussed possible feature: Sys admin chat with ME.

Feedback on Wireframes from week 5

  • Learning objectives need to be more prominent/ given equal weight as questions.
  • Search should show the related LOs for a question result
  • Coko to work on more detailed wireframe for the search functionality on 'Discover' page. E.g. what does user see when he first arrives on page (latest published questions at top of list); what does search area show when users is searching; how are questions displayed when search is completed.
  • Delete is preferred over archiving a question or list of questions.
  • future iterations could have user-created tags for questions on lists page.
  • Users can add the same questions to multiple lists