NCBI testing release 19 July 2022: Book versions
Hi @lathrops1 cc @DioneMentis
Before you proceed with testing of this issue, please note that commenting in the Review chat or Help chat on PDF and XML wholebooks and Word One docs currently crashes the app. We determined that getting book versions to NCBI to test without resolving that error first was worth the gain in testing time. Please avoid using the review chat in PDF and XML wholebooks until that is resolved.
Background book versions documentation
Epic &47 documents the scoping done for Book versions. This issue outlines the current status compared to the ideal target (to outline which use cases are met now and which will be handled by separate development). This issue also outlines where current development differs from what was originally discussed, which is in some cases because development raised questions that were not solved in the original scoping with NCBI, and in some cases is due to compromises made with NCBI in order to satisfy a short timeline. The priority has been to get this to NCBI to test as soon as possible, with the option to make further improvements.
Current status of book versions development
Book versions is limited to books using the ‘wholebooks’ template, these are PDF and XML wholebooks.
NCBI has confirmed that Book versions should also apply to ‘One Doc’ books in the Word workflow (which use the chapter-processed template for processing), so that their processing can change to ‘wholebooks’ in the PDF or XML workflow. This case is not developed. This requires a separate proposal so that all related open issues, especially the design improvement requested in #1138 (closed), can be considered together (related to issues linked to #215 (closed)).
Additionally, NCBI expects that in the future users will need book versions to support changing from wholebook-processing to chapter-processing. This case is not developed.
What is a Book Version in the BCMS?
All versions of the same book have the same domain name. Each version has a BCMS ID in format {ID.version-number}
.
Example for new book:
Book V1
- BCMS ID = bcms10002833.1
- domain name = bcms10002833
- BCMS Database ID = (unique per version)
Book V2
- BCMS ID = bcms10002833.2
- domain name = bcms10002833
- BCMS Database ID = (unique per version)
Book V3
- BCMS ID = bcms10002833.3
- domain name = bcms10002833
- BCMS Database ID = (unique per version)
Migrated book example:
Book V1
- BCMS ID = bcms10002833.1
- domain name = vasleepap
- BCMS Database ID = (unique per version)
Book V2
- BCMS ID = bcms10002833.2
- domain name = vasleepap
- BCMS Database ID = (unique per version)
Book V3
- BCMS ID = bcms10002833.3
- domain name = vasleepap
- BCMS Database ID = (unique per version)
Each Book version can have one or more file versions.
How and when does the BCMS create a Book Version?
The FTP path to fully support FTP source submissions with book versions will need to be finalised with #438 (closed), to introduce the relevant FTP errors where necessary, so FTP works as it did previously to create new books, but not to create subsequent versions yet. The ultimate FTP path for book versions is described in the Epic, but for now this handover document for testing focusses on the BCMS path.
BCMS Path: create or update Book Version
Creating a new book version becomes relevant after the current version is published. This is in line with how chapter versions are currently created in the BCMS.
User experience in the BCMS (explained further with images in the section below titled 'The UI to support the current book versions development'):
- When the latest Book version is “Published”, the “New version” button is active in the UI.
- In the “New version” UI the users set the parameters of the new version. The options include:
- Specify the Content type (either 'Author manuscript', 'Prepublication draft', 'Published PDF', or 'Final full text')
- Choose a workflow from either PDF or XML
- Select the version number (integer).
- The front end validates that the combination of Content type and Version number doesn't already exist for that book.
- Select the button “Create new version”.
- The user is redirected to the new version. The Status is ‘New version’. The next step would be to upload the files.
- The user can go back to any previous version to update its files and republish.
We have added 'Final full text' to the 'Content type' dropdown in the New book and New version modals, alongside the first three content types which were supported previously. Content type is now a required field for the new book and new version steps.
The Epic documenting our scoping outcomes originally said "‘Final-full-text’ is the default content type for Word and XML workflow so users do not need to choose this in the UI", however in current development this is a required field, since the Content type places restrictions on which Conversion workflow users can choose and is required for the 'Version name' as the [content type] + [version number]. If BCMS users who require the 'Final full-text' option cannot be expected to provide this value in the BCMS, and it does need to be used as a default when 'Content type is not provided, we could improve this further to introduce the default, but current development requires a Content type to be filled out in New book and New version steps.
NCBI notes the following use cases for the future (not currently supported):
- An Investigator (future BCMS role) tries to submit a manuscript, prepub or published pdf version when the final full text version already exists. In this case, whether the final full text is published or not, the BCMS does not create a new version. Instead the required metadata is collected from the user.
- A publisher tries to submit a published pdf version when the prepub version already exists but has not been published yet. This is highly unlikely but NCBI should consider what result they would expect for this case. (For example: block submission? Delete current version?).
These cases above are not currently supported.
How do Book versions affect Book Files?
- When BCMS sends files as a wholebook package to Load to PMC, the json file includes the
version-name
andversion-number
as shown below. - The converted file name must remain stable for the same Book version (although this will be changed for the MVP deployment by #1278 (closed), to allow converted files to have unique names within one book version). It’s expected that the converted file name can change between Book versions.
Update to json sent in wholebook Load to PMC packages:
"package_id": 1234567890,
"domain": "vasleepap",
"version-name": "prepub": // Book version name
"version-number": "2": // Book version number
"main_xml": "vasleepap.bxml",
"thumb": "vasleepap.png",
"package": "vasleepap.123457890.2020_05_15-09_30_19.zip",
"target_database": "prod",
"release": true,
"notification_recipients":
How do Book Versions affect Book Settings?
Currently Book Settings are maintained in one template (see book\_settings\_template
table in db model). There's a template type per processing type (chapter-processed and wholebook). This template includes all book settings, whether they are domain settings (NCBI-systems) or bcms-specific settings. Books inherit these settings at creation point.
To support book versioning, we had two options, NCBI confirmed we should take the workaround approach (the second option in the Epic) for simplicity. So we went with the second option which is:
We version all settings, but only allow the settings of the current version to be edited. Note: this excludes the settings ‘workflow’ and ‘submission type’ – a new book version is created in order to change these settings. From the UI perspective that means that the form in the Settings modal has moved to a tab per version.
Note that when you create a new version, if you change book settings related to the domain for that new version, it updates the domain settings for the book. Therefore if you go back to a previous version of the book and make changes to it, it shares the same domain as the new version and therefore the same domain settings. We need to further analyse with NCBI whether there are any cases in which updating domain settings and then republishing an old version will result in content issues.
How do Book Versions affect Book Metadata?
Book metadata is versioned because it is contained in the converted xml files. From the UI perspective that means that the form in the Metadata modal has moved to a tab per version.
How do Book Versions affect Book Team?
The original scoping outlined that all versions of the book share the same team. For simplicity during development we confirmed with NCBI that from a technical perspective, each version will have a unique team.
System admins and Org admins have access to all versions of the book. PDF2XML vendors have access to all versions with PDF as the Conversion workflow. The person who creates the new version, for example an Editor on book version 'Prepublication draft version 1' who then creates version 'Final full-text version 1', will have access to that new version, while all other editors will need to be given access by being added to the version of that book, via the Wholebook Book manager's Team tab.
How does Book Versions affect Chapter Versions and Chapter-processed books?
Current development does not include Book versions for chapter-processed books. NCBI has also confirmed that when this support is required, this will not include support for a book that has both book versions and chapter versions.
However the current implementation for chapter versions has been brought inline with book versions. This means the following changes:
- All versions of a chapter have a BCMS ID in format
{ID.version-number}
. - Current value
version
in json files changes toversion-number
- Chapter versions support
version-name
andversion-number
in the json file for all workflows. - In the UI, chapter processed books display the
version-name
on the dashboard - The BCMS sends both values in the json file for all chapter ingest sessions (Load to PMC), even though PMC does not support both at this time.
- version-name (ignored by NCBI-systems)
- version-number (read by NCBI-systems)
The UI to support the current book versions development
A demo of the book versions UI has been provided via Mattermost.
The dashboard and collection manager
On the dashboard and collection manager all versions of a book appear. We have not implemented search of BCMS ID yet so searching by the book title is the best way to find all versions of a book.
The New book step
As mentioned above, in the new book step Content type is now a required field, but we could do further work to apply ‘Final-full-text’ as the default if needed. The Content type is used to differentiate versions with the same number. Once the book is created, Content type will display in Book settings and is disabled after book creation and cannot be changed by any user, since it delineates the version in combination with the version number.
The ability to add a version number in the New book step only appears when Conversion workflow is set to 'PDF' or 'XML', and Submission type is set to 'Complete books and documents', since those are the workflows supported by current development of Book versions. In those cases a version number input appears at the bottom of the New book step. The version number need not be 1 in this step (the user can select any number greater than 1), since versions may be created non-consecutively, but the default is 1 for new books.
The Wholebook Book manager
Once you go into a book via the Dashboard, Collection manager or once the New book step is completed, on the Wholebook book manager page all versions will appear in the dropdown on the top-right of the screen, and the latest version is marked with '(latest version)', while on versions which are not the latest, a warning appears on the book manager: '
Books created prior to book versions that did not have 'Content type' filled out at the book creation step will not show a Content type in the version dropdown, they only show a number in the dropdown at present. We could do a migration to apply 'Final full text' to all existing wholebooks which didn't have a content type specified when they were created before book versions, if that is desired by NCBI. For now testing of book versions is best done in new books (created after book versions is released).
The 'New version' button becomes active on the latest version's book manager once that latest version is published. As in the screenshot below, it's not active before the latest version is published.
Once a new version is created, the settings of the previous versions are locked, so only the settings of the latest version are editable.
Metadata is versioned such that versions can have unique metadata (which is populated from the converted file, besides for covers, book source type, and grants information which are filled in manually by the user). The Metadata tab of a newly created version will be pre-populated with the existing metadata from the previous version, until a converted file is received or uploaded for the new version, at which time it's metadata will update with the contents of that file.
The New version modal
Once the latest version is published, the 'New version' button is active.
Clicking on the 'New version' button brings up the New version modal. In the modal you need to specify Conversion workflow and Content type, as well as a version number for this new version. The version number defaults to the next consecutive number after the latest version, but can be changed to any other number greater than 1.
There is a validation in place such that if you try create an existing version of that book again (with the same combination of [content type] + [version number]) you get an error and the duplicate version is not created. The error is 'A version of this book already exists with this combination of content type and version number'. So, for example, if you have created Prepublication draft v1, you can go on to create Final full-text v1 for example, but can't create Prepublication draft v1 in that same book as a new version again.
Once a new version is created, you will see it in the versions dropdown in the Wholebook Book manager, with the '(latest)' marker in the Dropdown.
Navigating to older versions in the dropdown, you will see '
Files for older (non-latest) versions of a book can be updated and the older version can be republished if needed.
In terms of permissions, the version Dropdown disables access to versions when the Editor is not added to the team of that version, but the user can still see the version name (Content type + version number) in the dropdown, so they are aware it exists. For example in the screenshot below, I am an Editor with access to Final full-text v1, and I can't navigate to Prepublication draft v1, since I have only been added to the Final full-text v1 team.
Issues related to this release
The issues below have been dealt with in preparation for this release. Note that the first issue outlines the original proposal. The last issue in the list is an issue which is still open that we are still working on.
Testing of delivered work is best done against this testing ticket, and #1151 (closed), instead of testing of individual issues below.
URL | Coko notes | Title | Milestone | Labels |
---|---|---|---|---|
#1154 (closed) | This issue was promoted to become the Book versions Epic, and informs the current development of book versions as outlined in these release notes. | Support for Book Versions (Maintaining each scientifically or partially updated version of a book) | ||
#1151 (closed) | The BCMS path in the issue should be supported by current development | PDF First Match to Later Full Text Version | MVP deployment,blocking: migration,feature,prod:: deploy 1,release-20220719 | |
#972 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Refactoring WholeBook Conversion case | 2022-May-A | MVP deployment,code refactor,dev:: done,prod:: deploy 1,release-20220719 |
#1191 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Book Version : Update Data Model | 2022-May-A | MVP deployment,dev:: done,feature,prod:: deploy 1,release-20220719 |
#1192 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Book Versions Updating requests to the database to include book_version
|
2022-May-B | Documentation,MVP deployment,code refactor,dev:: done,prod:: deploy 1,release-20220719 |
#1193 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | “New version” status at book level | 2022-May-B | MVP deployment,dev:: done,feature,prod:: deploy 1,release-20220719 |
#1194 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Book settings version: move settings per version | 2022-Jun-A | MVP deployment,dev:: done,feature,prod:: deploy 1,release-20220719 |
#1196 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Update Teams for Book versions | 2022-Jun-B | MVP deployment,code refactor,dev:: done,prod:: deploy 1,release-20220719,scope:: approved |
#1197 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Remove the one default bookComponent for the whole book case | 2022-Jun-A | MVP deployment,code refactor,dev:: done,prod:: deploy 1,release-20220719 |
#1199 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Update bcms_id relation from books, bookComponents, Collections, to errors, and notification Messages. | 2022-May-A | MVP deployment,code refactor,dev:: done,prod:: deploy 1,release-20220719 |
#1200 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Pass version_number and version_name On load to PMC. | 2022-Jun-B | MVP deployment,dev:: done,feature,prod:: deploy 1,release-20220719 |
#1201 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Writing migrations for the books that exist | 2022-Jun-B | MVP deployment,code refactor,dev:: done,prod:: deploy 1,release-20220719 |
#1242 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Change bcms structure to include the version of the book | 2022-Jun-A | MVP deployment,dev:: done,release-20220719 |
#1264 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation. | Frontend development to support book versions | 2022-Jun-B | MVP deployment,UI,dev:: done,feature,prod:: deploy 1,release-20220719 |
#1267 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Fix mutation to support versionNumber as input | 2022-Jun-B | MVP deployment,dev:: done,release-20220719 |
#1270 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Remove dependency of domain createCommand from domain newVersion command | 2022-Jun-B | MVP deployment,dev:: done,release-20220719 |
#1271 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | jest tests on the new version command functionality | 2022-Jun-B | MVP deployment,dev:: done,release-20220719 |
#1280 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Update getBook query to support for book versions | dev:: done,release-20220719 | |
#1283 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Status of the book is not changing from "Previewing" to "Publishing" | 2022-July-A | MVP deployment,dev:: done,release-20220719 |
#1284 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Support creation of vendor issues for Whole books | 2022-July-A | MVP deployment,back-end,bug,dev:: done,release-20220719 |
#1287 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Make sure "Files" and "Errors" tabs show on Word One docs in Book versions feature branch | 2022-July-A | MVP deployment,dev:: done,release-20220719 |
#1289 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | App crashes with the error "uncaughtException: Cannot read properties of undefined (reading 'ownerId') " | MVP deployment,dev:: done,release-20220719 | |
#1291 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | book version testing: Jobs can't retry actions | MVP deployment,bug,dev:: done,release-20220719 | |
#1292 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Support saving metadata updates to older versions in book versions branch | 2022-July-A | MVP deployment,dev:: done,release-20220719 |
#1293 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | QA testing results bugs for new version feature | MVP deployment,bug,dev:: done,release-20220719 | |
#1298 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Book version feature: FTP submissions for XML books not ingested | MVP deployment,bug,dev:: done,release-20220719 | |
#1300 (closed) | This issue was for Coko development work and NCBI is not required to test it in isolation | Book version feature: Cannot change the collection of a book to a Funded Collection | MVP deployment,dev:: done,release-20220719 | |
#1286 (closed) | This issue is not yet resolved and therefore a note to avoid using the chat in this testing release has been added to the top of the release notes. | Support chat for whole books | 2022-July-A | QA: passed,MVP deployment, back-end,bug |
Possible further improvements
Some further improvements that may be desired (there may be others) are:
- Fix the error in the Review and Help chat for PDF and XML wholebooks
- Ensure the files display
{book-version.file-version}
for all relevant sections of the files tab. - Ensure that the zip file created when downloading files changes from
{bcms-id}-{book-component-version}-{timestamp of user}
to:{bcms-id}-{timestamp of user}
- Introduce a restriction that when 'Prepublication draft' or 'Published PDF' are chosen as the Content type, we only allow 'PDF' as the conversion workflow.
- Only show the latest book version on the dashboard
- Make 'Final full-text' the default content type for Word and XML, when content type is not filled in.
- Do a migration to apply 'Final full-text' as the Content type to any existing testing books which are missing a value for Content type (so that this content type displays in the Dropdown and on the dashboard for any existing books which were created without this value before book versions was introduced).
- Implement search by BCMS ID on the dashboard, and possibly extend that further to include searching for specific versions by appending the version number to the BCMS ID
- Extend FTP submission support for subsequent book versions by resolving related issues (for example #438 (closed))
- Extend book versions to cater for going from chapter-processed books to wholebooks and vice versa (related to #783 (closed) and user story 4 in the Scoping sheet)
- Introduce Org Editor permissions so that Editors of an Org have access to all books and book versions in an Org.