Should not be able to add Content type for chapter-processed books, but should be able to add Content type for chapter versions, and other related changes
1. Removing 'versions' indicators at the chapter-processed-books book-level
1.1 Remove Content type (version-name
) and Version number (version
) from the book level for PDF, XML and Word chapter-processed books:
⠀⠀⠀⠀1.1.1 Changes to new book step required
⠀⠀⠀⠀1.1.2 Changes to what we send when we load to PMC for conversions and ingest, in the book-part-wrapper
. We won't send Content type (version-name
) and/or Version number (version
) from the book level. The JSON contains info for the chapter, so going forward we will add related values there when applied to the chapter only (see next section).
⠀⠀⠀⠀1.1.3 Further scoping in future to determine how to support #783 (closed) once this is done.
⠀⠀⠀⠀1.1.4 Further scoping in future to determine an approach for Word books which use the chapter-processed template but should in some cases be wholebooks. Note these changes to chapter-processed books will affect Word One docs as well since they use NCBI's chapter-processed template.
Reason for these changes
Reason for this:
a) PMC architecture and data model will only support versioning (of any version type) at the book OR book part wrapper unit, and never both, and we agreed the BCMS data model would do the same
b) For some books, Manuscript currently applied at book level, written into the JSON for all chapters. But that value is only relevant to some of the chapters, but now appears in all of them.
c) If we wrote a migration to remove the manuscript value from all chapters, it is still relevant to some chapters so shouldn't be removed for those chapters, the migration gets more complex and content type would need to be added to more chapters later
d) Migrations are prone to error, especially for 10,000+ books quality control is impossible, and causes data integrity issues
book-part-wrapper
for relevant chapters
2. Adding 'versions' indicators at the chapter-processed-books chapter-level, into the 2.1. Content type options at the chapter level for all chapter-processed books will be restricted to allowed types per flow and per user type:
⠀⠀⠀⠀2.1.1. For PDF chapter-processed books chapters for system admins: manuscript, prepub, published-pdf, final-full-text are shown to System admins: therefore we need a choice made by the user for that chapter and all subsequent chapters as soon as they're created in the UI or at least before we load to PMC for conversions and ingest.
⠀⠀⠀⠀2.1.2. For PDF chapter-processed books chapters for all other users: manuscript and final-full-text are shown to these users (Org admin, Editor, etc.): therefore we need a choice made by the user for that chapter and all subsequent chapters as soon as they're created in the UI or at least before we load to PMC for conversions and ingest. Note: We must have this content type for all chapters, even if 'Support multiple-published versions' is not on in settings, so that that setting could be turned on later if needed and to standardize the data.
⠀⠀⠀⠀2.1.3. For XML: final-full-text (therefore choice never required or allowed in UI)
⠀⠀⠀⠀2.1.4. For Word: final-full-text (therefore choice never required or allowed in UI)
2.2 Adding version numbers which are accurate to the content to chapter versions:
⠀⠀⠀⠀2.2.1. We did say we would treat chapters like books to maintain the 'real' version number which may correspond to real content version numbers of the content which exist outside the BCMS. Therefore custom version numbers for chapters are required for all workflows.
⠀⠀⠀⠀2.2.2. We also need to have the setting 'Support multiple-published versions' on when appropriate, to activate that functionality in the UI and to link the chapter versions to each other on Bookshelf accurately. But PMC only supports one published version per number, because content type (version-name
) is not supported.
⠀⠀⠀⠀2.2.3. For XML and Word this won't cause any issues since the content type (version-name
) is always the same, and version numbers need to be unique per content type (version-name
) in the BCMS UI and data model. For PDF there could be issues though, if you have manuscript
v1, v2, v3, v3, then create v1 final-full-text
, it could seem on Bookshelf that manuscript
v3 is the latest but final-full-text
v1 is the latest.
⠀⠀⠀⠀2.2.4. To prevent ambiguity around what is the latest chapter in the PDF workflow, we could possibly restrict subsequent versions with different content types to require them to be greater than or equal to the latest version number. We would expect the user to use accurate version number, but they can't create versions for older version numbers (only forward or equal to existing versions, not back). But if they were to go back to udpate an older version, it creates a problem because that older-created version which is now the latest published will display on Bookshelf.
⠀⠀⠀⠀2.2.5. The question then is if we would need to prevent going back, even for System admins? OR for new version, System admins would need to make exact copy of version 1 and make changes, and give it a new version number >/= highest number, but that would defeat the objective of having accurate version info relating to outside archiving information so is not ideal. The only time it would be a problem to go back, is if you had two versions of the same number, with different content types, because an update to an older version of the same number would replace the latest created (not updated) version of that same number on the Bookshelf site, since NCBI always shows the latest published since content type is ignored.
⠀⠀⠀⠀2.2.6. Therefore as a workaround, if an older version of a duplicate number requires an update, you could go back and update (for example manuscript v1) for archiving of the content, then re-publish the latest created final-full-text v1 to reset it back as the latest on Bookshelf. We need to really to think about all the points in 2.2. more, and talk to Martin about it.
2.3. We noted that Word can stay as is for version numbers in chapter versions, incrementing version numbers consecutively. Although this introduces a lot of complexity in the code to treat them differently, so doesn't sense from a technical perspective to treat them differently. We would suggest treating them in the same way and the user can accept a default of incrementing version numbers consecutively, which will be the default in the UI that is editable.
2.4. We need to make content type and version number editable for chapter versions, for all users at new version status. When editing these values, the same validation checks (to be determined based on the above info) should be applied as when first adding these values. In order to prevent incorrect version info before conversion or injest to PMC, we need to:
⠀⠀⠀⠀2.4.1. Introduce a double check of whether the version number and content type is acurrate.
⠀⠀⠀⠀2.4.2. Turn off auto-publishing globally.
⠀⠀⠀⠀2.4.3. At all statuses other than New version, since the values have been sent to PMC for conversion or injest, we need to scope and do separate development to delete or suppress content.
2.5. Update what we check and send to PMC for chapters:
⠀⠀⠀⠀2.5.1. For all chapter-processed books, when we send a chapter to load to PMC for conversions and ingest, we need to add the Content type (version-name
) and Version number (version-number
) to the JSON in the book-part-wrapper
.
⠀⠀⠀⠀2.5.2. In #73 (closed) we had version
for the file version, but at some point in development at NCBI's request we did rename version
to version-name
to describe file versions of a chapter, and that is no longer compatible if we have to use version-name
for the UI's content type of the chapter.
⠀⠀⠀⠀2.5.3. We there propose that we use either all three of the below:
⠀⠀⠀⠀⠀⠀⠀⠀2.5.3.1. version-number
for the chapter version number (the major in the BCMS file version number).
⠀⠀⠀⠀⠀⠀⠀⠀2.5.3.2. file-version
for file versions (the minor in the BCMS file version number).
⠀⠀⠀⠀⠀⠀⠀⠀2.5.3.3. version-name
as the BCMS content type.
⠀⠀⠀⠀2.5.4. Or we could:
⠀⠀⠀⠀⠀⠀⠀⠀2.5.4.1. Use content-type
instead of version-name
in wholebooks and chapter-processed books to describe the BCMS content type.
⠀⠀⠀⠀⠀⠀⠀⠀2.5.4.2. Continue to use version-name
for the file version (the minor in the BCMS file version).
⠀⠀⠀⠀⠀⠀⠀⠀2.5.4.3. Use version-number
for the chapter version number (the major in the BCMS file version).
2.6. Corresponding changes to wholebooks:
⠀⠀⠀⠀2.6.1. For wholebook-processed books: Corresponding restrictions need to be made for the PDF options (manuscript, prepub, published-pdf, final-full-text allowed for System admins; manuscript and final-full-text allowed for all other users) and the default for XML (always final-full-text so no choice should be required) SL: what we said is we'll need in the future to open this up for all users based on settings for the content, but at deployment only System admin would be selecting these as we can't onboard any other PDF content providers
⠀⠀⠀⠀2.6.2. For wholebook-processed books: Corresponding restrictions need to be made to limit wholebook version restrictions based on Org settings (e.g. if PDF is not allowed in Org settings they can't create that as a new book or new version of the book.)
⠀⠀⠀⠀2.6.3. For wholebook-processed books: We need to make BCMS content type and version number editable for versions, for all users at New book or New version status. When editing these values, the same validation checks (to be determined based on the above info and existing duplicate checks) should be applied as when first adding these values. In order to prevent incorrect version info before conversion or injest to PMC, we need to:
⠀⠀⠀⠀⠀⠀⠀⠀2.6.3.1. Introduce a double check of whether the version number and content type is acurrate.
⠀⠀⠀⠀⠀⠀⠀⠀2.6.3.2. Turn off auto-publishing globally.
⠀⠀⠀⠀⠀⠀⠀⠀2.6.3.3. At all statuses other than New version, since the values have been sent to PMC for conversion or injest, we need to scope and do separate development to delete or suppress content.
⠀⠀⠀⠀2.6.4. We also need to make corresponding changes to the JSON info sent to PMC, based on values to be decided on in section 2.5.
3. How point 2 will be implemented in the UI: Changes required to uploading of chapters and introducing of content type and version number
3.1. For the 'first' version of a chapter in the Book manager's bulk upload modal:
⠀⠀⠀⠀3.1.1. We need to introduce a check whether that filename already exists for a source file of an existing chapter, if it doesn't exist, it is considered a 'first' version of a new chapter.
⠀⠀⠀⠀3.1.2. You specify content type and version number for each file which will become each chapter. Restrictions per workflow and user are in place for the workflow and if there is a compulsory default for the workflow it is shown but not editable.
⠀⠀⠀⠀3.1.3. If a chapter version doesn't exist for that chapter (determined based on new check in 3.1.1) we use the default chapter version number 1, which can be edited in that same place it is shown in the UI.
3.2. For new files of an existing chapter version from the files tab of that existing chapter version, they are either incremented or replaced depending on the status of the latest file version in that chapter version:
⠀⠀⠀⠀3.2.1. These new files do not need content type and version number since that already exists for this version.
⠀⠀⠀⠀3.2.2. The existing chapter version's file version is replaced when the existing file is 'New upload' status.
⠀⠀⠀⠀3.2.3. In 'Converting' status no upload is possible for that chapter version.
⠀⠀⠀⠀3.2.4. In 'Previewing' or 'Published' status, the new upload file version has it's minor version incremented based on the last file's status.
3.3. For new files of an existing chapter version via the bulk upload modal:
⠀⠀⠀⠀3.3.1. We need to introduce a check in the Book manager's bulk upload modal to check whether that filename already exists for a source file of an existing chapter
⠀⠀⠀⠀3.3.2. If that filename already exists for a source file of an existing chapter, and is in 'Converting' status we need prevent uploading of that chapter's source file.
⠀⠀⠀⠀3.3.3. If the source filename already exists in an existing chapter version but is not in 'Converting' status, we will accept the source file in the bulk upload modal and increment file versions as described in points 3.2.2 and 3.2.4, but we don't accept a content type and version number value for that chapter's source file there since it exists for that chapter version already and it can't be changed there. See next the sections for how to change these values for existing versions, and how to create new chapter versions.
3.4. The content type and version number values for an existing chapter version can be changed in 'New version' status:
⠀⠀⠀⠀3.4.1. These values are changed from the settings of the chapter version in the relevant status.
⠀⠀⠀⠀3.4.2. At all statuses other than New version, since the values have been sent to PMC for conversion or injest, we need to scope and do separate development to delete or suppress content.
3.5 For new chapter versions:
⠀⠀⠀⠀3.5.1. New chapter versions has to be done from the 'New published version' button which becomes active when the current version is published.
⠀⠀⠀⠀3.5.2. When 'New published version' is clicked, a modal will request content type and version number and restrictions are in place depending on the existing workflow and the type of user. When a workflow requires a default, that default is always shown but can't be edited. We use the next consecutive version number as the default version number based on the last published version of that content type, or use 1 if the content type is not in use already, and the version number can be edited.
4. Edge cases and agreed methods for dealing with them: Required changes to submission workflow between chapters, not to support ideal use case, but to support rectifying mistakes:
4.1. Mistakenly created PDF chapter-processed book, when it was meant to be XML chapter-processed book.
⠀⠀⠀⠀4.1.1. PDF book created, and v1 of chapter sent as PDF manuscript to PMC, v2 of chapter needs to be XML final-full-text but submissions workflow type cannot change within one book across chapters or chapter versions.
⠀⠀⠀⠀4.1.2. You can only link a domain to a domain, not a chapter to a chapter. So history of chapter to chapter in different books can't be kept, only book to book.
⠀⠀⠀⠀4.1.3. We can't simply add a history tab to the BCMS for a chapter version, because that can't be integrated into PMC book database. They have the same IDs so they should be versions of the chapter and not separate books. Merging the records as versions of the same chapter woud be a better solution but isn't clear. We could provide a comments field for history of chapter.
⠀⠀⠀⠀4.1.4. The agreed solution for this case is to:
⠀⠀⠀⠀⠀⠀⠀⠀4.1.4.1. Suppress the PDF book created which includes the v1 of the chapter (since as per steps below it will become a duplicate).
⠀⠀⠀⠀⠀⠀⠀⠀4.1.4.2. Re-create v1 of the book's chapter in the correct workflow (in this case XML)
⠀⠀⠀⠀⠀⠀⠀⠀4.1.4.3. Create subsequent versions of the chapter, now in the correct book.
4.2. Mistakenly created XML chapter-processed book, when it was meant to be PDF chapter-processed book.
⠀⠀⠀⠀4.2.1 Publisher accidentally creates XML chapter-processed book, and adds v1 of chapter as XML (with the default) final-full-text
⠀⠀⠀⠀4.2.2 Publisher then says 'you are not allowed to have this final full-text version'
⠀⠀⠀⠀4.2.3 The agreed solution for this case is to:
⠀⠀⠀⠀⠀⠀⠀⠀4.2.3.1. Suppress the XML book created which includes the v1 of the chapter as final-full-text (since as per steps below it will become a duplicate, and moreover is not allowed with this content type and therefore submission workflow).
⠀⠀⠀⠀⠀⠀⠀⠀4.2.3.2. Re-create v1 of the book's chapter in the correct workflow (in this case PDF) with the correct content type (in this case manuscript).
⠀⠀⠀⠀⠀⠀⠀⠀4.2.3.3. Create subsequent versions of the chapter, now in the correct book.
5. More consideration is needed
5.1 For TOC building
5.2 For how preview support will work
6. Separate but related topics convered:
6.1. For PDF books in general (whole and chapter-processed) sometimes NCBI gets content that can only be released to the public on a certain date, restrict publishing before that date (e.g 2022-09-09). This is sometimes called an 'embargo' but is actually a restriction on release date and we should call it that. Prevent pressing publish until release date. Could be for a chapter or a book, not for collection level.
6.2 Allow system admins to change who can see and do what (may be restricted based on user type, org type, particular org), without doing permissions dev on Coko's side. Admin level management inside the BCMS from within Org (and possibly user) settings.
7. MVP Priority based on the above
7.1. Remove content type and version number at the book level for chapter-processed books:
⠀⠀⠀⠀7.1.1. Changes required to 'New book' step based on the MVP Priority, including related Wholebook options improvements to reduce errors:
⠀⠀⠀⠀⠀⠀⠀⠀7.1.1.2. Move the 'Submission type' choice up in the new book step, to sit below the existing 'Collection' choice, the options for 'Submission type' are 'Complete books and documents' and 'Individual chapters'.
⠀⠀⠀⠀⠀⠀⠀⠀7.1.1.3. Move 'Conversion workflow' choice up, to be below the 'Submission type' choice and above the 'Content type' choice. Restrict 'Conversion workflow' options to what the Organisation creating the book is allowed to create based on the organisation's settings – #1310 (closed).
⠀⠀⠀⠀⠀⠀⠀⠀7.1.1.4. Only show 'Content type' when 'Submission type' = 'Complete books and documents' and 'Conversion workflow' = PDF or XML.
⠀⠀⠀⠀⠀⠀⠀⠀7.1.1.5. When 'Submission type' = 'Complete books and documents' and 'Conversion workflow' = XML, make 'Content type' automatically = 'Final full-text', and that default Content type in this case shouldn't be editable.
⠀⠀⠀⠀⠀⠀⠀⠀7.1.1.6. When 'Submission type' = 'Complete books and documents' and 'Conversion workflow' = PDF, make the following 'Content type' options available, in this order: 'Author manuscript', 'Prepublication draft', 'Published PDF', 'Final full-text'.
⠀⠀⠀⠀⠀⠀⠀⠀7.1.1.7. Only show version number input when 'Submission type' = 'Complete books and documents' and 'Conversion workflow' ≠ Word ('Conversion workflow' = PDF or XML).
⠀⠀⠀⠀7.1.2. Changes required to JSON sent for chapters:
⠀⠀⠀⠀⠀⠀⠀⠀7.1.2.1 Do not send a content type (version-name
) for chapter processed books in the chapters' JSON.
7.2. Not overwriting content-type related processing instructions as per #1378 (closed) (@lathrops1 to check if they're all there and then we can handle development in that issue.)
7.3 Note: For now we will leave chapter version numbers as is for existing chapter versions (incrementing consecutively), and in future new chapter versions can use custom numbers when development in point 8 is done to cover other points in this issue. For now hold off on onboarding users who need custom version numbers.
8. In future:
8.1. Migrate PI from data to be developed in #1378 (closed), into the content type field, and send that in chapter JSON when applicable as the version-name
.
8.2. Do the development described in the rest of this issue for chapter versioning changes.
Archived discussion
@ChristinaTromp - cc @jordandc
Expected behaviour
User should only be able to create content and numbered book versions on wholebooks, never chapter-processed books.
Current behaviour
User can add a content type to chapter-processed books. This should not done, because user must be able to create and change content types on the chapter published versions based on the agreed data model. And you can never do both. We have real use we need to migrated blocked by this.
Steps to reproduce
Go to create book and see you have options to create add a content type in these cases:
NCBI's priority feedback
Y, blocking migration and deployment
Proposed solution
Changes to 'New book' step
- Move the 'Submission type' choice up in the new book step, to sit below the existing 'Collection' choice, the options for 'Submission type' are 'Complete books and documents' and 'Individual chapters'.
- Move 'Conversion workflow' choice up, to be below the 'Submission type' choice and above the 'Content type' choice. Restrict 'Conversion workflow' options to what the Organisation creating the book is allowed to create based on the organisation's settings – #1310 (closed).
- Only show 'Content type' when 'Submission type' = 'Complete books and documents' and 'Conversion workflow' = PDF or XML.
- When 'Submission type' = 'Complete books and documents' and 'Conversion workflow' = XML, make 'Content type' automatically = 'Final full-text', and that default Content type in this case shouldn't be editable.
- When 'Submission type' = 'Complete books and documents' and 'Conversion workflow' = PDF, make the following 'Content type' options available, in this order: 'Author manuscript', 'Prepublication draft', 'Published PDF', 'Final full-text'.
- Only show version number input when 'Submission type' = 'Complete books and documents' and 'Conversion workflow' ≠ Word ('Conversion workflow' = PDF or XML).
Changes to 'New version' step
- 'Conversion workflow' options are 'PDF' or 'XML', restrict the choice to what is allowed for the Organisation who created the book – #1310 (closed).
- When 'Conversion workflow' = 'XML', make 'Content type' = 'Final full-text' and not editable.
- When 'Conversion workflow' = 'PDF', make the following 'Content type' options available, in this order: 'Author manuscript', 'Prepublication draft', 'Published PDF', 'Final full-text'.
#1340 (closed)
Changes to PDF and XML Wholebook Settings –- When 'Submission type' = 'Complete books and documents' and 'Conversion workflow' = PDF, make 'Content type' in Settings editable, and shown in this order: 'Author manuscript', 'Prepublication draft', 'Published PDF', 'Final full-text'.
- Show version number in settings and make it editable in all cases.
- When saving a change to 'Content type' or 'Version number', perform the same validation we do for the 'New version' modal saving, where we check that the combination of 'Content type' and 'Version number' doesn't already exist, and if it does, we throw an error and do not save the settings.
- When 'Content type' is edited and settings are saved, update the dropdown which shows all book versions, and the name of the version shown on the dashboard and collection manager, to show the new 'Content type' value.
- When 'Version number' is edited and settings are saved, update the dropdown which shows all book versions, and the name of the version shown on the dashboard and collection manager, to show the new 'Version number' value.
- Make edits to PDF and XML Wholebook Settings (after book is already created) limited to Sys admins only. Note for Wholebook book versions, Settings for all versions which are not the latest will already be locked and not editable by anyone, and that will stay in place.
New chapter version step
- In a chapter-processed book with 'Support multiple published versions' turned on in settings, when a user presses the 'New published version' button from a published chapter, if the 'Conversion workflow' for the chapter-processed book = 'PDF', open a modal before creating the new chapter version. The modal should look similar to the Wholebook 'New version' modal, except only show 'Content type' dropdown, with modal heading 'Create new published chapter version'
- In the new modal (which only shows when conditions are as explained above), give the user a choice in a 'Content type' dropdown between 'Author manuscript', 'Prepublication draft', 'Published PDF', 'Final full-text', in that order.
- Change the carousel with chapter version numbers to a dropdown, now showing the 'Content type' and automatically incremented version number for the chapter versions.
- Send
version-name
andversion-number
in the json file for all workflows (should already be in place, but is currently getting inherited from the book 'Content type' forversion-name
which is wrong). For individual chapters in the Word and XML workflows, the defaultversion-name="final-full-text"
is sent. For PDF workflow the value provided in the FTP or BCMS submission is sent for that chapter version, eithermanuscript
,prepub
,published-pdf
,final-full-text
depending on the option chosen in the UI or in the submission.
Question for NCBI @lathrops1, since we are making 'Content type' editable for PDF and XML wholebooks, do you need it to be editable for chapter published versions too? We don't show settings per chapter version, so I don't know how that would be possible, if required. Just want to double check you don't expect that. – Answer: yes for Sys admins. Follow up question: what is the priority for this?
QA Steps
[To be completed by Coko once dev is done]