Book Manager Performance Issues
Context / User Story
The MVP BCMS must support Bookshelf staff and other existing Silverlight CMS users to complete all standard tasks supported by current Bookshelf operations and content management with comparable speed and minimal disruption (such as freezing or crashing) as they are capable using the Silverlight CMS. Because of an improvement made to the BCMS data model from the existing Silverlight CMS to support editorial teams to manage their book part contents at the same time themselves, Bookshelf requires the BCMS to have functionality and integration to support the storage, tracking, and processing of converted individual Bookshelf BITS book part wrappers of front matter parts, chapters, parts, and back matters parts and their associated source, image, supplementary, and support files. These files should be grouped for processing integration and files management, not only by the chapter content components, but also under their book title, book metadata, and settings forms, with important historical details, such as status, last updated and last published dates, to facilitate user’s ability to successfully manage their content. As with the Silverlight CMS, Bookshelf requires that the BCMS can load this page and conduct each transaction on that page within 20 sec, with notification to the users that the BCMS is loading or conducting a transaction. This user story should also meet the terms of the general Performance User Story: &103
Book Manager Performance issues that need to be solved for the MVP: Coko implemented lazy load scroll bars to the Book Manager pages to address nonresponsive page browser error. However, these scroll bars and all other functionality on the page, such as TOC Build drag and drop bars, are still extremely slow in responding and create issues in BCMS users ability to build their Tables of Contents (see TOC Building user story); it has also resulted in other performance issues, including the inability of users to see and multi-select components of a similar status or last updated or last published timestamp for a bulk action, such as to resubmit to conversion, reload a preview, or to publish them (frequently necessary for up to batches of 50 book part components), all required files management user stories for these users. Even if users try to select files individually via the Book Manager page, bulk check boxes are not responsive.
Impact if these Book Manager Performance issues are not solved: If this performance issue is not met, it will seriously impact a set of high NIH stakeholders from meeting their daily processing needs, and Bookshelf’s ability to mitigate the issues by assisting these users.
Proposal
If necessary to meet TOC Building and Performance user stories according to its related and dependent development path, redesign the existing Book Manager page using existing main BCMS Dashboard design and functionality.
Design - IF NECESSARY TO ADDRESS USER STORY
For processing, provide a chapter-processed dashboard of all book part components, whether front matter, part sections (including parts with metadata and body content), chapter, or back matter, that uses the same existing code for the main user dashboard for books, with same component and file history, ordering, and search and filtering, e.g.,
For filtering, see user story:
Search / Findability Story: The Silverlight CMS permits its users to search and find content by search by domain name, title, collection, publisher. It also permits users to sort and filter its dashboard tables by critical details, such as filename, filetype, last updated, last published, and status of the content. The BCMS should ideally permit the same level of search / findability by its users so that they may easily find the content they are required to manage.
Issues that need to be solved for the MVP: Significant chapter-processed book NIH-funded stakeholder editorial teams report they cannot easily find the chapters they need to download or process individually or in bulk to update, fix errors, or QA by title, filename, history details, and status. Bookshelf staff also have provided feed back that it is difficult to find legacy content when testing the BCMS migrations, because it lacks support for search by domain, which has been replaced by a search only by BCMS ID. Impact if this issue is not solved: If this issue is not resolved, it puts at risk Bookshelf’s ability to onboard its users to the BCMS without great resistance from them and adds significant time in adding new and updated content to Bookshelf.
Path to resolution:
IF the current BCMS Book Manager pages must be redesigned to meet Performance user stories, that redesign should also meet the criteria for this Search / Findability user story for chapter-processed books, OR IF the current BCMS Book Manager pages do not need to be redesigned to meet MVP Performance user stories, then Coko will make this user story a priority to address post-MVP once resources permit.
Implementation (if applicable)
Alternative approaches (if applicable)
APPROACH 1 WITHOUT FULL RESDESIGN:
@deniskar will provide full technical details, but performance audit shows:
- addressing the UI having a reference to ~every book in the database to determine roles
- removing the lazy load grid if we are content with current behavior
@lathrops1 test results based on Denis' performance audits. Test that was run on - fully processed GeneReviews bulk updates from 20220110 to NCBI fully migrate book of ~900 chapters in it deployed environment . Findings:
- Bulk upload of 18 files - extremely slow; had to refresh page to see files and status updates - note, some providers report they cannot bulk upload at all blocking their use of test BCMS - see #14444 even when the batch bulk uploaded is under the agreed 50 files.
- Very hard to find all files in New Upload status to process in bulk and not miss any - need following requested features estimated for this page: (a) ability to filter by component status and (b) ability to filter / sort by most recently updated and (c) ability to filter / sort by most recently published
- Current search of body hard to use and does not appear to permit wildcard use - it hides all contents if no search and then took two minutes until I could undo that and have to start my action all over again from scratch - this frustrates users already based on weekly feedback
- Errors logged as Tagging Errors but should be Loading Errors - this is not performance issue and will be reported separately after further testing.
- Bulk publishing takes for less than 15 chapters more than 15 minutes and if you navigate away from the page once bulk action starts, the bulk action stops and you have to start over - this happened several times. This is unstainable if you try to publish ALL ~900 chapters to update metadata or files for integration with outer databases (both frequent tasks), which I could not manage to do given my timing and work schedules given this issue, so this needs to be addressed. Also, though it is hard to reproduce on Coko test site repeatedly, sometimes got JSON errors when trying to publish, and then the action was stopped and I had to start over as well.
- Drag and Drop of files is not functional and must be minimally addressed per #1458 (closed), such as by alternative 1 that would not require full splitting out of TOC manager and dashboard for this page.
- Cannot publish TOC because of validation errors - see #1460
- Converted XML files are getting edited by BCMS when load to ingest in non-book metadata elements, which is outside of XML writing requirements - @lathrops1 needs to create tests for this to see if it is problematic and alters meaningful data in any way, particularly for special characters and will report in a new issue as necessary as it is outside of scope of this immediate issue.
APPROACH 2 WITH FULL REDESIGN:
See proposed design above.
Scheduling
Schedule this feature to be scoped and designed for Epic: