Implement pagination on main chapter-processed Book Manager dashboard page to resolve performance issues
Context
The context for the performance issues are in the Epic description: &106. This issue also relates to the proposal TOC building in #1458 (closed) and supporting parts with Body content in #1455. Also see NCBI's TOC analysis of their current chapter content and TOC analysis of current part content
Proposal
NCBI has chosen the performance improvements below, in combination with Design option 1.
For future-proofing beyond MVP we can look to known large book examples that NCBI has provided and confirmed we should not rule out as a possibility. The largest example provided to date is a book with 35 405 chapters. It is therefore a requirement to design for very large lists of data.
Design
Design option 1 (chosen)
- Improve performance by splitting long lists over pages -- Development estimate 3 days
- The amount of items per page can be experimented with, taking into account:
- the expected number of files being processed at one time (50 components).
- the expected highest number of chapters within a part
- the expected highest number of chapters within a sub-part, and highest number number of sub-parts within a part
- A good starting point is having 100 top-level components per page.
- Having the same amount of items per page introduces consistency which provides a short-hand for users to quickly jump to a particular section on the book.
Data from NCBI on TOC structures
- The highest known number of chapters within a part is currently 271 chapters in a part for the domain mtb and that number gets larger about ~1-5 new chapters every month. This book will be processed in the BCMS as chapter-processed. The parts are Languages, so you get the same drug record in every part.-- see comment reference #1480 (comment 119260)
- The highest known nested levels of parts is three, with the highest known number of parts within parts about ~20, and the highest known entire set of a top level part with all its descendant parts and chapters 613 components. This is for the domain micad.
Acceptance criteria
-
On the main Book Manager dashboard page for chapter-processed books, on the Front, Body, and Back tabs, users will see a pagination at the bottom of the page that displays at least the first and last page number (and however many page numbers in between the design specifies). Clicking on one of these page numbers brings the user to that page within whichever division they are in (Front, Body, Back). The pagination navigation also includes a button for the user to navigate to the previous page, and a button for the user to navigate to the next page.
Pagination Rules and Functionality
-
The initial implementation of the pagination will display 100 records per paginated page on the main Book Manager page for chapter-processed books. If we find during implementation or review that the pages do not load within 20 sec (the Silverlight CMS current baseline) in their native environments (within institutional networks or home systems; benchmark is Bookshelf staff and GeneReviews, LiverTox, LactMed editorial team's environments in which they manage their contents) or select and expand/collapse functionality on each paginated page is not responsive in transactions within 5 seconds, then this default number may need to be changed. -
The default view of a paginated Book Manager page is that all parts are collapsed. -
When a part is collapsed either when a user clicks on collapse all or using the collapse toggle for a particular part, only the top-level part counts toward the 100-record limit (not the subparts or chapters contained within that top-level part). -
For instance, in the case of 'mtb` (as of 20230809) a user when entering the Book Manager page would see two collapsed parts, which count as two records toward the 100-record limit per page. When they expand these two parts, they would see all ~ 542 chapters of that book on the first page, with each chapter nested within its associated parent part. -
For instance, in the case of 'micad` (as of 20230809) a user when entering the Book Manager page would see 7 collapsed top-level parts, which count as 7 records toward the 100-record limit per page. When they "Expand All", they would see all ~ 148 children parts grouped within one of those 7 parent parents they belong, and all ~2888 chapters grouped within one of the 148 second-level parts they belong. The user would see all of this content on page one of the Book Manager. -
For instance, in the case of gtrbook
(as of 20230809) a user when entering the Book Manager page would see the Introduction chapter component that doesn't belong to any part, followed by two collapsed top-level parts ("Genetic variants and drug responses" AND "Genetic variants and disease"), which count as 3 records toward the 100-record limit per page. When they "Expand All", they would see all other ~ 70 chapters grouped within their parent part. The user would see all of this content on page one of the Book Manager, even if another batch of ~ 25 new chapter records were added and moved within one of the top-level parts.
-
-
Each standalone chapter that is not grouped within a part always counts as one record on the page. -
Anything past the 100 record limit per page according to these pagination rules will get pushed to the next page.
-
-
When a System Admin, Org Admin, Editor clicks on the Previous button it will go to the page previous to the one they are currently on and display that page's 100 results -
When a System Admin, Org Admin, Editor clicks on the Next button it will go to the page next to the one they are currently on and display that page's 100 results -
When a System Admin, Org Admin, Editor clicks on a page number it will go to that page and display that page's 100 results
Performance of Paginated Pages
-
Each paginated page in the main, unfiltered / searched Book Manager dashboard page will load within 20 sec (the Silverlight CMS current baseline) in their native environments (within institutional networks or home systems; benchmark is Bookshelf staff and GeneReviews, LiverTox, LactMed editorial team's environments in which they manage their contents) -
System Admin, Org Admin, Editors should expect select functionality and expand/collapse functionality on each paginated page to be responsive for transactions within 5 seconds
Related Book Manager page functionality to test at the same time
-
A user will have the capability to expand all parts via a link at the top right of each paginated page. This link will only expand all parts on the paginated page the user is situated. -
A user will have the capability to collapse all parts via a link at the top right of each paginated page. This link will only collapse all parts on the paginated page the user is situated. -
In a Manual Sort Order for chapter components, System Admin, Org Admin, Editors should expect to see content in a New Upload status at the top of the first paginated page -
In a Manual Sort Order for Parts, System Admin, Org Admin, Editors should expect to see newly created parts at the top of the first paginated page -
Chapters and parts sorted and grouped according automated rules will remain where it has been placed according to those automated rules in the structure of the book regardless of which paginated page that might be -
Each paginated page will show all supported BCMS bulk action buttons, currently permitting System Admin, Org Admin, Editors to successfully: -
Bulk Upload files -
Add a part title -
Bulk Submit files -
Bulk Reload Preview files -
Bulk Publish files -
Move components in a TOC or within a Part -
Repeat components in a TOC
-
-
The Select all functionality on each paginated page of the Book Manager dashboard will only select all component items only on the paginated page the user is currently situated.
Definition of ready
-
BCMS User Story / Context has been well defined -
The priority of the user story is specified and agreed -
Digital assets added (design, database scheme, mockups etc if relevant) -
Coko Technical Proposal approved by NCBI -
Testable Acceptance Criteria approved by NCBI -
Estimate of effort to complete (time or points) -
The issue has been broken down into development tasks (if necessary) -
Requirements Clarified -
The product owner and development team agree that the user story is ready for development -
NCBI adds “Dev_Ready”
Definition of done
-
All coding tasks are finished and implemented by Coko -
All end-to-end tests written and integrated in pipeline by Coko -
QA approved by Coko -
Deployed and tested on “ncbidev” (by Coko team) -
Deployed and tested on “ncbi” (by NCBI team) -
Deployed and tested to NCBI hosted sites and tested by NCBI power users in their native environments -
Acceptance Criteria Met
Implementation
- Introduce pagination -- #1508 (closed)
- Test with data from NCBI (see above)
- Review testing results with NCBI before going any further
Alternative approaches (if applicable)
Alternative approaches (not chosen)
Alternative approaches (not chosen)
Design option 2
- Improve performance by allowing user to load more chapters into view when needed. -- Development estimate 3 days
- The amount of items shown prior to the user having to select 'load more' can be experienced with, taking into account:
- the expected number of files being processed at one time (50 components).
- the expected number of chapters with a part in most cases (?)
- suggest starting with 100 top-level components per page.
There's an assumption here that users are unlikely to load all the chapters within very large because they are automatically ordered and they can find what they're looking for via the search functionality. Eventually we'll reach a threshold where loading too many chapters will cause a performance issue so we should users reason for loading high amount of chapters at one time when considering this design option.
Design option 3
The third option combines two approaches and splits the file processing actions from the table of contents building actions
- Improve performance by splitting long lists over pages in "Dashboard style" (for file processing) -- Development estimate 5 days
- Items are ordered by same default as the Dashboard: last updated
- Nesting is removed
- The amount of items per page can be experienced with, taking into account the expected number of files being processed at one time (50 components).
- suggest starting with 100 components per page.
- Improve performance by simplifying list requirements (for TOC building).
- page accessed via "TOC" button in top right
- only be necessary when manual ordering is required
- requires only
id
andtitle
Scheduling
- Milestone:
- Iteration:
- Dependencies: (list issue numbers if relevant)
- Development estimate (hours):