ncbi issueshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues2024-02-07T15:42:54Zhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1654Upgrade to the latest @coko/server2024-02-07T15:42:54ZGiannis Kopanasjkopanas@gmail.comUpgrade to the latest @coko/server<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, ...<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, and goals? -->
Upgrading now is in line with the request in #1628 because we can make use of the fileStorage and AWS s3 Service, making the migration issue to AWS s3 easier.
## Proposal
<!-- A precise statement of the proposed feature. -->
Upgrade to the latest @coko/server. Note: there are breaking changes so we propose refactoring in BCMS step-by-step, alongside feature development, to bring the code closer to coko/server current version, then complete the upgrade.
## Design
<!-- Include sketches or wireframes of the UI suggested for this feature. -->
## Acceptance criteria
<!-- Provide the criteria that should be met for this feature. These criteria must be clearly defined customer requirements aligned to NCBI’s user stories in its Statement of Work. These criteria must be testable by either user testing, unit tests or integration tests.-->
## Definition of ready
<!-- A checklist of what needs to be done to a product backlog item before the team can start implementing it in the next sprint. -->
- [ ] BCMS User Story / Context has been well defined
- [ ] The priority of the user story is specified and agreed
- [ ] Digital assets added (design, database scheme, mockups etc if relevant)
- [ ] Coko Technical Proposal approved by NCBI
- [ ] Testable Acceptance Criteria approved by NCBI
- [ ] Estimate of effort to complete (time or points)
- [ ] The issue has been broken down into development tasks (if necessary)
- [ ] Requirements Clarified
- [ ] The product owner and development team agree that the user story is ready for development
- [ ] NCBI adds “Dev_Ready”
## Definition of done
<!-- A checklist of criteria that must be completed for a story to be considered “done.” -->
- [ ] All coding tasks are finished and implemented
- [ ] QA approved
- [ ] Deployed and tested on “ncbidev” (by Coko team)
- [ ] Deployed and tested on “ncbi” (by NCBI team)
- [ ] Acceptance Criteria Met
## Implementation
<!-- A description of the steps to implement the feature. To be completed by the lead dev. If there are multiple tasks, then break these down into "task" below.-->
## Alternative approaches (if applicable)
<!-- Include any alternatives to meet this use case. -->
## Scheduling
* [ ] Milestone is linked
* [ ] Iteration is linked
* [ ] Dependencies: ("None" or list issue numbers if relevant)
* [ ] Development estimate is added to issue time trackingP05: Address MVP Files Management and Processing Issues to support all current Bookshelf submitters and NCBI Integration specificationshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1653Add Loaders to Resolvers2024-02-07T15:45:51ZGiannis Kopanasjkopanas@gmail.comAdd Loaders to Resolvers- Check the loaders we already have
- Add new loaders where is necessary- Check the loaders we already have
- Add new loaders where is necessaryP05: Address MVP Files Management and Processing Issues to support all current Bookshelf submitters and NCBI Integration specificationshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1652Migrate all the parsing from Cheerio to SaxParser2024-01-29T09:05:26ZGiannis Kopanasjkopanas@gmail.comMigrate all the parsing from Cheerio to SaxParserAt the moment we have two different ways for parsing the Xml .
Cheerio for the smaller files and SaxParser for xml Files bigger than 10mb.
We need to move to SaxParser.
This can be done in parallel with issues about xml File Parsing .At the moment we have two different ways for parsing the Xml .
Cheerio for the smaller files and SaxParser for xml Files bigger than 10mb.
We need to move to SaxParser.
This can be done in parallel with issues about xml File Parsing .P03: Support valid and compliant metadata for all migrated Bookshelf contenthttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1651Improve SubmitComponent Controller2024-02-07T15:43:57ZGiannis Kopanasjkopanas@gmail.comImprove SubmitComponent ControllerRefactor / Clean the submit controller.
- Check the possibility to split the controller into two seperate controllers for book and chapters
- Improve the cases for pdf to separate functionality Since we need all the files . (Move that t...Refactor / Clean the submit controller.
- Check the possibility to split the controller into two seperate controllers for book and chapters
- Improve the cases for pdf to separate functionality Since we need all the files . (Move that to the model)
- Move the Creation of Xml files For pdf case. to the XmlFactory Service.P03: Support valid and compliant metadata for all migrated Bookshelf contenthttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1624Remove BookComponentVersionId from url and the codebase2024-02-07T15:40:03ZGiannis Kopanasjkopanas@gmail.comRemove BookComponentVersionId from url and the codebaseRemove BookComponentVersionId of tables (BookComponent and FileVersion) from the codebase, because it is not needed anymore and use instead the id of the model.
- [ ] Use the id of the bookComponent for the resolvers and remove the Boo...Remove BookComponentVersionId of tables (BookComponent and FileVersion) from the codebase, because it is not needed anymore and use instead the id of the model.
- [ ] Use the id of the bookComponent for the resolvers and remove the BookComponentVersionId .
- [ ] Remove the BookComponentVersionId from queries (UI) and use instead the id.
- [ ] Remοve the BookComponentVersionId from FileVersion tables everywhere it should be set as null if that is true the we can remove it .
- [ ] Let's check the resolver getNextPrevious of the BookComponent because it is used for some reason that we need to clarify and check that if this will be removed the functionality for moving to previous next still works.P02: Performance and Infrastructure MVP improvements for expected production stress loadsPokhiBela TociPokhihttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1614Search by "new upload" status finds "Part" in results even though "Parts" cur...2024-01-16T17:47:49ZDione Mentisdione@coko.foundationSearch by "new upload" status finds "Part" in results even though "Parts" currently have no Status displaying in these components<!-- Required. Provide a general summary of the issue in the title above -->
## Expected behaviour
<!-- Required. Tell us what should happen -->
All Components can be filtered correctly by displayed component status.
## Current behav...<!-- Required. Provide a general summary of the issue in the title above -->
## Expected behaviour
<!-- Required. Tell us what should happen -->
All Components can be filtered correctly by displayed component status.
## Current behaviour
<!-- Required. Tell us what happens instead of the expected behaviour -->
Parts don't have statuses currently and are retrieved when a user Filters by "New Upload" status
## Steps to reproduce
<!-- Required. Provide a link to a live example or screenshots, and the steps to reproduce this bug.]-->
1.
2.
3.
4.
## Environment
<!-- Required. Provide relevant information such as browser name and version, PC or Mac use, internet speed, etc.]-->
## Possible solution
<!-- If known, provide details on how to fix the bug.-->
To be scoped with #1455
<!-- After creating this issue you can link other related or blocking issues with the Gitlab's Linked issues functionality. -->
## QA Steps
[To be completed by Coko once dev is done]
## Scheduling
<!-- Select all the relevant options -->P07 - Support Current Bookshelf Chapter-Processed contents with Parts with Bodyhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1549Refactor graphql type "MetadataOutput"2023-09-27T11:02:26ZDuncan BennettRefactor graphql type "MetadataOutput"This is the current definition of "MetadataOutput":
```
type MetadataOutput {
total: String!
skip: String!
take: String!
}
```
"String!" should be "Int!" but correcting this will probably break a lot of dependent code.
TODO: Cor...This is the current definition of "MetadataOutput":
```
type MetadataOutput {
total: String!
skip: String!
take: String!
}
```
"String!" should be "Int!" but correcting this will probably break a lot of dependent code.
TODO: Correct the definition and fix any code that breaks as a result.Giannis Kopanasjkopanas@gmail.comGiannis Kopanasjkopanas@gmail.comhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1529Show and updated list of successfully uploaded files on all file lists2024-01-16T17:58:44ZDione Mentisdione@coko.foundationShow and updated list of successfully uploaded files on all file lists<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, ...<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, and goals? -->
This issue is for "Files lists" on all Files tabs. This relates to the new feature in #1497
## Proposal
<!-- A precise statement of the proposed feature. -->
The file list (i.e. where the user uploaded the file in the first place) shows completed uploads by all users.
## Design
<!-- Include sketches or wireframes of the UI suggested for this feature. -->
### Current design
Currently when users upload files on the Files tab UI they can see some basic information about the progress in the upload when there are lots of files or very big files.
#### Example for the supplementary files sections
*Step 1: User selects "Upload", choose the file, and then selects "Save" button*
![Screenshot_2023-08-11_at_15.29.12](/uploads/2b578fa010e3deca4e69096c70509126/Screenshot_2023-08-11_at_15.29.12.png)
*Step 2: User See button change to "Saving"*
Issues here include:
* The upload can take long and there is limited progress indicated here (resolved by #1497)
* The UI incorrectly shows the "Ready for Upload" at this point
![Screenshot_2023-08-11_at_15.29.22](/uploads/6f7698da9fc382e0d0ace6c35d40d324/Screenshot_2023-08-11_at_15.29.22.png)
*Step 2: User See status change to 'Uploading file'*
Issues here: It can take a long time to get to this point and it's not clear to the user what happens if they navigate away from the page (resolved by #1497)
![Screenshot_2023-08-11_at_15.30.46](/uploads/f859f7ab3865082736eb4f53254c2677/Screenshot_2023-08-11_at_15.30.46.png)
### Design amendment
* Step 1 above remains the same
* After the user selects "save" the modal will close
* The user will see the global files UI (#1497) appear
* The user will see files that upload successfully listed below
*Step 1: User selects "Upload", chooses the file, and then selects "Save" button*
![Screenshot_2023-08-11_at_15.29.12](/uploads/2b578fa010e3deca4e69096c70509126/Screenshot_2023-08-11_at_15.29.12.png)
*Step 2: users sees file list unchanged until the file has uploaded successfully.*
![Screenshot_2023-08-11_at_16.06.04](/uploads/c0fd27c5452b67f1e2f9c5eed553672d/Screenshot_2023-08-11_at_16.06.04.png)
## Acceptance criteria
<!-- Provide the criteria that should be met for this feature. These criteria must be clearly defined customer requirements aligned to NCBI’s user stories in its Statement of Work. These criteria must be testable by either user testing, unit tests or integration tests.-->
- [ ] A BCMS user may navigate away from any book part or book component any of its files upload areas once files have been validated and confirmed by the user that they should be completely uploaded into the BCMS file storage, including from:
- [ ] A Bulk Upload page from a chapter-processed Book Manager page
- [ ] Any Source, Converted, Supplementary, Images, Support, Display PDFs Upload page from the Files tab of a Book Part or Book Component
- [ ] A BCMS user may expect after they have pressed the Upload button via the Bulk Upload page of the Book Manager and the Save button via any Upload page from a Files tab that
- [ ] Status of all uploaded files will be reported within 20 sec via a Upload Status bar according to the acceptance criteria in #1497
- [ ] All completed uploads via a Bulk Upload from a Book Manager page will
- [ ] Create a new book part component if a new source file and load that source file to that component with the correct uploaded time and user details
- [ ] Create a new source file version in an existing book part component with the same file name as uploaded with the correct uploaded time and user details
- [ ] All completed uploads via a Upload from a Book Part or Book Component Files page will
- [ ] Create a new file version in the file list within the files folder in which it was uploaded with the correct uploaded time and user details
[*COMMENT: I DON’T KNOW WHAT WEBSOCKET NOTIFICATIONS ARE, BUT IF THEY ARE POPUP MESSAGES LIKE STREAMING MOBILE MESSAGES, I THINK THAT WILL ANNOY USERS IF THEY MOVE TO ANOTHER BOOK TO DO SOMETHING ELSE WHILE HUNDREDS OF FILES ARE UPLOADING – THEY WON’T WANT TO GET 100S OF POP UP MESSAGES*]
## Definition of ready
<!-- A checklist of what needs to be done to a product backlog item before the team can start implementing it in the next sprint. -->
- [ ] BCMS User Story / Context has been well defined
- [ ] The priority of the user story is specified and agreed
- [x] Digital assets added (design, database scheme, mockups etc if relevant)
- [ ] Coko Technical Proposal approved by NCBI
- [ ] Testable Acceptance Criteria approved by NCBI
- [ ] Estimate of effort to complete (time or points)
- [ ] The issue has been broken down into development tasks (if necessary)
- [ ] Requirements Clarified
- [ ] The product owner and development team agree that the user story is ready for development
- [ ] NCBI adds “Dev_Ready”
## Definition of done
<!-- A checklist of criteria that must be completed for a story to be considered “done.” -->
- [ ] All coding tasks are finished and implemented
- [ ] QA approved
- [ ] Deployed and tested on “ncbidev” (by Coko team)
- [ ] Deployed and tested on “ncbi” (by NCBI team)
- [ ] Acceptance Criteria Met
## Implementation
<!-- A description of the steps to implement the feature. To be completed by the lead dev. If there are multiple tasks, then break these down into "task" below.-->
Subscription Management
[Lead Devs to complete]
## Alternative approaches (if applicable)
<!-- Include any alternatives to meet this use case. -->
## Scheduling
* [ ] Milestone is linked
* [ ] Iteration is linked
* [ ] Dependencies: ("None" or list issue numbers if relevant)
* [ ] Development estimate is added to issue time tracking
---P09: Book Manager and other MVP BCMS improvements and Lower Priority Bug Fixeshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1483Replace drag and drop component with position up/down component2023-08-16T06:50:10ZDione Mentisdione@coko.foundationReplace drag and drop component with position up/down component<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, ...<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, and goals? -->
The context is provided in the Epic https://gitlab.coko.foundation/groups/ncbi/-/epics/2 and Table of contents building user stories in #1458.
## Proposal
<!-- A precise statement of the proposed feature. -->
Depending on threshold testing feedback proposed in the Book Manager Performance User Story, another minimal change is to replace drag and drop with a simpler position up/position down component to improve the user experience of moving chapters a short distance from their current position.
## Design
<!-- Include sketches or wireframes of the UI suggested for this feature -->
The user clicks on the up or down icon to move a component one position up or down.
![Screenshot_2023-02-16_at_17.04.22](/uploads/cab841af17630d025d68b3978781d265/Screenshot_2023-02-16_at_17.04.22.png)
## Implementation (if applicable)
<!-- A description of the steps to implement the feature.-->
## Alternative approaches (if applicable)
<!-- Include any alternatives to meet this use case. -->
## Scheduling
<!-- Assignee and labels are added automatically. After creating this issue you can link other related or blocking issues with the Gitlab's Linked issues functionality. -->P09: Book Manager and other MVP BCMS improvements and Lower Priority Bug Fixeshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1475Don't run Coko checks on metadata before saving uploaded converted file2024-01-08T23:38:23ZStacy LathropDon't run Coko checks on metadata before saving uploaded converted file<!-- Required. Provide a general summary of the issue in the title above -->
## Expected behaviour
Per specifications - https://docs.google.com/spreadsheets/d/1M4ZdBbzr2s4-PUXqEblfsKoPBUuk4aG343u9ZAUn_Tk/edit#gid=625909583
User should...<!-- Required. Provide a general summary of the issue in the title above -->
## Expected behaviour
Per specifications - https://docs.google.com/spreadsheets/d/1M4ZdBbzr2s4-PUXqEblfsKoPBUuk4aG343u9ZAUn_Tk/edit#gid=625909583
User should be able to upload valid converted xml files to new books. NCBI's converted BXML files for migration include two DTDs:
* BITS:https://jats.nlm.nih.gov/extensions/bits/
* NLM 2.0: https://dtd.nlm.nih.gov/book/2.0/
## Current behaviour
<!-- Required. Tell us what happens instead of the expected behaviour -->
User is unable to upload valid converted BITS xml file to new book, because the upload is blocked when a Coko check on `pub date` fails even though the file loads successfully from NCBI Silverlight CMS.
## Steps to reproduce
See recording: ![video1575604341](/uploads/79b8d27b44269e5e1bc4a317f5113d74/video1575604341.mp4)
Book recorded: https://ncbi.cloud68.co/organizations/308db55a-21c9-4a64-9a70-d5aa7574726c/bookmanager/31a23f23-aaac-49e8-abf4-f0bc23b006cd
Test files resupplied in https://gitlab.coko.foundation/ncbi/ncbi/-/issues/1475#note_108986 [bcms6772.590b65.2023_02_03-08_48_39.output.zip](/uploads/d23200821bc5a102f5dc2fa74c510a7a/bcms6772.590b65.2023_02_03-08_48_39.output.zip)
## Environment
<!-- Required. Provide relevant information such as browser name and version, PC or Mac use, internet speed, etc.]-->
## Agreed solution
Remove Coko checks on metadata that would prevent ANY converted XML in the BITS or NLM 2.3 DTD from being sent to load to PMC so that all errors are reported to user / system admin to know how to report to successfully fix the errors.
Note, Coko should still validate all XML THEY WRITE against the BITS DTD and PMC style checker to ensure it is valid against the DTD and PMC style compliant, and there should be a process for fixing any errors against those NCBI supported scripts as part of any Coko maintenance.
**Caveat**
* NCBI acknowledges if we remove checks when saving data, this is likely to lead to errors down the line when the BCMS writes book metadata (in the case of TOCs)
* We agree to follow a clear process to resolve these errors, summarized below:
* BCMS runs ALL XML IT WRITES against the PMC Style checker - see #1643
* If XML that the BCMS writes fails the PMC Style checker, a bug is immediately issued and triaged as maintenance
* In all other cases if there are XML issues, NCBI identifies which aspect of metadata is causing an error
* NCBI reproduces the case on ncbi.cloud68.co if it doesn't already exist there.
* NCBI opens a "Metadata feature extension" issue with the relevant information
* Coko team reviews and estimates the issue for development
* Coko/NCBI teams review scope, sign off, and schedule in the dev cycle according to priority.
## QA Steps
[To be completed by Coko once dev is done]
## Scheduling
## Possible alternatives
Provide all possible `pub date` types to save as accepted values so that Coko's existing check can remain.P05: Address MVP Files Management and Processing Issues to support all current Bookshelf submitters and NCBI Integration specificationshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1453Remove failed state and allow System admin to retry any in progress session2024-01-08T23:35:04ZDione Mentisdione@coko.foundationRemove failed state and allow System admin to retry any in progress session<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, ...<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, and goals? -->
It is suspected it could take more than four hours to load to PMC ~1000 files and so a Failed state would prevent all files from having their metadata updated. For example, it currently takes over 12 hours to load to PMC all GeneReviews chapters from Silverlight.
## Proposal
<!-- A precise statement of the proposed feature. -->
Remove "Failed" state and allow System admin to retry any in progress session for the Jobs UI without waiting. Note that in the Silverlight CMS users can reload a job that is in any status in PMC TM and this has not been a major issue to date.
## Design
<!-- Include sketches or wireframes of the UI suggested for this feature -->
## Implementation (if applicable)
<!-- A description of the steps to implement the feature.-->
## Alternative approaches (if applicable)
<!-- Include any alternatives to meet this use case. -->
## Acceptance Criteria
- [ ] System Admin can use the /Jobs UI (https://ncbi.cloud68.co/jobs) to retrigger any active state (Converting, Loading to Preview, Publishing)
- [ ] BCMS will never automatically "Fail" any active jobs after four hours
## Scheduling
<!-- Delete the irrelevant options -->P05: Address MVP Files Management and Processing Issues to support all current Bookshelf submitters and NCBI Integration specificationshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1322Multiple versions appearing on Collection pages and Dashboards2024-01-02T17:28:43ZStacy LathropMultiple versions appearing on Collection pages and Dashboards
## Expected behaviour
1. The Dashboard should display a list of all collections and the current version of all Books
2. Collection pages should display as how a Collection is built and that only has the most recent book version.
## ...
## Expected behaviour
1. The Dashboard should display a list of all collections and the current version of all Books
2. Collection pages should display as how a Collection is built and that only has the most recent book version.
## Current behaviour
1. All book version appear on the Dashboard
2. All book version appear on a collection page
## Steps to reproduce
Go to Org at: https://ncbi.cloud68.co/organizations/3293395b-7809-4bed-8b84-8bdc6d86f895
![image](/uploads/7c365fdccbd3b04cc657b90a0c36d536/image.png)
See multiple book versions.
See latest feedback from testing on Collection pages in: https://gitlab.coko.foundation/ncbi/ncbi/-/issues/1322#note_95880
## Implementation
**Option 1**
1. On Dashboard, roll up all versions of the same book into one row
2. On Collection manager page, roll up all versions of the same book into one row
**Option 2**
Keep the functionality as it is and just add a filter that lets the user hide the previous versions
Take into account, for this option: If a book has a high number of versions, this could affect loading performance. This testing should be done to validate the threshold and if necessary make amendments. (Since we already have pagination on both UIs, we expect low effort to resolve any performance issues).
## NCBI's priority feedback
Y - need to test the extent of this damage and its repercussions, but this is not a faithful collection list
## QA Steps
[To be completed by Coko once dev is done]P09: Book Manager and other MVP BCMS improvements and Lower Priority Bug Fixes