ncbi issueshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues2024-01-30T12:16:33Zhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1628Switch from FTP to AWS S3 for data exchange with NCBI2024-01-30T12:16:33ZEvgenySwitch from FTP to AWS S3 for data exchange with NCBINCBI Systems is planning to prohibit ftp access from the Kubernetes cluster where BCMS is deployed, because they want to control all outgoing network connections via Istio service mesh, which does not support ftp protocol. Instead of FTP...NCBI Systems is planning to prohibit ftp access from the Kubernetes cluster where BCMS is deployed, because they want to control all outgoing network connections via Istio service mesh, which does not support ftp protocol. Instead of FTP, AWS S3 (which is accessed via HTTPS REST API) will be used for data exchange between NCBI and Coko BCMS.
NCBI will create two buckets in AWS S3 for each environment (NCBI-AWS-DEV and NCBI-AWS-PROD), one for APEX and another one for everything else (conversion, submission, package ingest etc). NCBI will specify buckets name/URL/ARN and provide AWS IAM access keys with permissions to upload/download objects in these buckets. Directory structure within S3 buckets and file naming convention will be mapped to object prefixes / names in S3 and remain the same. Kafka messages / data exchange protocol should also remain the same, except that the `package` attribute in Kafka message JSON would be used to construct an object key within a bucket, for example `"S3://ncbi-books-bcms-dev/AHRQ/hcup_sb261.zip"`
CC @latternm @lathrops1P05: Address MVP Files Management and Processing Issues to support all current Bookshelf submitters and NCBI Integration specificationshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1648Support book versioning to support changes in workflow2024-01-16T17:43:28ZStacy LathropSupport book versioning to support changes in workflow<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, ...<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, and goals? -->
In original NCBI specifications the ability to change workflow type of a book by System Admin was required because this happens frequently in Bookshelf production.
## Proposal
<!-- A precise statement of the proposed feature. -->
Support users to create a new version in order to change the processing workflow. The complication here is that this use case does not constitute a new scientific record (published version) so it would not be correct to change either the `content type` (version name) or `version number`. For example, the following would be valid in this use case:
book version 1:
* conversion workflow: PDF
* content type: final-full-text
* version number: V1
* flag: locked
book version 2:
* conversion workflow: XML
* content type: final-full-text
* version number: V1
* flag: current
#### workflow only change
| Change from | Change to |
|--------------------|--------------------|
| XML | PDF |
| XML | Word |
| PDF | XML |
| PDF | Word |
| Word | XML |
| Word | PDF |
There is no difference in complexity between the possible conversion workflow options, if the processing type remains the same.
#### Minimal workflow for Option 1
1. User publishes OR locks the current book version
2. User chooses to create a new version
3. User chooses the workflow type for the new version
4. BCMS shows confirmation that previous versions will be locked (because of the conversion workflow change)
5. User confirms decision
* new book version is created
* previous version is locked
* user is redirected to the new version
6. User uploads the relevant files to new version (i.e. the BCMS does *not* attempt to match files from the previous version to the current one)
#### Caveats
1. On initial review we don't see a technical issue for books created in the BCMS, however since repeating `content type` and `version number` across BCMS book versions is a departure from the original scope, there may be implications we're not thinking of now. We're adding 2 days for code review and scope sign-off.
1. The [updated request above (option 1)](https://gitlab.coko.foundation/ncbi/ncbi/-/issues/783#option-1-change-in-conversion-workflow-type) asks the BCMS to accommodate a scenario that results in two book versions with the same `book-submit-id` + `version-name` + `version-number`. Therefore the current FTP spec would need to be reconsidered if NCBI plans to introduce FTP support at a later date. Since we also have the `workflow` information in the meta.xml file, a reasonable solution is to add `workflow` as a require value to match book versions. This is the best proposal we can give at this time and it would need to be validated by Coko and NCBI at the start of the project. As an integration point, this spec change requires input from NCBI and Coko teams.
2. We think it would be confusing to users to see two versions in the BCMS with the same name. Ideally we should hide all locked versions from all users except Sys Amins. This is not included in this estimate.
## Design
<!-- Include sketches or wireframes of the UI suggested for this feature. -->
Pending
## Acceptance criteria
<!-- Provide the criteria that should be met for this feature. These criteria must be clearly defined customer requirements aligned to NCBI’s user stories in its Statement of Work. These criteria must be testable by either user testing, unit tests or integration tests.-->
- [ ] System Admin can delete or lock a Book Version in any one Workflow and create a new Workflow Book Version in a different Workflow as long as the processing type remains the same
- [ ] When System Admin successfully modifies the Workflow Type for a Book by creating a new Workflow Book Version, then the Files UI will appear according to the specifications correctly for each Workflow Book Version type, AND
- [ ] Users can only download files in the Locked Workflow versions
- [ ] When System Admin successfully modifies the Workflow Type for a Book by creating a new Workflow Book Version, THEN
- [ ] The Locked Workflow version can no longer be Submitted, Reload to Preview, or Published
- [ ] The new Workflow version will Convert according to the new Workflow Type Processing Integration specifications
## Definition of ready
<!-- A checklist of what needs to be done to a product backlog item before the team can start implementing it in the next sprint. -->
- [ ] BCMS User Story / Context has been well defined
- [ ] The priority of the user story is specified and agreed
- [ ] Digital assets added (design, database scheme, mockups etc if relevant)
- [ ] Coko Technical Proposal approved by NCBI
- [ ] Testable Acceptance Criteria approved by NCBI
- [ ] Estimate of effort to complete (time or points)
- [ ] The issue has been broken down into development tasks (if necessary)
- [ ] Requirements Clarified
- [ ] The product owner and development team agree that the user story is ready for development
- [ ] NCBI adds “Dev_Ready”
## Definition of done
<!-- A checklist of criteria that must be completed for a story to be considered “done.” -->
- [ ] All coding tasks are finished and implemented
- [ ] QA approved
- [ ] Deployed and tested on “ncbidev” (by Coko team)
- [ ] Deployed and tested on “ncbi” (by NCBI team)
- [ ] Acceptance Criteria Met
## Implementation
<!-- A description of the steps to implement the feature. To be completed by the lead dev. If there are multiple tasks, then break these down into "task" below.-->
## Alternative approaches (if applicable)
<!-- Include any alternatives to meet this use case. -->
## Scheduling
* [ ] Milestone is linked
* [ ] Iteration is linked
* [ ] Dependencies: ("None" or list issue numbers if relevant)
* [ ] Development estimate is added to issue time tracking
---P06: MVP Data Model and Integrity Issues to Ensure Data Quality in Bookshelf and NLM Databaseshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1497A global uploads UI to show all uploads that in progress for that user2024-01-08T17:11:03ZDione Mentisdione@coko.foundationA global uploads UI to show all uploads that in progress for that user<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, ...<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, and goals? -->
This idea came up in scoping #1495. When users uploads a lot of files at once this can take some time. Ideally users should not be blocked from doing other work in the BCMS while they wait for the files to complete uploading.
## Proposal
<!-- A precise statement of the proposed feature. -->
Add a "global uploads" UI to show all bulk uploads that are in progress for that user. This includes:
- chapter source files
- wholebook source files
- files associated to chapter or book: images, supplementary, display PDFs
This would allow users to navigate away for from a page that has a large amount of uploads that are taking some time, so that they can continue work somewhere else in the BCMS, and then see when those uploads are complete so that they can come back to the book in question.
This should be developed in tandem with #1496
![mermaid-diagram-2023-06-06-170402](/uploads/4a9ff4abde03536a6378bc32fb5e623d/mermaid-diagram-2023-06-06-170402.png)
## Design
<!-- Include sketches or wireframes of the UI suggested for this feature -->
### Wireframe: Example from Dashboard page
This global uploads UI will be visible from all BCMS pages so that users can see a progress indicator and expand/open it to view details of the progress on all their uploads.
*Placement of upload icon*
![image](/uploads/4be4a7969e96b80c035cb51959d1415e/image.png)
The icon gives the user information before they select to detail. Icon:
* Uploads complete
* Uploads in Progress
* Uploads with errors
![Complete](/uploads/1c90837b0d1098e02298a5d6aaf15ae3/Complete.jpeg)
![In_progress](/uploads/e9a5baeb1ebc3d9a4c5b39eacf40a725/In_progress.jpeg)
![At_least_one_has_failed](/uploads/c5c621013d1b6120ede0aebdffe6a201/At_least_one_has_failed.jpeg)
*Detail shown when user clicks on Icon*
* The UI will include a list of files names and an indication of upload stage: uploading, complete, and failed.
* In the case of a failed upload, there will be text to help the user resolve the issue, for example: “filename.jpeg failed to upload to [folder name] in `[bcms ID](link to page)`”
![image](/uploads/eacf904371d22e20b021ca30a9e568c1/image.png)
## Acceptance criteria
<!-- Provide the criteria that should be met for this feature. These criteria must be clearly defined customer requirements aligned to NCBI’s user stories in its Statement of Work. These criteria must be testable by either user testing, unit tests or integration tests.-->
## Definition of ready
<!-- A checklist of what needs to be done to a product backlog item before the team can start implementing it in the next sprint. -->
- [ ] BCMS User Story / Context has been well defined
- [ ] The priority of the user story is specified and agreed
- [ ] Digital assets added (design, database scheme, mockups etc if relevant)
- [ ] Coko Technical Proposal approved by NCBI
- [ ] Testable Acceptance Criteria approved by NCBI
- [ ] Estimate of effort to complete (time or points)
- [ ] The issue has been broken down into development tasks (if necessary)
- [ ] Requirements Clarified
- [ ] The product owner and development team agree that the user story is ready for development
- [ ] NCBI adds “Dev_Ready”
## Definition of done
<!-- A checklist of criteria that must be completed for a story to be considered “done.” -->
- [ ] All coding tasks are finished and implemented
- [ ] QA approved
- [ ] Deployed and tested on “ncbidev” (by Coko team)
- [ ] Deployed and tested on “ncbi” (by NCBI team)
- [ ] Acceptance Criteria Met
## Implementation
<!-- A description of the steps to implement the feature.-->
Dev tasks:
- getting progress of the upload in percentage (backend)
- show percentage progress of the upload to visually (frontend)
- getting file status of the upload (backend)
- showing file status of the upload (frontend)
- we can keep the data of all uploads in the backend so that if there is an activity stream functionality in the future, we can expose it. (**note** we didn't include this in the original estimate)
- uploads list is per user session. The drawer is emptied when the the user signs out or refreshes the page
- user can dismiss failed or successful uploads
- **Question:** Should users be able to cancel an upload in progress? **Note** We think this is preferable however we didn't include this in estimate.
## Future extensions (if needed)
<!-- Include any alternatives to meet this use case. -->
2. Implement chunked file uploads: Especially for big files we will consider splitting the file into smaller chunks and upload them individually. Implement logic on the client-side to divide the file into chunks and send them to the server sequentially.
4. Handle interruptions and resume uploads: Implement logic to handle interruptions during file uploads, such as network failures or user-initiated pauses. Store the upload progress on the server and allow users to resume interrupted uploads from where they left off. (This is an extension to the main feature/not included in estimate)
## Scheduling
Dev estimate: 15 days
* [ ] Milestone is linked
* [ ] Iteration is linked
* [ ] Dependencies: ("None" or list issue numbers if relevant)
* [ ] Development estimate is added to issue time trackingP09: Book Manager and other MVP BCMS improvements and Lower Priority Bug Fixeshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1647Invalid TOC XML - repeated chapter tagged twice instead of a PI used2024-01-17T06:31:28ZStacy LathropInvalid TOC XML - repeated chapter tagged twice instead of a PI used<!-- Required. Provide a general summary of the issue in the title above -->
## Expected behaviour
<!-- Required. Tell us what should happen -->
TOC XML should be valid to BITS DTD and PMC style checker compliant.
## Current behaviou...<!-- Required. Provide a general summary of the issue in the title above -->
## Expected behaviour
<!-- Required. Tell us what should happen -->
TOC XML should be valid to BITS DTD and PMC style checker compliant.
## Current behaviour
<!-- Required. Tell us what happens instead of the expected behaviour -->
Many TOC XML are invalid because when a repeated chapter is indicated that chapter is getting tagged twice in the TOC XML rather than the correct PI being used in the repeated part.
## Steps to reproduce
<!-- Required. Provide a link to a live example or screenshots, and the steps to reproduce this bug.]-->
1. Validate attached TOC.xml -[TOC_1647.xml](/uploads/5a46b9da648e10c5197ba149224fa558/TOC_1647.xml)
2. See this file is tagged twice completely when it should only have a repeat PI - `<?xml_file epigeneticmechanismscontrollingmesode.xml?>`
## Environment
<!-- Required. Provide relevant information such as browser name and version, PC or Mac use, internet speed, etc.]-->
This is occurring in domain `stembook`. I cannot reproduce it on Coko site. @deniskar can you understand why?
## Possible solution
<!-- If known, provide details on how to fix the bug.-->
<!-- After creating this issue you can link other related or blocking issues with the Gitlab's Linked issues functionality. -->
## QA Steps
[To be completed by Coko once dev is done]
## Scheduling
<!-- Select all the relevant options -->P04: Support valid and compliant TOC XML for all migrated Bookshelf contentDenis KaramyshevDenis Karamyshevhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1640Invalid TOC XML - id cannot start with a number2024-01-16T21:19:14ZStacy LathropInvalid TOC XML - id cannot start with a number<!-- Required. Provide a general summary of the issue in the title above -->
## Expected behaviour
<!-- Required. Tell us what should happen -->
TOC XML should be valid to BITS DTD and PMC style checker compliant.
## Current behaviou...<!-- Required. Provide a general summary of the issue in the title above -->
## Expected behaviour
<!-- Required. Tell us what should happen -->
TOC XML should be valid to BITS DTD and PMC style checker compliant.
## Current behaviour
<!-- Required. Tell us what happens instead of the expected behaviour -->
Many TOC XML are invalid because IDs are beginning with a number when this is not permitted.
## Steps to reproduce
<!-- Required. Provide a link to a live example or screenshots, and the steps to reproduce this bug.]-->
1. Validate attached TOC.xml -[TOC_1640.xml](/uploads/7b815bde8d1f14a1253a39ee594e3e64/TOC_1640.xml)
2. See error -
```
System ID: C:\Users\lathrops\OneDrive - National Institutes of Health\Desktop\Unconfirmed 2202.crdownload
Main validation file: C:\Users\lathrops\OneDrive - National Institutes of Health\Desktop\Unconfirmed 2202.crdownload
Schema: \\fridge\pmcprod\load\converter4\DTD\dtd-niso-z39.96\extensions\bits\2.0\BITS-book2.dtd
Engine name: Xerces
Severity: error
Problem ID: IDInvalidWithNamespaces
Description: Attribute value "17q12foundationbreakgrimesiowa" of type ID must be an NCName when namespaces are enabled.
Start location: 545:1993
End location: 545:2025
```
## Environment
<!-- Required. Provide relevant information such as browser name and version, PC or Mac use, internet speed, etc.]-->
This is occurring in domain `gene`. I cannot reproduce it on Coko site. @deniskar can you understand why? Affiliations should not get tagged in TOC.XML but they are in content in NCBI deployment but not Coko deployment.
## Possible solution
<!-- If known, provide details on how to fix the bug.-->
<!-- After creating this issue you can link other related or blocking issues with the Gitlab's Linked issues functionality. -->
## QA Steps
[To be completed by Coko once dev is done]
## Scheduling
<!-- Select all the relevant options -->P04: Support valid and compliant TOC XML for all migrated Bookshelf contentDenis KaramyshevDenis Karamyshevhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1639Invalid TOC XML - duplicate aff ids are not valid2024-01-16T21:17:42ZStacy LathropInvalid TOC XML - duplicate aff ids are not valid<!-- Required. Provide a general summary of the issue in the title above -->
## Expected behaviour
<!-- Required. Tell us what should happen -->
TOC XML should be valid to BITS DTD and PMC Style Checker compliant.
## Current behaviou...<!-- Required. Provide a general summary of the issue in the title above -->
## Expected behaviour
<!-- Required. Tell us what should happen -->
TOC XML should be valid to BITS DTD and PMC Style Checker compliant.
## Current behaviour
<!-- Required. Tell us what happens instead of the expected behaviour -->
TOC XML is invalid in numerous cases because duplicate aff ids are tagged.
## Steps to reproduce
<!-- Required. Provide a link to a live example or screenshots, and the steps to reproduce this bug.]-->
1. Validate attached TOC.xml [TOC_1639.xml](/uploads/7924fbba1cc8ad7dd108e4f36fe4d919/TOC_1639.xml)
2. See error -
```
System ID: C:\Users\lathrops\Downloads\Unconfirmed_2202.crdownload
Main validation file: C:\Users\lathrops\Downloads\Unconfirmed_2202.crdownload
Schema: \\fridge\pmcprod\load\converter4\DTD\dtd-niso-z39.96\extensions\bits\2.0\BITS-book2.dtd
Engine name: Xerces
Severity: error
Problem ID: IDNotUnique
Description: Attribute value "seattlevamedicalcenterbreakdepartmentsofneurologyandmedicinebreakuniversityofwashingtonbreakseattlewashington" of type ID must be unique within the document.
Start location: 4684:169
End location: 4684:280
```
## Environment
<!-- Required. Provide relevant information such as browser name and version, PC or Mac use, internet speed, etc.]-->
This is occurring in domain `gene`. I cannot reproduce it on Coko site. @deniskar can you understand why? Affiliations should not get tagged in TOC.XML but they are in content in NCBI deployment but not Coko deployment.
## Possible solution
<!-- If known, provide details on how to fix the bug.-->
<!-- After creating this issue you can link other related or blocking issues with the Gitlab's Linked issues functionality. -->
## QA Steps
[To be completed by Coko once dev is done]
## Scheduling
<!-- Select all the relevant options -->P04: Support valid and compliant TOC XML for all migrated Bookshelf contentDenis KaramyshevDenis Karamyshevhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1475Don't run Coko checks on metadata before saving uploaded converted file2024-01-08T23:38:23ZStacy LathropDon't run Coko checks on metadata before saving uploaded converted file<!-- Required. Provide a general summary of the issue in the title above -->
## Expected behaviour
Per specifications - https://docs.google.com/spreadsheets/d/1M4ZdBbzr2s4-PUXqEblfsKoPBUuk4aG343u9ZAUn_Tk/edit#gid=625909583
User should...<!-- Required. Provide a general summary of the issue in the title above -->
## Expected behaviour
Per specifications - https://docs.google.com/spreadsheets/d/1M4ZdBbzr2s4-PUXqEblfsKoPBUuk4aG343u9ZAUn_Tk/edit#gid=625909583
User should be able to upload valid converted xml files to new books. NCBI's converted BXML files for migration include two DTDs:
* BITS:https://jats.nlm.nih.gov/extensions/bits/
* NLM 2.0: https://dtd.nlm.nih.gov/book/2.0/
## Current behaviour
<!-- Required. Tell us what happens instead of the expected behaviour -->
User is unable to upload valid converted BITS xml file to new book, because the upload is blocked when a Coko check on `pub date` fails even though the file loads successfully from NCBI Silverlight CMS.
## Steps to reproduce
See recording: ![video1575604341](/uploads/79b8d27b44269e5e1bc4a317f5113d74/video1575604341.mp4)
Book recorded: https://ncbi.cloud68.co/organizations/308db55a-21c9-4a64-9a70-d5aa7574726c/bookmanager/31a23f23-aaac-49e8-abf4-f0bc23b006cd
Test files resupplied in https://gitlab.coko.foundation/ncbi/ncbi/-/issues/1475#note_108986 [bcms6772.590b65.2023_02_03-08_48_39.output.zip](/uploads/d23200821bc5a102f5dc2fa74c510a7a/bcms6772.590b65.2023_02_03-08_48_39.output.zip)
## Environment
<!-- Required. Provide relevant information such as browser name and version, PC or Mac use, internet speed, etc.]-->
## Agreed solution
Remove Coko checks on metadata that would prevent ANY converted XML in the BITS or NLM 2.3 DTD from being sent to load to PMC so that all errors are reported to user / system admin to know how to report to successfully fix the errors.
Note, Coko should still validate all XML THEY WRITE against the BITS DTD and PMC style checker to ensure it is valid against the DTD and PMC style compliant, and there should be a process for fixing any errors against those NCBI supported scripts as part of any Coko maintenance.
**Caveat**
* NCBI acknowledges if we remove checks when saving data, this is likely to lead to errors down the line when the BCMS writes book metadata (in the case of TOCs)
* We agree to follow a clear process to resolve these errors, summarized below:
* BCMS runs ALL XML IT WRITES against the PMC Style checker - see #1643
* If XML that the BCMS writes fails the PMC Style checker, a bug is immediately issued and triaged as maintenance
* In all other cases if there are XML issues, NCBI identifies which aspect of metadata is causing an error
* NCBI reproduces the case on ncbi.cloud68.co if it doesn't already exist there.
* NCBI opens a "Metadata feature extension" issue with the relevant information
* Coko team reviews and estimates the issue for development
* Coko/NCBI teams review scope, sign off, and schedule in the dev cycle according to priority.
## QA Steps
[To be completed by Coko once dev is done]
## Scheduling
## Possible alternatives
Provide all possible `pub date` types to save as accepted values so that Coko's existing check can remain.P05: Address MVP Files Management and Processing Issues to support all current Bookshelf submitters and NCBI Integration specificationshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1465Support manual deletion of book / chapter components that have never been pub...2024-01-09T03:31:41ZStacy LathropSupport manual deletion of book / chapter components that have never been published and lock book / chapter components that have been published<!-- Required. Provide a general summary of the issue in the title above -->
## Context
Bookshelf staff can delete content (component) and file in the Silverlight CMS to avoid data integrity issues and should be able to do so in the BC...<!-- Required. Provide a general summary of the issue in the title above -->
## Context
Bookshelf staff can delete content (component) and file in the Silverlight CMS to avoid data integrity issues and should be able to do so in the BCMS too.
Currently Bookshelf staff have no way to correct the easy-to-make errors in creating components in the BCMS, which means they must create a new duplicate record to successfully process it.
Having duplicates with the same title means that the version of record is not clear to Bookshelf staff and other supported users when managing content, both by book or chapter and its package of version of record files. Based on extensive BCMS testing to date, an incorrect version often gets processed instead, causing data integrity issues in the PMC Books database of record, and possible duplicates displayed and indexed in Bookshelf and other NLM products, such as PubMed.
## Proposal
Support ability for Bookshelf staff (System Admin) to manually delete components or files and one or more of their versions IF they are NOT and HAVE NEVER BEEN in a published status, and if they are OR once were in a published status, to lock them from future use, in cases they are a duplicate. This does not need to be in synch with the PMC Books Domain Service, as Bookshelf staff can have that duplicate suppressed in the PMC Books database by manual means when necessary.
## Design
<!-- Include sketches or wireframes of the UI suggested for this feature -->
Design pending. Notes:
- The user should be able to "Lock" a version on any statuses except in-progress status: converting, loading, tagging.
- The choice to "Lock" is not dependent on creating a new version.
- Delete is button in bottom right of the files tab on the individual chapter version or book version
- Deleting chapter versions in bulk is button in bottom right of of Book Manager page
- Deleting book versions in bulk is button in bottom right of of Dashboard page (this should be implemented with #1322)
## Implementation (if applicable)
<!-- A description of the steps to implement the feature.-->
### Delete book or chapter version
Rules:
* Users cannot delete any version that has been published before.
* Users cannot delete any version that is not the current version.
Query last published value of the current version:
- wholebook or complete document: does the book version have a last published value?
- No: allow user to delete
- Yes: Show "Lock" option
- chapter-processed book: do any of the chapters within the book have a last published value?
- No: allow user to delete
- Yes: Show "Lock" option
- chapter: does the chapter version have a last published value?
- No: allow user to delete
- Yes: Show "Lock" option
### Lock book or chapter version
When locked, this means that no actions can be taken except downloading files. Once a version has been locked, allow sys admin to lock/unlock at any time.
Rules:
1. Previous versions: Show a "Lock" option on all previous versions for Sys Admin.
2. Current version: If current version is *not published*, then user can delete OR Lock. If current version *is published*, then user can choose to:
* lock current version only OR
* lock current version and create new version OR
* **Do not** lock current version and create new version
How does this effect the creation of chapter versions from bulk upload?
* check on backend: if filename exists and the current version of chapter is locked, ignore the file
* on frontend: display error in bulk upload modal "The chapter with this file name is locked and cannot be updated."
## Acceptance Criteria
- [ ] If any book or chapter component version has never been published, System Admin has the ability with a Delete button to Delete that component. When the component is deleted it is removed from the BCMS database and does not show on any dashboards or searches / filters.
- [ ] If any book or chapter component version has at some point been published, System Admin has the ability with a Lock button to Lock that component. When the component is locked no user can upload files or process files until / unless they unlock the component. Users can see the component is locked on the component page and all dashboards / book manager pages.
## Alternative approaches (if applicable)
<!-- Include any alternatives to meet this use case. -->
## Scheduling
### Estimates
* Implementing delete functionality at the backend is necessary to all the following cases. 3d
* delete or lock on individual version (FrontEnd work) 1d
* delete chapters in bulk on Book manager page (FrontEnd work) 1d
* delete books in bulk on Dashboard page (FrontEnd work) 1d
<!-- Assignee and labels are added automatically. After creating this issue you can link other related or blocking issues with the Gitlab's Linked issues functionality. -->P06: MVP Data Model and Integrity Issues to Ensure Data Quality in Bookshelf and NLM Databaseshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1439CPB: Data for contributors missing when there are affiliations tags2024-01-16T17:32:41ZSidorela UkuCPB: Data for contributors missing when there are affiliations tags## Expected behaviour
For chapter-proccessed books when the file goes to loading preview or when we upload a converted file, the content on the metadata tab of the book component should be updated, for all the related fields.
## Curren...## Expected behaviour
For chapter-proccessed books when the file goes to loading preview or when we upload a converted file, the content on the metadata tab of the book component should be updated, for all the related fields.
## Current behaviour
For chapter-processed books, when file has contributors with affiliations, the contributors do not appear on the metadata tab of the book component. This happens only for the files where the content for contributors is structured as below:
```
<contrib contrib-type="author">
<name>
<surname>Devanarayan</surname>
<given-names>Viswanath</given-names>
</name>
<aff>Eli Lilly & Company, IN</aff>
</contrib>
```
## Steps to reproduce
Example book [here](https://ncbidev.cloud68.co/organizations/a76c7f2c-870d-4d21-9211-435db4499460/bookmanager/d6828545-22da-4067-815d-516cb83b630f/e8fe0e53-fdf1-457b-9975-6d75a290e242#91348fdb-1b13-4942-a819-2a444e4837ee)
On the example link if you download the converted file, you will see there are a couple of authors. Which if you check on the metadata tab of this book component the contributors section is empty.
1. Create a word chapter processed book
2. Upload the same source file as in the book above ([qbiogloss.docx](/uploads/9aac971d8402b81c392f3a686a0e0617/qbiogloss.docx))
3. Submit the file and wait for it to go to loading preview
4. Check the metadata tab that the contributor info is not there
5. If you try to upload the converted file you will face the bug on issue #1422
## Environment
<!-- Required. Provide relevant information such as browser name and version, PC or Mac use, internet speed, etc.]-->
## Possible solution
<!-- If known, provide details on how to fix the bug.-->
<!-- After creating this issue you can link other related or blocking issues with the Gitlab's Linked issues functionality. -->
## QA Steps
[To be completed by Coko once dev is done]
## Scheduling
<!-- Select all the relevant options -->
- [ ] This issue is blocking current user testing
- [ ] This issue is blocking current migration testing
- [ ] Fixing this issue is required for [Priority 1: Deploy MVP](https://gitlab.coko.foundation/groups/ncbi/-/epics/56)
- [ ] This issue should be triaged into later epics in the PM weekly meetingP03: Support valid and compliant metadata for all migrated Bookshelf contenthttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1422Chapter processed books: Error on upload of converted file because of affilia...2024-01-16T17:32:34ZSidorela UkuChapter processed books: Error on upload of converted file because of affiliation tags## Expected behavior
If the conversion of a file is successful, we should be able to upload the same converted file without issues again, from 'File upload' button of converted section.
## Current behavior
When I try to upload some co...## Expected behavior
If the conversion of a file is successful, we should be able to upload the same converted file without issues again, from 'File upload' button of converted section.
## Current behavior
When I try to upload some converted files which have contributors and aff in their meta, I get an error `The converted file is not correct1` on UI. While is exactly the same file that came from a successful conversion [here](https://ncbidev.cloud68.co/organizations/73814fa2-d9fc-44f8-a12a-d3e52c589230/bookmanager/7ee8f0ec-0702-4282-8941-902f782cf01b/44c0f58b-39bb-47f4-97d0-b96958925b21#b48bab98-9995-46ea-a340-23249ff11cdc)
Also if you check the first converted xml file there are data about contributors which are missing on the ui at the metadata tab of the book component. (this is another bug, but the issues seem related)
## Steps to reproduce
[Provide a link to a live example or screenshots, and the steps to reproduce this bug]
1. Create a chapter processed book
2. Upload the source file: [ch-a.docx](/uploads/36b2227704f7c39bdeb88914d00deb6e/ch-a.docx)
3. Go to files tab and upload the converted file: [ch-a.xml](/uploads/d4f540e46e7593d627b871c420a32b41/ch-a.xml)
4. Click Save and the UI error will appear.
## Environment
[Provide browser name and version and if you're working from a PC or Mac]
## Possible solution
[Not required. Suggest a fix for the bug]
## NCBI's priority feedback
[Select "Y" or "N" and provide an explanation]
1. This bug is blocking NCBI's work on migration planning for deployment 1 (Y)
2. This can be prioritised after deployment 1 (N)
## QA Steps
[To be completed by Coko once dev is done]P03: Support valid and compliant metadata for all migrated Bookshelf contenthttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1149Previously published chapter that has a replacement chapter file version in p...2024-01-16T17:36:23ZDiana jordanPreviously published chapter that has a replacement chapter file version in previewing status does not show original published version of chapter on toc@ChristinaTromp @lathrops1
## Expected behaviour
toc should be accurate:
- a chapter should be included in the TOC.xml build when it is published and should remain on the TOC.xml even if the current *file version* is not published
...@ChristinaTromp @lathrops1
## Expected behaviour
toc should be accurate:
- a chapter should be included in the TOC.xml build when it is published and should remain on the TOC.xml even if the current *file version* is not published
## Current behaviour
a previously published chapter is no longer showing in the TOC if an update is previewing
## Steps to reproduce
https://ncbi.cloud68.co/organizations/32d78686-4b9a-46af-b52a-f5d0c4052177/bookmanager/18e972c9-df07-4dee-975d-00230567ea8f
1. publish a chapter (in example see chapter LM584.docx)
2. publish a TOC and see chapter shows in TOC
3. update the chapter so that it is now in previewing status
4. publish toc again (user was working on and published other chapters hence reloaded toc)
5. see that original chapter that is now previewing is no longer in TOC
## NCBI's priority feedback
y, for deployment - need to be able to process and have accurate data
## QA Steps
[To be completed by Coko once dev is done]P04: Support valid and compliant TOC XML for all migrated Bookshelf contenthttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1657Unable to upload Converted xml into a chapter for glycopodv22024-01-31T00:22:58ZKin Man NgUnable to upload Converted xml into a chapter for glycopodv2We are getting an error when trying to upload an Converted XML into the glycopodv2 book.
BCMS link: https://ncbi.cloud68.co/organizations/810011d3-1093-4550-a882-c7cbe0ae33c6/bookmanager/22544eb7-0452-4b48-9675-e9975fe23705/f2ce654d-066...We are getting an error when trying to upload an Converted XML into the glycopodv2 book.
BCMS link: https://ncbi.cloud68.co/organizations/810011d3-1093-4550-a882-c7cbe0ae33c6/bookmanager/22544eb7-0452-4b48-9675-e9975fe23705/f2ce654d-0667-47cf-8041-cb4f45461931#undefined
After a user selects the file to upload and when they click on 'Save', a red pop window appears on the upper right hand corner with the error:
"Error uploading file;
The converted file you uploaded is not correct or parsing failed with error: Running command: Chapter Update Command Failed'
![glycopodv2_coko_upload_converted_error.png](/uploads/93ba7d6aed2385475d0f95dd809cce4e/glycopodv2_coko_upload_converted_error.png)
The converted file would need to be removed.Kin Man NgKin Man Nghttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1656pub-date not displaying when pub-history isn't present2024-01-26T23:45:29ZKin Man Ngpub-date not displaying when pub-history isn't presentIn the production TOC for ahrqlsr, pub-history date is displayed under the contrib list. For chapters that does not have pub-history tagged, the pub-date is displayed instead.
https://www.ncbi.nlm.nih.gov/books/NBK568855/
![ahrqlsr_pro...In the production TOC for ahrqlsr, pub-history date is displayed under the contrib list. For chapters that does not have pub-history tagged, the pub-date is displayed instead.
https://www.ncbi.nlm.nih.gov/books/NBK568855/
![ahrqlsr_prod_TOC_dates](/uploads/7ecafcccce3be77945357ab3a35b4fe7/ahrqlsr_prod_TOC_dates.PNG)
'**Published:** ' is the pub-history created date
'**Published online**: ' is the pub-date
I loaded the chapter 'Living Systematic Review ... Pain: Surveillance Report 4 ..' into a test book and there is no date.
https://ncbi.cloud68.co/organizations/4387b034-27b8-4bd8-9b86-4ddd34aacd15/bookmanager/3dd24a0c-4130-467f-a8d9-665af2c181fb/toc/354201df-655e-45d3-94df-29f3f23940e8
I checked the TOC.xml (see attached ahrqlsr_coko_pub-date_TOC.xml) and the pub-date is available as an pi `?epub 20220601?.`
[ahrqlsr_coko_pub-date_TOC.xml](/uploads/de6b377b17ccb509a1fb7ea9148d6264/ahrqlsr_coko_pub-date_TOC.xml)
This is tagged as `<pub-date publication-format="electronic" date-type="pub"><month>06</month><year>2022</year></pub-date>` within the book-part-meta of the converted XML.Kin Man NgKin Man Nghttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1655Upgrade to latest version of coko/client2024-02-07T15:46:18ZDione Mentisdione@coko.foundationUpgrade to latest version of coko/client* Upgrade libraries in app first (to be in line with Coko/client)
* Then upgrade to latest version of coko/client* Upgrade libraries in app first (to be in line with Coko/client)
* Then upgrade to latest version of coko/clientP07 - Support Current Bookshelf Chapter-Processed contents with Parts with Bodyhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1654Upgrade to the latest @coko/server2024-02-07T15:42:54ZGiannis Kopanasjkopanas@gmail.comUpgrade to the latest @coko/server<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, ...<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, and goals? -->
Upgrading now is in line with the request in #1628 because we can make use of the fileStorage and AWS s3 Service, making the migration issue to AWS s3 easier.
## Proposal
<!-- A precise statement of the proposed feature. -->
Upgrade to the latest @coko/server. Note: there are breaking changes so we propose refactoring in BCMS step-by-step, alongside feature development, to bring the code closer to coko/server current version, then complete the upgrade.
## Design
<!-- Include sketches or wireframes of the UI suggested for this feature. -->
## Acceptance criteria
<!-- Provide the criteria that should be met for this feature. These criteria must be clearly defined customer requirements aligned to NCBI’s user stories in its Statement of Work. These criteria must be testable by either user testing, unit tests or integration tests.-->
## Definition of ready
<!-- A checklist of what needs to be done to a product backlog item before the team can start implementing it in the next sprint. -->
- [ ] BCMS User Story / Context has been well defined
- [ ] The priority of the user story is specified and agreed
- [ ] Digital assets added (design, database scheme, mockups etc if relevant)
- [ ] Coko Technical Proposal approved by NCBI
- [ ] Testable Acceptance Criteria approved by NCBI
- [ ] Estimate of effort to complete (time or points)
- [ ] The issue has been broken down into development tasks (if necessary)
- [ ] Requirements Clarified
- [ ] The product owner and development team agree that the user story is ready for development
- [ ] NCBI adds “Dev_Ready”
## Definition of done
<!-- A checklist of criteria that must be completed for a story to be considered “done.” -->
- [ ] All coding tasks are finished and implemented
- [ ] QA approved
- [ ] Deployed and tested on “ncbidev” (by Coko team)
- [ ] Deployed and tested on “ncbi” (by NCBI team)
- [ ] Acceptance Criteria Met
## Implementation
<!-- A description of the steps to implement the feature. To be completed by the lead dev. If there are multiple tasks, then break these down into "task" below.-->
## Alternative approaches (if applicable)
<!-- Include any alternatives to meet this use case. -->
## Scheduling
* [ ] Milestone is linked
* [ ] Iteration is linked
* [ ] Dependencies: ("None" or list issue numbers if relevant)
* [ ] Development estimate is added to issue time trackingP05: Address MVP Files Management and Processing Issues to support all current Bookshelf submitters and NCBI Integration specificationshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1653Add Loaders to Resolvers2024-02-07T15:45:51ZGiannis Kopanasjkopanas@gmail.comAdd Loaders to Resolvers- Check the loaders we already have
- Add new loaders where is necessary- Check the loaders we already have
- Add new loaders where is necessaryP05: Address MVP Files Management and Processing Issues to support all current Bookshelf submitters and NCBI Integration specificationshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1652Migrate all the parsing from Cheerio to SaxParser2024-01-29T09:05:26ZGiannis Kopanasjkopanas@gmail.comMigrate all the parsing from Cheerio to SaxParserAt the moment we have two different ways for parsing the Xml .
Cheerio for the smaller files and SaxParser for xml Files bigger than 10mb.
We need to move to SaxParser.
This can be done in parallel with issues about xml File Parsing .At the moment we have two different ways for parsing the Xml .
Cheerio for the smaller files and SaxParser for xml Files bigger than 10mb.
We need to move to SaxParser.
This can be done in parallel with issues about xml File Parsing .P03: Support valid and compliant metadata for all migrated Bookshelf contenthttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1651Improve SubmitComponent Controller2024-02-07T15:43:57ZGiannis Kopanasjkopanas@gmail.comImprove SubmitComponent ControllerRefactor / Clean the submit controller.
- Check the possibility to split the controller into two seperate controllers for book and chapters
- Improve the cases for pdf to separate functionality Since we need all the files . (Move that t...Refactor / Clean the submit controller.
- Check the possibility to split the controller into two seperate controllers for book and chapters
- Improve the cases for pdf to separate functionality Since we need all the files . (Move that to the model)
- Move the Creation of Xml files For pdf case. to the XmlFactory Service.P03: Support valid and compliant metadata for all migrated Bookshelf contenthttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1650Hide all locked versions on all dashboard, collection, book manager, search /...2024-01-16T18:06:04ZStacy LathropHide all locked versions on all dashboard, collection, book manager, search / filter pages from all users except Sys Amins<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, ...<!-- Required. Provide a general summary of the issue in the title above -->
## Context
<!-- Give the necessary context for your proposal. For example, what problem will this feature solve for users? What are the use cases, benefits, and goals? -->
It will likely be confusing after #1465 and #1648 are implemented for non-system admin to see or be able to retrieve from search / filter locked component versions.
## Proposal
<!-- A precise statement of the proposed feature. -->
Hide from all dashboard, collection, book manager, search / filter pages locked book and chapter versions.
## Design
<!-- Include sketches or wireframes of the UI suggested for this feature. -->
## Acceptance criteria
<!-- Provide the criteria that should be met for this feature. These criteria must be clearly defined customer requirements aligned to NCBI’s user stories in its Statement of Work. These criteria must be testable by either user testing, unit tests or integration tests.-->
## Definition of ready
<!-- A checklist of what needs to be done to a product backlog item before the team can start implementing it in the next sprint. -->
- [ ] BCMS User Story / Context has been well defined
- [ ] The priority of the user story is specified and agreed
- [ ] Digital assets added (design, database scheme, mockups etc if relevant)
- [ ] Coko Technical Proposal approved by NCBI
- [ ] Testable Acceptance Criteria approved by NCBI
- [ ] Estimate of effort to complete (time or points)
- [ ] The issue has been broken down into development tasks (if necessary)
- [ ] Requirements Clarified
- [ ] The product owner and development team agree that the user story is ready for development
- [ ] NCBI adds “Dev_Ready”
## Definition of done
<!-- A checklist of criteria that must be completed for a story to be considered “done.” -->
- [ ] All coding tasks are finished and implemented
- [ ] QA approved
- [ ] Deployed and tested on “ncbidev” (by Coko team)
- [ ] Deployed and tested on “ncbi” (by NCBI team)
- [ ] Acceptance Criteria Met
## Implementation
<!-- A description of the steps to implement the feature. To be completed by the lead dev. If there are multiple tasks, then break these down into "task" below.-->
## Alternative approaches (if applicable)
<!-- Include any alternatives to meet this use case. -->
## Scheduling
* [ ] Milestone is linked
* [ ] Iteration is linked
* [ ] Dependencies: ("None" or list issue numbers if relevant)
* [ ] Development estimate is added to issue time tracking
---P09: Book Manager and other MVP BCMS improvements and Lower Priority Bug Fixeshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1649Update FTP File Specifications to add `workflow` as a required value to matc...2024-01-09T03:25:29ZStacy LathropUpdate FTP File Specifications to add `workflow` as a required value to match book versionsAfter #1648 the BCMS will need to accommodate a scenario that results in two book versions with the same `book-submit-id` + `version-name` + `version-number`. Therefore the current FTP spec should be updated to add `workflow` as a requir...After #1648 the BCMS will need to accommodate a scenario that results in two book versions with the same `book-submit-id` + `version-name` + `version-number`. Therefore the current FTP spec should be updated to add `workflow` as a required value to match book versions.