ncbi issueshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues2024-01-30T12:16:33Zhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1628Switch from FTP to AWS S3 for data exchange with NCBI2024-01-30T12:16:33ZEvgenySwitch from FTP to AWS S3 for data exchange with NCBINCBI Systems is planning to prohibit ftp access from the Kubernetes cluster where BCMS is deployed, because they want to control all outgoing network connections via Istio service mesh, which does not support ftp protocol. Instead of FTP...NCBI Systems is planning to prohibit ftp access from the Kubernetes cluster where BCMS is deployed, because they want to control all outgoing network connections via Istio service mesh, which does not support ftp protocol. Instead of FTP, AWS S3 (which is accessed via HTTPS REST API) will be used for data exchange between NCBI and Coko BCMS.
NCBI will create two buckets in AWS S3 for each environment (NCBI-AWS-DEV and NCBI-AWS-PROD), one for APEX and another one for everything else (conversion, submission, package ingest etc). NCBI will specify buckets name/URL/ARN and provide AWS IAM access keys with permissions to upload/download objects in these buckets. Directory structure within S3 buckets and file naming convention will be mapped to object prefixes / names in S3 and remain the same. Kafka messages / data exchange protocol should also remain the same, except that the `package` attribute in Kafka message JSON would be used to construct an object key within a bucket, for example `"S3://ncbi-books-bcms-dev/AHRQ/hcup_sb261.zip"`
CC @latternm @lathrops1P05: Address MVP Files Management and Processing Issues to support all current Bookshelf submitters and NCBI Integration specificationshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/1096Allow new user registration when first and last name are not unique2024-01-16T18:11:02ZChristinaTrompAllow new user registration when first and last name are not uniqueHey @andynicholson cc @DioneMentis
There is one use case related to #1089 that we still need to cater for which I confirmed with Dione.
When a user signs up with unique details, except for their first and last name, they should still b...Hey @andynicholson cc @DioneMentis
There is one use case related to #1089 that we still need to cater for which I confirmed with Dione.
When a user signs up with unique details, except for their first and last name, they should still be able to sign up. For example, if two different people named John Smith exist, they should both be able to use the BCMS without making their *name* unique (but they will need to use a unique email and username).
At the moment, if all your details are unique except your first and last name, you get the following error:
1. First I registered a new unique account
![Screen_Shot_2022-03-15_at_08.58.44](https://gitlab.coko.foundation/ncbi/ncbi/uploads/fe7203ae43c9d98151e0772d462716ed/Screen_Shot_2022-03-15_at_08.58.44.png)
1. Then I tried to register another account with the same first and last names, with all other details (email, username) unique, and got this error:
![Screen_Shot_2022-03-15_at_09.06.48](https://gitlab.coko.foundation/ncbi/ncbi/uploads/65bdd7e0749b0ec0ce3c2d39162a2ad7/Screen_Shot_2022-03-15_at_09.06.48.png)
When all details are unique, except first and last name, we should allow them to register.P10: BCMS Production Deployment IssuesStacy LathropStacy Lathrophttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/804Inconsistent behavior in permitting ingest of tagged XML from PDF2024-01-16T17:29:20ZStacy LathropInconsistent behavior in permitting ingest of tagged XML from PDFcc @douglassue
## Expected behaviour
System creates error if chapter-processed files do not meet file naming specifications regardless if posted to FTP by taggers or manually uploaded.
## Current behaviour
System ingests chapter-pro...cc @douglassue
## Expected behaviour
System creates error if chapter-processed files do not meet file naming specifications regardless if posted to FTP by taggers or manually uploaded.
## Current behaviour
System ingests chapter-processed tagged files from PDF that are not named the same as the source file. This file is processed all the way to the publish state.
System creates error if user tries to manually upload a edited tagged file that is not named the same as the source file.
## Steps to reproduce
See: https://ncbi.cloud68.co/organizations/4dd138c1-1baa-437b-8a0e-d25b91319687/bookmanager/821733d2-6057-425e-bc8f-955457f6c36c/59f88edc-5f7c-4aa8-b39b-6258570f3608
![image](/uploads/ceeb5f3075c20309f83946e9c43150c9/image.png)
1. Download files and package as tagged files from PDF2XML taggers
2. Once published, download converted file and try to reupload
## Solution
Since this problem occurs when the Vendor submit a converted file that should not be accepted, this should be considered a "Tagging error". This solution described here is [documented as user story 5a in the BCMS scoping outcomes list](https://docs.google.com/spreadsheets/d/1ksBkzJjzTsmW3vtaTgrSL7nqzZEvOP0dSwaAscAX9d8/edit?usp=sharing)
1. Apply a new Status "Tagging error" to the PDF workflow (with the user actions defined in the [Status/Action sheet](https://docs.google.com/spreadsheets/d/1M4ZdBbzr2s4-PUXqEblfsKoPBUuk4aG343u9ZAUn_Tk/edit?usp=sharing))
2. When Vendor submits Chapter package with a converted file that does not have the same filename as the existing source file, put the chapter into a "Tagging error" state.
3. Report the following error on the errors tab
| Name | Category | Severity | Assignee | Error Message |
|-------------|----------------------|------|---|-------|
| file naming | Tagging error | error | Vendor | The converted file name must match the source file name |
(This is included on the [error tracking sheet](https://docs.google.com/spreadsheets/d/1bcOKaEihkrX4LR6cAWcKjQelcIolLiHH1e4AEbDR_4g/edit?usp=sharing))
4. Email the Sys admin assigned to the Org and the PDF2XML Vendor (this will only be done by #1296)
5. The Vendor submits a corrected package, and the status of the chapter updates accordingly, or a System admin can download the converted file, rename and upload it, to resolve the Tagging errors status.
## Priority
Y, if there is no workaround to prevent file version and content integrity issues on Bookshelf
## Remaining tasks
Note NCBI task remaining to complete in https://gitlab.coko.foundation/ncbi/ncbi/-/issues/804#note_97990
### QA Steps
- For PDF chapter processed books, we validate the filename that is uploaded on the converted section via UI. So if it doesn't match with the source it will not be uploaded.
- Tagging-errors status can be caused by the submission via ftp. Steps to reproduce:
1. Create a PDF chapter-processed book
2. Upload any source file
3. Download the vendor-meta.xml from the book component file tab
4. Connect with APEX ftp
5. Create a package similar to this [compress18aug.zip](/uploads/c02e67669acd61317e2480b7eb2d7a69/compress18aug.zip). Where the filename on the converted folder doesn't match with the source. Replace the vendor-meta.xml with the one you downloaded on step 3.
6. Submit the source file for tagging
7. The package which was created on step 5, submit it to ftp folder /done/testbcms
8. When the package is sent to ncbi, the converted file will appear on the UI and the status will be tagging-errors.P05: Address MVP Files Management and Processing Issues to support all current Bookshelf submitters and NCBI Integration specificationsStacy LathropStacy Lathrop