ncbi issueshttps://gitlab.coko.foundation/ncbi/ncbi/-/issues2021-11-29T06:47:01Zhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/679abstract parsing blocks conversion error status2021-11-29T06:47:01ZStacy Lathropabstract parsing blocks conversion error status@DioneMentis @John.kopanas @Kireev @deniskar @jordandc
Per Mattermost -
It looks like we are having Kafka issues again. See: https://ncbi.cloud68.co/organizations/32d78686-4b9a-46af-b52a-f5d0c4052177/bookmanager/7eaa27d6-984c-4a48-89f...@DioneMentis @John.kopanas @Kireev @deniskar @jordandc
Per Mattermost -
It looks like we are having Kafka issues again. See: https://ncbi.cloud68.co/organizations/32d78686-4b9a-46af-b52a-f5d0c4052177/bookmanager/7eaa27d6-984c-4a48-89fd-49d8e12bb069/ec8e96e5-d8a1-4f46-9645-b8542bd72890
But there were errors that should have gotten passed and the status updated in the BCMS - from our TM session:
http://ipmc-prod.be-md.ncbi.nlm.nih.gov:5701/internal/utils/tm/index.fcgi?s=monitor&sel=1&sessid=11085850
```
ERROR :[2021-09-08 09:32:52]: extyles-metatables.eXtylesOutput-invalid.XML: Invalid XML after conversion step.
ERROR :[2021-09-08 09:32:52]: -:3: element xref: validity error :
ERROR :[2021-09-08 09:32:52]: IDREFS attribute rid references an unknown ID "extyles-metatables.Td.aff1"
INFO :[2021-09-08 09:32:52]: STS ID: 6138bb84080301a5401618fb TYPE: notice
INFO :[2021-09-08 09:32:52]: extyles-metatables.eXtylesOutput.xml: FAILED [processed in 5.23 sec.]
INFO :[2021-09-08 09:32:52]: STS ID: 6138bb84080301a5401618fc TYPE: xml_map
INFO :[2021-09-08 09:32:52]: SUMMARY: Number of files successfully processed = 0; with errors = 1; with warnings = 0
```
My team testing says pretty much everything is getting stuck as of earlier today, though worked yesterday OK.Nov 02.Andy NicholsonAndy Nicholsonhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/812bad gateway error when downloading individual files in chapter-processed books2021-11-17T14:37:44ZDione Mentisdione@coko.foundationbad gateway error when downloading individual files in chapter-processed books
## Expected behaviour
Clicking on any individual file should download the file
## Current behaviour
I get a bad gateway when clicking on files within chapter-processed books
## Steps to reproduce
On ncbi dev:
1. click on source f...
## Expected behaviour
Clicking on any individual file should download the file
## Current behaviour
I get a bad gateway when clicking on files within chapter-processed books
## Steps to reproduce
On ncbi dev:
1. click on source file in chapter component, the file doesn't download
2. result: https://ncbidev.cloud68.co/downloadFileVersion?id=a98cac31-960a-45d5-8671-aad3d0dd3f0d&category=source
![Screenshot_2021-11-17_at_09.25.34](/uploads/f4ab661ac2fb6ee5b2bf122299c15cbc/Screenshot_2021-11-17_at_09.25.34.png)
## Environment
Mac; Chrome Version 95.0.4638.69Nov 02.Giannis Kopanasjkopanas@gmail.comGiannis Kopanasjkopanas@gmail.comhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/685Check abstract parsing in case of conversion Error2021-12-06T13:03:04ZGiannis Kopanasjkopanas@gmail.comCheck abstract parsing in case of conversion Error@andynicholson I was checking the issue https://gitlab.coko.foundation/ncbi/ncbi/-/issues/679. I realized the following things:
1) At line https://gitlab.coko.foundation/ncbi/ncbi/-/blob/develop/server/services/handleNcbiMessage/bookCom...@andynicholson I was checking the issue https://gitlab.coko.foundation/ncbi/ncbi/-/issues/679. I realized the following things:
1) At line https://gitlab.coko.foundation/ncbi/ncbi/-/blob/develop/server/services/handleNcbiMessage/bookComponentConversion.js#L72 we don't do the same checks as we do higher in the code for the abstract field.
2) Checking the bookComponent of issue #679 we can see that two upload versions happened. At the first one, we had a successful conversion but for the second one, a conversion error occurred. The second time failed because the abstract was an array (this should be a string) and it was coming from the DB. (which means that the first version saved the abstract inside the metadata column as an array which it shouldn't )
We need to simulate a chapter with the same files and check that abstract is saved correctly in case of a conversion error.
@andynicholson this is a tricky case. Please let's discuss it if the issue is not clear.Nov 02.Sidorela UkuSidorela Ukuhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/355Collection metadata UI and backend2022-05-06T12:27:56ZDione Mentisdione@coko.foundationCollection metadata UI and backend*Updated 10 Nov by @lathrops1 to address Coko questions*
*Updated 8 Nov for NCBI review*
## Workflow for collection metadata
See flow diagram in #678 and the related [Collection-meta sheet](https://docs.google.com/spreadsheets/d/1nldl...*Updated 10 Nov by @lathrops1 to address Coko questions*
*Updated 8 Nov for NCBI review*
## Workflow for collection metadata
See flow diagram in #678 and the related [Collection-meta sheet](https://docs.google.com/spreadsheets/d/1nldlGVs9uCq9CGB9efKzTJhZ_iKTO2unfjSXjLRWZZo/edit?usp=sharing).
NCBI has provided these samples:
1. **Column B in sheet**: Sample COLLECTION.XML [bookhelpcollect.xml](/uploads/bded97b23a29f05ba79419389ca83ae6/bookhelpcollect.xml) [bookiarcmonocollect.xml](/uploads/7234c8c6132a2bc82269debf2029ad12/bookiarcmonocollect.xml)
1. Sample `<book>` file with `<collection-meta>` - [bookiarcmono125.xml](/uploads/a425856dd92cae4c8c34cec11070e5c9/bookiarcmono125.xml) -- @lathrops1 please explain this sample in more detail. It's unclear to me why there are two collection-meta nodes. Is this for a wholebook in the PDF workflow?
@DioneMentis - This would be for any wholebook processed converted file.
There are two collection-meta nodes because each collection-type requires its own collection-meta wrapper.
This is collection information provided by a publisher about their Published Book Series (noted by the attribute `book-series`)
```
<collection-meta collection-type="book-series">
<title-group>
<title>IARC Monographs on the Identification of Carcinogenic Hazards to Humans</title>
</title-group>
<volume-in-collection>
<volume-number>125</volume-number>
</volume-in-collection>
</collection-meta>
```
This is collection metadata that comes from the UI because it is NCBI-only collection information (note by the attribute 'ncbi-books-collection')
```
<collection-meta collection-type="ncbi-books-collection">
<collection-id collection-id-type="pmcid">iarcmonocollect</collection-id>
<title-group>
<title>IARC Monographs on the Identification of Carcinogenic Hazards to Humans</title>
</title-group>
</collection-meta>
```
Required from NCBI:
@DioneMentis I took real Bookshelf samples - we did not have any one with all fields.
- [x] complete samples for all fields in Column B (collection.xml)
- [booksbuhtacollect.xml](/uploads/3f95313390b154c6ae51587d51f17507/booksbuhtacollect.xml)
- [bookwhohencollect.xml](/uploads/2b5816aa89d0842b308972c457e56400/bookwhohencollect.xml)
- [x] complete samples for all fields in Column E (writing node into chapter.cml files)
- [x] complete samples for all fields in Column C (see examples and questions per section below)
Samples for `collection-meta` and `book-meta` follow; questions addressed below.
- [bookwt606051.xml](/uploads/b4b2f891c7afdffae33554c179f1876d/bookwt606051.xml)
- [bookpartwrapperch5.xml](/uploads/a68455c819c78881a89c700a012e87d1/bookpartwrapperch5.xml)
- [bookntpcar13.xml](/uploads/6f3b28c13de9b572642099bd89a84f7c/bookntpcar13.xml)
## Metadata UI form and related XML output
PDF version of all updated UI tabs: [collection-meta-UIs-20211110.pdf](/uploads/7efc9a2baf3aa15eda19bdc25941b10b/collection-meta-UIs-20211110.pdf)
### Tab 1: Title and publisher information
![Collect-meta-tab01-V4](/uploads/79c22b4de1db6f4b972bbd2e2b064a21/Collect-meta-tab01-V4.png)
1. Title and IDs
```xml
<collection-id>domain name</collection-id>
<collection-name>NCBI collection title</collection-name>
```
ISSN
'type' is not required at this stage
@DioneMentis - if you are not pulling ISSN from converted XML here, we can keep to just one value and add types later.
```xml
<issn>0959-8138</issn>
```
2. NCBI's custom meta 'Source type'
```xml
<custom-meta-wrap>
<custom-meta>
<meta-name>books-source-type</meta-name>
<meta-value>Collection</meta-value>
</custom-meta>
</custom-meta-wrap>
```
* Dropdown values for Wholebook and Chapter-processed:
* report
* book
* database
* documentation
3. Publisher
```xml
<publisher>
<publisher-name>National Toxicology Program</publisher-name>
<publisher-loc>Research Triangle Park (NC)</publisher-loc>
</publisher>
```
4. Publication date
Only print and electronic range is needed, where the end date is optional.
```xml
<pub-date pub-type="epubr">
<year>2012</year>
</pub-date>
```
5. Abstract
Allow for multiple paragraphs.
```xml
<abstract>
<p>NTP Monographs contain literature-based evaluations of the evidence that environmental substances are associated with noncancer health effects or communicate the state of the science.</p>
</abstract>
```
### Tab 2: Contributors
![Screenshot_2021-10-07_at_20.18.32](/uploads/2e1594e6fc0b63a961f0e1ec43f2f5fe/Screenshot_2021-10-07_at_20.18.32.png)
No changes to the UI
1. Series editors and collaborative authors
```xml
<contrib-group>
<contrib contrib-type="editor">
<name>
<surname>Foster</surname>
<given-names>Bill</given-names>
</name>
<role>Series editor</role>
</contrib>
<contrib contrib-type="author">
<collab>National Academy of Sciences (US)</collab>
</contrib>
</contrib-group>
```
### Tab 3: Permissions
![Collect-meta-tab03-V3](/uploads/f50a7152b2bdc62c1ad58eae819b842a/Collect-meta-tab03-V3.png)
Within the `<permissions>` element, in order:
1. Copyright statement:
Copyright year should be added in the statement. It is not required as a separate `<copyright-year>`
```xml
<copyright-statement>1997 Cold Spring Harbor Laboratory Press</copyright-statement>
```
2. Licence information:
Below is what Denis is using to match CC license types in the UI to values in URLs. I don't know if this is sufficent for you to use. In his code he is missing the CC Public Domain value, which is the example you have in your markup and should be a drop down value. You can get all the URLs from this page by license type: https://creativecommons.org/about/cclicenses/
It matches the text in the url after the \licenses\ to the dropdown value
const ccMatch = ccLicense && ccLicense.match(/licenses\/(.*?)\//)
const ccLicenseType =
ccMatch && ccMatch.length === 2
? (matched => {
switch (matched) {
case 'by':
return 'CC BY'
case 'by-nc':
return 'CC BY-NC'
case 'by-nc-nd':
return 'CC BY-NC-ND'
case 'by-nc-sa':
return 'CC BY-NC-SA'
case 'by-nd':
return 'CC BY-ND'
case 'by-sa':
return 'CC BY-SA'
default:
return null
}
})(ccMatch[1])
: null
* `license-type` is "open access" when "Yes" is selected. when "No", then `license-type` or `xlink:href` is not included.
* Depending on type chosen, the relevant `xlink:href` is populated (information provided above)
```xml
<license license-type="open-access" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://creativecommons.org/publicdomain/zero/1.0/">
<license-p>This is a work of the US government and distributed under the terms of the Public Domain</license-p>
</license>
```
### Tab 4: Notes
Note: currently there is a bug when selecting the note, type which makes the fields disappear.
![Collect-meta-tab04-a-V3](/uploads/d93aaec96412144b025e39ab0674720e/Collect-meta-tab04-a-V3.png)
* Type: disclaimer, author notes, generic, funder courtesy note
* Title is optional
* allow multiple paragraphs
* When funder courtesy note is selected, user has option to all to all PDF books in the collection
![Collect-meta-tab04-b-V3](/uploads/f541ad3db642da307d01e16a06b7abda/Collect-meta-tab04-b-V3.png)
**Disclamer**
```xml
<notes notes-type="disclaimer">
<title>title</title>
<p>para</p>
</notes>
```
**Author notes**
`<fn>` is required by the BITS DTD as a child of `<author-notes>` to produce valid XML - see https://jats.nlm.nih.gov/extensions/bits/tag-library/2.0/element/author-notes.html. It does not need a label or necessary link / be related to anything, but is required as part of the markup.
```xml
<author-notes>
<fn><p>para</p></fn>
</author-notes>
```
**Generic**
```xml
<notes>
<p>para</p>
</notes>
```
**Funder courtesy note**
```
<notes notes-type=“courtesy-note”>
<p>Monographs, or book chapters, which are outputs of Wellcome Trust funding have been made freely available as part of the <ext-link xmlns:xlink=“http://www.w3.org/1999/xlink” ext-link-type=“uri” xlink:href=“https://wellcome.ac.uk/funding/guidance/open-access-policy”>Wellcome Trust’s open access policy</ext-link></p></notes>
```
### Tab 5: Funding
See full Granthub spec in #433
![Collection-meta-tab_05-Funding_C](/uploads/569aaa9b37f01808f5147c2761745af6/Collection-meta-tab_05-Funding_C.png)
### Acceptance criteria
See relevant sections in https://gitlab.coko.foundation/ncbi/ncbi/-/issues/776Nov 02.https://gitlab.coko.foundation/ncbi/ncbi/-/issues/738Contents are not getting autosorted by label and in the correct structural fr...2022-04-07T14:25:30ZStacy LathropContents are not getting autosorted by label and in the correct structural front / body sectionscc @jordandc
## Expected behaviour
Contents are auto-sorted after loaded to PMC by book metadata
## Current behaviour
Contents are not getting auto-sorted by label and the correct structural front / body sections on book manager pag...cc @jordandc
## Expected behaviour
Contents are auto-sorted after loaded to PMC by book metadata
## Current behaviour
Contents are not getting auto-sorted by label and the correct structural front / body sections on book manager page OR in toc.xml
See toc.xml - [toc__9_.xml](/uploads/899e51f572d3c269be28a028c5578b74/toc__9_.xml) - note, **for example**:
```
-<toc-entry>
<title>Preface</title>
-<nav-pointer>
-<related-object document-type="preface" document-id="preface">
<?xml_file preface.xml?>
<?version 1?>
</related-object>
</nav-pointer>
</toc-entry>
```
document-type="preface" so should not be in book-body but front-matter content type
AND
contents are not ordered by chapter label and in this case where chapter labels are in the book part metadata they have not been correctly tagged in the xml
## Steps to reproduce
See:
![image](/uploads/1454347298b11134f3c22ab90afd41fd/image.png)
https://ncbi.cloud68.co/organizations/9f638bd9-4226-4d99-a9fb-579dba128443/bookmanager/84b75936-7136-4312-8174-8accde58ab33
1. Download contents of book
2. Upload contents of book
3. Submit contents
4. Make sure seetings match link above
## Environment
N/A
## Priority
Yes - this is a blocker for testingNov 02.Dione Mentisdione@coko.foundationDione Mentisdione@coko.foundationhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/827domain error when adding book to collection (or updating existing book in col...2021-12-14T22:20:09ZDione Mentisdione@coko.foundationdomain error when adding book to collection (or updating existing book in collection## Expected behaviour
I should be able to add a book to collection when I created it, and update a book's settings.
## Current behaviour
- In new book step: I cannot create a book if I add it to a collection. I get error "Could not c...## Expected behaviour
I should be able to add a book to collection when I created it, and update a book's settings.
## Current behaviour
- In new book step: I cannot create a book if I add it to a collection. I get error "Could not create book: Update domain request failed"
- on NCBI dev, for all existing books that belong to a collection I cannot update settings. I get error "Could not create book: Update domain request failed"
## Steps to reproduce
1. Go to "new book", create any book and select an existing collection -- see error
2. Try updating setting in this [existing book](https://ncbidev.cloud68.co/organizations/a2c1e381-f348-41a9-9d76-e4f11f761033/bookmanager/709b89aa-5fa3-4736-be37-436f0cfb1d01/) -- see error
## Priority
Y -- required for release tomorrowNov 02.Yannis BarlasDenis KaramyshevYannis Barlashttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/822Don't allow upload to "support" folder when status is 'loading errors' or 'pr...2021-12-08T23:58:46ZDione Mentisdione@coko.foundationDon't allow upload to "support" folder when status is 'loading errors' or 'preview' (PDF wf)@danjela
@lathrops1 and I confirmed the changes shown in **red** boxes below. At the moment, users can upload files to the support folder when status is:
- loading error
- preview
This is incorrect because we don't allow "submit" f...@danjela
@lathrops1 and I confirmed the changes shown in **red** boxes below. At the moment, users can upload files to the support folder when status is:
- loading error
- preview
This is incorrect because we don't allow "submit" for these statuses. I have corrected the [BCMS Status/Actions sheet](https://docs.google.com/spreadsheets/d/1M4ZdBbzr2s4-PUXqEblfsKoPBUuk4aG343u9ZAUn_Tk/edit?usp=sharing)
![NCBI-PDF-workflow-actions-fixes-20211117](/uploads/569b0cd3b130caa9e1d55e1d6027942a/NCBI-PDF-workflow-actions-fixes-20211117.jpeg)Nov 02.Danjela Shehidanjelashehi@gmail.comDanjela Shehidanjelashehi@gmail.comhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/824Errors table for TOC sometimes doesn't show unless refresh browser2021-12-02T18:48:25ZDione Mentisdione@coko.foundationErrors table for TOC sometimes doesn't show unless refresh browser
## Expected behaviour
The UI should update to match the status
## Current behaviour
When Status = "publishing failed" I see text "There are no errors". If I refresh, the errors are shown
## Steps to reproduce
Happened in [this b...
## Expected behaviour
The UI should update to match the status
## Current behaviour
When Status = "publishing failed" I see text "There are no errors". If I refresh, the errors are shown
## Steps to reproduce
Happened in [this book](https://ncbidev.cloud68.co/organizations/a2c1e381-f348-41a9-9d76-e4f11f761033/bookmanager/709b89aa-5fa3-4736-be37-436f0cfb1d01/toc/1b347347-fc81-4edd-b8b4-f00c7460f2de)
1. On the TOC page, select "update published version" to send the toc to Load to PMC (this needs to be a toc with errors, for example you can add stying to a part since this #737 is not implemented yet)
2. Wait for status to update to "publishing failed" -- see that error table doesn't show.
![Screenshot_2021-11-29_at_10.17.32](/uploads/b6e2a29a58d4c26a14b35e05016483e8/Screenshot_2021-11-29_at_10.17.32.png)Nov 02.Sidorela UkuDanjela Shehidanjelashehi@gmail.comSidorela Ukuhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/737How styled metadata in the UI translates to xml2022-08-10T06:25:08ZDione Mentisdione@coko.foundationHow styled metadata in the UI translates to xml## Styling
Users can add style, to the fields defined below, in collection metadata and chapter-processed book metadata UIs. The BCMS needs to translate the html tags to xml tags. The BCMS stying possibilities are limited to:
* bold ...## Styling
Users can add style, to the fields defined below, in collection metadata and chapter-processed book metadata UIs. The BCMS needs to translate the html tags to xml tags. The BCMS stying possibilities are limited to:
* bold and italics
* superscript and subscript
* an ordered list (numbered) of one level
* an unordered list (bulleted) of one level
* an external hyperlink
These styles can be applied to following metadata fields
**Styling requirements**
| Metadata filed name in UI | Styling required | Context | relevant xml tag where style can be applied |
| -------------- | ---------------- | ---------------- | ---------------- |
| NCBI collection title | bold, italic, superscript, subscript | collection-meta node | `<collection-name>NCBI collection title</collection-name>`; `<title>NCBI collection title</title>` |
| Book series title | bold, italic, superscript, subscript | collection-meta node | `<title>Book series title</title>` |
| title | bold, italic, superscript, subscript | book-meta node | `<book-title>Book title</book-title>` |
| subtitle | bold, italic, superscript, subscript | book-meta node | `<subtitle>Book subtitle</subtitle> ` |
| alternative title | bold, italic, superscript, subscript | book-meta node | `<alt-title>Book alternative title</alt-title>` |
| abstract | bold, italic, hyperlink, lists | collection-meta node; book-meta node | `<abstract><p>This is an abstract</p></abstract>`|
| notes: disclaimer; generic; author notes; Funder Courtesy Note; Manuscript Final Version Note; Editorial notes (with content types `genetic_counseling` and `resources`) | bold, italic, hyperlinks, lists | collection-meta node; book-meta node | `<notes><p>This is a note without a type defined</p></notes>`; `<author-notes><fn><p>This is an author note</p></fn></author-notes>` ; `<notes notes-type="ednotes"><p content-type="genetic_counseling"><italic>Genetic counseling is the process of p....</p><p content-type="resources"><italic>GeneReviews staff has selected ....</italic></p></notes>` |
| permissions: copyright statement; license statement | bold, italic, hyperlinks | collection-meta node; book-meta node | `<license xmlns:xlink="http://www.w3.org/1999/xlink" license-type="open-access" xlink:href="http://creativecommons.org/licenses/by-nc-nd/4.0/"><license-p>This work is also available in an Open Access edition, which is licensed under a Creative Commons Attribution–NonCommercial–NoDerivatives 4.0 International License: <ext-link xmlns:xlink="http://www.w3.org/1999/xlink" ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by-nc-nd/4.0/">http://creativecommons.org/licenses/by-nc-nd/4.0/</ext-link>.</license-p></license>` ; `<copyright-statement>This is a copyright statement</copyright-statement>`|
The xml tags are:
```xml
<bold>bold</bold>
```
```xml
<italic>italic</italic>
```
```xml
<sub>subscript</sub> (same as html)
```
```xml
<sup>superscript</sup> (same as html)
```
Ordered list
```xml
<list list-type="order">
<list-item><p>item one</p></list-item>
<list-item><p>item two</p></list-item>
</list>
```
Unordered list
```xml
<list list-type="bullet">
<list-item><p>item one</p></list-item>
<list-item><p>item two</p></list-item>
</list>
```
Link
```xml
<ext-link xmlns:xlink="http://www.w3.org/1999/xlink" ext-link-type="uri" xlink:href="https://example.com">text to link</ext-link>
```
## Acceptance criteria
- [ ] HTML2XML translation will generate valid BITS 2.0 XML that meets Bookshelf tagging guidelines (so pass style checker rules)
- [ ] BCMS Coded XML will render on the Bookshelf site accuratelyNov 02.Dione Mentisdione@coko.foundationDione Mentisdione@coko.foundationhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/828new collection form logic errors2021-11-30T12:57:43ZDione Mentisdione@coko.foundationnew collection form logic errors@yannis
(see how these currently function on NCBI site)
- [ ] step 2: I should not be able to select `Group books by` if the toggle `Group books` is off
- [ ] step 2: `order books by` should be required@yannis
(see how these currently function on NCBI site)
- [ ] step 2: I should not be able to select `Group books by` if the toggle `Group books` is off
- [ ] step 2: `order books by` should be requiredNov 02.Yannis BarlasYannis Barlashttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/809refactor search filters on Dashboard to list all error status types2021-12-06T20:24:13ZDione Mentisdione@coko.foundationrefactor search filters on Dashboard to list all error status types@andynicholson
As @sidorelauku reported in here https://gitlab.coko.foundation/ncbi/ncbi/-/issues/494#note_64731:
![image](/uploads/be01d3c174ca0c03dbc17515f74206da/image.png)
Previously we had one "error" status. There are now thre...@andynicholson
As @sidorelauku reported in here https://gitlab.coko.foundation/ncbi/ncbi/-/issues/494#note_64731:
![image](/uploads/be01d3c174ca0c03dbc17515f74206da/image.png)
Previously we had one "error" status. There are now three:
1. Conversion errors
2. Loading errors
3. Submission errors (not yet developed)
The user should be able to filter by each of these.Nov 02.Sidorela UkuSidorela Ukuhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/751submit cover images separately2022-03-08T18:14:58ZMartin Latternersubmit cover images separately
## Context
In #739, we decided to decouple ingest and cover image submissions. This applies to both wholebook and chapter-processed domains. This also applies to collection covers (there is no difference between a collection and a who...
## Context
In #739, we decided to decouple ingest and cover image submissions. This applies to both wholebook and chapter-processed domains. This also applies to collection covers (there is no difference between a collection and a wholebook in terms of NCBI's processing).
NCBI needs two things:
1. the physical image (ex. `bcms123-cover.jpg`)
3. the domain name
## Workflow
1. User adds or updates a cover image in the book or collection metadata UI
2. BCMS creates package: Zip the image together with a JSON file detailing the domain: ex `bcms134-cover.zip` contains `foobar.jpg` and `foobar.json`. JSON contains `{domain: "bcms134"}`.
3. submit to FTP dir: `/covers/`
4. Processing errors NCBI-side should be very rare, but are no impossible. In these cases NCBI sends a receipt via Kafka to a dedicated topic
5. BCMS sends email notification for the errors reported (to be defined in #511
### Updated spec from https://gitlab.coko.foundation/ncbi/ncbi/-/issues/751#note_73067
Some refinements of the spec:
1. json/zip file names are expected to match `^([\w\.-]+)\.(\d{4}_\d\d_\d\d-\d\d_\d\d_\d\d)\.(json|zip)$`
2. zip package must contain exactly one image
3. JSON must include the fields `package`, `package_id`, `domain`
Receipts are sent to the Kafka topic `cover_receipt`. Notifications contain:
1. `package_id`: the BCMS reference ID for the package
2. `status`: 0=success, 3=failure
3. `thumb`: the name of the converted cover (not really useful, but included anyways)
4. `timestamp`
5. `notices`: a list of error and/or warning notices
In production, the FTP is polled once a day, at 6.30 pm.= In development, I'll keep it at every minute - to facilitate dev work.
**Important note:**
The complete process for cover thumbs looks like this:
1. Cover thumbs get submitted to a dedicated location on the PMC file system - on a rolling basis
2. Images get picked up from \#1 and submitted to a VCS - twice a day, at 10am and 7pm
3. Images get synced from the VCS to NCBI web fronts - once a day, at 3am
This process here only covers step \#1!
In development, steps \#2 and \#3 are not executed - so @lathrops1 , @DioneMentis: do not expect any thumbs for all those test domains to show up. You cannot verify that the process works by looking at the preview site. @John.kopanas can verify it based on what he gets from Kafka.
In production, note that \#3 is a nightly tasks - so @lathrops1 , @Kireev: expect the thumb to show up on preview one day after it has been submitted.
## Acceptance criteria
- [ ] Covers submitted to BCMS display on NCBI Bookshelf web pages per NCBI rendering rules after covers are passed to NCBI from BCMS and committed and distributed by NCBINov 02.https://gitlab.coko.foundation/ncbi/ncbi/-/issues/777Unable to Submit Newly Uploaded PDF Source File to Tagging2021-12-06T06:53:48ZStacy LathropUnable to Submit Newly Uploaded PDF Source File to Taggingcc @douglassue
Unable to Submit Newly Upload PDF Source File to Tagging possibly introducing costly and avoidable problems.
## Expected behaviour
Should be able to Submit New Upload files per: https://docs.google.com/spreadsheets/d/1...cc @douglassue
Unable to Submit Newly Upload PDF Source File to Tagging possibly introducing costly and avoidable problems.
## Expected behaviour
Should be able to Submit New Upload files per: https://docs.google.com/spreadsheets/d/1M4ZdBbzr2s4-PUXqEblfsKoPBUuk4aG343u9ZAUn_Tk/edit#gid=0
## Current behaviour
Unable to Submit New Upload PDF source files.
## Steps to reproduce
See https://ncbi.cloud68.co/organizations/308db55a-21c9-4a64-9a70-d5aa7574726c/bookmanager/3dd6021e-10c6-4585-919a-789a7868497b/c34cdaae-2f21-4400-aebd-ff900c2bc244
1. Upload file stated in "Tagging" status in link above
2. Submit that file
3. Upload file stated as "New Upload in link above
4. See you can submit
## Possible solution(s)
Any combination of solutions 1 and 2:
**Solution 1:**
1. Fail file in the "Tagging" status
1. Sent notification to System Admin, Submitter, PDF2XML taggers that a file and its package is "failed" and should NOT be tagged; email has package name, file name, date, and a clear message.
1. Permit submission of the New Upload to generate a Tagging package per PDF tagging integration specifications
**Solution 2**
1. Do NOT permit new PDF source file uploads when in "Tagging" status
1. System Admin | Org Admin | Editor can post a Vendor Query message to ignore a sent file / package and upload corrected file(s)
1. Corrected files are uploaded to Source Files folder with a note they were added via Vendor Query module with date / details
Solution 2 would require earlier implementation of Vendor Queries to Phase 1. Decision should be based on easiest implementation.
## Priority
Y, to avoid costly processing errors.Nov 02.https://gitlab.coko.foundation/ncbi/ncbi/-/issues/817User should be able to submit missing files to Apex during "tagging" work2021-12-06T20:29:13ZDione Mentisdione@coko.foundationUser should be able to submit missing files to Apex during "tagging" work## Expected behaviour
User should be able to submit missing files to Apex during their tagging work. This means "submit" action must be possible in "tagging" status. This will result in a second package send to Apex (package has timest...## Expected behaviour
User should be able to submit missing files to Apex during their tagging work. This means "submit" action must be possible in "tagging" status. This will result in a second package send to Apex (package has timestamp so Apex will know which package is most recent.)
See green "submit to apex" step
![NCBI-PDF-workflow-actions-fixes-20211117](/uploads/569b0cd3b130caa9e1d55e1d6027942a/NCBI-PDF-workflow-actions-fixes-20211117.jpeg)
## Current behaviour
The button "submit" is active, but it results in a UI error: "There are no files to send for conversion"
## Steps to reproduce
See video: ![resubmit-in-BCMS-to-Apex-20211117](/uploads/c694dac3bbb6dd3b128351395c426d9b/resubmit-in-BCMS-to-Apex-20211117.mp4)
1. Source files (with may include files added to suppl, images, and support folders)
2. Submit files (status changes to tagging)
3. Add missing files to "suppl, images, and support"
4. Select submit and see error
## Priority
Y -- because it's common for tagger to request missing files, and the "support" folders is being used as a easy way for BCMS staff to any any additional metadata not currently supported in the UI.Nov 02.Giannis Kopanasjkopanas@gmail.comSidorela UkuGiannis Kopanasjkopanas@gmail.comhttps://gitlab.coko.foundation/ncbi/ncbi/-/issues/814Write collection meta node to include in converted chapter.xml files2022-01-28T17:01:28ZDione Mentisdione@coko.foundationWrite collection meta node to include in converted chapter.xml filesThis is a sub-issue of #355
@andynicholson
Referring to the [meta flow diagram](https://miro.com/app/board/o9J_lEzF9oQ=/?moveToWidget=3074457365392162438&cot=14):
1. Write fields in Column E of [collection-meta sheet](https://docs.g...This is a sub-issue of #355
@andynicholson
Referring to the [meta flow diagram](https://miro.com/app/board/o9J_lEzF9oQ=/?moveToWidget=3074457365392162438&cot=14):
1. Write fields in Column E of [collection-meta sheet](https://docs.google.com/spreadsheets/d/1nldlGVs9uCq9CGB9efKzTJhZ_iKTO2unfjSXjLRWZZo/edit?usp=sharing) into `<collection-meta>` within `<book-part-wrapper>`
Sample file from NCBI: [bookpartwrapperch5.xml](/uploads/bac2877adaa8d4810f7eab1cc3ab71c7/bookpartwrapperch5.xml)
(note: I added a closing italics tag on line 7 that was missing from the sample provided)
`<italic>Genetic counseling is the process of p....</italic></p>`
Both collection types (whether "funded" or "book-series") will have this node with `collection-type=""ncbi-books-collection"`
```xml
<collection-meta collection-type="ncbi-books-collection">
<collection-id collection-id-type="pmcid">iarcmonocollect</collection-id>
<title-group>
<title>IARC Monographs on the Identification of Carcinogenic Hazards to Humans</title>
</title-group>
</collection-meta>
```
If the collection is a "book-series" it will have an additional collection-meta node
```xml
<collection-meta collection-type="book-series">
<title-group>
<title>IARC Monographs on the Identification of Carcinogenic Hazards to Humans</title>
</title-group>
</collection-meta>
```Nov 02.https://gitlab.coko.foundation/ncbi/ncbi/-/issues/815Write collection xml file for package to send to load to PMC2022-02-23T21:04:52ZDione Mentisdione@coko.foundationWrite collection xml file for package to send to load to PMCThis is a sub-issue of #355
@andynicholson
Referring to the meta flow diagram:
1. Create collection.xml file according to the `bookcollection.dtd` [bookcollection-dtd.zip](/uploads/9e4c50036a901b103c42b957af19b9bc/bookcollection-dtd.z...This is a sub-issue of #355
@andynicholson
Referring to the meta flow diagram:
1. Create collection.xml file according to the `bookcollection.dtd` [bookcollection-dtd.zip](/uploads/9e4c50036a901b103c42b957af19b9bc/bookcollection-dtd.zip)
2. Name the file with the collection's `bcmsid`
3. Write fields in Column B of [collection-meta sheet](https://docs.google.com/spreadsheets/d/1nldlGVs9uCq9CGB9efKzTJhZ_iKTO2unfjSXjLRWZZo/edit?usp=sharing) into `<collection-meta>` node
Sample files from NCBI:
1. [bookwhohencollect.xml](/uploads/dfe4bce725846c864a08b48a076ce8f6/bookwhohencollect.xml)
2. [booksbuhtacollect.xml](/uploads/da1c2ca1aba657aae429dbaa10d1bd90/booksbuhtacollect.xml)
## QA feedback
1. [x] In the generated xml file publication date type is always as `<pub-date pub-type="pubr">` in both cases for Print publication range and Electronic publication range.
2. [x] Validation feedback from NBCI in https://gitlab.coko.foundation/ncbi/ncbi/-/issues/815#note_74968
## Testing guidance
To check the xml file:
1. Create collection
2. Complete the fields in collection meta UI, and save
3. Copy the collection ID from the URL (e.g. `8e926dcf-362f-496f-9650-1de5582b14fe` in `https://ncbi.cloud68.co/organizations/23378725-638a-4329-b060-3b52d85a03c9/collectionmanager/8e926dcf-362f-496f-9650-1de5582b14fe`
4. Add the collection ID to this URL format: `https://ncbi.cloud68.co/downloadCollectionXml?id=` for example https://ncbi.cloud68.co/downloadCollectionXml?id=8e926dcf-362f-496f-9650-1de5582b14fe
**Do not add the a slash `/` at the end of the URL**Nov 02.