Switch from FTP to AWS S3 for data exchange with NCBI
NCBI Systems is planning to prohibit ftp access from the Kubernetes cluster where BCMS is deployed, because they want to control all outgoing network connections via Istio service mesh, which does not support ftp protocol. Instead of FTP, AWS S3 (which is accessed via HTTPS REST API) will be used for data exchange between NCBI and Coko BCMS.
NCBI will create two buckets in AWS S3 for each environment (NCBI-AWS-DEV and NCBI-AWS-PROD), one for APEX and another one for everything else (conversion, submission, package ingest etc). NCBI will specify buckets name/URL/ARN and provide AWS IAM access keys with permissions to upload/download objects in these buckets. Directory structure within S3 buckets and file naming convention will be mapped to object prefixes / names in S3 and remain the same. Kafka messages / data exchange protocol should also remain the same, except that the package
attribute in Kafka message JSON would be used to construct an object key within a bucket, for example "S3://ncbi-books-bcms-dev/AHRQ/hcup_sb261.zip"