fileStorage.list() truncates at 1000 results
Problem
fileStorage.list()
is limited to 1000 results, and doesn't provide a way to list the remaining stored objects. The problem is here.
According to AWS docs ListObjects
is limited to 1000 keys. The newer version of the action, ListObjectsV2, also has a default maximum of 1000 keys returned.
Proposed solution
The object returned by coko/server
's fileStorage.list()
will contain members:
-
IsTruncated
: true if the results were cut off -
NextMarker
: the Marker to use to continue retrieving further objects.
Given this, the quickest, least disruptive solution I can see is to permit an optional argument to fileStorage.list()
named marker
or keyToStartFrom
which, if supplied, will set params.Marker
here. Then it is the responsibility of the end application (e.g. Kotahi) to check for IsTruncated
and make further requests.
Convenience method
It may be worth also providing a listAll()
method that would handle the truncation/re-request logic internally and return a plain array of all file objects; though this could exhaust memory if there are very large numbers of files. I propose something like this (untested):
const listAll = async () => {
if (!s3) {
throw new Error(
's3 does not exist! Probably configuration is missing/invalid',
)
}
const { bucket } = config.get('fileStorage')
const params = { Bucket: bucket }
const allObjects = []
let shouldContinueListing = true
while (shouldContinueListing) {
const data = await new Promise((resolve, reject) => {
s3.listObjects(params, (err, data) => {
if (err) {
reject(err)
} else {
resolve(data)
}
})
})
allObjects.push(...data.Contents)
shouldContinueListing = data.IsTruncated
if (shouldcontinueListing) params.Marker = data.NextMarker
}
return allObjects
}