Integrated job queue
This closes #404 (closed).
The API exposed for jobs in pubsweet-server
matches pg-boss
's API almost exactly. The only difference comes from starting and connection to a queue.
When the pubsweet-server
starts, we will automatically call startQueue()
, which calls pg-boss
's start()
. This:
- sets up the PostgreSQL database (e.g. adds a job table, migrates to latest
pg-boss
schema, etc.) to support job queues. - is a way for
pg-boss
to keep an eye on jobs, and manages storage (archival, deletion, etc.).
All other job queue users should use the exported connectToJobQueue
:
const { jobs: { connectToJobQueue } } = require('pubsweet-server')
const jobQueue = await connectToJobQueue()
After this, the API is exactly the same as pg-boss
's: https://github.com/timgit/pg-boss
Check out the associated tests for examples of processing a job 'in-line' (https://gitlab.coko.foundation/pubsweet/pubsweet/blob/jobs/packages/server/test/jobs/jobs_test.js) or in a different process (https://gitlab.coko.foundation/pubsweet/pubsweet/blob/jobs/packages/server/test/jobs/jobs_different_process_test.js).
For completeness, one short, complete example (in a server component context) would be:
const { jobs: { connectToJobQueue } } = require('pubsweet-server')
const emailHandler = async job => {
return sendEmail(job.data)
}
const jobQueue = await connectToJobQueue()
const queueName = 'email-queue'
// Add job to the queue
await jobQueue.publish(queueName, {
subject: 'Hello!',
to: 'some@example.com',
content: 'Hi again!'
})
// Subscribe to the job queue with an async handler
await jobQueue.subscribe(queueName, emailHandler)
// Be notified on job completion with job result
jobQueue.onComplete(queueName, job => {
console.log('Email sent to: ', job.request.data.email)
console.log('Email job returned: ', job.response.data)
})
Containers
While it is easy to create a docker container that processes jobs (equally easy as processing from a separate process, as shown in this test: https://gitlab.coko.foundation/pubsweet/pubsweet/blob/jobs/packages/server/test/jobs/jobs_different_process_test.js), it's not easy to sensibly manage these docker containers as a part of PubSweet's functionality. As such, I've decided to keep managing docker containers out of the scope of this MR:
- it makes the MR quite complex and testing this the integration with Docker is not trivial
- it's not clear this integration is needed, as using
docker-compose
, as we currently do, is quite powerful - this MR provides significant and much needed utility as it is, and should land as soon as possible
We'll add Docker-based job runner examples (starting the xsweet docx to html processing in editoria), so it will be easy to implement a similar thing on your own.
That's about it! Let me know what you think :)