App is unstable - constantly crashes
Expected behaviour
The app should be stable without crashing.
Current behaviour
On the NCBI deployed dev site we are constantly getting "no health upstream" messages after the BCMS crashes and goes blank. @deniskar tried to troubleshoot the logs but says the stack error messages make it hard to trace where the app crashed and why. It is, he says, crashing all the time for a varietyof reasons. We would like to ask that you monitor your logs for this behavior in your instances for fixing.
Steps to reproduce
Unable to provide.
Update information from @deniskar in #1155 (comment 108472)
@John.kopanas @DioneMentis We do still have frequent crashes:
For example, today we had crashes at these times: (19:19, 18:12, 14:44, 10:57, 00:32, 23:32)
They typically have the same error that postgres Connection terminated unexpectedly
It could be related to the (excessive pg connections but no query) issue;
Here is some of the logs surrounding these crashes
January_25th_2023__19.19.14.582_2023-01-.txt
January_25th_2023__18.12.32.208_2023-01-.txt
January_25th_2023__18.12.29.144_2023-01-.txt
January_17th_2023__09.18.33.551_2023-01-.txt
Environment
N/A
Possible solution
[Not required. Suggest a fix for the bug]
NCBI's priority feedback
Y, making testing, migration, deployment all difficult
QA Steps
[To be completed by Coko once dev is done]