[Task] investigate why errors in code cause server to crash and are not found on test environment

[OriHoch]

there were a few cases where errors caused the server to crash (for example: #339)

need to investigate those cases (or the spark code in general) and find out:

• why errors in code cause the entire node process to crash
• why these errors are not reproduced on test environment

[noam-almog]

Not sure what the logs are, but if we don't dump them to /var/logs we should

node index.js 2>>/var/logs/spark/stderr.log >>/var/logs/spark/stdout.log

[noam-almog]

as for node crashing for no good reason, we are using node 7.4 which is by definition unstable, we should either consider downgrading to 6.10.2 or upgrade to latest unstable release (7.9)

[OriHoch]

regarding logs, we use systemctl which logs all output to /var/log/syslog

feel free to send a pull request or issue if you think we should change it to the method you suggested

ran this on production instance just now (censored sensitive details)-

$ tail /var/log/syslog
Apr 25 12:23:27 ip-172-31-32-142 start.sh[2020]: 25/04/17 12:23:27.668 INFO app.js : #033[0mGET /camps/*******/members/*******/approve #033[32m200 #033[0m25.175 ms - 72#033[0m
Apr 25 12:23:27 ip-172-31-32-142 start.sh[2020]: 25/04/17 12:23:27.827 INFO libs/mail.js : Mail sent successfully
Apr 25 12:23:36 ip-172-31-32-142 start.sh[2020]: approve from camp ******* of user ******* / mgr id: *******
Apr 25 12:23:36 ip-172-31-32-142 start.sh[2020]: approve from camp ******* of user ******* / status: approved
Apr 25 12:23:36 ip-172-31-32-142 start.sh[2020]: Trying to send mail to *******@gmail.com from support@midburn.org: Spark: you have been approved!, template emails/camps/member_approved
Apr 25 12:23:36 ip-172-31-32-142 start.sh[2020]: 25/04/17 12:23:36.531 INFO libs/mail.js : Sending email to ********@gmail.com
Apr 25 12:23:36 ip-172-31-32-142 start.sh[2020]: 25/04/17 12:23:36.532 INFO app.js : #033[0mGET /camps/***/members/****/approve #033[32m200 #033[0m33.025 ms - 72#033[0m
Apr 25 12:23:36 ip-172-31-32-142 start.sh[2020]: 25/04/17 12:23:36.661 INFO libs/mail.js : Mail sent successfully
Apr 25 12:23:49 ip-172-31-32-142 start.sh[2020]: 25/04/17 12:23:49.227 INFO app.js : #033[0mGET /he/login #033[32m200 #033[0m2.117 ms - 4585#033[0m
Apr 25 12:23:49 ip-172-31-32-142 start.sh[2020]: 25/04/17 12:23:49.368 INFO app.js : #033[0mGET /he/login #033[32m200 #033[0m1.088 ms - 4585#033[0m

[OriHoch]

@noam-almog opened an issue for changing the Node version - #346 - feel free to do it

[noam-almog]

@OriHoch what about error log, do you also see exceptions there ?

[OriHoch]

I think so

 #011{"isFulfilled":false,"isRejected":true,"rejectionReason":{"message":"Cannot read property 'email' of undefined","stack":"TypeError: Cannot read property 'email' of undefined\n    at _after_update (/opt/spark/latest/routes/api_camps_routes.js:129:100)\n    at tryCatcher (/opt/spark/latest/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/opt/spark/latest/node_modules/bluebird/js/release/promise.js:512:31)\n    at Promise._settlePromise (/opt/spark/latest/node_modules/bluebird/js/release/promise.js:569:18)\n    at Promise._settlePromise0 (/opt/spark/latest/node_modules/bluebird/js/release/promise.js:614:10)\n    at Promise._settlePromises (/opt/spark/latest/node_modules/bluebird/js/release/promise.js:693:18)\n    at Async._drainQueue (/opt/spark/latest/node_modules/bluebird/js/release/async.js:133:16)\n    at Async._drainQueues (/opt/spark/latest/node_modules/bluebird/js/release/async.js:143:10)\n    at Immediate.Async.drainQueues (/opt/spark/latest/node_modules/bluebird/js/release/async.js:17:14)\n    at runCallback (timers.js:651:20)\n    at tryOnImmediate (timers.js:624:5)\n    at processImmediate [as _immediateCallback] (timers.js:596:5)"}}

[OriHoch]

also related: #347

[asihud]

@OriHoch
Since the error are logged, my suggestion that all error that pop up to the exception point,
should be logged and send to email or to slack channel.
so everyone can see them, and open/fix issue about them.

[OriHoch]

@asihud #347 provides all that

[amotenko]

I guess it's not crashing anymore