java - Tomcat8 shutdown randomly with AbstractProtocol.pause -

1. problem

i deployed webapp in tomcat, , found shutdowns randomly, time varies 2 or 3 hours 2 or 3 days. log in catalina.out is:

26224 2015-06-10 13:59:04.110 {http-nio-8080-exec-3} info  com.timediff.controller.user.userprofilecontroller#getuserhome - /user/profile/home done, curuid: 889 26225 10-jun-2015 14:15:35.050 info [thread-11] org.apache.coyote.abstractprotocol.pause pausing protocolhandler ["http-nio-8080"] 26226 10-jun-2015 14:15:35.052 info [thread-11] org.apache.coyote.abstractprotocol.pause pausing protocolhandler ["ajp-nio-8009"] 26227 10-jun-2015 14:15:35.053 info [thread-11] org.apache.catalina.core.standardservice.stopinternal stopping service catalina 26228 10-jun-2015 14:15:35.058 info [localhost-startstop-2] org.springframework.web.context.support.xmlwebapplicationcontext.doclose closing webapplicationcontext namespace 'timediff-dispatcher-servlet': startup date [wed jun 10 13:38:14 cst 2015]; root of context hierarchy 26229 10-jun-2015 14:15:35.059 info [localhost-startstop-2] org.springframework.context.support.defaultlifecycleprocessor.stop stopping beans in phase 2147483647 26230 2015-06-10 14:15:35.061 {localhost-startstop-2} info  org.quartz.core.quartzscheduler#standby - scheduler timediffscheduler_$_izu1skaofy1z1433914696931 paused. 26231 10-jun-2015 14:15:35.072 info [localhost-startstop-2] org.springframework.scheduling.quartz.schedulerfactorybean.destroy shutting down quartz scheduler 26232 2015-06-10 14:15:35.072 {localhost-startstop-2} info  org.quartz.core.quartzscheduler#shutdown - scheduler timediffscheduler_$_izu1skaofy1z1433914696931 shutting down. 26233 2015-06-10 14:15:35.075 {localhost-startstop-2} info  org.quartz.core.quartzscheduler#standby - scheduler timediffscheduler_$_izu1skaofy1z1433914696931 paused. 26234 2015-06-10 14:15:35.077 {localhost-startstop-2} info  org.quartz.core.quartzscheduler#shutdown - scheduler timediffscheduler_$_izu1skaofy1z1433914696931 shutdown complete. 26235 10-jun-2015 14:15:35.082 info [localhost-startstop-2] org.springframework.scheduling.concurrent.threadpooltaskexecutor.shutdown shutting down executorservice 'quartzthreadpool' 26236 2015-06-10 14:15:35.103 {localhost-startstop-2} info  com.timediff.listener.stopmemoryleaklistener#lambda$contextdestroyed$0 - driver: com.mysql.jdbc.driver@7657b26d de-registered. 26237 2015-06-10 14:15:35.104 {localhost-startstop-2} info  com.timediff.listener.stopmemoryleaklistener#contextdestroyed - abandonedconnectioncleanupthread shutdown. 26238 10-jun-2015 14:15:35.150 info [thread-11] org.apache.coyote.abstractprotocol.stop stopping protocolhandler ["http-nio-8080"] 26239 10-jun-2015 14:15:35.152 info [thread-11] org.apache.coyote.abstractprotocol.stop stopping protocolhandler ["ajp-nio-8009"] 26240 10-jun-2015 14:15:35.154 info [thread-11] org.apache.coyote.abstractprotocol.destroy destroying protocolhandler ["http-nio-8080"] 26241 10-jun-2015 14:15:35.156 info [thread-11] org.apache.coyote.abstractprotocol.destroy destroying protocolhandler ["ajp-nio-8009"] 

on stackoverflow, this question , this question similar situation, i'm still stumbled.

now i'll give detailed description of problem:

2. more detail

2.1 tomcat , jdk version

tomcat: 8.0.22 jdk: 1.8.0_45 

2.2 jvm options in catalina.sh:

catalina_opts="-server -xms1g -xmx1g -xx:maxmetaspacesize=512m -xmn512m  -xx:survivorratio=8  -xx:+useconcmarksweepgc -xx:+cmsparallelremarkenabled  -xx:+usecmsinitiatingoccupancyonly -xx:cmsinitiatingoccupancyfraction=70 -xx:+scavengebeforefullgc  -xx:+cmsscavengebeforeremark -xx:+printgcdatestamps -verbose:gc -xx:+printgcdetails  -xloggc:/opt/logs/gc/timediff-gc.log  -xx:+usegclogfilerotation -xx:numberofgclogfiles=10 -xx:gclogfilesize=10m -dsun.net.inetaddr.ttl=120  -xx:+heapdumponoutofmemoryerror -xx:heapdumppath=/opt/logs/gc/timediff-oom.hprof   -djava.rmi.server.hostname=**.**.**.** -dcom.sun.management.jmxremote.port=1099  -dcom.sun.management.jmxremote.authenticate=false -dcom.sun.management.jmxremote.ssl=false" 

2.3 there no exception logs in webapp related abortion of tomcat, , i'm sure never called system.exit(), , there no code blocks like:

try {  } catch(exception e) {     // nothing } 

2.4 while found allocation failure in gc log:

2015-06-10t15:36:28.589+0800: 3099.795: [gc (allocation failure) 3099.795: [parnew: 419780k->382k(471872k), 0.0125816 secs] 469721k->50348k(996160k), 0.0126820 secs] [times: user=0.01 sys=0.00, real=0.01 secs] 2015-06-10t15:37:30.141+0800: 3161.347: [gc (allocation failure) 3161.347: [parnew: 419838k->372k(471872k), 0.0062445 secs] 469804k->50338k(996160k), 0.0063629 secs] [times: user=0.01 sys=0.00, real=0.01 secs] 2015-06-10t15:38:41.680+0800: 3232.886: [gc (allocation failure) 3232.886: [parnew: 419828k->369k(471872k), 0.0064920 secs] 469794k->50356k(996160k), 0.0066009 secs] [times: user=0.01 sys=0.00, real=0.01 secs] 2015-06-10t15:39:43.222+0800: 3294.428: [gc (allocation failure) 3294.428: [parnew: 419825k->384k(471872k), 0.0058772 secs] 469812k->50372k(996160k), 0.0059823 secs] [times: user=0.01 sys=0.00, real=0.01 secs] 2015-06-10t15:40:54.758+0800: 3365.964: [gc (allocation failure) 3365.964: [parnew: 419840k->388k(471872k), 0.0056674 secs] 469828k->50395k(996160k), 0.0069850 secs] [times: user=0.02 sys=0.00, real=0.00 secs] 

i think maybe cause, result of top , jvisualvm makes unclear:

web@izu1skaofy1z:/usr/local/apache-tomcat-8.0.22/logs$ free -m              total       used       free     shared    buffers     cached mem:          3951       3087        864          0        190        553 -/+ buffers/cache:       2343       1608 swap:            0          0          0  top - 15:50:05 16 days,  5:11,  2 users,  load average: 0.33, 0.17, 0.09 tasks: 128 total,   2 running, 126 sleeping,   0 stopped,   0 zombie %cpu(s):  0.8 us,  0.5 sy,  0.0 ni, 98.5 id,  0.0 wa,  0.2 hi,  0.0 si,  0.0 st kib mem:   4046820 total,  3161260 used,   885560 free,   194880 buffers kib swap:        0 total,        0 used,        0 free.   566984 cached mem    pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command 27307 web       20   0 2068604 865872  22048 s   0.7 21.4  20:20.28 java 16557 web       20   0 3680756 801708  13740 s   0.0 19.8   2:02.99 java 15597 mysql     20   0 1800972 526220   6636 s   0.0 13.0  36:26.08 mysqld 

2.4 deployed tomcat on same server, changed shutdown port , connector port, don't think conflict.

i've tried best, maybe forget during analysis, please give me tips, in advance!

update(2015-07-04): after switch user web user root when running tomcat, problem never occurs. doubt tomcat kill system because of user privilege, if have idea, please tell me, thanks！

this answer (from 1 of questions found) seems good.

something telling tomcat stop. , since doesn't happen when tomcat run root, think cause other (non-system) process (maybe script or cron job) sending signal (probably, sigterm) tomcat, kill <tomcat pid>. maybe other process run user web - explain why process can't kill root's tomcat. or maybe other process searches processes kill, , 1 of criteria "processes owned web".

i suggest read crontabs users root , web, system-wide crontab , in /etc/cron.*/ folders. can check if other processes owned web terminated. , building tomcat source, tracing added (as suggested in answer mentioned), seems idea.