We have a LCES 8.2.1 install on a WebSphere App Server 6.1.x on a windows platform.
The database used is DB2.
Recently we have not been able to start the LiveCycle ES app server. We issue the startServer command and it remains in start in-progress indefinitely. The SystemOut shows the following error.
It looks like its trying to recover some old job info, but then threads hang and the server never start.
What may have caused this problem?
How can we clean the Job store or DB2 database, if that is the cause of the problem?
Is there some table(s) used by the IDPSchedulerService during startup that can be examined...and maybe be cleaned up?
Any help or pointers appreciated.
Thanks in advance.
..............
..................
[6/29/10 12:26:31:264 EDT] 0000001c ArchiveFileMa I ArchiveFileManagerImpl:getLocalCacheRootDir():Adobe tmp directory is:C:\Adobe\lcestemp\adobews_lces82Db2Node01Cell_lces82Db2Node01_serv er1
[6/29/10 12:26:38:733 EDT] 0000001c SchedulerServ A com.adobe.idp.scheduler.SchedulerServiceImpl onLoad IDPSchedulerService onLoad
[6/29/10 12:26:42:093 EDT] 0000001c SchedulerServ A com.adobe.idp.scheduler.SchedulerServiceImpl startSchedulerServiceInternal Start scheduler service
[6/29/10 12:26:42:124 EDT] 0000001c SchedulerServ A com.adobe.idp.scheduler.SchedulerServiceImpl startSchedulerServiceInternal Context classloader is com.ibm.ws.classloader.CompoundClassLoader
[6/29/10 12:26:42:171 EDT] 0000001c SchedulerServ A com.adobe.idp.scheduler.SchedulerServiceImpl createSchedulerFactory Initializing SCHEDULER FACTORY from properties ...
[6/29/10 12:26:42:218 EDT] 0000001c SchedulerServ A com.adobe.idp.scheduler.SchedulerServiceImpl createSchedulerFactory SCHEDULER FACTORY initialized.
[6/29/10 12:26:42:233 EDT] 0000001c SchedulerServ A com.adobe.idp.scheduler.SchedulerServiceImpl startSchedulerServiceInternal Creating new scheduler. isInUpgrade:false
[6/29/10 12:26:42:405 EDT] 0000001c QuartzSchedul I org.quartz.core.QuartzScheduler <init> Quartz Scheduler v.1.6.0 created.
[6/29/10 12:26:42:421 EDT] 0000001c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport initialize Using thread monitor-based data access locking (synchronization).
[6/29/10 12:26:43:093 EDT] 0000001c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport cleanVolatileTriggerAndJobs Removed 0 Volatile Trigger(s).
[6/29/10 12:26:43:108 EDT] 0000001c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport cleanVolatileTriggerAndJobs Removed 0 Volatile Job(s).
[6/29/10 12:26:43:139 EDT] 0000001c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreTX initialize JobStoreTX initialized.
[6/29/10 12:26:43:139 EDT] 0000001c DSCSchedulerF I com.adobe.idp.scheduler.DSCSchedulerFactory instantiate Quartz scheduler 'IDPSchedulerService' initialized from an externally provided properties instance.
[6/29/10 12:26:43:155 EDT] 0000001c DSCSchedulerF I com.adobe.idp.scheduler.DSCSchedulerFactory instantiate Quartz scheduler version: 1.6.0
[6/29/10 12:26:43:171 EDT] 0000001c SchedulerServ A com.adobe.idp.scheduler.SchedulerServiceImpl createScheduler Scheduler loaded with name IDPSchedulerService
[6/29/10 12:26:43:202 EDT] 0000001c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport recoverJobs Freed 0 triggers from 'acquired' / 'blocked' state.
[6/29/10 12:26:43:218 EDT] 0000001c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport recoverMisfiredJobs Handling 46 trigger(s) that missed their scheduled fire-time.
[6/29/10 12:26:44:061 EDT] 0000001c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport recoverJobs Recovering 0 jobs that were in-progress at the time of the last shut-down.
[6/29/10 12:26:44:077 EDT] 0000001c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport recoverJobs Recovery complete.
[6/29/10 12:26:44:124 EDT] 0000001c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport recoverJobs Removed 0 'complete' triggers.
[6/29/10 12:26:44:139 EDT] 0000001c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport recoverJobs Removed 0 stale fired job entries.
[6/29/10 12:26:44:186 EDT] 0000001c QuartzSchedul I org.quartz.core.QuartzScheduler start Scheduler IDPSchedulerService_$_20 started.
[6/29/10 12:26:44:389 EDT] 0000002b ErrorLogger E org.quartz.core.ErrorLogger schedulerError An error occured while scanning for the next trigger to fire.
org.quartz.JobPersistenceException: Couldn't acquire next trigger: null [See nested exception: java.lang.NullPointerException]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobSt oreSupport.java:2700)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$35.execute(JobStoreSuppo rt.java:2658)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLoc k(JobStoreSupport.java:3662)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobSt oreSupport.java:2654)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java: 235)
Caused by: java.lang.NullPointerException
at java.util.Hashtable.getEntry(Hashtable.java:468)
at java.util.Hashtable.containsKey(Hashtable.java:354)
at com.ibm.db2.jcc.c.fg.setCursorName(fg.java:692)
at com.ibm.websphere.rsadapter.DB2UniversalDataStoreHelper.doStatementCl eanup(DB2UniversalDataStoreHelper.java:225)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.closeWrapper(WSJdbc PreparedStatement.java:435)
at com.ibm.ws.rsadapter.jdbc.WSJccPreparedStatement.closeWrapper(WSJccPr eparedStatement.java:274)
at com.ibm.ws.rsadapter.jdbc.WSJdbcObject.close(WSJdbcObject.java:185)
at com.ibm.ws.rsadapter.jdbc.WSJdbcObject.close(WSJdbcObject.java:144)
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.closeStatement(StdJDBCDe legate.java:3609)
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTriggerToAcquire(S tdJDBCDelegate.java:2931)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobSt oreSupport.java:2669)
... 4 more
[6/29/10 12:26:45:436 EDT] 0000002c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport recoverMisfiredJobs Handling 1 trigger(s) that missed their scheduled fire-time.
[6/29/10 12:27:44:218 EDT] 0000002c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport recoverMisfiredJobs Handling the first 20 triggers that missed their scheduled fire-time. More misfired triggers remain to be processed.
[6/29/10 12:27:44:608 EDT] 0000002c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport recoverMisfiredJobs Handling the first 20 triggers that missed their scheduled fire-time. More misfired triggers remain to be processed.
[6/29/10 12:27:44:968 EDT] 0000002c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport recoverMisfiredJobs Handling 8 trigger(s) that missed their scheduled fire-time.
[6/29/10 12:28:44:905 EDT] 0000002c DSCJobStoreTX I org.quartz.impl.jdbcjobstore.JobStoreSupport recoverMisfiredJobs Handling the first 20 triggers that missed their scheduled fire-time. More misfired triggers remain to be processed.
[6/29/10 12:34:47:468 EDT] 00000015 ThreadMonitor W WSVR0605W: Thread "server.startup : 1" (0000001c) has been active for 628610 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
[6/29/10 12:36:36:327 EDT] 00000016 TimeoutManage I WTRN0006W: Transaction 0000012984880F97000000010000001C7471DEC905A41DB7A0F30BD7DADA44AEB7E10 5B70000012984880F97000000010000001C7471DEC905A41DB7A0F30BD7DADA44AEB7E 105B700000001 has timed out after 600 seconds.
[6/29/10 15:23:03:186 EDT] 0000002e ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl initialize FFDC0009I: FFDC opened incident stream file c:\Program Files\IBM\WebSphere\AppServer\profiles\LCESProfile\logs\ffdc\server1_ 0000002e_10.06.29_15.23.03_0.txt