LiveCycle unable to access Cache Controller. Exception message - Error accessing the cache container - Error on GET action for cache Replicated:UM_CLUSTER_INVALIDATION

The incident starts after all the members are removed from the DCS Stack DefaultCoreGroup view and added back again by WebSphere Application Server. Following exceptions are seen after LiveCycle was added back in again to the view.

We have a clustered environment with two nodes 2A and 2B. Server 2A crashed and therefore all members on 2B node were removed from the DCS view. Later the new core group view was installed but LiveCycle did resume operations as expected.

Errors below:

Exception caught while dealing with cache : Action - Get, ObjectType - UM_CLUSTER_INVALIDATION_CACHE, Exception message - Error accessing the cache container - Error on GET action for cache Replicated:UM_CLUSTER_INVALIDATION_CACHE

ServiceRegist W Cache get failed for service EncryptionService Reason: Error accessing the cache container - Error on GET action for cache

Error accessing the cache container - Error on GET action for cache Local:SERVICE_FACTORY_CACHE

The following message appeared for several different services:

Cache put failed for service [CredentialService, ReaderExtensionsService,EncryptionService, etc]

SSOSessionCle W Error in cleaning stale sessions from the database. These sessions would be deleted in next trigger

java.lang.RuntimeException: com.adobe.livecycle.cache.CacheActionException: Error accessing the cache container - Error on ENTRYSET action for cache Local:UM_ASSERTION_CACHE

at com.adobe.idp.um.businesslogic.authentication.AssertionCacheManager.removeExpiredResults( AssertionCacheManager.java:125)

at com.adobe.idp.um.scheduler.SSOSessionCleanupJob.execute(SSOSessionCleanupJob.java:69)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)

Caused by: com.adobe.livecycle.cache.CacheActionException: Error accessing the cache container - Error on ENTRYSET action for cache Local:UM_ASSERTION_CACHE

at com.adobe.livecycle.cache.adapter.GemfireLocalCacheAdapter.entrySet(GemfireLocalCacheAdap ter.java:219)

at com.adobe.idp.um.businesslogic.authentication.AssertionCacheManager.removeExpiredResults( AssertionCacheManager.java:99)

... 3 more

Caused by: com.gemstone.gemfire.distributed.DistributedSystemDisconnectedException: GemFire on 2B<v1>:9057/51073 started at Sun Aug 17 08:57:23 EDT 2014: Message distribution has terminated, caused bycom.gemstone.gemfire.ForcedDisconnectException: This member has been forced out of the distributed system. Reason='this member is no longer in the view but is initiating connections'

at com.gemstone.gemfire.distributed.internal.DistributionManager$Stopper.generateCancelledEx ception(DistributionManager.java:746)

at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem$Stopper.generateCance lledException(InternalDistributedSystem.java:846)

at com.gemstone.gemfire.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(G emFireCacheImpl.java:1090)

at com.gemstone.gemfire.CancelCriterion.checkCancelInProgress(CancelCriterion.java:59)

at com.gemstone.gemfire.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:669 4)

at com.gemstone.gemfire.internal.cache.LocalRegion.checkReadiness(LocalRegion.java:2587)

at com.gemstone.gemfire.internal.cache.LocalRegion.entries(LocalRegion.java:1815)

at com.gemstone.gemfire.internal.cache.LocalRegion.entrySet(LocalRegion.java:7941)

at com.adobe.livecycle.cache.adapter.GemfireLocalCacheAdapter.entrySet(GemfireLocalCacheAdap ter.java:209)

... 4 more

Caused by: com.gemstone.gemfire.ForcedDisconnectException: This member has been forced out of the distributed system. Reason='this member is no longer in the view but is initiating connections'

at com.gemstone.org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleLeaveResponse(Particip antGmsImpl.java:106)

at com.gemstone.org.jgroups.protocols.pbcast.GMS.up(GMS.java:1289)