Recovery

Chapter 9: How to recover from service crashes and possibly lost events

Resgate and the RES protocol has built in ways of ensuring that data is synchronized, as well as ways to recover from lost connections and server crashes.

Whenever a microservice suspects that the resources stored in Resgate’s cache may no longer be up-to-date, the microservice must send a signal to Resgate, so that measures can be taken to resynchronize. This should be done in cases such as when:

  • microservice is disconnected and reconnects to NATS
  • microservice restarts from a crash
  • resources are modified without the microservice being able to send proper events to describe the mutations

The way to signal this is through a system reset event. Let’s have a look at it:

System reset event

A system reset event1 isn’t as drastic as it sounds. It just tells Resgate(s) which resources might be out of sync, so that it can fetch them again and resynchronize.

Subject

The subject of the event is:

system.reset

Event message

The event message is a JSON object containing the following properties:

  • resources - optional array of resource name patterns2 matching resources which might be out of sync
  • access - optional array of resource name patterns2 matching resources which might have changed access permissions

A resource name pattern2 is basically just a resource ID where you can use the asterisk (*) or greater than (>) wildcards, as described in Chapter 2 - Basic concepts.

Resetting all resources for service example could look like this:

nats.publish("system.reset", JSON.stringify({
	"resources": [ "example.>" ]
}));

Tip

The access property can used to revoke access to groups of resources.

Eg. if a list of inventory items may no longer accessible to customers, only to personel, you could send:

nats.publish("system.reset", JSON.stringify({
	"access": [ "inventory.items", "inventory.item.*" ]
}));

This would tell Resgate to send new access requests for clients subscribing to those resources. When the customer’s client gets access denied, Resgate will unsubscribe the resource and notify the client.