Serverless Database Backup Monitoring with Oracle Functions, Oracle Events Service & Oracle Notifications in Oracle Cloud Infrastructure

Persistence layers are what need backups

This is part-2 of the series, check out part-1 if you haven’t, link to Part-1

Part-1

Differential Functions — Not a Math Lesson!

Database backups are chosen as an example in this article to show how cloud-native principles discussed in the previous article can be applied.

The Ingredients & The Strategy

The Architecture

In The Context of Backups

Backups are long-running processes and if one has to follow every single stage of the backup, one has to have an always-on component and these processes cannot be modeled around short-running serverless components. Hence, we use

The Cycle

Events Service
Fully managed OCI Service that raises an event on based on conditions defined for any OCI Service Action

Functions
Get triggered from Events and enrich event information, with Work requests information.

Work Requests
The Interface that OCI API Provides to get the status of long running processes like backups. 

Notifications
Notification queue service can be used to subscribe to Webhooks, Email, Slack also a Trigger for Functions if required.

Build Guard Rails

Allow Application Owners to Create Databases but use guard rails

If Databases are created without auto backups enabled, enable it for them and decide exactly when to enable them

MATCH event WHERE (
eventType EQUALS ANY OF (
com.oraclecloud.databaseservice.autonomous.container.database.instance.create.end,
com.oraclecloud.databaseservice.autonomous.database.instance.create.end,
com.oraclecloud.databaseservice.createdatabase.end
)
AND (
autoBackupsEnabled MATCHES ANY OF (
false
)
)
)

Automate Obsessively but keep the human in the loop

There are two actions,

1) Automated action that needs to fix the issue of bad backup configuration, by updating the database configuration
2) Also notify the stakeholder over Slack / Email to know someone created a bad backup configuration.

When Backups are Created

Trigger an event when something unexpected happens, database backup end in the event that fires when the database backup try completes, on the success the lifecycle state reflects a successful flag and if it fails, it reflects the state of failure.

Coverage
Database Cloud Service on VMs
Database Cloud Service on Bare Metal
Database Cloud Service on Exadata 
Autonomous Database Serverless
Autonomous Database Dedicated

Capture only failed Events

MATCH event WHERE (
eventType EQUALS ANY OF (
com.oraclecloud.databaseservice.autonomous.container.database.backup.end,
com.oraclecloud.databaseservice.autonomous.database.backup.end,
com.oraclecloud.databaseservice.automaticbackupdatabase.end,
com.oraclecloud.databaseservice.backupdatabase.end
)
AND (
lifecycleState MATCHES ANY OF (
FAILED
)
)
)

Example of the Event Generated

{
  "eventID": "022953e2-ff77-11e9-99b8-f45c89b1cb17",
  "eventTime": "2019-11-05T02:50:21Z",
  "extensions": {
    "compartmentId": "ocid1.compartment.oc1..unique_id"
  },
  "eventType": "com.oraclecloud.databaseservice.backupdatabase.end",
  "eventTypeVersion": "2.0",
  "cloudEventsVersion": "0.1",
  "source": "databaseservice",
  "contentType": "application/json",
  "definedTags": {},
  "data": {
    "compartmentId": "ocid1.compartment.oc1..unique_id",
    "compartmentName": "example_name",
    "resourceName": "my_container_database",
    "resourceId": "ocid1.database.oc1.phx.unique_id",
    "availabilityDomain": "all",
    "freeFormTags": {},
    "definedTags": {},
    "additionalDetails": {
      "id": "ocid1.database.oc1.phx.unique_id",
      "timeCreated": "2019-06-27T21:15:59.000Z",
      "timeUpdated": "2019-06-27T21:16:04.389Z",
      "displayName": "Database - Create Backup End",
      "lifecycleState": "FAILED",
      "lifecycleDetails": "",
      "exadataInfrastructureId": "ocid1.autonomousexadatainfrastructure.oc1.phx.unique_id",
      "dbSystemId": "ocid1.dbsystem.oc1.phx.unique_id",
      "vmClusterId": "ocid1.vmcluster.oc1.phx.unique_id",
      "DbHomeId": "ocid1.dbhome.oc1.phx.unique_id",
      "dbUniqueName": "exampleDbName",
      "dbVersion": "19.0.0.0",
      "databaseEdition": "Enterprise Edition - Extreme Performance",
      "autoBackupsEnabled": "False",
      "recoveryWindow": "3 days",
      "backupDestinationId": "ocid1.database.oc1.phx.unique_id",
      "backupDestinationType": "Object store",
      "backupDestinationName": "bucket 2222",
      "backupType": "periodic",
      "databaseId": "ocid1.database.oc1.phx.unique_id",
      "licenseType": "BRING_YOUR_OWN_LICENSE",
      "workloadType": "Transaction Processing"
    }
  }
}

What could you do to extend the Sphere of Influence

Possibly you could dig into the Work requests API that is available for understanding a little bit further with functions and then notify the user, the next time?

So ..