23-06-2020

Go-Room, a version management utility for edge data stores.

15 min read

Whatsapp image 2020 06 23 at 12.13.05 pm

If you have written an application which runs on edge devices it is very likely that you used a data store. It could be to store user preferences, transaction logs or cached intelligence data for offline capabilities among many other such use cases. As developers focus on offline capabilities to address network issues and low bandwidth limits, data stores tend to be a crucial part of the application.

Android Room

SQLite is the weapon of choice in Android. A fast, reliable and yet hassle free database option in terms of contribution to application file size. It is through my work with SQLite on Android that I read about Room. It is an excellent utility that provides a very sleek interface for SQLite database creation and management.

Why make Go-Room?

Version Management

The major challenge that you face with a data store on edge apps is version management and migration. *Edge apps are often updated at the convenience of the user and network.* This results in a fractured ecosystem where multiple versions of your app exist in edge devices at a given point in time. A data store only adds to the complexity when you factor in changes to schema across various versions of the app.

The following figure demonstrates how different sections(by percentage) of users/edge devices can have different versions of your app as you release new versions over time.

An example of how fractured distribution of an edge app looks like across overall users

Distribution Dashboard of a Hypothetical Edge App

Migration Pains

The biggest risk that comes along is your data migration queries feel radioactive as you need to account for migration from every possible version of your app ever released. This leads to the risk of data corruption and loss. A simple strategy would be to perform destructive migrations on every update but data stored on edge device could be too valuable to lose. For instance they could be transaction logs of a payment gadget that are yet to be synced to the server.

How it works?

Room in Android defines the whole interaction with database. It is intended to be declarative where a user declares their intention and Room generates a standard/safe implementation at build time.

In this project my intention was to replicate the version management part while abstracting the data access layer. A typical usage of go-room would be as follows.

.


The following code snippets will illustrate the use of Go-Room on a database using the excellent GORM for DB access.

The snippet below shows an adapter I wrote to use Room in one of my projects. The focus is on the InitGormDB method.

  package main
  import (
  "fmt"
  "adonmo.com/goroom"
  "adonmo.com/goroom/orm"
  "adonmo.com/goroom/room"
  "adonmo.com/goroom/util/adapter"
  "github.com/jinzhu/gorm"
  )
   
  //RoomAdapter Adapter for Room DB Store management
  type RoomAdapter struct{}
   
  //InitGormDB Init DB
  func (r *RoomAdapter) InitGormDB(db *gorm.DB, entityList []interface{}, migrations []orm.Migration, versionNumber orm.VersionNumber, fallbackToDestructiveMigration bool) (err error) {
  gormAdapter := adapter.NewGORM(db)
  identityHashCalculator := &adapter.EntityHashConstructor{}
  roomDB, errList := room.New(entityList, gormAdapter, versionNumber, migrations, identityHashCalculator)
  if errList != nil {
  err = fmt.Errorf("Error while setting up DB props. Err: %v", errList)
  return
  }
  err = goroom.InitializeRoom(roomDB, fallbackToDestructiveMigration)
  if err != nil {
  err = fmt.Errorf("Error while initializing App Database. Err: %v", err)
  return
  }
  return
  }

goroom_init.go

We can now break down each argument passed to the InitGormDB function to understand the usage steps I mentioned before.

db

It is the ORM object created by the app for database access.

entityList

This is a list of entities(tables) that are expected to exist in the database. The following snippets show the schema of entities in oldest and latest app versions respectively.

import "github.com/jinzhu/gorm"
//User User Entity
type User struct {
gorm.Model
Name string
}

goroom_models_oldest.go

import "github.com/jinzhu/gorm"
//User User Entity
type User struct {
gorm.Model
Name string
Credits int
}
//Profile `Profile` belongs to `User`, `UserID` is the foreign key
type Profile struct {
gorm.Model
UserID int
User User `gorm:"foreignkey:UserRefer"`
Name string
}

goroom_models_latest.go

migrations

These are all the migrations rolled out with this app till date. Hence ideally Room is equipped to migrate the DB to the target version from any source version across a fractured ecosystem. In the gist below you can see the migrations defined for consecutive updates.

Room creates a graph to select the path for update from a given base to a given target version. This path could involve applying multiple migrations if required.

package main
 
import (
"fmt"
"github.com/adonmo/goroom/example/models/latest"
"github.com/adonmo/goroom/example/models/old"
"github.com/adonmo/goroom/logger"
"github.com/adonmo/goroom/orm"
"github.com/jinzhu/gorm"
)
 
//UserDBMigration Represents migration objects used for the example DB
type UserDBMigration struct {
BaseVersion orm.VersionNumber
TargetVersion orm.VersionNumber
MigrationFunc func(db interface{}) error
}
 
//GetBaseVersion …
func (m *UserDBMigration) GetBaseVersion() orm.VersionNumber {
return m.BaseVersion
}
 
//GetTargetVersion …
func (m *UserDBMigration) GetTargetVersion() orm.VersionNumber {
return m.TargetVersion
}
 
//Apply ….
func (m *UserDBMigration) Apply(db interface{}) error {
logger.Infof("Applying Migrations for %v to %v", m.BaseVersion, m.TargetVersion)
return m.MigrationFunc(db)
}
 
//GetMigrations Returns migrations applicable to the given DB over various version transistions
func GetMigrations() (migrations []orm.Migration) {
 
migration12 := &UserDBMigration{
BaseVersion: 1,
TargetVersion: 2,
MigrationFunc: func(db interface{}) error {
gormDB, ok := db.(*gorm.DB)
if !ok {
return fmt.Errorf("Unable to get the desired DB object")
}
return gormDB.CreateTable(old.Profile{}).Error
},
}
migration23 := &UserDBMigration{
BaseVersion: 2,
TargetVersion: 3,
MigrationFunc: func(db interface{}) error {
gormDB, ok := db.(*gorm.DB)
if !ok {
return fmt.Errorf("Unable to get the desired DB object")
}
return gormDB.AutoMigrate(latest.User{}).Error
},
}
var migration34 = &UserDBMigration{
BaseVersion: 3,
TargetVersion: 4,
MigrationFunc: func(db interface{}) error {
gormDB, ok := db.(*gorm.DB)
if !ok {
return fmt.Errorf("Unable to get the desired DB object")
}
return gormDB.AutoMigrate(latest.Profile{}).Error
},
}
migrations = append(migrations, migration12, migration23, migration34)
return
}

goroom_migrations.go

versionNumber

The expected Target Version in which the database must be for the App to function. This is a constant that is a part of your source code. You are expected to change it whenever you make changes to the schema.

fallbackToDestructiveMigration

A flag that forces recreation of database in case corruption or lack of migration path to target is detected.

Boilerplate

There are two other components that need to be initialised for Room.

gormAdapter

Implements standard ORM interface for GORM to function with Room. This ORM interface allows Room to perform clean up and version management.

identityHashCalculator

Implements a Hash calculation interface that allows Room to generate a unique stable signature for a database consisting of a given set of entities. This signature is coupled with version and serves as a straightforward check of compatibility.

In the snippet below you can see the final method call for initialising a GORM DB using Room.

package main
 
func main() {
roomAdapter := &RoomAdapter{}
db, err : gorm.Open("sqlite3", "test.db")
if err != nil {
panic(err)
}
 
err = roomAdapter.InitGormDB(db, []interface{}{User{}, Profile{}}, GetMigrations(), 4, false)
if err != nil {
err = fmt.Errorf("Error while initializing DB. Err %v", err)
return
}
}

goroom_final.go

When does Init fail exactly?

A database initialisation with room fails in following scenarios:

  • You make changes to schema and fail to update the versionNumber.
  • You make changes to schema and update the versionNumber but do not provide a migration. An empty migration is required even if you do not expect any database DDL to be run.
  • You mess around with the boilerplate components. **With great power comes great responsibility**

What next?

In this article we covered a basic usage pattern of Room with GORM. As we mentioned earlier in the article, Room is agnostic to the data access layer. In plain English you can use a different ORM of your choice or even a different kind of data store (lets say a file storage) with Room.

We intend to illuminate these points subsequently as we discuss further in my next two articles:

Part-2 -> “Go-Room: The Internals”

Part-3 -> “Go-Room: Advanced Usage”