GORM 框架研究
什么是 ORM
ORM(Object-relational mapping) 是一种在关系数据库和面向对象编程语言内存之间转换数据的编程技术。这实际上创建了一个可以在编程语言中使用的虚拟对象数据库。
在程序开发中,数据库保存的表、字段与程序中的实体类之间是没有关联的,在实现持久化时就比较不方便。那么,到底如何实现持久化呢?一种简单的方案是采用硬编码方式,为每一种可能的数据库访问操作提供单独的方法。这种方案存在以下不足:
- 持久化层缺乏弹性。一旦出现业务需求的变更,就必须修改持久化层的接口
- 持久化层同时与域模型与关系数据库模型绑定,不管域模型还是关系数据库模型发生变化,毒药修改持久化曾的相关程序代码,增加了软件的维护难度
ORM 提供了实现持久化层的另一种模式,它采用映射元数据来描述对象关系的映射,使得 ORM 中间件能在任何一个应用的业务逻辑层和数据库层之间充当桥梁,ORM 的方法论基于四个核心原则:
- 简单:ORM 以最基本的形式建模数据。比如 ORM 会将 MySQL 的一张表映射成一个 Go struct (模型),表的字段就是这个类的成员变量;
- 精确:ORM 使所有的 MySQL 数据表都按照统一的标准精确地映射成 Go struct,使系统在代码层面保持准确统一;
- 易懂:ORM 使数据库结构文档化。比如 MySQL 数据库就被 ORM 转换为了程序员可以读懂的 Go 结构体,程序员可以只把注意力放在他擅长的编程语言层面(当然能够熟练掌握 MySQL 更好);
- 易用:ORM 包含对持久类对象进行 CRUD 操作的 API,例如
create(), update(), save(), load(), find(), find_all(), where()
等,也就是讲 sql 查询全部封装成了编程语言中的函数,通过函数的链式组合生成最终的 SQL 语句。通过这种封装避免了不规范、冗余、风格不统一的SQL语句,可以避免很多人为 Bug,方便编码风格的统一和后期维护;
举例来说明 ORM 到底是做什么的:
/***************************************************/
/* 没有 ORM 的时候,实现所有 Persons 相关的数据库读写操作 */
/***************************************************/
db, err := sql.Open("mysql", "root:<password>@tcp(127.0.0.1:3306)/123begin")
if err != nil {
panic(err.Error())
}
defer db.Close()
personsFirstName, err := db.Query("SELECT first_name FROM persons WHERE id = 10")
if err !=nil {
panic(err.Error())
}
for persons.Next() {
var firstName string
err = results.Scan(&firstName)
if err !=nil {
panic(err.Error())
}
fmt.Println(firstName)
}
personsLastName, err := db.Query("SELECT last_name FROM persons WHERE id = 10")
if err !=nil {
panic(err.Error())
}
for persons.Next() {
var lastName string
err = results.Scan(&lastName)
if err !=nil {
panic(err.Error())
}
fmt.Println(lastName)
}
/***************************************************************/
/* ORM 抽象出 Persons 与数据库表的映射关系,然后直接用面向对象的方式访问 */
/***************************************************************/
type Persons struct {
gorm.Model
ID int64 `gorm:"primaryKey"`
FirstName string
LastName string
Phone string
Birth string
Age int32
}
orm, err := gorm.Open(mysql.Open("root:<password>@tcp(127.0.0.1:3306)/123begin"), &gorm.Config{
Logger: logger.Default.LogMode(logger.Silent),
})
var persons Persons
orm.Limit(1).Find(&persons, Persons{ID: 10})
fmt.Println(persons.FirstName)
fmt.Println(persons.LastName)
什么是 GORM
GORM 是专门针对 Go 语言的 ORM 库,以下是它的官方介绍:
The fantastic ORM library for Golang aims to be developer friendly.
- Full-Featured ORM
- Associations (Has One, Has Many, Belongs To, Many To Many, Polymorphism, Single-table inheritance)
- Hooks (Before/After Create/Save/Update/Delete/Find)
- Eager loading with Preload, Joins
- Transactions, Nested Transactions, Save Point, RollbackTo to Saved Point
- Context, Prepared Statement Mode, DryRun Mode
- Batch Insert, FindInBatches, Find/Create with Map, CRUD with SQL Expr and Context Valuer
- SQL Builder, Upsert, Locking, Optimizer/Index/Comment Hints, Named Argument, SubQuery
- Composite Primary Key, Indexes, Constraints
- Auto Migrations
- Logger
- Extendable, flexible plugin API: Database Resolver (Multiple Databases, Read/Write Splitting) / Prometheus…
- Every feature comes with tests
- Developer Friendly
GORM 原理研究
ORM 的原理就是在数据库和编程语言中 object(interface) 实例中间抽象出一个 ORM 层,ORM 层将 object(interface) 的字段与关系型数据库的库表进行映射,然后由 ORM 完成数据持久化的事情。如下图所示,红色线条表示传统的直接访问数据库做数据持久化的方式,蓝色线表示通过 ORM 层访问数据库做数据持久化的方式:
GORM 则是为了 Go 语言实现的 ORM,下面通过源码阅读来理解 GORM 的技术细节。
GORM 样例代码
以下是 GORM overview 里的一段样例代码,将 Product 结构体与 SQLite 数据库进行映射,可以实现 Product 实例 Product{Code: "D42", Price: 100}
的「增删改查」:
package main
import (
"gorm.io/gorm"
"gorm.io/driver/sqlite"
)
type Product struct {
gorm.Model
Code string
Price uint
}
func main() {
db, err := gorm.Open(sqlite.Open("test.db"), &gorm.Config{})
if err != nil {
panic("failed to connect database")
}
// Migrate the schema
db.AutoMigrate(&Product{})
// Create
db.Create(&Product{Code: "D42", Price: 100})
// Read
var product Product
db.First(&product, 1) // find product with integer primary key
db.First(&product, "code = ?", "D42") // find product with code D42
// Update - update product's price to 200
db.Model(&product).Update("Price", 200)
// Update - update multiple fields
db.Model(&product).Updates(Product{Price: 200, Code: "F42"}) // non-zero fields
db.Model(&product).Updates(map[string]interface{}{"Price": 200, "Code": "F42"})
// Delete - delete product
db.Delete(&product, 1)
}
GORM 源码阅读
首先 db, err := gorm.Open(sqlite.Open("test.db"), &gorm.Config{})
初始化一个 DB 对应的 GORM 实例,主要包括:
NamingStrategy
初始化用于将 Go struct 名称和各个 field 名称转换成数据库表的 table 和 column 名;callback
和Dialector
初始化,注册了默认的 callbacks 用于调用和执行数据库的操作;
// Open initialize db session based on dialector
func Open(dialector Dialector, opts ...Option) (db *DB, err error) {
/*****************/
/* 此处省略若干代码 */
/*****************/
if config.NamingStrategy == nil {
config.NamingStrategy = schema.NamingStrategy{IdentifierMaxLength: 64} // Default Identifier length is 64
}
/*****************/
/* 此处省略若干代码 */
/*****************/
db = &DB{Config: config, clone: 1}
db.callbacks = initializeCallbacks(db)
if config.ClauseBuilders == nil {
config.ClauseBuilders = map[string]clause.ClauseBuilder{}
}
if config.Dialector != nil {
err = config.Dialector.Initialize(db)
if err != nil {
if db, _ := db.DB(); db != nil {
_ = db.Close()
}
}
}
if config.PrepareStmt {
preparedStmt := NewPreparedStmtDB(db.ConnPool)
db.cacheStore.Store(preparedStmtDBKey, preparedStmt)
db.ConnPool = preparedStmt
}
db.Statement = &Statement{
DB: db,
ConnPool: db.ConnPool,
Context: context.Background(),
Clauses: map[string]clause.Clause{},
}
if err == nil && !config.DisableAutomaticPing {
if pinger, ok := db.ConnPool.(interface{ Ping() error }); ok {
err = pinger.Ping()
}
}
if err != nil {
config.Logger.Error(context.Background(), "failed to initialize database, got error %v", err)
}
return
}
其次看一下 db.Create(&Product{Code: "D42", Price: 100})
操作,主要包括:
stmt.Parse(stmt.Model)
解析 Product(Model) 转换它成数据库表以及对应 SQL 语句;db.Dialector.Explain(sql, vars...)
执行解析之后的 SQL 语句;
func (db *DB) Create(value interface{}) (tx *DB) {
if db.CreateBatchSize > 0 {
return db.CreateInBatches(value, db.CreateBatchSize)
}
tx = db.getInstance()
tx.Statement.Dest = value
return tx.callbacks.Create().Execute(tx)
}
func (p *processor) Execute(db *DB) *DB {
/*****************/
/* 此处省略若干代码 */
/*****************/
// parse model values
if stmt.Model != nil {
if err := stmt.Parse(stmt.Model); err != nil && (!errors.Is(err, schema.ErrUnsupportedDataType) || (stmt.Table == "" && stmt.TableExpr == nil && stmt.SQL.Len() == 0)) {
if errors.Is(err, schema.ErrUnsupportedDataType) && stmt.Table == "" && stmt.TableExpr == nil {
db.AddError(fmt.Errorf("%w: Table not set, please set it like: db.Model(&user) or db.Table(\"users\")", err))
} else {
db.AddError(err)
}
}
}
/*****************/
/* 此处省略若干代码 */
/*****************/
if stmt.SQL.Len() > 0 {
db.Logger.Trace(stmt.Context, curTime, func() (string, int64) {
sql, vars := stmt.SQL.String(), stmt.Vars
if filter, ok := db.Logger.(ParamsFilter); ok {
sql, vars = filter.ParamsFilter(stmt.Context, stmt.SQL.String(), stmt.Vars...)
}
return db.Dialector.Explain(sql, vars...), db.RowsAffected
}, db.Error)
}
if !stmt.DB.DryRun {
stmt.SQL.Reset()
stmt.Vars = nil
}
if resetBuildClauses {
stmt.BuildClauses = nil
}
return db
}
最后看一下 Parse
的实现细节,主要包括:
schema.ParseWithSpecialTableName(value, stmt.DB.cacheStore, stmt.DB.NamingStrategy, specialTableName)
将value(Product)
解析映射关系并保存在Schema
结构体中;stmt.Quote(stmt.Schema.Table)
按照Schema
中的映射关系以及调用的callback(clause)
生成 SQL 字符串;
func (stmt *Statement) Parse(value interface{}) (err error) {
return stmt.ParseWithSpecialTableName(value, "")
}
func (stmt *Statement) ParseWithSpecialTableName(value interface{}, specialTableName string) (err error) {
if stmt.Schema, err = schema.ParseWithSpecialTableName(value, stmt.DB.cacheStore, stmt.DB.NamingStrategy, specialTableName); err == nil && stmt.Table == "" {
if tables := strings.Split(stmt.Schema.Table, "."); len(tables) == 2 {
stmt.TableExpr = &clause.Expr{SQL: stmt.Quote(stmt.Schema.Table)}
stmt.Table = tables[1]
return
}
stmt.Table = stmt.Schema.Table
}
return err
}
通过 Create
的源码分析可以清楚的了解 GORM 是如何实现 struct 到 database 映射解析以及 SQL 执行的,其他操作 Update
、Delete
、Select
等也是类似的。另外作为 ORM 框架,GORM 还提供了很多自定义的能力,主要解决特殊场景的需求,例如复杂 SQL 操作等,主要是通过注册 Callback 和开发 Plugin:
/*
* register callback to extend GORM
*/
func cropImage(db *gorm.DB) {
if db.Statement.Schema != nil {
// crop image fields and upload them to CDN, dummy code
for _, field := range db.Statement.Schema.Fields {
switch db.Statement.ReflectValue.Kind() {
case reflect.Slice, reflect.Array:
for i := 0; i < db.Statement.ReflectValue.Len(); i++ {
// Get value from field
if fieldValue, isZero := field.ValueOf(db.Statement.Context, db.Statement.ReflectValue.Index(i)); !isZero {
if crop, ok := fieldValue.(CropInterface); ok {
crop.Crop()
}
}
}
case reflect.Struct:
// Get value from field
if fieldValue, isZero := field.ValueOf(db.Statement.Context, db.Statement.ReflectValue); !isZero {
if crop, ok := fieldValue.(CropInterface); ok {
crop.Crop()
}
}
// Set value to field
err := field.Set(db.Statement.Context, db.Statement.ReflectValue, "newValue")
}
}
// All fields for current model
db.Statement.Schema.Fields
// All primary key fields for current model
db.Statement.Schema.PrimaryFields
// Prioritized primary key field: field with DB name `id` or the first defined primary key
db.Statement.Schema.PrioritizedPrimaryField
// All relationships for current model
db.Statement.Schema.Relationships
// Find field with field name or db name
field := db.Statement.Schema.LookUpField("Name")
// processing
}
}
// Register the callback for the Create operation
db.Callback().Create().Register("crop_image", cropImage)
/*
* register plugin to extend GORM
*/
// Example of registering a plugin
db.Use(MyCustomPlugin{})
// Access a registered plugin by its name
plugin := db.Config.Plugins[pluginName]
// Registering the Prometheus plugin
db.Use(prometheus.New(prometheus.Config{
// Configuration options here
}))
GORM 竞品
Go 支持 ORM 有很多框架实现,没有太多时间做竞品分析,我查到了一个 ORM for Go 的列表(数据资料源自 ORMs for Go, most starred on GitHub.):
Project Name | Stars | Forks | Open Issues | Description | Last Update |
---|---|---|---|---|---|
gorm | 34675 | 3844 | 293 | The fantastic ORM library for Golang, aims to be developer friendly | 2024-01-27 23:26:55 |
beego | 30603 | 5702 | 16 | beego is an open-source, high-performance web framework for the Go programming language. | 2024-01-27 23:47:19 |
sqlx | 14912 | 1063 | 346 | general purpose extensions to golang's database/sql | 2024-01-28 00:27:48 |
ent | 14553 | 918 | 412 | An entity framework for Go | 2024-01-27 18:22:58 |
sqlc | 9904 | 673 | 251 | Generate type-safe code from SQL | 2024-01-27 22:31:10 |
xorm | 6659 | 766 | 307 | Simple and Powerful ORM for Go, support mysql,postgres,tidb,sqlite3,mssql,oracle, Moved to https://gitea.com/xorm/xorm | 2024-01-26 05:39:47 |
sqlboiler | 6259 | 566 | 92 | Generate a Go ORM tailored to your database schema. | 2024-01-27 22:31:18 |
pg | 5524 | 406 | 115 | Golang ORM with focus on PostgreSQL features and performance | 2024-01-27 18:12:36 |
gorp | 3704 | 413 | 146 | Go Relational Persistence - an ORM-ish library for Go | 2024-01-26 05:05:26 |
xo | 3514 | 313 | 41 | Command line tool to generate idiomatic Go code for SQL databases supporting PostgreSQL, MySQL, SQLite, Oracle, and Microsoft SQL Server | 2024-01-27 17:31:08 |
db | 3437 | 243 | 153 | Data access layer for PostgreSQL, CockroachDB, MySQL, SQLite and MongoDB with ORM-like features. | 2024-01-26 23:43:36 |
bun | 2719 | 176 | 139 | SQL-first Golang ORM | 2024-01-27 03:50:17 |
gormt | 2297 | 376 | 56 | database to golang struct | 2024-01-26 10:06:19 |
prisma-client-go | 1821 | 92 | 96 | Prisma Client Go is an auto-generated and fully type-safe database client | 2024-01-26 18:27:25 |
jet | 1786 | 101 | 39 | Type safe SQL builder with code generation and automatic query result data mapping | 2024-01-27 01:48:42 |
reform | 1429 | 73 | 86 | A better ORM for Go, based on non-empty interfaces and code generation. | 2024-01-26 09:56:56 |
pop | 1391 | 247 | 95 | A Tasty Treat For All Your Database Needs | 2024-01-26 10:02:10 |
go-sqlbuilder | 1154 | 106 | 6 | A flexible and powerful SQL string builder library plus a zero-config ORM. | 2024-01-26 10:01:53 |
go-queryset | 717 | 72 | 20 | 100% type-safe ORM for Go (Golang) with code generation and MySQL, PostgreSQL, Sqlite3, SQL Server support. GORM under the hood. | 2024-01-10 01:31:59 |
rel | 711 | 59 | 26 | :gem: Modern ORM for Golang - Testable, Extendable and Crafted Into a Clean and Elegant API | 2024-01-27 22:30:48 |
qbs | 548 | 101 | 10 | QBS stands for Query By Struct. A Go ORM. | 2024-01-24 06:27:27 |
bob | 520 | 25 | 10 | SQL query builder and ORM/Factory generator for Go with support for PostgreSQL, MySQL and SQLite | 2024-01-27 19:15:37 |
zoom | 304 | 28 | 2 | A blazing-fast datastore and querying engine for Go built on Redis. | 2024-01-04 19:38:09 |
pggen | 247 | 22 | 18 | Generate type-safe Go for any Postgres query. If Postgres can run the query, pggen can generate code for it. | 2024-01-17 12:32:25 |
grimoire | 160 | 18 | 0 | Database access layer for golang | 2023-09-25 03:44:37 |
GoBatis | 118 | 17 | 1 | An easy ORM tool for Golang, support MyBatis-Like XML template SQL | 2023-12-12 08:07:15 |
go-store | 112 | 9 | 1 | A simple and fast Redis backed key-value store library for Go | 2023-09-25 03:42:25 |
marlow | 82 | 7 | 2 | golang generator for type-safe sql api constructs | 2024-01-25 13:28:04 |
beeorm | 55 | 8 | 0 | Golang ORM | 2024-01-09 19:00:44 |
go-firestorm | 47 | 8 | 0 | Simple Go ORM for Google/Firebase Cloud Firestore | 2023-09-25 03:41:53 |
lore | 14 | 3 | 0 | Light Object-Relational Environment (LORE) provides a simple and lightweight pseudo-ORM/pseudo-struct-mapping environment for Go | 2023-09-25 08:03:17 |
裸写 SQL V.S. GORM
相比裸写 SQL,使用 GORM:
优点
与传统的数据库访问技术相比,ORM 有以下优点:
- 开发效率更高
- 使开发更加对象化,数据访问更抽象、轻便
- 支持面向对象封装
- 可移植
- 可以很方便地引入数据缓存之类的附加功能
缺点
- 自动化进行关系数据库的映射需要消耗系统性能,降低程序的执行效率
- 思维固定化,一些通过写 SQL 进行优化的技巧没法进行运用
- 采用 ORM 一般都是多层系统,系统的层次多了,效率就会降低
- ORM 所生成的代码一般不太可能写出很高效的算法,同时有可能会被误用,主要体现在对持久对象的提取和和数据的加工处理上,很有可能错误地将全部数据提取到内存对象中然后再进行过滤和加工处理