1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
|
# LogDB
An educational endeavor in implementing a log-structured database with a focus on simplicity, understandability and performance.
LogDB has the following features:
- Log-structured single-table storage, based on a durable append-only log
- In-memory indexes for fast lookups (primary and secondary)
- Log rotation and compaction for efficient storage even with larger databases
- Multiple concurrent readers and a single writer, using filesystem locks for synchronization
- Simple data types: `Int`, `Decimal`, `String`, `Bytes` (arbitrary bytestring), and `Null`
- A Rust API for interacting with the database, as well as Python bindings for the Rust API
- Transactions based on eager exclusive locking
- Batch read operations for improved performance
LogDB does not support:
- Authentication or authorization in any capacity
- Multiple tables
- Type checking or schema evolution. These are outsourced to the application layer.
See the [ARCHITECTURE.md](ARCHITECTURE.md) document for more details on the design and implementation of LogDB.
## Inspiration
The most significant sources of inspiration for LogDB are:
- [SQLite](https://www.sqlite.org/index.html) for its filesystem storage and
locking mechanisms.
- [Designing Data-Intensive Applications (book)](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/)
for its excellent overview of database internals and in-depth analysis of log-structured storage.
LogDB is heavily based on the design outlined in chapter 3.
## Usage in Rust
Add LogDB as a dependency in your `Cargo.toml`.
```toml
[dependencies]
log_db = { git = "https://github.com/jantuomi/log_db.git" }
```
Then use it in your code like so:
```rust
use log_db::*;
// Define a type that represents your fields (columns)
#[derive(Eq, PartialEq, Clone, Debug)]
enum Field {
Id,
Name,
}
// Define your data type that represents a database row
struct Inst {
pub id: i64,
pub name: Option<String>,
}
impl Inst {
// Describe how to convert the data type to a vector of `Value`s
fn into_record(self) -> Vec<Value> {
vec![
Value::Int(self.id),
match self.name {
Some(name) => Value::String(name),
None => Value::Null,
},
]
}
// Similarly, describe how to convert a vector of database values to the data type
fn from_record(record: Vec<Value>) -> Self {
let mut it = record.into_iter();
let inst = Inst {
id: match it.next().unwrap() {
Value::Int(id) => id,
other => panic!("Invalid value type: {:?}", other),
},
name: match it.next().unwrap() {
Value::String(name) => Some(name),
Value::Null => None,
other => panic!("Invalid value type: {:?}", other),
},
};
assert_eq!(it.next(), None);
inst
}
}
fn example() -> DBResult<()> {
// Initialize the database
let mut db = DB::configure()
// Set the directory where the database files are stored.
.data_dir("data")
// Select the database fields, i.e. columns.
.fields(vec![Field::Id, Field::Name])
// Select the primary key field.
.primary_key(Field::Id)
// Select the secondary key fields. All queries must be
// based on the primary key or secondary keys.
.secondary_keys(vec![Field::Name])
// Define the conversion functions between the data type and database values.
.from_record(Inst::from_record)
.into_record(Inst::into_record)
// Finish the builder pattern and initialize the database.
.initialize()?;
// Insert or update the record based on the primary key
db.upsert(Inst {
id: 1,
name: Some("Alice".to_string()),
})?;
// Get the record by primary key
let found = db.get(&Value::Int(1))?;
...
}
```
## Tests
Run the tests with:
```sh
cargo test
```
Generate the benchmark reports with:
```sh
cargo bench
```
## Python bindings
To build the Python bindings, run:
```sh
cd py_bindings
python -m venv venv
. venv/bin/activate
pip install maturin
maturin develop # for the development version, or
maturin build --release # for the release version
```
Then you can use the Python bindings like so:
```python
from log_db_py import DB, Value, Record
config = DB.configure() \
.data_dir("data") \
.fields(["id", "name"]) \
.primary_key("id") \
.secondary_keys(["name"]) \
.initialize()
db.upsert([Value.int(10)])
```
## Copyright and license
LogDB is licensed under the Apache License, Version 2.0. © 2024 Jan Tuomi.
|
