1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
|
# AutereDB
AutereDB is an educational endeavor in implementing a log-structured database engine with a focus on simplicity, understandability and performance.
AutereDB has the following features:
- Log-structured single-table storage, based on a durable append-only log
- In-memory indexes for fast lookups (primary and secondary)
- Log rotation and compaction for efficient storage even with larger databases
- Multiple concurrent readers and a single writer, using filesystem locks for synchronization
- Simple data types: `Int`, `Decimal`, `String`, `Bytes` (arbitrary bytestring), and `Null`
- A Rust API for interacting with the database, as well as Python bindings for the Rust API
- Transactions based on eager exclusive locking
- Batch read operations for improved performance
AutereDB does not support:
- Authentication or authorization in any capacity
- Multiple tables
- Type checking or schema evolution. These are outsourced to the application layer.
For production workloads, AutereDB would benefit from a porcelain layer that provides features such as query language, networking, and monitoring. AutereDB does not come with such a layer. See the [ARCHITECTURE.md](ARCHITECTURE.md) document for more details on the design and implementation.
## Inspiration
The most significant sources of inspiration for AutereDB are:
- [SQLite](https://www.sqlite.org/index.html) for its filesystem storage and
locking mechanisms.
- [Designing Data-Intensive Applications (book)](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/)
for its excellent overview of database internals and in-depth analysis of log-structured storage.
AutereDB is heavily based on the design outlined in chapter 3.
## In this repository
- `autere_db`: The core Rust library
- `py_bindings`: Python bindings for the Rust library API
- `demo`: A simple Python chat app that demonstrates the usage of AutereDB
## Usage in Rust
Add AutereDB as a dependency in your `Cargo.toml`.
```toml
[dependencies]
autere_db = { git = "https://github.com/jantuomi/autere_db.git" }
```
Then use it in your code like so:
```rust
fn example() -> DBResult<()> {
// Initialize the database
let mut db = DB::configure()
// Set the directory where the database files are stored.
.data_dir("data")
// Select the database fields, i.e. columns.
.fields(vec![Field::Id, Field::Name])
// Select the primary key field.
.primary_key(Field::Id)
// Select the secondary key fields. All queries must be
// based on the primary key or secondary keys.
.secondary_keys(vec![Field::Name])
// Define the conversion functions between the data type and database values.
.from_record(Inst::from_record)
.into_record(Inst::into_record)
// Finish the builder pattern and initialize the database.
.initialize()?;
// Insert or update the record based on the primary key
db.upsert(Inst {
id: 1,
name: Some("Alice".to_string()),
})?;
// Get the record by primary key
let found = db.get(&Value::Int(1))?;
...
}
```
With `Inst` etc. being defined like so:
```rust
use autere_db::*;
// Define a type that represents your fields (columns)
#[derive(Eq, PartialEq, Clone, Debug)]
enum Field {
Id,
Name,
}
// Define your data type that represents a database row
struct Inst {
pub id: i64,
pub name: Option<String>,
}
impl Inst {
// Describe how to convert the data type to a vector of `Value`s
fn into_record(self) -> Vec<Value> {
vec![
Value::Int(self.id),
match self.name {
Some(name) => Value::String(name),
None => Value::Null,
},
]
}
// Similarly, describe how to convert a vector of database values to the data type
fn from_record(record: Vec<Value>) -> Self {
let mut it = record.into_iter();
let inst = Inst {
id: match it.next().unwrap() {
Value::Int(id) => id,
other => panic!("Invalid value type: {:?}", other),
},
name: match it.next().unwrap() {
Value::String(name) => Some(name),
Value::Null => None,
other => panic!("Invalid value type: {:?}", other),
},
};
assert_eq!(it.next(), None);
inst
}
}
```
## Tests
Run the tests with:
```sh
cargo test
```
Generate the benchmark reports with:
```sh
cargo bench
```
## Python bindings
To build the Python bindings, run:
```sh
cd py_bindings
python -m venv venv
. venv/bin/activate
pip install maturin
maturin develop # for the development version, or
maturin build --release # for the release version
```
Then you can use the Python bindings like so:
```python
from autere_db import DB, Value, Record
config = DB.configure() \
.data_dir("data") \
.fields(["id", "name"]) \
.primary_key("id") \
.secondary_keys(["name"]) \
.initialize()
db.upsert([Value.int(10)])
# You can unwrap a database value like so:
db_value = Value.string("foo")
python_value = db_value.as_string()
```
## Copyright and license
AutereDB is licensed under the Apache License, Version 2.0. © 2024 Jan Tuomi.
|
