1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
|
# LogDB
An educational endeavor in implementing a log-structured database with a focus on simplicity, understandability and performance.
LogDB has the following features:
- Log-structured single-table storage, based on a durable append-only log
- In-memory indexes for fast lookups (primary and secondary)
- Log rotation and compaction for efficient storage even with larger databases
- Multiple concurrent readers and a single writer, using filesystem locks for synchronization
- Simple data types: `Int`, `Float`, `String`, `Bytes` (arbitrary bytestring), and `Null`
- A Rust API for interacting with the database, as well as Python bindings for the Rust API
LogDB does not support:
- Authentication or authorization in any capacity
- Multiple tables
- Schema evolution, other than adding new nullable fields
Possible future features:
- Transactions
See the [ARCHITECTURE.md](ARCHITECTURE.md) document for more details on the design and implementation of LogDB.
## Inspiration
The most significant sources of inspiration for LogDB are:
- [SQLite](https://www.sqlite.org/index.html) for its filesystem storage and
locking mechanisms.
- [Designing Data-Intensive Applications (book)](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/)
for its excellent overview of database internals and in-depth analysis of log-structured storage.
LogDB is heavily based on the design outlined in chapter 3.
## Usage in Rust
Add LogDB as a dependency in your `Cargo.toml`.
```toml
[dependencies]
log_db = { git = "https://github.com/jantuomi/log_db.git" }
```
Then use it in your code like so:
```rust
use log_db::*;
// Configure and initialize the database
let mut db = DB::configure()
.fields(vec![
(Field::Id, RecordField::int()),
(Field::Data, RecordField::bytes()),
])
.primary_key(Field::Id)
.initialize()?;
// Define a record matching the `fields` schema
let record = Record {
values: vec![
RecordValue::Int(1),
RecordValue::Bytes(vec![1, 2, 3, 4]),
],
};
// Insert or update the record based on the primary key (ID, first value)
db.upsert(&record)?;
// Get the record by primary key
let found = db.get(RecordValue::Int(1))?;
```
## Tests
Run the tests with:
```sh
cargo test
```
Generate the benchmark reports with:
```sh
cargo bench
```
## Python bindings
To build the Python bindings, run:
```sh
cd py_bindings
python -m venv venv
. venv/bin/activate
pip install maturin
maturin develop # for the development version, or
maturin build --release # for the release version
```
Then you can use the Python bindings like so:
```python
from log_db_py import DB, RecordValue, RecordField, Record
config = DB.configure();
config.primary_key = "id"
config.fields = [("id", RecordField.int().nullable())]
db = config.initialize()
db.upsert(Record(RecordValue.int(10)))
```
## Copyright and license
LogDB is licensed under the Apache License, Version 2.0. © 2024 Jan Tuomi.
|
