diff options
| -rw-r--r-- | README.md | 19 |
1 files changed, 13 insertions, 6 deletions
@@ -8,6 +8,8 @@ Ever wanted a zero-dependency, filesystem-backed, lock-free, durable, concurrent, retrying task queue implementation? No? +Note: This is pre-release software. Backwards compatibility will be guaranteed after v1.0. + ## Example ```python @@ -36,18 +38,23 @@ Or just chuck the `femtoqueue.py` file into your Python 3 project. There are no ## Features -This mini-library provides the `FemtoQueue` class with the standard queue interface: +This mini-library provides the `FemtoQueue` class with the standard queue interface, with some additions: -| Method | Description | -| :------------------------- | :---------------------------------- | -| `push(task: bytes) -> str` | Add a task to the queue, returns id | -| `pop() -> FemtoTask` | Get a task from the queue | +| Method | Description | +| :------------------------------------------- | :-------------------------------------- | +| `push(task: bytes) -> str` | Add a task to the queue, returns id | +| `pop() -> FemtoTask` | Get a task from the queue | +| `schedule(task: bytes, time_us: int) -> str` | Schedule a task for the given timestamp | +| `done(task: FemtoTask)` | Mark a task as done | +| `fail(task: FemtoTask)` | Mark a task as failed | Each task corresponds to one file in the `data_dir` directory. State changes are atomic since they use `rename()`. The task can contain whatever you want, the queue does not inspect it in any way. Each concurrent worker node (library user) must have a stable identifier `node_id`. This way workers can automatically retry a task if they unexpectedly crash in the middle of processing. -Stale tasks (i.e. in progress for too long) are moved back to `pending` automatically after a timeout is reached (default: 30s). +Scheduled tasks are moved to the back of the `pending` queue when the specified wall-clock time is less than the current wall-clock time. Stale tasks (i.e. in progress for too long) are moved back to `pending` automatically after a timeout is reached (default: 30s). These events are processed during a `pop()` call; the library does not run any background jobs by itself. + +Tasks are ordered using the system-provided monotonic clock to avoid issues such as NTP-related clock skew or daylight savings time moving the clock backwards in time, which would be possible if ordering was based on wall-clock time. As long as all worker nodes have the same clock source, pending tasks are guaranteed to be processed in insertion order, with microsecond precision. ## But isn't this slow? |