]> git.mikk.net Git - mtbl-rs/log
mtbl-rs
7 months agoLots of WIP on new lending "Cursor" abstraction
Chris Mikkelson [Fri, 2 May 2025 05:42:52 +0000 (00:42 -0500)]
Lots of WIP on new lending "Cursor" abstraction

Added merger of cursors and of duplicates. IntoIterator implementations
for Cursors which maintain a local copy of the value modified to use
take() instead of .clone()ing values. Getting closer to zero copy.

Merger uses Ord wrappers on the cursors themselves to provide duplicate
sorting or unsorting, which removes the need to copy values to wrap them
in an ordering wrapper.

Overall, code is getting simpler with fewer copies, a win-win.

7 months agoAdd "Cursor" abstraction for lending iterator.
Chris Mikkelson [Thu, 1 May 2025 22:44:51 +0000 (17:44 -0500)]
Add "Cursor" abstraction for lending iterator.

Uses immutable get() method to return borrowed key, value pair, mostly
to avoid copying keys to provide sorting in merger.

7 months agoRearranged source code to be more granular
Chris Mikkelson [Thu, 17 Apr 2025 21:44:46 +0000 (16:44 -0500)]
Rearranged source code to be more granular

Also made a bunch of other changes

7 months agoRefactor, again (WIP)
Chris Mikkelson [Wed, 16 Apr 2025 05:06:47 +0000 (00:06 -0500)]
Refactor, again (WIP)

Break up some larger files into smaller chunks, also decouple Iter
and Iterator again, in favor of implementing IntoIterator for
Iter implementations.

Still a few remaining nits to work out -- the IntoIterator trait
isn't coming into scope where I expect it, so may need to back
off of `impl Iter` in favor of an associated type on Source which
requires IntoIterator.

7 months agoAdd map capability to Source, SeekableIter back-to-impl master
Chris Mikkelson [Wed, 9 Apr 2025 19:51:06 +0000 (14:51 -0500)]
Add map capability to Source, SeekableIter

7 months agoCorrect RangeIter trait bounds
Chris Mikkelson [Wed, 9 Apr 2025 19:50:22 +0000 (14:50 -0500)]
Correct RangeIter trait bounds

Like get_range(), only PartialOrd<[u8]> is required, not AsRef<[u8]>

7 months agoImplement dyn-compatible source variant.
Chris Mikkelson [Tue, 8 Apr 2025 20:48:07 +0000 (15:48 -0500)]
Implement dyn-compatible source variant.

With appropriate boxing, allows creation of dynamically-dispatched
sources for heterogeneous collections of sources.

7 months agoRefactor again to take advantage of impl Trait.
Chris Mikkelson [Tue, 8 Apr 2025 18:13:17 +0000 (13:13 -0500)]
Refactor again to take advantage of impl Trait.

Will pursue mixed mergers through methods other than direct dyn
Compatibility of Source trait.

7 months agoAddress div_ceil clippy warnings dyn-source
Chris Mikkelson [Tue, 8 Apr 2025 05:16:33 +0000 (00:16 -0500)]
Address div_ceil clippy warnings

7 months agoReduce lifetime proliferation.
Chris Mikkelson [Tue, 8 Apr 2025 05:14:43 +0000 (00:14 -0500)]
Reduce lifetime proliferation.

Makes Source trait non-dyn-compatible again, but an improvement in
reducing lifetimes and function generics.

7 months agoWIP #2 - Remove Item associated type from Source
Chris Mikkelson [Mon, 7 Apr 2025 19:39:00 +0000 (14:39 -0500)]
WIP #2 - Remove Item associated type from Source

7 months agoWIP attempt to make dyn compatible source abstraction.
Chris Mikkelson [Mon, 7 Apr 2025 18:43:59 +0000 (13:43 -0500)]
WIP attempt to make dyn compatible source abstraction.

7 months agoStreamline sorter 'write' method again.
Chris Mikkelson [Mon, 7 Apr 2025 01:26:29 +0000 (20:26 -0500)]
Streamline sorter 'write' method again.

7 months agoRename merge_func, dupsort_func sources, iterators.
Chris Mikkelson [Mon, 7 Apr 2025 01:16:58 +0000 (20:16 -0500)]
Rename merge_func, dupsort_func sources, iterators.

8 months agocargo-clippy fixes
Chris Mikkelson [Sat, 5 Apr 2025 22:56:59 +0000 (17:56 -0500)]
cargo-clippy fixes

8 months agoRefactor to allow for dyn-compatible Source, iteration over arbitrary
Chris Mikkelson [Sat, 5 Apr 2025 22:26:47 +0000 (17:26 -0500)]
Refactor to allow for dyn-compatible Source, iteration over arbitrary
types.

8 months agoUpdate .gitignore
Chris Mikkelson [Thu, 3 Apr 2025 21:24:35 +0000 (16:24 -0500)]
Update .gitignore

8 months agoRename Iter -> SeekableIter
Chris Mikkelson [Thu, 3 Apr 2025 21:23:16 +0000 (16:23 -0500)]
Rename Iter -> SeekableIter

13 months agoMake Iter, Source generic on item type
Chris Mikkelson [Sat, 19 Oct 2024 12:22:33 +0000 (14:22 +0200)]
Make Iter, Source generic on item type

This is to allow mapping items to higher level types while preserving seek
functionality.

14 months agoFix Fileset bugs
Chris Mikkelson [Tue, 17 Sep 2024 22:14:38 +0000 (22:14 +0000)]
Fix Fileset bugs

Handle relative paths in fileset file, start last_reload at 0/epoch

14 months agoAdd Debug derives to mtbl::{Fileset,Reader}
Chris Mikkelson [Tue, 17 Sep 2024 22:13:34 +0000 (22:13 +0000)]
Add Debug derives to mtbl::{Fileset,Reader}

14 months agoVarious API tweaks
Chris Mikkelson [Mon, 16 Sep 2024 05:17:58 +0000 (00:17 -0500)]
Various API tweaks

Avoid need to import memmap, expose fileset

14 months agoAdd filter with seek capability
Chris Mikkelson [Fri, 13 Sep 2024 17:58:13 +0000 (12:58 -0500)]
Add filter with seek capability

14 months agoSimplify merge_func, dupsort helpers
Chris Mikkelson [Fri, 13 Sep 2024 17:55:46 +0000 (12:55 -0500)]
Simplify merge_func, dupsort helpers

Add the iterator wrapper methods to the Iter trait, allowing
the source implementations to apply them to the inner iterator
rather than constructing the implementation structs directly.

14 months agoHide inner Entry structure refactor-source
Chris Mikkelson [Thu, 12 Sep 2024 23:37:56 +0000 (18:37 -0500)]
Hide inner Entry structure

14 months agoRefactor fileset
Chris Mikkelson [Thu, 12 Sep 2024 23:36:50 +0000 (18:36 -0500)]
Refactor fileset

Implement lazy loading on first use and error reporting.

14 months agoMultiple cleanups; flesh out sorter logic
Chris Mikkelson [Fri, 6 Sep 2024 21:31:07 +0000 (16:31 -0500)]
Multiple cleanups; flesh out sorter logic

Give merge function a mutable reference to data only, to prevent
accidental mangling of keys.

Implement source on Vec<Entry> for sorter; migrate tests eventually.

14 months agoMore cleanup from "impl Iter" based sources
Chris Mikkelson [Fri, 6 Sep 2024 21:23:57 +0000 (16:23 -0500)]
More cleanup from "impl Iter" based sources

Not having to explicitly name the iterator return types saves
the need to name the returned iterator's reference lifetimes, if any.
This removes the need to implement Source on a reference to a type
in order to assign `&self` a lifetime.

Once again, we can implment Source on an owned type reliably, saving
the need for pointer indirections.

14 months agoRefactor Source without object safety
Chris Mikkelson [Fri, 6 Sep 2024 05:41:24 +0000 (00:41 -0500)]
Refactor Source without object safety

14 months agoWIP: attempt to refactor Source without object safety
Chris Mikkelson [Fri, 6 Sep 2024 00:24:59 +0000 (19:24 -0500)]
WIP: attempt to refactor Source without object safety

15 months agoWIP: sorter implementation
Chris Mikkelson [Wed, 4 Sep 2024 19:30:02 +0000 (14:30 -0500)]
WIP: sorter implementation

15 months agoblock_builder: use Vec::new() instead of Vec::default()
Chris Mikkelson [Mon, 2 Sep 2024 05:40:59 +0000 (00:40 -0500)]
block_builder: use Vec::new() instead of Vec::default()

15 months agoblock_builder: remove last bit of custom varint code
Chris Mikkelson [Mon, 2 Sep 2024 05:40:52 +0000 (00:40 -0500)]
block_builder: remove last bit of custom varint code

15 months agoAdd merge_func and dupsort_func methods to Source
Chris Mikkelson [Mon, 2 Sep 2024 05:28:53 +0000 (00:28 -0500)]
Add merge_func and dupsort_func methods to Source

15 months agoMore refactoring, natch.
Chris Mikkelson [Sat, 17 Aug 2024 22:07:12 +0000 (17:07 -0500)]
More refactoring, natch.

Replace DefaultSource with generic implementation on IterSource.

Ditch "prelude" import -- IterSource no longer required, importing
mtbl::Source is sufficient to use Source trait.

15 months agofirst cut of fileset support
Chris Mikkelson [Fri, 16 Aug 2024 15:47:59 +0000 (10:47 -0500)]
first cut of fileset support

15 months agoRemove blanket dead code warning suppression
Chris Mikkelson [Fri, 16 Aug 2024 15:47:36 +0000 (10:47 -0500)]
Remove blanket dead code warning suppression

15 months agoreader: support uncompression
Chris Mikkelson [Tue, 13 Aug 2024 00:37:33 +0000 (19:37 -0500)]
reader: support uncompression

15 months agoWeave CBuf into blocks for compression
Chris Mikkelson [Tue, 13 Aug 2024 00:30:07 +0000 (19:30 -0500)]
Weave CBuf into blocks for compression

15 months agoMake compression CBuf type generic
Chris Mikkelson [Tue, 13 Aug 2024 00:04:30 +0000 (19:04 -0500)]
Make compression CBuf type generic

15 months agoWrite index block with length, crc
Chris Mikkelson [Mon, 12 Aug 2024 19:21:14 +0000 (14:21 -0500)]
Write index block with length, crc

Remove dead code

15 months agoAdd file reader, rudimentary mtbl_dump
Chris Mikkelson [Thu, 8 Aug 2024 22:39:10 +0000 (16:39 -0600)]
Add file reader, rudimentary mtbl_dump

15 months agotests: use small blocksize in rwtest to exercise index
Chris Mikkelson [Thu, 8 Aug 2024 22:38:36 +0000 (16:38 -0600)]
tests: use small blocksize in rwtest to exercise index

15 months agoblock: use little-endian fixed representations
Chris Mikkelson [Thu, 8 Aug 2024 12:56:25 +0000 (06:56 -0600)]
block: use little-endian fixed representations

Consistent with C mtbl

15 months agoreader: handle seeking past end of index block
Chris Mikkelson [Thu, 8 Aug 2024 12:38:07 +0000 (06:38 -0600)]
reader: handle seeking past end of index block

15 months agowriter: fix logic error in offset tracking
Chris Mikkelson [Thu, 8 Aug 2024 12:37:16 +0000 (06:37 -0600)]
writer: fix logic error in offset tracking

15 months agoRefactor block code for clean-up, tests, seek() fixes
Chris Mikkelson [Thu, 8 Aug 2024 05:39:45 +0000 (23:39 -0600)]
Refactor block code for clean-up, tests, seek() fixes

15 months agoincrease size of rwtest data
Chris Mikkelson [Thu, 8 Aug 2024 02:28:58 +0000 (20:28 -0600)]
increase size of rwtest data

15 months agoUpdate API layout to use prelude
Chris Mikkelson [Thu, 8 Aug 2024 00:12:38 +0000 (18:12 -0600)]
Update API layout to use prelude

Add test for range iter, add missing linear search step to
BlockIter seek

15 months agoreader: remove debug print and unused CRC decoding
Chris Mikkelson [Wed, 7 Aug 2024 22:49:41 +0000 (16:49 -0600)]
reader: remove debug print and unused CRC decoding

15 months agocargo-clippy lints
Chris Mikkelson [Wed, 7 Aug 2024 22:48:45 +0000 (16:48 -0600)]
cargo-clippy lints

15 months agoGive ReaderIter a Reader Clone for flexibility
Chris Mikkelson [Wed, 7 Aug 2024 22:13:17 +0000 (16:13 -0600)]
Give ReaderIter a Reader Clone for flexibility

15 months agoWIP: working reader, with test
Chris Mikkelson [Wed, 7 Aug 2024 06:53:10 +0000 (00:53 -0600)]
WIP: working reader, with test

Squashed several bugs, will have to squash `.expect()`s next.

15 months agoiter.rs: cargo fmt
Chris Mikkelson [Wed, 7 Aug 2024 04:54:08 +0000 (22:54 -0600)]
iter.rs: cargo fmt

15 months agowriter: Move index update to data block write
Chris Mikkelson [Wed, 7 Aug 2024 04:53:00 +0000 (22:53 -0600)]
writer: Move index update to data block write

Fixes missing index write on last block.

15 months agoRefactor reader code around Arc<impl AsRef<[u8]>>
Chris Mikkelson [Wed, 7 Aug 2024 04:50:22 +0000 (22:50 -0600)]
Refactor reader code around Arc<impl AsRef<[u8]>>

API currently named 'DataSlice' provides convenient views of
ranges / slices of the underlying buffer, avoiding lifetime
entaglements.

16 months agoexperimental first cut at reader -- not working
Chris Mikkelson [Tue, 6 Aug 2024 01:38:49 +0000 (19:38 -0600)]
experimental first cut at reader -- not working

16 months agowriter: write in-progress block
Chris Mikkelson [Mon, 5 Aug 2024 06:02:28 +0000 (00:02 -0600)]
writer: write in-progress block

16 months agometadata: read from file-like object instaed of buf
Chris Mikkelson [Mon, 5 Aug 2024 06:01:41 +0000 (00:01 -0600)]
metadata: read from file-like object instaed of buf

16 months agoFlesh out writer, block builder, requirements
Chris Mikkelson [Mon, 5 Aug 2024 05:14:49 +0000 (23:14 -0600)]
Flesh out writer, block builder, requirements

16 months agomove block result, error to main lib
Chris Mikkelson [Mon, 5 Aug 2024 05:14:17 +0000 (23:14 -0600)]
move block result, error to main lib

16 months agoadd metadata, compression
Chris Mikkelson [Mon, 5 Aug 2024 05:12:22 +0000 (23:12 -0600)]
add metadata, compression

16 months agoRefactor block, block_builder for readability, magic numbers
Chris Mikkelson [Fri, 2 Aug 2024 23:42:45 +0000 (18:42 -0500)]
Refactor block, block_builder for readability, magic numbers

Fix "off by 4" bug caused by double call to block builder finish.

16 months agoBreak iter and source into separate modules.
chris mikkelson [Mon, 29 Jul 2024 19:00:38 +0000 (14:00 -0500)]
Break iter and source into separate modules.

16 months agosource.rs: Remove unneeded GenSource trait
chris mikkelson [Mon, 29 Jul 2024 18:45:20 +0000 (13:45 -0500)]
source.rs: Remove unneeded GenSource trait

Rename GenWrap to BoxWrap for clarity.

16 months agoRename GenericSource -> BoxedSource
Chris Mikkelson [Sun, 28 Jul 2024 18:23:12 +0000 (13:23 -0500)]
Rename GenericSource -> BoxedSource

16 months agoFlesh out 'generic merger' test
Chris Mikkelson [Wed, 24 Jul 2024 18:22:46 +0000 (13:22 -0500)]
Flesh out 'generic merger' test

16 months agoGeneric sources working after big refactor.
Chris Mikkelson [Tue, 23 Jul 2024 20:15:21 +0000 (15:15 -0500)]
Generic sources working after big refactor.

Root cause was Box<dyn Trait> defaults to Box<dyn Trait + 'static>,
leading to the 'static lifetime "infecting" the generated iterators,
causing the borrow to last beyond the end of the function which dropped
the box or its containing structure.

The quick fix was to inject and add an explicit lifetime to the Box.

While tuning that fix, opted to inject lifetimes by implementing
Source methods on references. This tracks more with the C mtbl API,
where:

m = mtbl_merger_init(mopt);
/* populate m */
mtbl_merger_source(m);

is roughly mirrored by:

let m = mtbl::Merger::from(...);
let source = &m;

16 months agogeneric sources working, sort of, in test
Chris Mikkelson [Mon, 22 Jul 2024 13:58:21 +0000 (08:58 -0500)]
generic sources working, sort of, in test

16 months agoWIP experiment dyn source
Chris Mikkelson [Mon, 22 Jul 2024 02:12:51 +0000 (21:12 -0500)]
WIP experiment dyn source

Lifetime making it hard to test, possibly implement at all

16 months agoRefactor Source trait hierarchy
Chris Mikkelson [Sun, 21 Jul 2024 00:40:23 +0000 (19:40 -0500)]
Refactor Source trait hierarchy

Source implements get/prefix/range, and is a subtrait of
IterSource which implements only iter(). The Default
implementation is now another subtrait of IterSource.

This avoids several of the previouas `Source<'a> + Ranges<'a>`
and 'Source<'a> + DefaultRanges' type constraints, and
simplifies type paths.

16 months agoGeneral cleanup, cargo-clippy fixes
Chris Mikkelson [Sat, 20 Jul 2024 02:02:01 +0000 (21:02 -0500)]
General cleanup, cargo-clippy fixes

16 months agopull out the non-working generic source bits
Chris Mikkelson [Sat, 20 Jul 2024 02:01:43 +0000 (21:01 -0500)]
pull out the non-working generic source bits

16 months agoWIP of attempt to make a generic source
Chris Mikkelson [Sat, 20 Jul 2024 01:56:22 +0000 (20:56 -0500)]
WIP of attempt to make a generic source

16 months agoAdd generic source wrapper for mixed mergers
Chris Mikkelson [Fri, 19 Jul 2024 20:02:20 +0000 (15:02 -0500)]
Add generic source wrapper for mixed mergers

16 months agofixed lifetime issue! ?
Chris Mikkelson [Fri, 19 Jul 2024 18:41:45 +0000 (13:41 -0500)]
fixed lifetime issue! ?

16 months agoWIP commit: attempt to make Source object safe
Chris Mikkelson [Fri, 19 Jul 2024 17:10:19 +0000 (12:10 -0500)]
WIP commit: attempt to make Source object safe

Use associated iterator types instead of `impl Iter`. Move
get, get_prefix, get_range to separate "Ranges" trait, with
a generic default implementation based on iter() enabled by
"DefaultRanges" trait.

WIP due to one pesky lifetime issue.

16 months agomerger: Remove all internal use of Box<dyn Iter>
Chris Mikkelson [Fri, 19 Jul 2024 03:28:00 +0000 (22:28 -0500)]
merger: Remove all internal use of Box<dyn Iter>

Adding Iter implementation for Box<dyn Iter> allows it
to be used as a concrete type in case of a mixed merger.

16 months agoAdd support for Boxed dyn Iter objects
Chris Mikkelson [Fri, 19 Jul 2024 03:27:11 +0000 (22:27 -0500)]
Add support for Boxed dyn Iter objects

16 months agomerger: remove Source paramater from Iter template.
Chris Mikkelson [Fri, 19 Jul 2024 01:02:20 +0000 (20:02 -0500)]
merger: remove Source paramater from Iter template.

16 months agoBeginnings of reader and writer
Chris Mikkelson [Tue, 16 Jul 2024 06:14:44 +0000 (01:14 -0500)]
Beginnings of reader and writer

16 months agoAnother refactor: back to mtbl C idioms
Chris Mikkelson [Sun, 14 Jul 2024 20:31:15 +0000 (15:31 -0500)]
Another refactor: back to mtbl C idioms

Iter (includes seek + iterator), static entry type, Source,
merger, etc.

17 months agomap values only; generic keys too complex
Chris Mikkelson [Mon, 17 Jun 2024 02:08:38 +0000 (21:08 -0500)]
map values only; generic keys too complex

19 months agoSeekable: relax Ord requirement on keys
Chris Mikkelson [Fri, 3 May 2024 05:54:15 +0000 (00:54 -0500)]
Seekable: relax Ord requirement on keys

This allows implementations to return Result<> types to propagate
errors. Those errors will need to be handled prior to merging
or coalescing.

19 months agoSeekableVec: relax Copy requirement to Clone
Chris Mikkelson [Mon, 29 Apr 2024 01:48:59 +0000 (20:48 -0500)]
SeekableVec: relax Copy requirement to Clone

19 months agoModify filter_map to only map values
Chris Mikkelson [Mon, 29 Apr 2024 01:44:00 +0000 (20:44 -0500)]
Modify filter_map to only map values

Mapping keys requires a conversion on seek from the
mapped key to the inner key. Since the keys are passed
to seek by reference to avoid moves, this complicates
consuming the key for a full conversion to the inner
key.

For now, restrict mapping to values only. This required
changing the filter function signature to take a reference
to the key so its caller could retain the key for returning.

19 months agocargo-clippy fixes
Chris Mikkelson [Sun, 28 Apr 2024 20:57:27 +0000 (15:57 -0500)]
cargo-clippy fixes

Remove redudant Some(foo()?).

Make PartialOrd more canonical.

19 months agoAdd coalesce (merge function) support
Chris Mikkelson [Sun, 28 Apr 2024 20:54:28 +0000 (15:54 -0500)]
Add coalesce (merge function) support

19 months agoSeekableVec: add value type.
Chris Mikkelson [Sun, 28 Apr 2024 20:46:14 +0000 (15:46 -0500)]
SeekableVec: add value type.

19 months agoRefactor imports for visibility.
Chris Mikkelson [Sun, 28 Apr 2024 19:18:23 +0000 (14:18 -0500)]
Refactor imports for visibility.

19 months agoMove SeekableVec to module for test support
Chris Mikkelson [Sun, 28 Apr 2024 18:52:14 +0000 (13:52 -0500)]
Move SeekableVec to module for test support

19 months agoRefactor to generic Seekable IntoIterator
Chris Mikkelson [Sun, 28 Apr 2024 18:43:04 +0000 (13:43 -0500)]
Refactor to generic Seekable IntoIterator

19 months agoRemove lender dependency
Chris Mikkelson [Sun, 28 Apr 2024 18:18:18 +0000 (13:18 -0500)]
Remove lender dependency

19 months agoformat iter.rs
Chris Mikkelson [Thu, 18 Apr 2024 01:16:10 +0000 (20:16 -0500)]
format iter.rs

19 months agoConvert AsRef template functions to use impl AsRef
Chris Mikkelson [Thu, 18 Apr 2024 01:15:48 +0000 (20:15 -0500)]
Convert AsRef template functions to use impl AsRef

19 months agoiter: Genericize repeated IntoIterator implementations
Chris Mikkelson [Fri, 12 Apr 2024 19:15:10 +0000 (14:15 -0500)]
iter: Genericize repeated IntoIterator implementations

Introduce 'Seekable<T: Entries>' wrapper to bring common
'return Iter<T>' logic to one place.

19 months agoEntries::filter_seek -> Entries::filter
Chris Mikkelson [Sun, 7 Apr 2024 20:04:53 +0000 (15:04 -0500)]
Entries::filter_seek -> Entries::filter

No longer need separate name due to Iterator co-implementation

19 months agoRename Iter to "entries" to avoid confusion with iterator.
Chris Mikkelson [Sun, 7 Apr 2024 20:01:41 +0000 (15:01 -0500)]
Rename Iter to "entries" to avoid confusion with iterator.

Entries implements IntoIterator.

19 months agoIter -> Entries, prep for Iter: IntoIterator
Chris Mikkelson [Sun, 7 Apr 2024 19:09:25 +0000 (14:09 -0500)]
Iter -> Entries, prep for Iter: IntoIterator

19 months agoiter: IterBox -> IterCell, private struct
Chris Mikkelson [Sun, 7 Apr 2024 18:37:03 +0000 (13:37 -0500)]
iter: IterBox -> IterCell, private struct