A domain-agnostic R6 class that provides an interface to a folder-based data storage system. An in-memory DuckDB instance attaches a DuckLake catalog, so that writes are atomic and readers see snapshot-isolated data even while other processes are writing. This class can be inherited by domain-specific database classes.
By default a database is one self-contained folder: the catalog lives in SQLite at
<path>/catalog.sqlite and the parquet data files at <path>/data/. Passing catalog and/or
data_path puts either half somewhere else, such as a shared PostgreSQL catalog or an S3 bucket.
Public fields
connectionDBI connection object to an in-memory DuckDB database
catalogCharacter string, the DuckLake catalog connection, e.g.
"sqlite:<path>/catalog.sqlite"data_pathCharacter string, where DuckLake writes its data files
read_onlyIf true, the catalog is attached read-only
retry_timeoutNumeric, how long in seconds to keep retrying a contended catalog operation before giving up.
retry_waitNumeric, base wait in seconds for the retry backoff, which doubles per attempt up to 64 times this
retry_maxInteger, maximum number of attempts in case something retries faster than
retry_timeoutcan measure.
Methods
ducklake_db$new()
Initialize a new ducklake_db object
Usage
ducklake_db$new(
path = NULL,
read_only = FALSE,
extensions = character(0),
catalog = NULL,
data_path = NULL,
sqlite_journal_mode = "wal",
expire_older_than = NULL,
delete_older_than = NULL
)Arguments
pathCharacter string. Path to the data folder. Only needed for the halves of the database that are derived from it, so it may be omitted when both
cataloganddata_pathare given.read_onlyLogical. If true, the catalog is attached read-only.
extensionsCharacter vector of additional DuckDB extensions to load
catalogCharacter string. DuckLake catalog connection; defaults to SQLite at
<path>/catalog.sqlite. Anything DuckLake accepts works, e.g."postgres:dbname=evoland host=...". See Choosing a catalog databasedata_pathCharacter string. Where DuckLake writes its data files; defaults to
<path>/data/. May be remote, e.g."s3://bucket/prefix/". See Choosing storagesqlite_journal_modeCharacter string, the journal mode to put a SQLite catalog in;
walby default, because the default rollback journal lets one writer block every other process's reads of the catalog.NULLleaves the catalog as it is. Ignored for catalogs that are not SQLite or when attaching read-only. Readers adapt to whatever journal mode they find.expire_older_than, delete_older_thanRetention for
$checkpoint(), as an interval DuckDB reads, e.g."7 days": how old a snapshot must be before it is expired, and how old an unreferenced file must be before it is deleted. Stored in the catalog but can be overwritten by passing. Default to NULL, discarding nothing.
ducklake_db$execute()
Execute a SQL statement. Retried on catalog lock contention, except inside a
$transaction(), which is retried as a whole instead.
Arguments
statementA SQL statement, interpolated with
glue::glue_sql()in the caller's environment:{value}is quoted as a value,{`name`}as an identifier,{`names`*}as a comma-separated list of them, and aDBI::SQL()object – such as$get_read_expr()returns – inserted as it stands. A statement already built withglue::glue()orglue::glue_sql()is passed through as it is....Values for the interpolation, as with
glue::glue_sql()..open, .closeInterpolation delimiters, for a statement whose own syntax needs the braces.
ducklake_db$get_query()
Execute a SQL query and return results. Reading contends for the catalog lock too; queries are retried on the same terms as writes.
Arguments
statementA SQL query statement, interpolated as in
$execute()....Values for the interpolation, as with
glue::glue_sql().as_atomicLogical. If true, return the single column the query selects as a plain vector rather than a one-column data.table.
.open, .closeInterpolation delimiters, as in
$execute().
ducklake_db$fetch()
Fetch data from a table
ducklake_db$get_table_metadata()
Get table metadata, stored as a comment on the catalog table. A table with no metadata, and one that does not exist, both return an empty list
ducklake_db$delete_from()
Delete rows from a table
ducklake_db$commit()
Commit data using overwrite, append, or upsert modes. Handles partitioning,
key identity columns, and list-to-MAP conversion. Which columns serve which
purpose is read from x's attributes, and otherwise from the metadata the
target table carries – so a character x inherits the target's specs,
and a table first created from one has none.
Usage
ducklake_db$commit(x, table_name, method = c("overwrite", "append", "upsert"))Arguments
xIf data.table, the data to commit; use as_ducklake_db_t for setting constraints. If character, treated as an in-DuckDB-memory table or view name.
table_nameTarget table name to commit to.
methodCharacter, one of "overwrite", "append", "upsert" (upsert being an update for existing rows, and insert for new rows). Only "overwrite" can change a table's schema; even if a table is empty, the others reject columns the table does not have.
ducklake_db$transaction()
Evaluate expr as a single DuckLake transaction, so that a block of reads/writes
either all land or none do. On failure part-way through, DB state is rolled back
and no harm is done. Contention is retried by re-evaluating the whole block, so
expr is safe to run more than once – it re-reads whatever it derived its
writes from.
ducklake_db$checkpoint()
Run DuckLake's CHECKPOINT, which flushes inlined data, expires
snapshots, merges adjacent files, rewrites files with many deletes, and
removes the files left unreferenced.
Every commit adds a snapshot and leaves the files it superseded in place, and DuckLake deletes neither on its own, so a long run grows without bound until this is called.
How much is discarded is governed by the expire_older_than and
delete_older_than options, set on the database (see $new()). Left
unset they keep everything, so checkpointing an unconfigured database
compacts but deletes nothing. Expiring a snapshot gives up time travel
to it, and anything still reading at one loses the files under it, so
set a retention that covers the readers you expect.
ducklake_db$print()
Print method for ducklake_db
Usage
ducklake_db$print(subheaders = character(0), ...)ducklake_db$next_id()
Use inside $transaction(): Safely hand out the next value of an integer id,
such that two processes allocating at once cannot both get the same one. Is not a
DuckLake native feature; employs an auxiliary table for row-level locks to ensure
uniqueness.
ducklake_db$get_read_expr()
Get SQL expression to read a table, potentially with additional filtering applied. evoland_db uses it to inject its own state based filter.
Arguments
table_nameCharacter string table name
...Ignored here; lets subclasses add their own filters, see evoland_db