5.1.0
Migration from 5.0 (optional)
- Run
init
command. Now you have two conflicting hooks:on_rollback
andon_index_rollback
. Follow the guide below to perform the migration.ConflictingHooksError
exception will be raised until then.
What's New
Per-index rollback hook
In this release, we continue to improve the rollback-handling experience, which became much more important since the Ithaca protocol reached mainnet. Let's briefly recap how DipDup currently processes chain reorgs before calling a rollback hook:
- If the
buffer_size
option of a TzKT datasource is set to a non-zero value, and there are enough data messages buffered when a rollback occurs, data is just dropped from the buffer, and indexing continues. - If all indexes in the config are
operation
ones, we can attempt to process a single-level rollback. All operations from rolled back block must be presented in the next one for rollback to succeed. If some operations are missing, theon_rollback
hook will be called as usual. - Finally, we can safely ignore indexes with a level lower than the rollback target. The index level is updated either on synchronization or when at least one related operation or bigmap diff has been extracted from a realtime message.
If none of these tricks have worked, we can't process a rollback without custom logic. Here's where changes begin. Before this release, every project contained the on_rollback
hook, which receives datasource: IndexDatasource
argument and from/to levels. Even if your deployment has thousands of indexes and only a couple of them are affected by rollback, you weren't able to easily find out which ones.
Now on_rollback
hook is deprecated and superseded by the on_index_rollback
one. Choose one of the following options:
- You haven't touched the
on_rollback
hook since project creation. Runinit
command and removehooks/on_rollback
andsql/on_rollback
directories in project root. Default action (reindexing) has not changed. - You have some custom logic in
on_rollback
hook and want to leave it as-is for now. You can ignore introduced changes at least till the next major release. - You have implemented per-datasource rollback logic and are ready to switch to the per-index one. Run
init
, move your code to theon_index_rollback
hook and deleteon_rollback
one. Note, you can access rolled back datasource viaindex.datasource
.
Token transfer index
Sometimes implementing an operation
index is overkill for a specific task. An existing alternative is to use a big_map
index to process only the diffs of selected big map paths. However, you still need to have a separate index for each contract of interest, which is very resource-consuming. A widespread case is indexing FA1.2/FA2 token contracts. So, this release introduces a new token_transfer
index:
indexes:
transfers:
kind: token_transfer
datasource: tzkt
handlers:
- callback: transfers
The TokenTransferData
object is passed to the handler on each operation, containing only information enough to process a token transfer.
config env
command to generate env-files
Generally, It's good to separate a project config from deployment parameters, and DipDup has multiple options to achieve this. First of all, multiple configs can be chained successively, overriding top-level sections. Second, the DipDup config can contain docker-compose-style environment variable declarations. Let's say your config contains the following content:
database:
kind: postgres
host: db
port: 5432
user: ${POSTGRES_USER:-dipdup}
password: ${POSTGRES_PASSWORD:-changeme}
database: ${POSTGRES_DB:-dipdup}
You can generate an env-file to use with this exact config:
$ dipdup -c dipdup.yml -c dipdup.docker.yml config env
POSTGRES_USER=dipdup
POSTGRES_PASSWORD=changeme
POSTGRES_DB=dipdup
The environment of your current shell is also taken into account:
$ POSTGRES_DB=foobar dipdup -c dipdup.yml -c dipdup.docker.yml config env
POSTGRES_USER=dipdup
POSTGRES_PASSWORD=changeme
POSTGRES_DB=foobar # <- set from current env
Use -f <filename>
option to save output on disk instead of printing to stdout. After you have modified the env-file according to your needs, you can apply it the way which is more convenient to you:
With dipdup --env-file / -e
option:
dipdup -e prod.env <...> run
When using docker-compose:
services:
indexer:
...
env_file: prod.env
Keeping framework up-to-date
A bunch of new tags is now pushed to the Docker Hub on each release in addition to the X.Y.Z
one: X.Y
and X
. That way, you can stick to a specific release without the risk of leaving a minor/major update unattended (friends don't let friends use latest
😉). The -pytezos
flavor is also available for each tag.
FROM dipdup/dipdup:5.1
...
In addition, DipDup will poll GitHub for new releases on each command which executes reasonably long and print a warning when running an outdated version. You can disable these checks with advanced.skip_version_check
flag.
Pro tip: you can also enable notifications on the GitHub repo page with 👁 Watch -> Custom -> tick Releases -> Apply to never miss a fresh DipDup release.
Changelog
See full 5.1.0 changelog here.