Backends

By default Pyncette runs without persistence. This means that the schedule is maintained in-memory and there is no coordination between multiple instances of the app.

Enabling persistence allows the aplication to recover from restarts as well as the ability to run multiple instances of an app concurrently without duplicate executions of tasks.

SQLite

SQLite is the default peristence engine and is included in the base Python package.

from pyncette import Pyncette, Context

app = Pyncette(sqlite_database="pyncette.db")

@app.task(schedule='* * * * * */10')
async def foo(context: Context):
    print('This will run every 10 seconds')

if __name__ == '__main__':
    app.main()

Redis

Redis can be enabled by passing redis_repository() as repository_factory parameter to the Pyncette constructor.

from pyncette import Pyncette, Context
from pyncette.redis import redis_repository

app = Pyncette(repository_factory=redis_repository, redis_url='redis://localhost')

Optionally, the tasks can be namespaced if the Redis server is shared among different Pyncette apps:

app = Pyncette(repository_factory=redis_repository, redis_url='redis://localhost', redis_namespace='my_super_app')

PostgreSQL

Redis can be enabled by passing postgres_repository() as repository_factory parameter to the Pyncette constructor.

from pyncette import Pyncette, Context
from pyncette.postgres import postgres_repository

app = Pyncette(
    repository_factory=postgres_repository,
    postgres_url='postgres://postgres@localhost/pyncette'
    postgres_table_name='pyncette_tasks'
)

The table will be automatically initialized on startup if it does not exists unless postgres_skip_table_create is set to True.

MySQL

MySQL can be configured by passing mysql_repository() as repository_factory parameter to the Pyncette constructor.

The MySQL backend requires MySQL version 8.0+.

from pyncette import Pyncette, Context
from pyncette.postgres import mysql_repository

app = Pyncette(
    repository_factory=mysql_repository,
    mysql_host="localhost",
    mysql_database="pyncette",
    mysql_user="pyncette",
    mysql_password="password",
    mysql_table_name='pyncette_tasks'
)

The table will be automatically initialized on startup if it does not exists unless mysql_skip_table_create is set to True.

Caution

MySQL backend currently does not work with Python 3.10 due to an issue with an upstream library.

Amazon DynamoDB

Amazon DynamoDB backend can be configured with dynamodb_repository().

from pyncette import Pyncette, Context
from pyncette.dynamodb import dynamodb_repository

app = Pyncette(
    repository_factory=dynamodb_repository,
    dynamodb_region_name="eu-west-1",
    dynamodb_table_name="pyncette",
)

DynamoDB repository will use ambient credentials, such as environment variables, ~/.aws/config or EC2 metadata service if e.g. running on EC2 or a Kubernetes cluster with kiam/kube2iam.

For convenience, an appropriate DynamoDB table will be automatically created on startup if it does not exist. The created table uses on-demand pricing model. If you would like to customize this behavior, you can manually create the table beforehand and pass dynamodb_skip_table_create=True in parameters.

Expected table schema should look something like this

{
    "AttributeDefinitions": [
        { "AttributeName": "partition_id", "AttributeType": "S" },
        { "AttributeName": "ready_at", "AttributeType": "S" },
        { "AttributeName": "task_id", "AttributeType": "S" }
    ],
    "KeySchema": [
        { "AttributeName": "partition_id", "KeyType": "HASH" },
        { "AttributeName": "task_id", "KeyType": "RANGE" }
    ],
    "LocalSecondaryIndexes": [
        {
            "IndexName": "ready_at",
            "KeySchema": [
                { "AttributeName": "partition_id", "KeyType": "HASH" },
                { "AttributeName": "ready_at", "KeyType": "RANGE" }
            ],
            "Projection": {
                "ProjectionType": "ALL"
            }
        }
    ]
}