Checkpointers store (small) key-value records in a meta table (in the specified project). They are useful for maintaining consumer and pipeline state, such as the cursor for a subscription or the last time a scraper ran.
Checkpoint keys are strings and values are serialized with msgpack (supports most normal Python values, but not custom classes).
New checkpointed values are flushed at regular intervals (every 30 seconds by default). Checkpointers should not be used for storing large amounts of data. Checkpointers are not currently suitable for synchronizing parallel processes.
The meta-table instance that checkpoints are written to
get(key: str, default: Optional[Any] = None) → Any¶
Gets a checkpointed value
set(key: str, value: Any)¶
Sets a checkpoint value. Value will be encoded with msgpack.