openwpm.utilities.rediswq module

class openwpm.utilities.rediswq.RedisWQ(name: str, max_retries=2, **redis_kwargs: Any)

Bases: object

Simple Finite Work Queue with Redis Backend

This work queue is finite: as long as no more work is added after workers start, the workers can detect when the queue is completely empty.

The items in the work queue are assumed to have unique values.

This object is not intended to be used by multiple threads concurrently.

From: https://kubernetes.io/docs/tasks/job/fine-parallel-processing-work-queue

check_expired_leases()

Return to the work queue

If the lease key is not present for an item (it expired or was never created because the client crashed before creating it) then move the item back to the main queue so others can work on it.

complete(job)

Complete working on the job.

If the lease expired, the item may not have completed, and some other worker may have picked it up. There is no indication of what happened.

empty()

Return True if the queue is empty, including work being done, False otherwise.

False does not necessarily mean that there is work available to work on right now,

get_retry_number(job)

Return the number of retries for the given job.

This returns 0 if the job has never been retried. We do not attempt to verify that the job is valid, is currently leased by the worker, etc The caller is responsible for ensuring that there is currently a lease on this job (thus ensuring the retry count reflects the current state).

lease(lease_secs=60, block=True, timeout=None)

Begin working on an item the work queue.

Lease the item for lease_secs. After that time, other workers may consider this client to have crashed or stalled and pick up the item instead.

If optional args block is true and timeout is None (the default), block if necessary until an item is available.

renew_lease(job: Any, lease_secs=60) → bool

Checks if the item is currently leased by this client and if so renews that lease by lease_secs Return false if the lease was already expired

sessionID()

Return the ID for this session.