openwpm.storage.storage_providers module

This module contains all base classes of the storage provider hierarchy Any subclass of these classes should be able to be used in OpenWPM without any changes to the rest of the code base

class openwpm.storage.storage_providers.StorageProvider

Bases: ABC

Base class that defines some general helper methods Do not inherit from this class directly Inherit from StructuredStorageProvider or UnstructuredStorageProvider instead

abstract async flush_cache() → None

Blockingly write out any cached data to the respective storage

abstract async init() → None

Initializes the StorageProvider for use

Guaranteed to be called in the process the StorageController runs in.

abstract async shutdown() → None

Close all open resources After this method has been called no further calls should be made to the object

class openwpm.storage.storage_providers.StructuredStorageProvider

Bases: StorageProvider

Structured Storage Providers are responsible for handling all structured data that OpenWPM emits. This includes: - All data that is collected by the WebExtension instrumentation - Data about browser configuration and - Any data that custom commands send to the Storage Controller

See docs/Schema-Documentation.md to see what an unmodified OpenWPM will attempt to store

abstract async finalize_visit_id(visit_id: VisitId, interrupted: bool = False) → Task[None] | None

This method is invoked to inform the StructuredStorageProvider that no more records for this visit_id will be submitted

This method returns once the data is ready to be written out. If the data is immediately written out nothing will be returned. Otherwise an awaitable will returned that resolve onces the records have been saved out to persistent storage

abstract async store_record(table: TableName, visit_id: VisitId, record: Dict[str, Any]) → None

Submit a record to be stored The storing might not happen immediately

class openwpm.storage.storage_providers.UnstructuredStorageProvider

Bases: StorageProvider

Unstructured Storage Providers are responsible for handling the unstructured data that OpenWPM emits. This is primarily content loaded by websites. Don’t make any assumptions about the data (especially don’t assume it’s valid unicode)

In the future this interface will be expanded to address the needs of https://github.com/openwpm/OpenWPM/issues/232

abstract async store_blob(filename: str, blob: bytes, overwrite: bool = False) → None

Stores the given bytes under the provided filename